Tag: November

  • Incident Post Mortem: November 19, 2021

    Incident Post Mortem: November 19, 2021

    [ad_1]

    Summary

    On November 19, 2021, Coinbase learned that it had erroneously credited some customers transacting in GYEN and POWR either 100x or 1/100th the amount they purchased. Coinbase promptly disabled trading in POWR and GYEN, worked around the clock to resolve the underlying technical issue, and then made adjustments in customer accounts to reflect the amount of GYEN and POWR that customers actually purchased. This incident affected approximately 0.0072% of Coinbase’s total verified users.

    What happened?

    On November 19 at approximately 4:00 p.m. EST, Coinbase updated an internal data source related to POWR and GYEN precision. The update was tested through our standard automated testing and deployment monitoring procedures. However, the testing didn’t detect that the update would propagate at various speeds through a number of internal systems and would result in customers being credited either 100x or 1/100th the amount of GYEN or POWR they purchased.

    The data rollout error was identified through our position risk monitoring systems shortly after the November 19 4:00 p.m. EST update. At 5:35 p.m. EST, we disabled transacting in GYEN and POWR pending resolution of the underlying issue. At 7:26 p.m. EST, we identified accounts that transacted in GYEN or POWR during the data rollout, and temporarily restricted these accounts pending further investigation. By November 21, restrictions were removed for 98.8% of these accounts and, by December 13, Coinbase restored full trading for GYEN and POWR.

    What did Coinbase do to correct the problem?

    Coinbase immediately devoted substantial engineering resources to quickly correct the problem, ensuring our customers received the correct amount of GYEN and POWR that they purchased. For customers who were erroneously over-credited 100x the GYEN and POWR they purchased, we ensured that they received the correct amount of assets that they paid for. For those who still had GYEN and POWR in their accounts, this was relatively straightforward — we notified customers of the error and simply debited those customers’ accounts, removing the extra GYEN or POWR that was erroneously credited.

    Some customers had already converted their GYEN and POWR to other digital assets, such as Bitcoin. Other customers sent their GYEN and POWR to wallets off the Coinbase platform, but kept other digital assets on the Coinbase platform. For these customers, we notified them of the error and, in accordance with the Coinbase User Agreement, withdrew other assets from these customers’ Coinbase accounts equal to the amount of GYEN or POWR they had been over-credited.

    When determining how much to debit from these customers’ accounts, we used the most favorable exchange rate for our customers. Specifically, we calculated the USD value of the GYEN or POWR owed to Coinbase by using the lowest exchange rate on the Coinbase Exchange from the time this incident began until trading was halted ($0.00825/GYEN, $0.4742/POWR). This minimized the amount owed to Coinbase by these customers. We then debited funds from user accounts up to this USD value, starting with their fiat balances, then USDC and other stablecoin balances, followed by other digital asset balances ranked by descending market cap. The value of these digital assets was calculated using the market rate at the time user accounts were debited.

    A small group of customers who were erroneously over-credited GYEN or POWR sent these digital assets off-platform and left no other assets on the Coinbase platform. Coinbase has been reaching out to those customers individually and appreciates our customers’ cooperation returning the erroneously credited GYEN and POWR. Repayment of the over-credited funds is required under the Coinbase User Agreement.

    For customers who were undercredited GYEN or POWR, receiving a lower amount than they purchased, Coinbase first determined the amount of GYEN or POWR owed to these customers. Coinbase then calculated the USD value of the GYEN or POWR owed to customers by using the highest exchange rate from the start of the incident until the remediation process was completed ($0.009799/GYEN, $0.9617/POWR), which was the most favorable exchange rate for our customers. That means that regardless of the price customers purchased at, we assumed that the customers would have sold these assets at the highest price while trading was disabled. After calculating this USD value, we credited customers an equivalent amount of Bitcoin. We credited these customers in Bitcoin because GYEN and POWR trading was still suspended, and Bitcoin is used in every country where customers were affected.

    To further benefit our customers we used an exchange rate of $55,000/BTC, which was lower than the market rate of BTC at the time these BTC payments were made. This exchange rate ensured our customers received more Bitcoin than they would have received had we used the actual BTC-USD exchange rate at the time.

    Additionally, for all customers whose accounts were restricted, Coinbase provided a customer experience credit of up to $100 in BTC.

    What happens next?

    Many customers still have questions about how their accounts were credited or debited because of this incident. For questions specific to your account, please feel free to reach out to Coinbase Support.

    Coinbase is also revising the information in our customers’ account statements and tax forms to correctly reflect our customers’ GYEN and POWR transactions. If your statements or tax forms appear incorrect, please reach out to Coinbase Support, but know that we are working to correct that information as well.

    GYEN Values Before the Data Rollout Error.

    In the days leading up to the data rollout error, between November 16 and November 19, Coinbase Exchange observed GYEN-USD break parity when compared to JPY/USD. We have seen customers speculate on social media that this incident was somehow related to this break in parity. We have also seen customers speculate that this break in parity was somehow caused by Coinbase. These allegations are false and reflect a misunderstanding about what GYEN is and how Coinbase works.

    The Break in Parity Occurred Before and Was Not Related to the Incident. This break in parity occurred days before the incident. At the peak of this break in parity, on November 17, 1 GYEN traded for approximately ¥7.48.

    The price of GYEN (blue) in Yen (red) during the break in parity, with a high of ¥7.48 (Red Line Indicating the time of the Data Rollout Error)

    By the time the data rollout error occurred, on November 19, GYEN’s price stability had recovered and GYEN was trading at approximately ¥0.96–0.98. When Coinbase unrestricted impacted customers’ accounts, GYEN was trading at approximately ¥0.98. In other words, the break in parity occurred before the data rollout error and the two issues had nothing to do with each other.

    How the Break in Parity Occurred. When Coinbase listed GYEN, there was significant demand for GYEN that could not be matched by supply. The surge in buyer demand for GYEN, coupled with the insufficient supply of GYEN across all markets (not just Coinbase), ultimately caused the break in parity. From November 17 through November 19, Coinbase implemented an alert, informing its customers who were buying, selling and trading GYEN of “Unusual Market Activity — Due to unusual market activity for GYEN, you may have trouble trading GYEN on Coinbase.com. We apologize for any inconvenience caused by this.” The break in parity occurred because of these market conditions specific to the GYEN digital asset unrelated to Coinbase operations.


    Incident Post Mortem: November 19, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    [ad_2]

    Source link

  • Incident Post Mortem: November 23, 2021

    Incident Post Mortem: November 23, 2021

    [ad_1]

    Summary

    Between 4:00 pm and approximately 5:36 pm PT on Tuesday, November 23rd, we experienced an outage across most Coinbase production systems. During this outage, users were unable to access Coinbase using our websites and apps, and therefore were unable to use our products. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

    The Incident

    On November 23rd, 2021, at 4:00pm PT (Nov 24, 2021 00:00 UTC) an SSL certificate for an internal hostname in one of our Amazon Web Services (AWS) accounts expired. The expired SSL certificate was used by many of our internal load balancers which caused a majority of inter-service communications to fail. Due to the fact that our API routing layer connects to backend services via subdomains of this internal hostname, about 90% of incoming API traffic returned errors.

    Error rates returned to normal once we were able to migrate all load balancers to a valid certificate.

    Chart depicting overall 90% error rate at our API routing layer for duration of incident.

    Context: Certificates at Coinbase

    It’s helpful to provide some background information about how we manage SSL certificates at Coinbase. For the most part, certificates for public hostnames like coinbase.com are managed and provisioned by Cloudflare. For certificates for internal hostnames used to route traffic between backend services, we historically leveraged AWS IAM Server Certificates.

    One of the downsides of IAM Server Certificates is that certificates must be generated outside of AWS and uploaded via an API call. So last year, our infrastructure team migrated from IAM Server Certificates to AWS Certificate Manager (ACM). ACM solves the security problem because AWS generates both the public and private components of the certificate within ACM and stores the encrypted version in IAM for us. Only connected services like Cloudfront and Elastic Load Balancers will get access to the certificates. Denying the acm:ExportCertificate permission to all AWS IAM Roles ensures that they can’t be exported.

    In addition to the added security benefits, ACM also automatically renews certificates before expiration. Given that ACM certificates are supposed to renew and we did a migration, how did this happen?

    Root Cause Analysis

    Incident responders quickly noticed that the expired certificate was an IAM Server Certificate. This was unexpected because the aforementioned ACM migration had been widely publicized in engineering communication channels at the time; thus we had been operating under the assumption that we were running exclusively on ACM certificates.

    As we later discovered, one of the certificate migrations didn’t go as planned; the group of engineers working on the migration uploaded a new IAM certificate and postponed the rest of the migration. Unfortunately, the delay was not as widely communicated as it should have been and changes to team structure and personnel resulted in the project being incorrectly assumed complete.

    Migration status aside, you may ask the same question we asked ourselves: “Why weren’t we alerted to this expiring certificate?” The answer is: we were. Alerts were being sent to an email distribution group that we discovered only consisted of two individuals. This group was originally larger, but shrank with the departure of team members and was never sufficiently repopulated as new folks joined the team.

    In short, the critical certificate was allowed to expire due all of three factors:

    1. The IAM to ACM migration was incomplete.
    2. Expiration alerts were only being sent via email and were filtered or ignored.
    3. Only two individuals were on the email distribution list.

    Resolution & Improvements

    In order to resolve the incident we migrated all of the load balancers that were using the expired IAM cert to the existing auto-renewing ACM cert that had been provisioned as part of the original migration plan. This took longer than desired due to the number of load balancers involved and our cautiousness in defining, testing, and applying the required infrastructure changes.

    In order to ensure we don’t run into an issue like this again, we’ve taken the following steps to address the factors mentioned in the RCA section above:

    1. We’ve completed the migration to ACM, are no longer using IAM Server Certificates and are deleting any legacy certificates to reduce noise.
    2. We’re adding automated monitoring that is connected to our alerting and paging system to augment the email alerts. These will page on impending expiration as well as when ACM certificates drop out of auto-renewal eligibility.
    3. We’ve added a permanent group-alias to the email distribution list. Furthermore, this group is automatically updated as employees join and leave the company.
    4. We’re building a repository of incident remediation operations in order to reduce time to define, test and apply new changes.

    We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving challenges like those listed here, come work with us.


    Incident Post Mortem: November 23, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    [ad_2]

    Source link

  • Bitcoin ‘Moonvember’ begins as data shows November is best month for S&P 500

    Bitcoin ‘Moonvember’ begins as data shows November is best month for S&P 500

    [ad_1]

    Bitcoin (BTC) bulls are betting on a solid month for BTC price action, as November traditionally sees strong gains for United States stocks.

    Data shows that November has been the best performing month for the S&P 500 since 1985.

    November 2021 has stiff competition

    With “Uptober” already the biggest month in terms of gains for Bitcoin in 2021, odds are stocks could act as a catalyst for further upside in “Moonvember.”

    Median S&P 500 progress in November over the past 35 years has been just over 2% — making it the only month to achieve those median returns.

    At the same time, over 70% of years have seen positive returns, and Bitcoin’s history is similar.

    In November, BTC/USD has ended up higher than when it started with the exception of just two years: 2018 (-36.5%) and 2019 (-17.2%).

    2020 conversely saw 43% gains, leaving the door open for a rematch in line with expectations.

    BTC/USD monthly returns table. Source: Bybt

    As Cointelegraph reported, these predict a volatile but ultimately extremely beneficial month as Bitcoin approaches its Q4 peak.

    “Highest monthly close in history. Congrats Bitcoin and congrats y’all,” an optimistic TechDev summarized on Nov. 1.

    “We are now headed toward our second monthly RSI peak like every cycle before. Nowhere near a top. Trust the indicators.”

    TechDev is eyeing either copycat retrace of 2017’s top sequence or that of 1970s gold, both apt to send BTC/USD far beyond $100,000.

    Bumps in the road for stocks and Bitcoin

    Bitcoin’s relationship to traditional markets has come into examination in recent months as the cryptocurrency begins to carve out its own path away from macro.

    Related: ‘Uptober’ closes at record high in best month of 2021 — 5 things to watch in Bitcoin this week

    A test of trader resolve may come as soon as this week as the Federal Reserve prepares new comments on asset purchase tapering.

    For Bitcoin, the decision whether or not to allow a regulated exchange-traded fund (ETF) in the U.S. this month may yet steer price action far away from predictions — especially if a rejection is seen.

    As proponents point out slowness in following other countries, VanEck — one of over 40 applicants — has revealed it is mulling applying to launch a spot ETF in Australia.