Tag: Post

  • Asian Firm HashKey Unveils Global Exchange Post Bermuda Licensing

    Asian Firm HashKey Unveils Global Exchange Post Bermuda Licensing

    [ad_1]

    The HashKey Group, an Asian entity specializing in digital
    asset services, has unveiled the HashKey Global exchange after securing a
    license in Bermuda to provide regulated digital asset trading services. The
    announcement was made today (Monday), marking a milestone for the firm
    headquartered in Hong Kong, with operational presence in Singapore and Tokyo.

    With the unveiling of HashKey Global, the firm is poised to
    expand its offerings, starting with spot trading services for 21 digital
    assets. Among the featured assets are popular cryptocurrencies such as bitcoin, ether, Tether’s USDT, and Circle’s USDC. Additionally, the
    exchange has revealed plans to introduce futures trading product services in
    the coming weeks, further diversifying its portfolio and catering to the needs
    of its clientele.

    “HashKey Group aims to establish one of the world’s largest
    clusters of licensed exchanges within the next 5 years, surpassing all current
    regulated exchanges,” said Livio Weng, COO of HashKey Group.

    The HashKey Group attained unicorn status earlier this year
    following a fundraising round. The infusion of capital, which brought the
    company “nearly” to its $100 million fundraising objective, bolstered
    its position in the industry.

    Establishing HashKey Global in Bermuda’s Favorable
    Regulatory Landscape

    The choice to set up HashKey Global in Bermuda highlights
    the firm’s emphasis on operating within a regulated framework, with a
    commitment to compliance with industry standards and the cultivation of trust
    among investors and stakeholders. Bermuda’s favorable regulatory environment
    has positioned it as an appealing jurisdiction for companies exploring
    opportunities in the digital asset sector while maintaining adherence to
    rigorous regulatory protocols.

    Earlier, HashKey
    obtained all necessary licenses, making it the first Hong Kong firm to
    offer crypto retail trading, as reported by Finance Magnates. This achievement
    marks a notable milestone in legal regulations, as it updated Type 1 and Type 7
    licenses issued by the Securities and Futures Commission, allowing it to
    operate a virtual asset trading platform and provide automatic trading services
    to both institutional and retail users.

    The HashKey Group, an Asian entity specializing in digital
    asset services, has unveiled the HashKey Global exchange after securing a
    license in Bermuda to provide regulated digital asset trading services. The
    announcement was made today (Monday), marking a milestone for the firm
    headquartered in Hong Kong, with operational presence in Singapore and Tokyo.

    With the unveiling of HashKey Global, the firm is poised to
    expand its offerings, starting with spot trading services for 21 digital
    assets. Among the featured assets are popular cryptocurrencies such as bitcoin, ether, Tether’s USDT, and Circle’s USDC. Additionally, the
    exchange has revealed plans to introduce futures trading product services in
    the coming weeks, further diversifying its portfolio and catering to the needs
    of its clientele.

    “HashKey Group aims to establish one of the world’s largest
    clusters of licensed exchanges within the next 5 years, surpassing all current
    regulated exchanges,” said Livio Weng, COO of HashKey Group.

    The HashKey Group attained unicorn status earlier this year
    following a fundraising round. The infusion of capital, which brought the
    company “nearly” to its $100 million fundraising objective, bolstered
    its position in the industry.

    Establishing HashKey Global in Bermuda’s Favorable
    Regulatory Landscape

    The choice to set up HashKey Global in Bermuda highlights
    the firm’s emphasis on operating within a regulated framework, with a
    commitment to compliance with industry standards and the cultivation of trust
    among investors and stakeholders. Bermuda’s favorable regulatory environment
    has positioned it as an appealing jurisdiction for companies exploring
    opportunities in the digital asset sector while maintaining adherence to
    rigorous regulatory protocols.

    Earlier, HashKey
    obtained all necessary licenses, making it the first Hong Kong firm to
    offer crypto retail trading, as reported by Finance Magnates. This achievement
    marks a notable milestone in legal regulations, as it updated Type 1 and Type 7
    licenses issued by the Securities and Futures Commission, allowing it to
    operate a virtual asset trading platform and provide automatic trading services
    to both institutional and retail users.

    [ad_2]

    Source link

  • Incident Post Mortem: November 19, 2021

    Incident Post Mortem: November 19, 2021

    [ad_1]

    Summary

    On November 19, 2021, Coinbase learned that it had erroneously credited some customers transacting in GYEN and POWR either 100x or 1/100th the amount they purchased. Coinbase promptly disabled trading in POWR and GYEN, worked around the clock to resolve the underlying technical issue, and then made adjustments in customer accounts to reflect the amount of GYEN and POWR that customers actually purchased. This incident affected approximately 0.0072% of Coinbase’s total verified users.

    What happened?

    On November 19 at approximately 4:00 p.m. EST, Coinbase updated an internal data source related to POWR and GYEN precision. The update was tested through our standard automated testing and deployment monitoring procedures. However, the testing didn’t detect that the update would propagate at various speeds through a number of internal systems and would result in customers being credited either 100x or 1/100th the amount of GYEN or POWR they purchased.

    The data rollout error was identified through our position risk monitoring systems shortly after the November 19 4:00 p.m. EST update. At 5:35 p.m. EST, we disabled transacting in GYEN and POWR pending resolution of the underlying issue. At 7:26 p.m. EST, we identified accounts that transacted in GYEN or POWR during the data rollout, and temporarily restricted these accounts pending further investigation. By November 21, restrictions were removed for 98.8% of these accounts and, by December 13, Coinbase restored full trading for GYEN and POWR.

    What did Coinbase do to correct the problem?

    Coinbase immediately devoted substantial engineering resources to quickly correct the problem, ensuring our customers received the correct amount of GYEN and POWR that they purchased. For customers who were erroneously over-credited 100x the GYEN and POWR they purchased, we ensured that they received the correct amount of assets that they paid for. For those who still had GYEN and POWR in their accounts, this was relatively straightforward — we notified customers of the error and simply debited those customers’ accounts, removing the extra GYEN or POWR that was erroneously credited.

    Some customers had already converted their GYEN and POWR to other digital assets, such as Bitcoin. Other customers sent their GYEN and POWR to wallets off the Coinbase platform, but kept other digital assets on the Coinbase platform. For these customers, we notified them of the error and, in accordance with the Coinbase User Agreement, withdrew other assets from these customers’ Coinbase accounts equal to the amount of GYEN or POWR they had been over-credited.

    When determining how much to debit from these customers’ accounts, we used the most favorable exchange rate for our customers. Specifically, we calculated the USD value of the GYEN or POWR owed to Coinbase by using the lowest exchange rate on the Coinbase Exchange from the time this incident began until trading was halted ($0.00825/GYEN, $0.4742/POWR). This minimized the amount owed to Coinbase by these customers. We then debited funds from user accounts up to this USD value, starting with their fiat balances, then USDC and other stablecoin balances, followed by other digital asset balances ranked by descending market cap. The value of these digital assets was calculated using the market rate at the time user accounts were debited.

    A small group of customers who were erroneously over-credited GYEN or POWR sent these digital assets off-platform and left no other assets on the Coinbase platform. Coinbase has been reaching out to those customers individually and appreciates our customers’ cooperation returning the erroneously credited GYEN and POWR. Repayment of the over-credited funds is required under the Coinbase User Agreement.

    For customers who were undercredited GYEN or POWR, receiving a lower amount than they purchased, Coinbase first determined the amount of GYEN or POWR owed to these customers. Coinbase then calculated the USD value of the GYEN or POWR owed to customers by using the highest exchange rate from the start of the incident until the remediation process was completed ($0.009799/GYEN, $0.9617/POWR), which was the most favorable exchange rate for our customers. That means that regardless of the price customers purchased at, we assumed that the customers would have sold these assets at the highest price while trading was disabled. After calculating this USD value, we credited customers an equivalent amount of Bitcoin. We credited these customers in Bitcoin because GYEN and POWR trading was still suspended, and Bitcoin is used in every country where customers were affected.

    To further benefit our customers we used an exchange rate of $55,000/BTC, which was lower than the market rate of BTC at the time these BTC payments were made. This exchange rate ensured our customers received more Bitcoin than they would have received had we used the actual BTC-USD exchange rate at the time.

    Additionally, for all customers whose accounts were restricted, Coinbase provided a customer experience credit of up to $100 in BTC.

    What happens next?

    Many customers still have questions about how their accounts were credited or debited because of this incident. For questions specific to your account, please feel free to reach out to Coinbase Support.

    Coinbase is also revising the information in our customers’ account statements and tax forms to correctly reflect our customers’ GYEN and POWR transactions. If your statements or tax forms appear incorrect, please reach out to Coinbase Support, but know that we are working to correct that information as well.

    GYEN Values Before the Data Rollout Error.

    In the days leading up to the data rollout error, between November 16 and November 19, Coinbase Exchange observed GYEN-USD break parity when compared to JPY/USD. We have seen customers speculate on social media that this incident was somehow related to this break in parity. We have also seen customers speculate that this break in parity was somehow caused by Coinbase. These allegations are false and reflect a misunderstanding about what GYEN is and how Coinbase works.

    The Break in Parity Occurred Before and Was Not Related to the Incident. This break in parity occurred days before the incident. At the peak of this break in parity, on November 17, 1 GYEN traded for approximately ¥7.48.

    The price of GYEN (blue) in Yen (red) during the break in parity, with a high of ¥7.48 (Red Line Indicating the time of the Data Rollout Error)

    By the time the data rollout error occurred, on November 19, GYEN’s price stability had recovered and GYEN was trading at approximately ¥0.96–0.98. When Coinbase unrestricted impacted customers’ accounts, GYEN was trading at approximately ¥0.98. In other words, the break in parity occurred before the data rollout error and the two issues had nothing to do with each other.

    How the Break in Parity Occurred. When Coinbase listed GYEN, there was significant demand for GYEN that could not be matched by supply. The surge in buyer demand for GYEN, coupled with the insufficient supply of GYEN across all markets (not just Coinbase), ultimately caused the break in parity. From November 17 through November 19, Coinbase implemented an alert, informing its customers who were buying, selling and trading GYEN of “Unusual Market Activity — Due to unusual market activity for GYEN, you may have trouble trading GYEN on Coinbase.com. We apologize for any inconvenience caused by this.” The break in parity occurred because of these market conditions specific to the GYEN digital asset unrelated to Coinbase operations.


    Incident Post Mortem: November 19, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    [ad_2]

    Source link

  • Incident Post Mortem: November 23, 2021

    Incident Post Mortem: November 23, 2021

    [ad_1]

    Summary

    Between 4:00 pm and approximately 5:36 pm PT on Tuesday, November 23rd, we experienced an outage across most Coinbase production systems. During this outage, users were unable to access Coinbase using our websites and apps, and therefore were unable to use our products. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

    The Incident

    On November 23rd, 2021, at 4:00pm PT (Nov 24, 2021 00:00 UTC) an SSL certificate for an internal hostname in one of our Amazon Web Services (AWS) accounts expired. The expired SSL certificate was used by many of our internal load balancers which caused a majority of inter-service communications to fail. Due to the fact that our API routing layer connects to backend services via subdomains of this internal hostname, about 90% of incoming API traffic returned errors.

    Error rates returned to normal once we were able to migrate all load balancers to a valid certificate.

    Chart depicting overall 90% error rate at our API routing layer for duration of incident.

    Context: Certificates at Coinbase

    It’s helpful to provide some background information about how we manage SSL certificates at Coinbase. For the most part, certificates for public hostnames like coinbase.com are managed and provisioned by Cloudflare. For certificates for internal hostnames used to route traffic between backend services, we historically leveraged AWS IAM Server Certificates.

    One of the downsides of IAM Server Certificates is that certificates must be generated outside of AWS and uploaded via an API call. So last year, our infrastructure team migrated from IAM Server Certificates to AWS Certificate Manager (ACM). ACM solves the security problem because AWS generates both the public and private components of the certificate within ACM and stores the encrypted version in IAM for us. Only connected services like Cloudfront and Elastic Load Balancers will get access to the certificates. Denying the acm:ExportCertificate permission to all AWS IAM Roles ensures that they can’t be exported.

    In addition to the added security benefits, ACM also automatically renews certificates before expiration. Given that ACM certificates are supposed to renew and we did a migration, how did this happen?

    Root Cause Analysis

    Incident responders quickly noticed that the expired certificate was an IAM Server Certificate. This was unexpected because the aforementioned ACM migration had been widely publicized in engineering communication channels at the time; thus we had been operating under the assumption that we were running exclusively on ACM certificates.

    As we later discovered, one of the certificate migrations didn’t go as planned; the group of engineers working on the migration uploaded a new IAM certificate and postponed the rest of the migration. Unfortunately, the delay was not as widely communicated as it should have been and changes to team structure and personnel resulted in the project being incorrectly assumed complete.

    Migration status aside, you may ask the same question we asked ourselves: “Why weren’t we alerted to this expiring certificate?” The answer is: we were. Alerts were being sent to an email distribution group that we discovered only consisted of two individuals. This group was originally larger, but shrank with the departure of team members and was never sufficiently repopulated as new folks joined the team.

    In short, the critical certificate was allowed to expire due all of three factors:

    1. The IAM to ACM migration was incomplete.
    2. Expiration alerts were only being sent via email and were filtered or ignored.
    3. Only two individuals were on the email distribution list.

    Resolution & Improvements

    In order to resolve the incident we migrated all of the load balancers that were using the expired IAM cert to the existing auto-renewing ACM cert that had been provisioned as part of the original migration plan. This took longer than desired due to the number of load balancers involved and our cautiousness in defining, testing, and applying the required infrastructure changes.

    In order to ensure we don’t run into an issue like this again, we’ve taken the following steps to address the factors mentioned in the RCA section above:

    1. We’ve completed the migration to ACM, are no longer using IAM Server Certificates and are deleting any legacy certificates to reduce noise.
    2. We’re adding automated monitoring that is connected to our alerting and paging system to augment the email alerts. These will page on impending expiration as well as when ACM certificates drop out of auto-renewal eligibility.
    3. We’ve added a permanent group-alias to the email distribution list. Furthermore, this group is automatically updated as employees join and leave the company.
    4. We’re building a repository of incident remediation operations in order to reduce time to define, test and apply new changes.

    We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving challenges like those listed here, come work with us.


    Incident Post Mortem: November 23, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    [ad_2]

    Source link

  • Incident Post Mortem: October 27, 2021

    Incident Post Mortem: October 27, 2021

    [ad_1]

    Summary

    Between approximately 6:40 am and 10:42 am PT, and again between 12:20 pm and 2:32 pm PT on Wednesday, October 27th, we experienced intermittent outages on Coinbase.com, Coinbase mobile apps, and Coinbase Pro. During these outages, many users experienced slow loading times and errors while attempting to access Coinbase, or were unable to use features like buying, selling, and trading through our Retail and Pro websites and apps. The Exchange itself was not materially impacted. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

    We’re continuing to learn more about these events, and will continue to update this post with additional details that may be of interest.

    The Incident

    On the morning of October 27th PT, we experienced a significant increase in traffic. As traffic increased, our engineers were alerted about elevated error rates appearing across a number of services.

    The following functionality was affected:

    • Logged-out experience: users that were not logged in experienced errors when visiting coinbase.com or our mobile apps.
    • Coinbase Pro: users were temporarily unable to log in to Coinbase Pro.
    • Transfers: There was a higher rate of cancelled and refunded transfers during this time, as well as delays in processing on-chain money movements. Users may have been unable to see their latest transfer history.

    Root Cause Analysis

    These issues were caused by two separate but related outages. Both were triggered by system bottlenecks caused by the elevated traffic.

    Traffic to Coinbase — 10/27/2021

    In the first outage, we observed traffic patterns that were several times greater than previous peaks. This increase in traffic began to overload a datastore responsible for our rewards functionality. As latency increased on this database, related services became saturated and started to deplete resources as well. This resulted in a chain of failures and a more widespread outage.

    Query capacity to key database cluster

    The second outage was also triggered by a spike in traffic levels. In the early afternoon, engineers were alerted that our payment processing was being similarly overloaded. Unfortunately, an automated maintenance event that was already underway slowed our ability to scale this cluster up to meet with demand, and a set of failures similar to those that occurred during the first outage followed.

    Elevated query latency for Payments cluster

    In this instance, the servers that power our logged-out experience were also affected. As these servers became overwhelmed, they were unable to serve new traffic and were ultimately marked by our load balancer as unhealthy and removed from its pool, causing coinbase.com to become unavailable to users who were logged out or who were attempting to log in. Other impacted functionality included the ability to buy, sell, and trade in both Coinbase’s retail application as well as Coinbase Pro.

    At 2:32pm PT, our services returned to normal operation.

    Resolution & Improvements

    For the first outage, once the caching changes were deployed, the rewards database was scaled up, and additional replicas became available. Afterwards, our system was able to resume normal operation.

    To resolve the second outage, we upgraded the under-capacity payments cluster to a larger instance size and introduced additional read-only replicas.

    To prevent similar issues in the future, we are taking several additional actions:

    1. Reorganizing our largest services: we will continue to shard and isolate our largest services to avoid hitting limits like those mentioned previously.
    2. Enhanced load testing: we’re enhancing our load testing framework to be more representative of new traffic patterns that we saw during this event.
    3. Additional scaling: we are further scaling several of our databases that we observed operating close to limits at Wednesday’s elevated traffic levels.

    We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving scaling challenges like those presented here, come work with us.


    Incident Post Mortem: October 27, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

    [ad_2]

    Source link