Starting today, COTI (COTI) is available on Coinbase.com and in the Coinbase Android and iOS apps. Coinbase customers can now trade, send, receive, or store COTI in most Coinbase-supported regions, with certain exceptions indicated here. Trading for these assets is also supported on Coinbase Pro.
COTI is a token that powers Coti, a project that describes itself as a “DAG (directed acyclic graph) protocol optimized for creating decentralized payment networks and stable coins.” Coti uses Trustchain, a decentralized ledger that can process over 100,000 transactions per second, to power online and offline payments, loyalty payments, stablecoins, and more. Note: Coinbase currently only supports COTI running on the Ethereum blockchain (ERC-20).
One of the most common requests we hear from customers is to be able to buy and sell more cryptocurrencies on Coinbase. We announced a process for listing assets, designed in part to accelerate the addition of more cryptocurrencies. We are also investing in new tools to help people understand and explore cryptocurrencies. We launched informational asset pages (see COTI), as well as a new section of the Coinbase website to answer common questions about crypto.
Customers can sign up for a Coinbase account here to buy, sell, convert, send, receive, or store COTI today.
Please note: Coinbase Ventures may be an investor in the crypto projects mentioned here, and additionally, Coinbase may hold such tokens on its balance sheet for operational purposes. A list of Coinbase Ventures investments is available at https://ventures.coinbase.com/. Coinbase intends to maintain its investment in these entities for the foreseeable future and maintains internal policies that address the timing of permissible disposition of any related digital assets, if applicable. All assets, regardless of whether Coinbase Ventures holds an investor or Coinbase holds for operational purposes, are subject to the same strict review guidelines and review process.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Crypto is a new type of asset. Besides potential day to day or hour to hour volatility, each crypto asset has unique features. Make sure you research and understand individual assets before you transact.
All images provided herein are by Coinbase.
COTI (COTI) is now available on Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
As part of our mission to increase economic freedom in the world, we’ve been thinking more about how we expand our products internationally. To get 1 billion people using crypto through our products, we will need to have much broader reach in more countries.
I recently caught up with the WhatsApp founders because I wanted to get their perspective on how WhatsApp became so pervasive globally. Almost anywhere you go in the world, even in remote places, you will see billboards and kiosks with businesses listing their WhatsApp number as the main way to contact their business, in addition to individuals using the product regularly. How did they do this? And more importantly, how can Coinbase do for payments what they did for messaging? This is what it will take to build a truly global economy.
Here are the main lessons they shared with me:
1. Develop good global accessibility and empathy
Reduce barriers to entry (such as in onboarding). Instill strong localization that feels natural in native languages. Make the product work seamlessly in adverse environments (low/intermittent network connectivity, old devices, etc). Understand that not everyone is affluent or an expert. Understand that people may use your product differently due to cultural norms (Brazil used a lot of voice messaging, Arabic nations used a lot of broadcast messaging). Learning from your users is key. They will guide you to what to build.
2. Partner with key players
There is an interesting anecdote of WhatsApp history where they signed a global marketing relationship with Nokia. The writing was on the wall for Nokia’s demise but at the same time, they had the marketing heft to put the WhatsApp logo all over the world. It was a big win for WhatsApp. In addition, they signed key deals with phone OEM manufacturers to get WhatsApp pre-installed on a number of devices.
3. Don’t overly fixate on one market
WhatsApp never focused on the US or India or Mexico or Brazil exclusively. They focused on building a global product and global process. Their strategy was more “breadth first” rather than “depth first”.
Conclusion
At Coinbase we will be thinking about these lessons to see how we can further accelerate our ambitious international expansion goals. How could our app perform better in low connectivity environments? What equivalent partnership deals should we be pursuing? How can we reduce friction in onboarding or to make payment? Etc.
There is a piece of the puzzle here for us that is different from WhatsApp: we need the blockchains to scale several orders of magnitude to reach this many users globally. Coinbase to Coinbase transactions are fast (because they’re off chain) today, but on chain transactions are still much too slow. We are spinning up a new Protocol Team of engineers at Coinbase to work on the underlying blockchains, including layer 2 solutions, to see what we can do to help accelerate this.
It will take a number of parts of the company and industry coming together to build a new more fair, free, and efficient financial system for the world. But when we’re successful it will have far reaching implications.
Lessons from WhatsApp on international expansion was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Nick Johnson, lead developer at the Ethereum Name Service (ENS), talks about making Ethereum human-readable, why Budweiser purchased an NFT, how purchasing an ENS works, and more. Show highlights:
what ENS does and why it is important
why Budweiser bought “beer.eth”
how ENS names can be used to make Ethereum easier to use
what business use cases ENS names make possible
what made Nick want to create a naming service on Ethereum
how people are using their ENS names
why people are purchasing ENS names
what privacy issues arise from people attaching their name to an ETH address
what happens when Ethereum addresses get spammed with unwanted tokens
how to buy an ENS name and how much it costs
how ENS handles squatters
what happens when you lose access to an address holding an ENS name
how ENS is integrating normal domains (ex: “.com”) to Ethereum
When Millenials and Gen Z representatives gather on social media to speculatively increase Dogecoin (DOGE) prices or Wall Street stocks (remember GameStop?), your CFD offering. limited to the tired MetaTrader 4. may not be enough. Today’s trading for younger generations is all about memes, simple mobile apps, gamification, following trends seen on TikTok and Twitter and seamlessly adapting to dynamic market changes.
CFD Offering: Give Volatility to Those who Seek Volatility
Since November 2020, Bitcoin grew by an average of 30%, falling 35% in one month in May 2021, and the volatility spurred investor activity and boosted the earnings of cryptocurrency exchanges. However, June brought stagnation while BTC and other digital assets which were wildly volatile became very quiet: the oldest cryptocurrency closed the last month of Q2 with a decline of less than 6%, and in July, it decreased again to the $30,000 level completely resetting this year’s appreciation
. According to Filip Kaczmarzyk, Director of Trading Department, Member of the Management Board of XTB, “Cryptocurrency investors are divided into two groups. Let’s call the first one ‘crypto buyers’ and the second ‘volatility seekers’.” The second group is permanently looking for volatility.
“They might have been trading cryptos during turbulent times, or cryptos might have even been their first traded instrument, but now they have switched to the markets where the action is, such as bullion or indices. At XTB we feel it is of utmost importance to allow clients to swiftly move between all asset classes in seconds to exploit all possible chances for profit,” Kaczmarzyk added.
Many other asset classes have turned out to be more volatile than Bitcoin during this period, e.g., the shares of the US stock giants (for example, Amazon rose by 6.7%), or the quotes of the commodity market (gold lost more than 7%, and oil rose by 9%).
Indirect Exposure to Cryptocurrencies and Leverage Can be Your Edge
The stocks of publicly traded companies might be another good direction in the search for market volatility. They will undeniably encourage many investors to abandon the cryptocurrency exchanges offerings in favor of a CFD broker. The list of publicly traded companies (e.g., on Wall Street) that are indirectly or directly linked to the digital asset market is currently quite significant and still growing.
Suggested articles
Connect to the Markets with Exclusive Capital’s Bridging SolutionsGo to article >>
One of the most famous examples besides Elon Musk’s Tesla (which at one point in 2020 was more volatile than Bitcoin itself) is MicroStrategy (ticker: MSTR). The company, whose CEO is Michael Saylor, has invested billions of dollars in buying BTC in the last 12 months.
You can also add Riot Blockchain, Hut 8 Mining, Bitcoin group, Coinbase Global, Marathon Digital Holdings, Square, Voyager Digital, Galaxy Digital Holdings, and many more to the long list. They are among the publicly listed companies that hold the most BTC in their accounts.
“CFD brokers already have an advantage as CFDs allow traders to trade both rising and falling markets. CFDs essentially involve speculating on the future price of a specific asset, which means you can go long when you expect prices to rise and go short when prices are expected to fall. This means more trading opportunities. Millennials make up the largest demographic of traders today, with Gen Z hot on their heels. Once you understand your target audience and region, pitching your services to them becomes easier,” said Charlotte Day, Director at Contentworks Agency.
Promoting leverage and focusing on education about what it is and how it works (a cryptocurrency trader often has less experience and knowledge than a trader from a more traditional CFD market) can be an additional positive factor.
To get the full article and the bigger picture on the opportunities among cryptocurrency traders, get our latest Quarterly Intelligence Report.
Bears pressed harder on the market on Aug. 26 after Bitcoin (BTC) continued its pullback to an intra-day low at $46,250 and altcoins nursed near double-digit losses.
While most of the market is seeing red, a few altcoins managed to notch notable gains due to exchange listings and new staking opportunities.
Top seven coins with the highest 24-hour price change. Source: Cointelegraph Markets Pro
Data from Cointelegraph Markets Pro and TradingView shows that the biggest gainers over the past 24 hours were COTI, XYO Network (XYO) and Conflux Network (CFX).
COTI rallies after listing on Coinbase
The top-performing coin over the past 24 hours was COTI, an enterprise-grade fintech platform focused on decentralized payments.
VORTECS™ data from Cointelegraph Markets Pro began to detect a bullish outlook for COTI on Aug. 22, prior to the recent price rise.
The VORTECS™ Score, exclusive to Cointelegraph, is an algorithmic comparison of historic and current market conditions derived from a combination of data points including market sentiment, trading volume, recent price movements and Twitter activity.
VORTECS™ Score (green) vs. COTI price. Source: Cointelegraph Markets Pro
As seen in the chart above, the VORTECS™ Score for COTI turned solidly green on Aug. 22 and proceeded to climb to a high of 79 on Aug. 25, around two hours before the price increased 81% over the next day.
The burst of momentum for COTI came after it was announced that the token would be listed on Coinbase Pro and Huobi, two of the largest cryptocurrency exchanges by daily trading volume.
XYO volume surges
The XYO Network is comprised of a network of devices that anonymously collect and validate data with a geographic component (geospatial). All data obtained from the tracking devices on the network are stored on the XYO blockchain.
VORTECS™ data from Cointelegraph Markets Pro began to detect a bullish outlook for XYO on Aug. 21, prior to the recent price rise.
VORTECS™ Score (green) vs. XYO price. Source: Cointelegraph Markets Pro
As seen in the chart above, the VORTECS™ Score for XYO climbed into the green zone on Aug. 21 and reached a high of 72, around 67 hours before its price increased 36% over the next two days.
Related: Forget Lambos, NFTs are the new crypto status symbol
Conflux Network expands its NFT ecosystem
The Conflux Network is a public, permissionless blockchain network that has the goal of bridging the communities and economies of Asian and Western societies in order to facilitate the secure and interoperable flow of assets and data.
Data from Cointelegraph Markets Pro and TradingView shows that after hitting a low of $0.269 on Aug. 25, the price of CFX rallied 40% to an intraday high at $0.377 on Aug. 26 as its 24-hour trading volume surged by 242% to $66 million.
CFX/USDT 4-hour chart. Source: TradingView
The sudden price surge comes as the CFX ecosystem works on building out its NFT ecosystem, and the token appears to have also benefited from recently being listed on the Mexo and Tokocrypto exchanges.
The overall cryptocurrency market capitalization now stands at $1.993 trillion, and Bitcoin’s dominance rate is 41.2%.
The views and opinions expressed here are solely those of the author and do not necessarily reflect the views of Cointelegraph. Every investment and trading move involves risk, and you should conduct your own research when making a decision.
Starting Today, Wednesday August 25, transfer COTI into your Coinbase Pro account ahead of trading. Support for COTI will generally be available in Coinbase’s supported jurisdictions with certain exceptions as indicated in each asset page here. Trading will begin on or after 9AM Pacific Time (PT) Thursday August 26, if liquidity conditions are met.
One of the most common requests we receive from customers is to be able to trade more assets on our platform. Per the terms of our listing process, we anticipate supporting more assets that meet our standards over time. Most recently we have added trading support for Axie Infinity (AXS), Request (REQ), TrueFi (TRU), Wrapped Luna (WLUNA), Harvest Finance (FARM) Fetch.ai (FET) Paxos Standard (PAX) and Polymath Network (POLY), Clover Finance (CLV), Mask Network (MASK), Rally (RLY), BarnBridge (BOND), Livepeer (LPT), Quant (QNT), Chiliz (CHZ) Keep Network (KEEP), Polkadot (DOT), Solana (SOL), Gitcoin (GTC), Enzyme Token (MLN), Amp (AMP), Dogecoin (DOGE), Internet Computer (ICP), Cartesi (CTSI), iExec (RLC), Mirror Protocol (MIR), Tellor (TRB), Tether (USDT) and Ampleforth Governance Token (FORTH).
Starting Today, Wednesday August 25, we will begin accepting inbound transfers of COTI to Coinbase Pro. Trading will begin on or after 9AM Pacific Time (PT) Thursday August 26, if liquidity conditions are met.
Once sufficient supply of COTI is established on the platform, trading on our COTI-USD order book will launch in three phases, post-only, limit-only and full trading. If at any point one of the new order books does not meet our assessment for a healthy and orderly market, we may keep the book in one state for a longer period of time or suspend trading as per our Trading Rules.
We will publish tweets from our Coinbase Pro Twitter account as each order book moves through the phases.
COTI (COTI) is a token that powers Coti, a project that describes itself as a “DAG (directed acyclic graph) protocol optimized for creating decentralized payment networks and stable coins.” Coti uses Trustchain, a decentralized ledger that can process over 100,000 transactions per second, to power online and offline payments, loyalty payments, stablecoins, and more. Note: Coinbase currently only supports COTI running on the Ethereum blockchain (ERC-20).
COTI is not yet available on Coinbase.com or via our Consumer mobile apps. We will make a separate announcement if and when this support is added.
You can sign up for a Coinbase Pro account here to start trading. For more information on trading COTI on Coinbase Pro, visit our support page.
### Please note: Coinbase Ventures may be an investor in the crypto projects mentioned here, and additionally, Coinbase may hold such tokens on its balance sheet for operational purposes. A list of Coinbase Ventures investments is available at https://ventures.coinbase.com/. Coinbase intends to maintain its investment in these entities for the foreseeable future and maintains internal policies that address the timing of permissible disposition of any related digital assets, if applicable. All assets, regardless of whether Coinbase Ventures holds an investor or Coinbase holds for operational purposes, are subject to the same strict review guidelines and review process. This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Crypto is a new type of asset. Besides potential day to day or hour to hour volatility, each crypto asset has unique features. Make sure you research and understand individual assets before you transact.
All images provided herein are by Coinbase.
COTI (COTI) is launching on Coinbase Pro was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
On August 23rd, Derivatives Exchange Bitget announced its new logo designed to indicate a place where people can exchange freely and realize their financial dreams with the help of leverage.
According to Bitget, Bit is a blended word coming from Binary and digit and the smallest unit of information. It implies a business that is based on digitalization and IT. “Get” represents interaction, a kind of connection between the platform and the external world. In combination, the name Bitget expresses its mission “Better Trading Better Life”, and its commitment to offer liberal, ultimate and fair trading services.
The new LOGO uses two interactive arrows to symbolize the liquidity and activity of Bitget and its pursuit of freedom and fairness. It also reflects the nature of platform trading. The interactive image shows that Bitget will always put users first and lead users to success and fortune through transactions. The newly adopted blue color represents the energy and vitality of the crypto industry and delivers a sense of boldness and creativity — an image the platform wants to show to its users.
The new logo is a testament to Bitget’s achievements in the past and an embodiment of its vision for the future. In just three years since its establishment, the platform has successfully made itself into the Top6 derivative exchange, closely after the professional ones FTX and Bybit. Bitget now has more than 1.6 million registered users in 48 countries and regions, including Japan, South Korea, Vietnam, Russia and Turkey, with an average daily trading volume of $5.6 billion. It has become one of the fastest-growing derivatives exchanges in the world.
According to its CEO Sandra, Bitget will continue to dive deeper into derivatives trading in the remaining months of the year. It aims to bring users products and services focused on futures, and expand to global markets such as South America, Japan, Central Europe, and Russia under the principle of compliant and localized operations. As Bitget steps up its globalization efforts and connects with users from more countries, a simpler and unique logo to display a new corporate culture and values become a necessity.
Last month, it released a new slogan “Better Trading Better Life”, indicating that every user across the world can enjoy a liberal, ultimate and fair trading experience on Bitget and realize their financial dreams with the help of leverage. “The new logo conveys not only a change of appearance but also a stronger inner spirit. The updated logo and slogan deliver a message that is in line with our corporate culture. In the future, Bitget will bring more surprises to users with our brand new image.” says Sandra.
About Bitget
Established in July 2018, Bitget is one of the world’s fastest-growing derivatives exchanges that support futures trading, spot trading and crypto asset purchasing. As a platform in pursuit of perfection and innovation, Bitget has pioneered to launch three flagship products of USDT-Margined Futures, One-Click Copy Trade, and Quanto Swap Futures to provide the best services to users. It is now the Top6 derivatives exchange and the largest crypto copy trading platform in the world.
Singapore-based crypto exchange KuCoin is launching a mining pool aimed at providing revenue to proof-of-work miners after integrating their rigs.
In a Wednesday announcement, the exchange said its KuCoin Pool product would allow miners around the world to contribute to the Bitcoin (BTC) and Bitcoin Cash (BCH) and share rewards. At the moment, miners are required to install and run the necessary hardware themselves to join the pool, but KuCoin said it would introduce mining in the cloud in the future.
KuCoin CEO Johnny Lyu also claimed the pool would be encouraging miners to participate in environmentally-friendly solutions — people using renewable energy sources for mining will receive discounts on fees. The move is seemingly part of a shift in many mining firms beginning to transition to cleaner or renewable energy.
“For existing KuCoin users, it will become straightforward to set up their mining devices to generate passive income right away,” said Lyu. “Miners can benefit from the one-stop mining service platform and its features to get up and running very quickly.”
The exchange is coming in late to mining when compared with major firms like Binance, which launched its mining pool in April 2020. According to blockchain data, some of the largest BTC miners include Antpool — owned by Chinese mining giant Bitmain — Poolin, ViaBTC, and F2Pool.
Related: Are KuCoin Shares overvalued after KCS price gains 100% in one month?
Launched in 2017, KuCoin reported this week that it had reached 10 million users, having risen by 1,114% in the last year. Last year, hackers stole roughly $275 million from the exchange before KuCoin was able to recover the majority of the funds.
For ethics-based investment enthusiasts, here are two events to check out, solely because speakers from the first Shariah-compliant DeFi project, MRHB DeFi will be speaking there.
Islamic Fintech Leaders Forum 2021
To access event platform:
More info on event: https://emnesevents.com/fintech-islamic-leaders-forum/
Spotlight on:
MRHB DeFi’s Chairman of Shariah Board, Dr. Farrukh Habib, who will be holding a blockchain panel discussion today 25th August at 12:05 PM (UAE Time) at Islamic Fintech Leaders Forum, and Shariah Leaders Discussion session at 04:00 PM (UAE Time).
And MRHB DeFi’s Chairman of Governance Board, Khalid Howlader, who is also the Senior Managing Director and Head of Credit & Sukuk for R.J. Fleming & Co. for institutional and sovereign clients. With his global perspectives, he is a recognised authority in his field and has addressed investors worldwide as well as audiences at the World Bank, IMF, ECB and IIF.
ICCIA Leadership Talks “Digital Banking: The Way Forward”
ICCIA (Islamic Chamber of Commerce, Industry and Agriculture), a sub-organization of OIC (Organization of Islamic Cooperation), representing 56 Muslim countries, is organizing the event.
Spotlight on:
MRHB DeFi’s Chairman of Shariah Board, Dr. Farrukh Habib, who will be presenting followed by a Q & A session today.
Tl;dr: Coinbase is leveraging AWS’ Managed Streaming for Kafka (MSK) for ultra low latency, seamless service-to-service communication, data ETLs, and database Change Data Capture (CDC). Engineers from our Data Platform team will further present this work at AWS’ November 2021 Re:Invent conference.
Abstract
At Coinbase, we ingest billions of events daily from user, application, and crypto sources across our products. Clickstream data is collected via web and mobile clients and ingested into Kafka using a home-grown Ruby and Golang SDK. In addition, Change Data Capture (CDC) streams from a variety of databases are powered via Kafka Connect. One major consumer of these Kafka messages is our data ETL pipeline, which transmits data to our data warehouse (Snowflake) for further analysis by our Data Science and Data Analyst teams. Moreover, internal services across the company (like our Prime Brokerage and real time Inventory Drift products) rely on our Kafka cluster for running mission-critical, low-latency (sub 10 msec) applications.
With AWS-managed Kafka (MSK), our team has mitigated the day-to-day Kafka operational overhead of broker maintenance and recovery, allowing us to concentrate our engineering time on core business demands. We have found scaling up/out Kafka clusters and upgrading brokers to the latest Kafka version simple and safe with MSK. This post outlines our core architecture and the complete tooling ecosystem we’ve developed around MSK.
Configuration and Benefits of MSK
Config:
TLS authenticated cluster
30 broker nodes across multiple AZs to protect against full AZ outage
Multi-cluster support
~17TB storage/broker
99.9% monthly uptime SLA from AWS
Benefits:
Since MSK is AWS managed, one of the biggest benefits is that we’re able to avoid having internal engineers actively maintain ZooKeeper / broker nodes. This has saved us 100+ hours of engineering work as AWS handles all broker security patch updates, node recovery, and Kafka version upgrades in a seamless manner. All broker updates are done in a rolling fashion (one broker node is updated at a time), so no user read/write operations are impacted.
Moreover, MSK offers flexible networking configurations. Our cluster has tight security group ingress rules around which services can communicate directly with ZooKeeper or MSK broker node ports. Integration with Terraform allows for seamless broker addition, disk space increases, configuration updates to our cluster without any downtime.
Finally, AWS has offered excellent MSK Enterprise support, meeting with us on several occasions to answer thorny networking and cluster auth questions.
Performance:
We reduced our end-to-end (e2e) latency (time taken to produce, store, and consume an event) by ~95% when switching from Kinesis (~200 msec e2e latency) to Kafka (<10msec e2e latency). Our Kafka stack’s p50 e2e latency for payloads up to 100KB averages <10 msec (in-line with LinkedIn as a benchmark, the company originally behind Kafka). This opens doors for ultra low latency applications like our Prime Brokerage service. Full latency breakdown from stress tests on our prod cluster, by payload size, presented below:
Proprietary Kafka Security Service (KSS)
What is it?
Our Kafka Security Service (KSS) houses all topic Access Control Lists (ACLs). On deploy, it automatically syncs all topic read/write ACL changes with MSK’s ZooKeeper nodes; effectively, this is how we’re able to control read/write access to individual Kafka topics at the service level.
KSS also signs Certificate Signing Requests (CSRs) using the AWS ACM API. To do this, we leverage our internal Service-to-Service authentication (S2S) framework, which gives us a trustworthy service_id from the client; We then use that service_id and add it as the Distinguished Name in the signed certificate we return to the user.
With a signed certificate, having the Distinguished Name matching one’s service_id, MSK can easily detect via TLS auth whether a given service should be allowed to read/write from a particular topic. If the service is not allowed (according to our acl.yml file and ACLs set in ZooKeeper) to perform a given action, an error will occur on the client side and no Kafka read/write operations will occur.
Also Required
Parallel to KSS, we built a custom Kafka sidecar Docker container that: 1) Plugs simply into one’s existing docker-compose file 2) Auto-generates CSRs on bootup and calls KSS to get signed certs, and 3) Stores credentials in a Docker shared volume on user’s service, which can be used when instantiating a Kafka producer / consumer client so TLS auth can occur.
Rich Data Stream Tooling
We’ve extended our core Kafka cluster with the following powerful tools:
Kafka Connect
This is a distributed cluster of EC2 nodes (AWS autoscaling group) that performs Change Data Capture (CDC) on a variety of database systems. Currently, we’re leveraging the MongoDB, Snowflake, S3, and Postgres source/sink connectors. Many other connectors are available open-source through Confluent here
Kafdrop
We’re leveraging the open-source Kafdrop product for first-class topic/partition offset monitoring and inspecting user consumer lags: source code here
Cruise Control
This is another open-source project, which provides automatic partition rebalancing to keep our cluster load / disk space even across all broker nodes: source code here
ConfluentSchema Registry
We use Confluent’s open-source Schema Registry to store versioned proto definitions (widely used along Coinbase gRPC): source code here
Internal Kafka SDK
Critical to our streaming stack is a custom Golang Kafka SDK developed internally, based on the segmentio/kafka release. The internal SDK is integrated with our Schema Registry so that proto definitions are automatically registered / updated on producer writes. Moreover, the SDK gives users the following benefits out of the box:
Consumer can automatically deserialize based on magic byte and matching SR record
Message provenance headers (such as service_id, event_time, event_type) which help conduct end-to-end audits of event stream completeness and latency metrics
These headers also accelerate message filtering and routing by avoiding the penalty of deserializing the entire payload
Streaming SDK
Beyond Kafka, we may still need to make use of other streaming solutions, including Kinesis, SNS, and SQS. We introduced a unified Streaming-SDK to address the following requirements:
Delivering a single event to multiple destinations, often described as ‘fanout’ or ‘mirroring’. For instance, sending the same message simultaneously to a Kafka topic and an SQS queue
Receiving messages from one Kafka topic, emitting new messages to another topic or even a Kinesis stream as the result of data processing
Supporting dynamic message routing, for example, messages can failover across multiple Kafka clusters or AWS regions
Offering optimized configurations for each streaming platform to minimize human mistakes, maximize throughput and performance, and alert users of misconfigurations
Upcoming
On the horizon is integration with our Delta Lake which will fuel more performant, timely data ETLs for our data analyst and data science teams. Beyond that, we have the capacity to 3x the number of broker nodes in our prod cluster (30 -> 90 nodes) as internal demand increases — that is a soft limit which can be increased via an AWS support ticket.
Takeaways
Overall, we’ve been quite pleased with AWS MSK. The automatic broker recovery during security patches, maintenance, and Kafka version upgrades along with the advanced broker / topic level monitoring metrics around disk space usage / broker CPU, have saved us hundreds of hours provisioning and maintaining broker and ZooKeeper nodes on our own. Integration with Terraform has made initial cluster configuration, deployment, and configuration updates relatively painless (use 3AZs for your cluster to make it more resilient and prevent impact from a full-AZ outage).
Performance has exceeded expectations, with sub 10msec latencies opening doors for ultra high-speed applications. Uptime of the cluster has been sound, surpassing the 99.9% SLA given by AWS. Moreover, when any security patches take place, it’s always done in a rolling broker fashion, so no read/write operations are impacted (set default topic replication factor to 3, so that min in-sync replicas is 2 even with node failure).
We’ve found building on top of MSK highly extensible having integrated Kafka Connect, Confluent Schema Registry, Kafdrop, Cruise Control, and more without issue. Ultimately, MSK has been beneficial for both our engineers maintaining the system (less overhead maintaining nodes) and unlocking our internal users and services with the power of ultra-low latency data streaming.
If you’re excited about designing and building highly-scalable data platform systems or working with cutting-edge blockchain data sets (data science, data analytics, ML), come join us on our mission building the world’s open financial system: careers page.
How we scaled data streaming at Coinbase using AWS MSK was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
