They focused particularly on audit logging data Edgestore’s top cold metadata use case. “If you’re writing data that’s not meant to be read often, it’s extremely expensive to use-not to mention pointless to store-in mediums that are optimized for retrieval speed,” says Jonathan Lee, tech lead for Dropbox’s Alki team.Īs a result, two employees split off from the database team to build Alki, the solution that would cost-effectively store metadata. Additionally, it no longer made sense to store cold metadata in the same database as hot-or frequently use -metadata. Yet increasing the capacity of the on-premises database would require splitting existing partitions and buying new machines to host them, which would double the cost of Edgestore by adding millions of dollars per year. By mid-2018, the rapidly growing cold metadata-data that is accessed infrequently but needs to be stored durably and available instantly-was less than 2 years away from overwhelming Edgestore. It has become one of the most successful startups in the world, with over 600 million users uploading more than 400 billion pieces of content.ĭropbox’s metadata stores were originally housed solely within the company’s main data store, Edgestore, hosted in an on-premises distributed database built on top of sharded MySQL clusters. This made room for virtually unlimited user metadata and not only saved the company millions of dollars-since it would not have to increase on-premises storage-but also reduced the cost per gigabyte by a factor of 5.5.įounded in 2007 by two Massachusetts Institute of Technology students, Dropbox is a global collaboration tool and file sharing service. Using Amazon DynamoDB, a fully managed, flexible NoSQL database that delivers single-digit millisecond performance at any scale, and Amazon Simple Storage Service (Amazon S3), a cloud object storage service, Dropbox rapidly developed a new managed storage system called Alki. Those circumstances pushed Dropbox to pursue a managed solution from Amazon Web Services (AWS).
Dropbox had less than 2 years until its on-premises system would reach maximum capacity, and the implementation team for the project was made up of just two employees. The third option was the best, but achieving it would be a challenge. Dropbox’s database team had three choices: double the on-premises storage capacity (which would cost millions of dollars), delete swaths of metadata, or find a new, highly scalable yet cost-effective solution. In the summer of 2018, Dropbox experienced a capacity crunch in its on-premises metadata store due to fast data growth in some of the partitions.
0 kommentar(er)
