Tech Dive on DynamoDB

We'll talk about the history of DynamoDB, it's architecture, downsides and more. Plus, how Dropbox selects data centers, a history of C compilers and more.

May 05, 2024

Hey Everyone!

Today we’ll be talking about

  • Tech Dive - DynamoDB

    • History behind DynamoDB and why Amazon created it

    • The Dynamo Project and SimpleDB

    • DynamoDB Data Model

    • Partitioning Scheme and Durability Guarantees

    • Downsides of DynamoDB

    • Case Studies at Uber, Instacart and Canva

  • Tech Snippets

    • How Dropbox selects Data Centers

    • Why Complexity sells better than Simplicity

    • A History of C Compilers

    • The Fallacy of having Developers Split Time on Multiple Projects

    • Time-based CSS Animations

DynamoDB Tech Dive

DynamoDB is a fully managed NoSQL database available on AWS. It’s proprietary, so the source code is private and it can only be run on AWS infrastructure. Although it’s closed source, AWS has published a bunch of talks and papers describing how it works.

In this article, we’ll be talking about the history behind DynamoDB, it’s data model, pros/cons, details on the architecture and more.

This is the first part of a tech dive we did on DynamoDB.

You can read the full article by subscribing here.

So, why use DynamoDB? Some of the core selling points are:

  • Scalability - DynamoDB is known for it’s ability to scale to extremely high usage while providing predictable performance around latency.

    Amazon “dogfoods” their services heavily and DynamoDB has been able to handle loads of 80 million+ requests per second from Amazon use-cases. It’s also heavily used by companies like Netflix, Zoom, Lyft, Snapchat, Dropbox and more.

  • Database as a Service - DynamoDB has a serverless model where you don’t have to do any maintenance or auto-scaling. You can just use AWS’ API for creating tables, sending reads/writes, managing indexes, etc. and they handle everything internally. DynamoDB will accommodate your workload as it ramps up or down and you’ll be billed per request.

History

DynamoDB is the culmination of learnings from multiple previous projects at Amazon.

The two sources of inspiration were:

  • Dynamo

  • SimpleDB

Dynamo

Dynamo was the first NoSQL database system developed at Amazon. It was a key-value store built for internal use (Dynamo was built in the early 2000s so this was before AWS).

With Dynamo, Amazon was able to gain experience around building highly scalable storage systems with techniques like

  • consistent hashing for partitioning data across multiple nodes

  • object versioning for handling concurrent updates

  • quorums for maintaining consistency amongst replicas

and much more.

They discussed lessons learned in their 2007 paper on the Dynamo system. This paper was crucial to distributed databases and helped to popularize the idea of “eventual consistency”.

(Note - despite the similar names, DynamoDB has a completely different architecture than Dynamo)

One of the main downsides of Dynamo was that it was single-tenant. Each team at Amazon had to manage and scale their own Dynamo installation, leading to a lot of wasted time and inefficiency.

This helped led to SimpleDB.

SimpleDB

SimpleDB was announced in 2007 as a fully managed NoSQL database. The goal was to provide the same scalability and performance of Dynamo with a managed/easier-to-use experience.

Unlike Dynamo, SimpleDB was publicly available as one of AWS’ cloud offerings.

However, there were quite a few issues with SimpleDB:

  • No Consistent Reads - At launch, SimpleDB only provided eventually consistent reads. Consistent reads were added a few years after launch but Amazon discouraged their use due to higher latency and lower throughput.

  • Storage Limitations - SimpleDB had a strict storage limitation of 10 GB per domain. It was meant for small, less-demanding workloads.

  • Lack of Complex Querying and Transactions - SimpleDB didn’t have support for joins or aggregations. It also didn’t provide ACID transaction guarantees.

Despite being very limited, SimpleDB showed the developer demand for a fully managed, highly scalable, NoSQL database.

Amazon took their lessons from Dynamo and SimpleDB and created DynamoDB.

This is the first part of our Tech Dive on DynamoDB.

In the rest of the article, we’ll talk about

  • DynamoDB Data Model

  • Partitioning Scheme and Durability Guarantees

  • Downsides of DynamoDB

  • Case Studies at Uber, Instacart and Canva

You’ll also get access to a PDF of all our recent tech summaries, spaced-repetition flash cards on all the content covered in Quastor and more.

Tech Snippets

How Dropbox Selects Data Centers

This is a fantastic read that delves into all the factors Dropbox considers when they’re finding data centers. They look at things like electricity costs, power redundancy, network speeds, security and more and create a score card.

The article also discusses how Dropbox negotiates leases for its data centers.

dropbox.tech/infrastructure/how-the-data-center-site-selection-process-works-at-dropbox

Simplicity is An Advantage but Sadly Complexity Sells Better

Everyone loves to talk about how they prefer simplicity, but complexity often sells better in the real world. Having lots of complexity signals effort, mastery and innovation. This can lead to a bias towards complex systems.

Rewarding complexity often incentivizes people to create unnecessarily complicated solutions

This is a really insightful article that delves into this phenomena.

eugeneyan.com/writing/simplicity

A History of C Compilers

This is a really terrific history of C compilers and how they’ve evolved.

In 1972, Dennis Ritchie developed the C language and a compiler for the DEC PDP-11. By the early 1980s, there were around 20 C compilers available for various platforms. The open source GNU C compiler (GCC) was released in the late 1980s and went on to become a dominant compiler.

This article delves into the evolution of all of these, why different compilers were created and more.

thechipletter.substack.com/p/a-history-of-c-compilers-part-1-performance

The Fallacy of Having Developers Split Time on Multiple Projects

Managers may sometime assume that if a developer’s time is split between two projects, then each project will effectively get half of the dev’s total time.

However, this overlooks the overhead associated with each project, where you have to read emails, attend meetings and do basic coordination to understand the current state of things. Project overhead will significantly reduce a developer’s productivity. Because of this, it’s usually a bad idea to have developers split across multiple projects.

www.bennorthrop.com/Essays/2022/fallacy-of-splitting-time.php

Time-based CSS Animations

This is an in-depth dive into time-based CSS animations with practical examples and codepen demos. It’s a useful resource if you want to learn about advanced CSS animation techniques.

yuanchuan.dev/time-based-css-animations