Reducing DynamoDB read costs by 60% using Sync

As a NoSQL database, StatelyDB exposes all the classic CRUD APIs you’re used to - Put, Get, Delete, and List. But there’s one other core API that we think is a game changer - SyncList. To explain why this is such a big deal, let me revisit how Destiny Item Manager (DIM) runs on StatelyDB, and how utilizing the SyncList API drastically reduced its costs and latency.

In a previous post I explained how I was able to migrate DIM’s data model from Postgres over to StatelyDB. I chose my key paths carefully so that I could retrieve all of a user’s profile data with a single List request instead of the multiple queries I used in Postgres. This is already way more efficient than before, and allows me to add new data types to users’ profiles without having to add more and more queries.

The Problem

There’s still room for improvement. DIM’s client refreshes the user’s profile from its API every few minutes, and every time that happens the API fetches potentially thousands of items. The client polls for changes every few minutes in case the user is using DIM on both their laptop and a phone, so that a change on one device shows up on the other. You might be surprised by how fast this is (less than 50ms), but we still have to pay DynamoDB costs to retrieve all those items, pay for CPU to serialize them all to JSON, and each client has to parse all the returned data. That’s wasted money, wasted bandwidth, and wasted energy. And 99.9% of the time, nothing has changed—we did all that work even though there were no new changes to show. What a waste!

Sync to the Rescue

StatelyDB has a built-in SyncList API because we recognized that this was a common problem. It’s better to ask the server “what’s new?” and for it to respond with “Nothing!” than to re-fetch all of a user’s data. The difference between SyncList and BeginList is that BeginList gets all the data, but SyncList gets only the changes since the last time you got new data. It might respond with “no changes!” which is very easy to handle. Or, it could return some items that have been changed, or the keys of some items that were deleted.

Whenever you list data with the BeginList API, you’re given a token that you can save along with the items it returns. The token is like a “bookmark” that keeps track of where you are in the timeline of the database. You can save this token to disk, or in memory, depending on how you’re keeping track of data. DIM’s client saves all of a user’s profile data to IndexedDB so that a user’s data is always available, even when their device is offline. It saves the list token corresponding to the profile data in the same place.

Whenever the client requests new profile data, it sends the list token with the request:

• If the client doesn’t have any locally-cached data yet, the token is empty, and the API calls BeginList to fetch all the profile data. The client replaces all its local data with the remote data.
• If the API gets a list token, it passes that token to SyncList, which returns only the changed items plus a new token that can be used for the next SyncList call. The client can take this list of changes and merge them in with its cached data—updated items overwrite the local copy, and deleted items are removed from the local copy.

With this pattern in place, clients go from having to fetch thousands of items at a time, to mostly getting zero changes, or occasionally a handful of updates and deletes. This is way faster (down to an average of 20ms per request), saves bandwidth, and saves CPU on both the server and on users’ devices. But best of all, it saves DynamoDB costs—see if you can tell where in this RRU (Read Request Unit) graph I switched the DIM client over to using SyncList:

DIM is now using 60% fewer read units as before! After this change rolled out, I increased the client’s refresh rate from every 10 minutes to every 2 minutes—an improvement in user experience and responsiveness that I couldn’t have done without sync. And my AWS bill is still smaller than before!

How does it work?

If you weren’t using StatelyDB, how would you implement SyncList?

1. First, you would start tracking whenever items were last modified, then add an index on that last modified time.
2. Then you keep track of when you made your initial query (that’s your equivalent to our list token).
3. To sync, you query for items whose last updated time is greater than that timestamp.

That’s not too hard, but the real complication comes when you want to find out if an item has been deleted. You have to include information about deleted items in the sync response, or your local cached data might still show data that’s been deleted remotely. If you just query your database on timestamp, you won’t find out about deleted items, because they’re gone—they’re not going to be returned at all.

To solve this, you’d need to implement “tombstones” 🪦. Whenever an item is deleted, instead of completely deleting it, you need to replace it with a placeholder item that still tracks a last updated time, but just represents that an item used to be there. These tombstones are hard to manage. You probably don’t want to keep them around forever, so you need to clean them up after a while. And you need to filter them out from your other APIs—if you call Get or List, you don’t want to see the tombstones. It turns out there’s a lot of tricky book-keeping involved to make sure these tombstones work, just so you can find out about deleted items in a sync.

All of that tombstone logic would need to be built into your application and repeated for each of your data types. It’s enough of a headache that you might put that work on your backlog, and never get around to it, and end up wasting money fetching too much data. At least, that’s what I did with DIM until I moved it to StatelyDB. And that’s why StatelyDB offers SyncList as a first-class API, and manages all the tombstone logic for you.

Of course, this book-keeping doesn’t come entirely for free. There are certainly more DynamoDB operations than would be required if we didn’t support Sync at all. And while I hope I’ve convinced you that it’s worth it to adopt Sync, we give you fine-grained configurability of which parts of your data model have sync enabled and which don’t, so you can optimize costs.