AssetPanda

The constraint everything else followed from

DynamoDB is fast and predictable when you ask it for data the way it wants to be asked — by key. Step outside that and the cost model turns against you: filtering on arbitrary attributes means scanning, and a scan gets slower and more expensive as the table grows, not as the result set grows. A query that is fine in staging quietly becomes a problem in production.

That single property shaped all three pieces of work below. Each feature had to either fit the key model, or grow its own read model designed for the access pattern it actually needed. Deciding which — per feature — was most of the design work.

Access control that runs before the query, not after

The straightforward way to build permissions is to fetch the records, then filter out the ones the caller isn't allowed to see. It produces correct answers and it is the wrong shape. You pay to read rows you are about to throw away, pagination becomes incoherent — a page of 20 can return 3 — and every list endpoint drifts toward a scan.

So the permission check happens before the data access. Each request resolves the caller's grants once, up front, and those grants are then used to construct the query rather than to filter its results. What comes back is already the set the caller is entitled to.

Enforcement lives in one place at the API boundary rather than being repeated inside each handler. That matters less for the endpoints that exist today than for the ones added six months from now: a new endpoint is covered by default, and forgetting the check has to be a deliberate act rather than an oversight.

Resolving grants once per request is also what keeps this off the N+1 path. The alternative — checking permissions per record as you walk the result set — is the version that looks fine on a page of 20 and falls over on a page of 2,000.

Search: a second read model, fed by the write path

Search is exactly the access pattern DynamoDB is bad at. Users want to search across fields, combine filters, and sort by things that aren't the sort key. There is no key-shaped answer to that, so search got its own read model in OpenSearch, kept current by the write path itself:

1. A write commits to DynamoDB.
2. DynamoDB Streams emits the change.
3. An indexing consumer transforms the record and upserts it into OpenSearch.

New data becomes searchable within seconds, with no rebuild step, and search traffic never touches the primary table. The interesting part isn't the happy path — it's the three properties of Streams that dictate how the consumer has to be written:

• Delivery is at-least-once. The same change can arrive twice, so indexing has to be idempotent. Upserting the whole document against a stable id is safe; anything that reads-then-modifies, like incrementing a counter, is not. Applying the same event twice must land in the same place.
• Ordering is per partition key, not global. Two updates to the same record can't overtake each other, which is the guarantee that actually matters. Updates to different records have no ordering relationship — which is fine, because nothing about a search index depends on one.
• Retention is 24 hours. That is the replay window, and it sets a hard operational deadline: a bug in the indexer has to be caught and fixed inside a day, or recovery stops being a replay and becomes a full reindex. Which is a very different afternoon.

Living with the lag

A second read model is eventually consistent by construction: there is a window between a write committing and it appearing in search. Pretending otherwise is how you end up with users who just saved something and can't find it.

The way out is to be deliberate about which store answers which question. Search answers discovery — find me things matching this. Detail views read from DynamoDB directly, by key, where the data is strongly consistent. A user who just saved a record and opens it sees their own write immediately; the search index catches up behind them.

When the index is unhappy

Indexing failures retry with backoff, and what still won't go through goes to a dead-letter queue rather than being dropped — a failed index is a repairable problem, but only if you kept the event.

The more important decision is what a broken index does to everything else: nothing. Indexing sits behind the write, not in front of it, so OpenSearch being slow or down degrades search while writes carry on landing normally. The alternative — writing to both stores inside the request — means a search cluster problem becomes a total outage, and buys consistency you don't need for a search box.

Pipelines that fail small

Bulk imports and multi-step data jobs are where the batch-versus-record distinction earns its keep. Written as a single long-running process, one malformed row anywhere in the file ends the run. You restart from the top, you have no idea how far it got, and the only evidence is whatever made it to the logs.

Running these through Step Functions changes the failure unit. Each stage declares its own retry policy with backoff, and explicit catch handlers route failures to defined error states rather than letting an exception unwind the whole thing. A bad record fails its own execution; the rest of the batch keeps moving.

The operational win is as large as the correctness one. Every execution keeps its own history — which step failed, with what input, after how many retries — so diagnosing a failed import is reading a state machine's execution history rather than grepping logs for a correlation id and hoping.

What it added up to

• Data searchable within seconds of a write, with no rebuild step.
• Search traffic isolated from the primary table, so one can't degrade the other.
• Search-cluster problems degrade search only — writes keep working.
• Import failures scoped to a single record, visible in execution history, and replayable.
• Permission checks that a new endpoint inherits by default.

Stack

• NestJS
• Go
• DynamoDB
• DynamoDB Streams
• OpenSearch
• Step Functions