Data quality as a contract: building Data Quality Observability Platform at Coinbase

By late 2024 Coinbase was not short of data quality checks. Teams had written thousands into pipelines, DAGs, and service code, and they had already prevented a material amount of potential financial loss. However, incidents kept happening, and the failures had little to do with detection and everything to do with what surrounded it:

When adding a check is expensive, people add them only after an incident, and coverage becomes a map of past failures rather than current risk.

DQOP launched in Q1 2025 as a control plane embedded in our internal data catalog: if you can see a table, you can see its checks, owners, failure history, and score, and add a check to it. Three design principles shaped it.

A check moves through five stages owned by four services. Execution, evaluation, and persistence hand off through Kafka, so any of those stages can be redeployed or backlogged without losing results.

Figure 1. A check's lifecycle. profiling-service appears twice because it both authors the configuration and runs the profiling that produces the metrics; judgement is a separate service.

One detail matters more than it looks: "this check did not pass" is three different events. A config error, such as bad SQL or a renamed column, goes to the check's author. A system error, such as a warehouse timeout, is retried and then pages the platform team. Only a failed expectation is a statement about the data, and checks that couldn't run are excluded from pass rates, so a broken check never quietly skews a quality score.

Figure 2. Run states. The three non-passing outcomes have three different owners, and only one of them is a statement about your data.

Every DQOP check is two configs: a profiling config that says what to measure (SQL, metrics, dimensions, schedule) and a ThirdEye config that says what makes it wrong (expectation function, thresholds, notifications). Picking a check type in the UI loads a template that pre-fills both, so the split costs authors nothing.

Figure 3. One profiling config measures; many ThirdEye configs judge. Because metrics are retained, a check that compares against yesterday reads the stored value instead of rescanning the warehouse.

The split pays off three ways. One measurement backs many checks, so a single query can feed a volume check, a per-region floor, and a completeness check. Thresholds change far more often than measurements, and they can be re-tuned without touching SQL. Most importantly, every run's metrics are kept, which makes historical comparisons nearly free: "today's volume within 20% of yesterday's" reads yesterday's row count from Postgres, and a 30-day trailing average reads thirty stored rows rather than scanning thirty days of a large table. Warehouse spend tracks how many metrics you collect, not how many comparisons you make.

The profiling config also decides how finely to measure. Partition metadata is nearly free but blind to a partition that arrived on time full of wrong values; record-level comparison is exact but surfaces more differences than anyone can triage. Grouping metrics by four to ten dimensions is usually enough to say where a regression lives, so that's where we push people to start.

Figure 4. The profiling granularity spectrum. Grouping by four to ten dimensions is where most issues become both detectable and diagnosable.

DQOP checks two kinds of assets. Table quality checks validate data at rest (freshness, volume, completeness, and business rules such as "settlement never precedes execution") from a Quality tab on every Snowflake and Databricks table. A dry run shows the rows a new check would flag before it can page anyone.

Figure 5. Authoring a table check. The dry run returns the rows the check would have flagged, with the run kept out of production reporting so nobody is paged.

Event quality checks invert the model: the consumer of a streaming event declares what it needs, field by field (present, typed, non-empty, in range, within a tolerated null-rate drift), and the platform compiles those declarations into one query and one alert with per-field attribution.

Figure 6. The event quality rule builder. A consumer toggles what they need per field; the platform generates the query.

Each asset family also gets a score. A table's score combines Health (pipeline reliability and check pass rate), Enrichment (metadata and check coverage), and Importance (downstream usage). A table with no checks scores badly rather than neutrally, and scores roll up by schema or tag into one view per data product.

Figure 7. Two asset families, two scoring models. The Event Quality Score spreads 100 points unevenly across seven dimensions, and that weighting decides which failures it can detect.

Events get an Event Quality Score from 0 to 100 across seven dimensions: metadata, freshness, and volume stability at 20 points each, and delivery latency, duplicate rate, null density, and adoption at 10 each. Each criticality tier has a gate, and a score below it alerts the producer: 80 for Tier 0, 60 for Tier 1, and 40 for Tiers 2 and 3.

Figure 8. The Event Quality Score panel. With six dimensions full and one degraded, the composite reads 96 against a Tier 0 gate of 80, the case examined in the next section.

Before relying on a composite score, do the arithmetic on what it can catch. Take one field in an otherwise healthy event that starts arriving as an empty string, climbing from 0.01% to over 30% of records. Three properties work against detecting it.

Figure 9. A total collapse of a 10-point dimension leaves the score at 90, above the Tier 0 gate of 80. Three 10-point dimensions have to fail at once to drop below it.

So we split the roles. Consumers declare field-level expectations, such as a not-empty or null-rate-drift rule, which evaluate the continuous per-field rate and catch this case directly. The composite still answers "how healthy is this overall" and still gates broad degradation, but it isn't what catches a single broken field.

We tell people to profile before they assert, and for most tables the platform already holds ninety days of per-column profiling history. Checks are structured documents, so something other than a human can write them. Put together, the next authoring interface is a prompt: a user describes what they care about, and the platform proposes checks, each justified by the profiling evidence behind it, dry-run before it is shown, and activated only after a human accepts it. Alongside that, we're working on lineage-aware alert routing, assisted incident triage, and adaptive thresholds.

Figure 10. Where check authoring is going. The user states what they care about; the platform proposes checks justified by profiling history, dry-run before they are offered.

Two lessons transfer if you're building something similar:

The composite score is also getting the job it is best at. We're putting together a dataset certification program: a badge in the catalog that answers one question about a table, whether it is trustworthy enough to depend on, earned through criteria for ownership, documentation, and data quality. DQOP supplies the quality evidence. Each table in the program gets freshness, uniqueness, completeness, and volume checks, created from templates on its certification tab and run daily against the production table consumers actually query, and owners can tag their own checks to count as well. Those checks start as advisory, so we can measure real pass rates before they gate a badge, and the composite quality score becomes a certification signal alongside them. It is the split the arithmetic above argued for: checks detect, and the score summarizes whether a table is still earning its badge.