building roazon · incident note
when monitoring became our biggest database user
a free-tier warning looked like a traffic problem. it was mostly our own Grafana dashboard, repeatedly asking one small database far too many expensive questions.
the warning
On 25 August, Cloudflare emailed us that D1 would begin actively enforcing the Workers Free daily limits on 1 September 2026. Roazon had no budget for a paid plan, and the account had just read 6,017,746 rows in one UTC day—past the 5 million free allowance.
The application was not suddenly popular enough to explain that number. The first useful move was to stop guessing from request counts and read the account-wide billing dataset. Cloudflare counts rows scanned, not just rows returned; a query can return one result and still pay for a table scan.
The bug was not “D1 is expensive.”
Our observability path was doing more database work than the product behavior it was supposed to observe.
what actually happened
Grafana polled a broad /api/metrics endpoint every minute. One HTTP request fanned out into dozens of aggregate statements across search, click, onboarding, map, and product telemetry. Most panels needed only one small slice, but each refresh rebuilt the whole dashboard response.
Several time filters also wrapped indexed timestamp columns in a function:
WHERE substr(created_at, 1, 10) >= ?
That expression made the predicate easy to read and hard for the query planner to index. Repeating broad aggregates and scans every 60 seconds turned monitoring into the dominant workload: 99,682 read queries on the baseline day.
the smallest fixes that mattered
- Poll every five minutes. The dashboard did not need minute-level freshness.
- Ask for one section at a time. Alert rules now call narrow views such as health or daily buys instead of rebuilding the full metrics document.
- Cache identical metrics responses. Repeated requests inside a short window reuse the edge response rather than rerunning D1 aggregates.
- Make time predicates indexable. Range comparisons keep the raw timestamp column visible to the planner.
- Add only the indexes the hot queries proved they needed. We ranked statements by total rows read and checked their plans instead of indexing every column.
WHERE created_at >= ? AND created_at < ?
This was not one magic index. Frequency, endpoint shape, caching, predicates, and indexes multiplied each other. Fixing the most repeated path first produced the useful result.
the result
The next two complete UTC days averaged 75,326 rows read—a 98.7% reduction from the 6,017,746-row baseline. Rows written fell by about 81% as the same cleanup removed unnecessary telemetry churn.
| UTC day | rows read | rows written | read queries |
|---|---|---|---|
| 25 Aug · before | 6,017,746 | 14,819 | 99,682 |
| 26 Aug · after | 70,314 | 2,812 | 21,665 |
| 27 Aug · after | 80,338 | 2,852 | 23,962 |
Those are account-wide Cloudflare billing rows from complete days, not a sample from one query or a rolling dashboard estimate.
how we know it stays fixed
A daily zero-cost monitor reads the previous complete UTC day from Cloudflare's d1AnalyticsAdaptiveGroups GraphQL dataset. It does not query D1, so the guard cannot create the usage it measures.
- warning: 2.5 million rows read or 50,000 written
- critical: 4 million rows read or 80,000 written
- hard free limits: 5 million rows read and 100,000 written
The job runs on our existing self-hosted runner, preserves its raw response and normalized report, and maintains one GitHub issue: it opens or updates on warning, critical, or unknown data, then closes itself after a healthy day. “No analytics returned” is unknown, never a reassuring zero.
what we would check first next time
- Use billing rows, not HTTP traffic, as the first signal.
- Rank queries by total rows read. A moderately bad query run constantly beats a terrible query run once.
- Inspect every dashboard refresh interval and every endpoint fan-out.
- Run
EXPLAIN QUERY PLAN; look forSCANwhere a targetedSEARCH ... USING INDEXshould be possible. - Verify improvement over complete UTC days and automate the regression threshold.
The useful lesson is mundane: monitoring is production traffic. Give it a budget, make it query only what it displays, and measure the monitor itself.