21 provinces went empty: the guard did not block bad data, it tripped over its own history
The incoming data was healthy the whole time. What stopped it being written was the guard’s own output from the day before.
The symptom was one sentence: “there are no pharmacy records in İstanbul today.” During on-duty hours. When measured, 21 provinces were empty — İstanbul, Ankara, İzmir, Antalya and Bursa among them. Users were seeing “no records found”.
The first elimination was easy: the other 61 provinces were fine. A provider outage would have emptied all of them. So the problem was on our side.
Why the guard existed
The data arrives through a daily fetch from the pharmacists’ chambers, and the provider sometimes returns a half list. Writing a half list means dropping a pharmacy that is genuinely on duty — so there is a guard: if the incoming record count falls below half of a reference, the write is skipped for that day.
The reference was “the previous day”. That was the bug.
Root cause
The day before was 30 August, a public holiday, and two things coincided:
- Two fetches ran that day (06:15 and 16:45).
- On holidays the duty shift rotates during the day, so two different shifts were written under the same duty date.
The result: İstanbul’s count for that day became 285 instead of 130. The next day’s perfectly normal 131 records fell under the 50% threshold of that inflated reference, looked like a collapse, and were skipped.
İstanbul’s last five days read: 130 · 130 · 130 · 131 · 129. The incoming data had been healthy the entire time.
The guard did not block bad data. It tripped over a single-day anomaly in its own history and blocked good data instead.
Any threshold anchored to a single sample will, sooner or later, start guarding against the truth.
Immediate response — data, not code
Production had to be fixed before any code change. The morning batch from 30 August (691 rows across 21 provinces) was backed up to a side table, deleted, the fetch stamp cleared, and a fetch forced.
Result: 1332 records across 82 provinces, nothing skipped. Verified from the live API — İstanbul 131, Ankara 49, İzmir 72, Antalya 39, Bursa 34.
Permanent fix — the median of the last five days
const recentDays = await prisma.dutyEntry.groupBy({
by: ['dutyDate'],
where: { provinceCode: code, dutyDate: { lt: dutyDate } },
_count: { _all: true },
orderBy: { dutyDate: 'desc' },
take: 5,
});
const counts = recentDays.map((d) => d._count._all).sort((a, b) => a - b);
const lastCount = counts.length > 0 ? counts[Math.floor(counts.length / 2)] : 0;A median swallows a single-day anomaly completely: a double fetch, a provider hiccup, a holiday rotation. A mean would still carry one fifth of the inflation — shrinking the problem without solving it.
The guard still works: a genuinely half-sized list is still low against a five-day median.
To verify, the deleted batch was restored from the backup and the incident reproduced exactly:
| Province | Incoming | Old reference | New reference | Old rule | New rule |
|---|---|---|---|---|---|
| 34 | 131 | 285 | 130 | skipped | passes |
| 35 | 72 | 147 | 71 | skipped | passes |
| 06 | 49 | 104 | 51 | skipped | passes |
The worst part was that it was silent
The worker logs “skipped” and exits successfully. No error alarm, monitoring stays green, the metrics look normal. Diagnosis depended entirely on somebody opening the app and noticing.
The right monitoring point is this: alert when the record count is zero in the large provinces. “The job finished successfully” and “the job did the right thing” are not the same statement.
What to take from it
- Do not anchor a threshold to a single sample. Yesterday is not a reliable measure of what today should be.
- Holidays carry a double-shift risk. Two shifts under one date does not mean the data is wrong — but no count-based guard should use that day as its reference.
- A guard that references its own output is a feedback loop. If today’s output sets tomorrow’s threshold, one inflated number corrupts the next day too.
- “Skipped” is not an error, it is a decision — and decisions need monitoring, not just logging. A system that quietly fails to do the right thing is more dangerous than one that loudly does the wrong thing.
Eczanow