This page is a dated, human-written history of notable operational events at
CashLab: incidents, what they cost, and what changed afterwards. Entries are
written after the fact, once an event is fully understood and
resolved, this is a record, not a live status feed (that's the
status page). We update it whenever something notable
happens, including things that went wrong; honesty about failures is the point.
Routine day-to-day operations are not logged here.
Delegation guide gets its own page; navigation unifiedRoutine
The delegation guide moved from a homepage section to a dedicated
Delegate page, including a widget that reads the
stake currently delegated to CashLab live from the WNat contract, so the
number on the page is the chain's, not ours. At the same time the navigation
bar was made identical across every page of the site, and the sitemap was
completed.
What changed
No operational impact, logged to show the working cadence: site changes ship
in small, verified steps, and every number a page displays should be readable
from the chain by your own browser.
Misconfigured security header broke the site's live widgetsIncident
While verifying a site deploy we found that a content-security-policy
header, set at our CDN rather than in the site's own code, was blocking the
site's own scripts: the live-number widgets on several pages sat stuck at
"checking…" and the mobile menu was dead on sub-pages. No wrong data was ever
shown, the widgets fail toward showing nothing rather than showing something
stale or invented.
Fixed the same day in two steps: page scripts were moved into standalone
files the policy permits, and the header itself was corrected so browsers may
read Flare's public API. Every widget was then re-verified live.
What changed
Header configuration that lives outside the site's code is now documented
alongside it, and deploy verification includes loading each page the way a
visitor's browser does, not just checking that files uploaded.
Failover for BTC and DOGE attestation data, rehearsed liveImprovement
Our FDC attestations for BTC and DOGE depend on full nodes of those chains
that we run ourselves. Applying the epoch 422 lesson (below) before it could
repeat elsewhere, we built automatic failure detection and failover for those
data sources and then rehearsed it live: deliberately pointed the system at a
dead endpoint and watched it detect, rotate, and recover on both chains.
What changed
No outage occurred, this was proactive. Detection of a dead BTC/DOGE data
source went from roughly an hour and a half to a few minutes, and the
recovery path is now something we have actually executed, not something we
believe would work.
Our paid RPC provider's endpoint shut down after our own indexer traffic
burned through the plan's request cap, a billing failure, and our fault for
putting a heavy workload on a metered service. Protocol submissions stopped
for 14 straight hours on August 9. Worse than the outage was the silence: we
had no alarm on the one thing that matters, whether submissions are actually
landing on-chain, and the stall alarms we did have were built to fire once
and never again.
The cost was real: epoch 422, our first reward-eligible epoch, settled at
zero. FTSO accuracy came in at 74.85% against an 80% gate (roughly 89.8%
outside the outage window, the outage alone flipped the gate). Validator
uptime was unaffected (99.999%, separate machine). Full per-gate numbers are
in the epoch report.
What changed
Our own node is now the primary RPC with automatic failover behind it; a
liveness alarm watches actual on-chain submissions and keeps re-firing until
the problem is fixed; and independent off-site monitoring alerts us even if
the main server dies entirely.
A background service died silently and stayed dead for a monthIncident
A routine security review in mid-August found that a background monitoring
service on one of our servers had been dead since July 10: an automatic
operating-system update had killed it mid-upgrade, and nothing noticed. There
was no reward or user-facing impact, but a detection gap of over a month is
the real finding, and we'd rather log it than pretend it didn't happen.
What changed
The service was restored on August 15, and it, like our other critical
background services, now has its own liveness check that restarts it and
alerts us if it dies. The general lesson we now apply everywhere: every
watcher gets a watcher, because a service that dies silently stays dead.
