A weather network, and the database underneath it
Ten weather stations streaming every second, plus a storage engine I wrote from scratch to keep up with them.
- Role
- Wrote the storage engine and the streaming pipeline.
- Built with
- Java, Spring Boot, Kafka, Kafka Streams, Parquet, Elasticsearch, Kubernetes
- Outcome
- The latest reading for any station comes back in a single disk read, and it survives crashes.
The mess
Ten stations each send a reading every second. Someone needs the latest status of every station right now, someone else needs the full history to analyze later, and someone needs to know the moment it starts raining.
What I built
Readings flow through Kafka into a central station that splits them three ways. A rain detector flags anything above 70% humidity in real time. Full history gets archived to Parquet files and indexed into Elasticsearch for dashboards. The latest reading per station goes into a key-value engine I wrote from scratch.
the storage engine is the fun part ↓
The engine
It’s a Bitcask: every write goes to the end of a file, and an in-memory map remembers where each station’s latest value lives. A read is one lookup and one disk read, however much history piles up. A background job compacts old files without ever blocking reads.
What happened
Under profiling, the longest garbage-collection pause was about 50 ms, which is well clear of anything that would upset Kafka. The same system runs with Docker Compose on a laptop or on Kubernetes.
Decision log
Every fork in the road, and which way I went.
- 01
Where should the latest reading per station live?
ruled out: an off-the-shelf key-value store- chose: a Bitcask engine written from scratch
Append-only writes and one lookup per read fit "the latest value per station" perfectly. Writing it myself was the only way to really know what crash recovery costs.
- 02
How do you clean up old data without stopping reads?
ruled out: lock everything while compacting- chose: compact only sealed files and swap pointers carefully
Reads never wait. A pointer only moves if it still points at the old file, so a fresh write that lands mid-cleanup is never overwritten.
the bug that never happens, on purpose.
- 03
How fast should it come back after a crash?
ruled out: rescan every file on startup- chose: write small index files as it goes
Startup reads the small index files and only scans the one file that was still being written.