What is Real-time Processing?
Real-time processing is a freshness SLO: the maximum time from an event happening to when a normal query can see the correct, up-to-date record.
What “queryable” means (in practice)
- The event is stored durably (it won’t disappear).
- It passed basic checks (schema validation, de-duplication).
- It’s visible as a complete write (readers see all of it or none of it).
- If your endpoint depends on them, any required indexes or materialized views have updated.
Where that time goes (end-to-end)
- Ingest: get the event from the source or CDC.
- Transform: light validation/enrichment only.
- Commit/visibility: make the write atomically readable.
- Query-ready: update indexes/MVs when the endpoint needs them.
Targets that usually work
- 0.1–1 s: key-based lookups and gates (fraud checks, rate limits, feature flags).
- 1–15 s: live ops dashboards, promo pacing, anomaly alerts, inventory signals.
- 15–60 s: near-live KPIs and partner views.
Most dashboards feel great at 5–30 s of freshness.
The key principle
Real time is not “we use streaming.” Streaming is a technique; real time is an outcome. You get it by combining:
- Fast, durable ingest.
- Low and predictable tail latency on reads (design to P95/P99, not averages).
- On-demand context: join new events with the right slice of history so decisions use both recency and memory.
How to make it measurable
- Write the SLO: for example, “95% of events are queryable in ≤10 s; 99% in ≤30 s.”
- Track freshness as event_time → first_queryable_time.
- Monitor ingestion lag, MV/index refresh time, and commit visibility delay.
- Isolate workloads (interactive vs. exploration) so spikes don’t break your P95/P99.
Bottom line: pick the freshness you need, prove it with metrics, and keep the pipeline only as complex as necessary—that’s real-time.