Windowed High Cardinality
Same tumbling 30s window and count + avg(latency_ms) aggregates as the low-cardinality bench, but grouped by user_id (~100k keys) — one open window per active user. Stresses the keyed-state store rather than framework overhead.
Last updated: 2026-06-18 12:57 · same-machine, relative comparison.
Bench characteristics
In-process computation — Tumbling 30s event-time window, GROUP BY user_id (~100k keys) → count + avg(latency_ms); one pane per key per window close.
| product | delivery | processing | engine | partition parallelism | state |
|---|---|---|---|---|---|
| turbine | at-least-once | batch (Arrow columnar) | in-process · Rust core + free-threaded CPython 3.14t | intra-partition (3 shards, hash-keyed) | own state store (msgpack on disk) |
| flink | at-least-once | batch (pipelined, JVM code-gen SQL) | in-process · JVM (PyFlink local MiniCluster, parallelism=1) | none (one task slot / partition, parallelism=1) | heap (HashMapStateBackend) |
| quix | at-least-once | row (per-record StreamingDataFrame) | in-process · pure-Python over librdkafka (C client) | none (one consumer per partition) | RocksDB (embedded) |
| risingwave | at-least-once | batch (vectorized SQL) | external server · Rust (single-node: meta+compute+frontend) | none (one source reader per partition) | server-managed (Hummock LSM store) |
All products are benched at at-least-once (each one's default): Turbine, Quix, RisingWave and Flink can opt into exactly-once (Kafka transactions / a transactional sink) but the bench keeps the default, so the comparison is iso-semantics. Input is a single Kafka partition (lz4), so partition parallelism is what each product can extract from that one partition; windows are tumbling 30 s event-time.
Throughput / resource usage
| product | offered_in/s | consumed_in/s | output_rate | cpu_avg | cpu_p95 | rss_avg_mb | rss_max_mb | run |
|---|---|---|---|---|---|---|---|---|
| turbine 0.1.0a5 | 200,000 | 195,601 | 13,214.36 | 118.2 | 121.0 | 2,252.4 | 2,503.4 | 2026-06-17 22:24 |
| flink 2.2.1 | 350,000 | 344,845 | 23,115.48 | 167.7 | 182.7 | 3,555.5 | 3,830.9 | 2026-06-18 12:57 |
| quix 3.23.6 | 27,000 | 26,504 | 1,776.05 | 78.7 | 84.1 | 526.6 | 556.5 | 2026-06-16 10:21 |
| risingwave v2.8.4 | 200,000 | 197,517 | 15,021.91 | 311.9 | 495.9 | 6,408.2 | 7,038.0 | 2026-06-16 16:14 |
output_rate is the window-emission cadence (≈ keys ÷ window size), not a throughput figure — it only moves when windows close, which needs the consumer to keep up with offered_in/s. Compare CPU/RSS; see the README on benching windowed.
flink — PyFlink local MiniCluster, parallelism=1, heap state (HashMapStateBackend), at-least-once checkpointing. CPU/RSS include the JVM child (see the stateless note); unlike single-core lowcard, the keyed aggregation parallelizes ACROSS cores (~1.7 cores at the 350k knee). Single-partition knee ≈ 350k (consume ≈ offered, lag ≈ 0); 400k decouples (consume ~381k, lag +1.7M).
Sanity — are we benching the same thing?
| product | output_rate | expected | sample |
|---|---|---|---|
| turbine | 13,214.36 | 13,333.00 | 500 |
| flink | 23,115.48 | 23,333.00 | 500 |
| quix | 1,776.05 | 1,785.00 | 500 |
| risingwave | 15,021.91 | 13,333.00 | 500 |
Aggregated output: checked against the expected pane cadence (expected.output_rate in bench.yaml), not per-record selectivity.
Verdict: ✓ matches theory — output_rate within 2× of the expected pane cadence.