High-Cardinality Telemetry Without the Bill: OpenTelemetry on ClickHouse

Every incident review eventually asks the same question: which customers were affected? If the telemetry does not carry a tenant or customer ID, nobody can answer it quickly. Yet tenant ID is often the first label teams remove, because of how many observability tools bill.

The cardinality math

In a time-series database, every unique combination of metric name and label values is a separate series. The theoretical number of series is the product of each label’s distinct values.

http_server_request_duration
  service      ×    40
  route        ×   150
  status_code  ×     8
  region       ×     4
  ─────────────────────
  = 192,000 possible series

add tenant_id  × 5,000
  = 960,000,000 possible series

Real systems never hit every combination, but even a small fraction of that is far beyond what per-series or per-custom-metric pricing can absorb. So teams drop tenant_id, sample traces aggressively, and lose the detail they need most. You can estimate your own numbers with our free cardinality calculator.

Why a columnar store changes the model

ClickHouse stores telemetry as rows in a columnar table rather than as pre-aggregated series. A high-cardinality attribute is just another value in a column, compressed alongside millions of similar values. Its cost is the bytes it adds after compression, not a new billable series. That moves cardinality from a pricing problem to a capacity-planning problem you control.

CREATE TABLE otel_traces
(
    Timestamp      DateTime64(9) CODEC(Delta, ZSTD(1)),
    TraceId        String CODEC(ZSTD(1)),
    SpanId         String CODEC(ZSTD(1)),
    ParentSpanId   String CODEC(ZSTD(1)),
    ServiceName    LowCardinality(String) CODEC(ZSTD(1)),
    SpanName       LowCardinality(String) CODEC(ZSTD(1)),
    Duration       UInt64 CODEC(ZSTD(1)),
    StatusCode     LowCardinality(String) CODEC(ZSTD(1)),
    SpanAttributes Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
TTL toDateTime(Timestamp) + INTERVAL 14 DAY;

• LowCardinality(String) dictionary-encodes columns like service and span name, which have few distinct values.
• The ORDER BY key puts spans for the same service and operation next to each other, which is what makes compression and range scans efficient.
• A bloom filter index on TraceId makes “show me this trace” fast without adding it to the sort key.
• TTL deletes old partitions automatically, so retention is a line of SQL rather than a pricing tier.

Asking the incident question

With tenant.id kept as a span attribute, the question from the incident review becomes one query:

SELECT
    SpanAttributes['tenant.id']           AS tenant,
    count()                                AS requests,
    countIf(StatusCode = 'Error')          AS errors,
    quantile(0.95)(Duration) / 1e6         AS p95_ms
FROM otel_traces
WHERE ServiceName = 'checkout'
  AND Timestamp >= now() - INTERVAL 1 HOUR
GROUP BY tenant
ORDER BY errors DESC
LIMIT 20;

The ingestion pipeline

1. Applications emit OTLP over gRPC or HTTP to an OpenTelemetry Collector.
2. The Collector batches, applies memory limits, and optionally tail-samples (for example, keep every error trace and 10% of successful ones).
3. Batches are written to Kafka, which absorbs spikes during deploys and incidents and lets you replay data after an outage.
4. Consumers write to ClickHouse in large batches. ClickHouse prefers fewer, bigger inserts; thousands of tiny inserts per second create too many parts to merge.
5. Materialized views roll spans up into per-minute RED metrics (rate, errors, duration) for dashboards and long retention.

The trade-offs

This is not free. Someone has to run Kafka and ClickHouse, plan disk, and watch merges. Schema changes need care. A managed SaaS tool is often the right choice for a small team with modest data volumes. The economics flip as telemetry grows, as cardinality grows, or as data residency rules make sending telemetry to a third party difficult.