ClickHouse architecture

Each MCSService custom resource declares one coordinated ClickHouse service data plane: stateful ClickHouse nodes ingest and manage data, managed Keeper coordinates metadata, CoreWeave AI Object Storage (CAIOS) provides durable capacity, and ElasticQuery supplies independently scalable read-only compute for tables that opt in to shared table storage. This object-storage-backed operating model is the ClickLake architecture.

The CR is a control-plane desired-state record. The ClickHouse, Keeper, ElasticQuery, storage, networking, and supporting Kubernetes resources reconciled from it are the data plane. The Runtime Operator is the executor that connects those two boundaries.

The product control plane that creates and operates these components is documented in MCS Control Plane Architecture.

Data-plane topology

ClickHouse nodes are the write and DDL authority. ElasticQuery does not accept writes, mutations, or table administration; it attaches eligible table data read-only.

The service stack is the isolation boundary. It has one runtime namespace, one Keeper root, one service-specific object-storage root, one primary ClickHouse endpoint, and an optional ElasticQuery endpoint. Increasing ClickHouse nodes is a stateful topology change; increasing ElasticQuery nodes adds read-only query capacity.

Storage model

Each service receives an object-storage root. Every ClickHouse node writes through its own stable replica prefix, so two nodes never accidentally own the same physical metadata:

<bucket>/<service-id>/<replica-id>/...

Managed table-disk data uses a table identity below that prefix:

<bucket>/<service-id>/<replica-id>/mcstables/<table-uuid>/

The service ID provides tenant/service isolation, the replica ID separates independent physical copies, and the table UUID gives one storage identity per table lifecycle. Dropping and recreating a table produces a new identity; schema sync removes the stale reader and creates the replacement.

Object storage alone does not make ClickHouse metadata shared. A normal MergeTree table can place blobs in S3-compatible storage while keeping metadata local to one ClickHouse process. ElasticQuery eligibility therefore depends on table-disk semantics, not merely on the disk type.

Table classes

Normal ClickHouse tables

Normal tables use the service’s default storage behavior. They support the ordinary ClickHouse DDL and mutation lifecycle and remain on the ClickHouse service.

They are intentionally not mounted by ElasticQuery.

ClickLake table-disk tables

A MergeTree-family table opts in to ClickLake read sharing with:

SETTINGS table_disk = true

The table must contain a reconstructable inline disk(...) definition in SHOW CREATE TABLE. Schema sync converts that definition to a read-only reader definition on ElasticQuery and enables frequent part refresh.

The table-disk contract keeps the table’s storage metadata with its object storage data. ElasticQuery can recreate a reader from that definition without using Writer-local filesystem metadata.

Replicated MergeTree-family definitions are read as their non-replicated local MergeTree equivalent on ElasticQuery. Keeper replication is a Writer concern; ElasticQuery reads the resulting table storage and does not join the Writer replication group.

Writer clusters

Release configuration exposes dynamic ClickHouse clusters:

• all_shards addresses one endpoint per ClickHouse node and is used for discovery and ON CLUSTER operations across independent Writers;
• all_replicas represents the replica topology for queries that should visit every ClickHouse replica.

Membership comes from runtime discovery rather than embedding Pod membership in the schema-sync script. Scaling the ClickHouse service updates cluster membership without baking node addresses into table metadata.

ElasticQuery schema synchronization

Every ElasticQuery Pod runs schema sync against the active ClickHouse nodes. The loop is idempotent and follows this sequence:

1. Discover active Writers through the dynamic all_shards cluster.
2. Resolve each Writer’s stable replica macro.
3. Read eligible, non-system MergeTree-family tables.
4. Reject table_disk=true tables whose inline disk definition cannot be reconstructed safely.
5. Create one physical reader table per source node in the original logical database:

   events__<replica-id>

1. Convert object storage to read-only mode and set refresh_parts_interval = 1.
2. Create one logical events table using the ClickHouse Merge engine over the compatible node reader tables.
3. Remove operator-managed reader tables and databases that no longer exist in the Writer catalog.

The Console labels the logical table All data and each physical reader Synced from <replica-id>. These labels are presentation metadata; the underlying objects remain normal ClickHouse tables.

Table identity

Writers may create the same logical table through ON CLUSTER. Their logical UUID can be the same while their table-disk endpoint expands the replica macro to distinct physical storage prefixes.

ElasticQuery cannot register the same table UUID repeatedly in one catalog. Schema sync therefore creates a deterministic reader UUID from:

writer table UUID + source replica ID + reader role

This preserves stable reconciliation while keeping every physical reader identity unique.

Query behavior

The logical table is the normal ElasticQuery entry point. The node-suffixed tables are available for diagnostics, data validation, and source-specific queries.

Metadata responsibilities

ElasticQuery is a derived catalog. It must never become the source of truth for table definitions.

Managed Keeper

Keeper coordinates replicated databases, distributed DDL, replica queues, sessions, and health. An MCSKeeper selects an odd-sized quorum, release, resources, and storage. The Operator renders one single-member StatefulSet per Raft member and rolls followers one at a time, verifies quorum health, and updates the leader last.

ClickHouse services reference a managed Keeper fleet but receive their own service-specific Keeper root:

/clickhouse/<service-id>

Keeper health is a prerequisite for Writer coordination. It does not own table data, ElasticQuery reader catalogs, or user schema.

Failure behavior

• Missing Writers or incomplete discovery makes schema sync fail readiness rather than serving an incomplete catalog.
• A table with table_disk=true but no usable inline disk definition is reported as unsupported.
• Schema incompatibility between node-local readers prevents a reliable all-data logical table and must be surfaced.
• Reader databases that are not operator-managed are never overwritten.
• A dropped or recreated Writer table removes the old managed reader during the next sync.
• ElasticQuery remains read-only even though it has its own ClickHouse catalog.

Scaling model

ClickLake separates two scaling axes:

Read-heavy workloads scale ElasticQuery independently. Increasing Writers is a stateful topology change and should not be used merely to add read concurrency.

Current constraints

• Only explicitly opted-in table-disk MergeTree-family tables are synchronized.
• Inline disk reconstruction is required until managed ClickHouse releases can inject the table-local disk definition safely.
• ElasticQuery does not trigger Writer-side materialized views or background ingestion.
• Schema changes originate on Writers and become visible after schema sync.
• Keeper remains a critical dependency for Writer coordination.