Testing

MCS needs more than controller unit tests. Its correctness depends on several systems agreeing about desired state: the control plane, Kubernetes, the MCS Operator, ClickHouse, Keeper, object storage, and ElasticQuery. The test strategy therefore uses progressively more realistic layers and keeps the expensive failure tests out of the fast pull-request path.

This document separates three states explicitly:

• automated today: checked-in tests that run without a person interpreting each step;
• manual today: required validation that an engineer can perform, but that does not yet have a reliable automated runner;
• automation target: the CI tier where each manual test must eventually run.

A command sequence written in a runbook is still manual testing if a person must provision the environment, choose the next action, or decide whether the result is correct.

Product modules and test ownership

MCS is one product, but it crosses several independently failing modules. A test result is meaningful only when it identifies which module it proves.

The two most important boundaries are:

1. management control-plane tests stop after proving the correct CR or directly owned bootstrap resource;
2. Runtime Operator tests start from that CR and prove the resulting runtime resources.

Only a live end-to-end test proves the whole path from a user action to a SQL result.

Test principles

1. Test policy and business behavior below the HTTP layer.
2. Test Kubernetes reconciliation against a real API server when possible.
3. Render every generated resource and Helm chart deterministically.
4. Use a real Kind cluster for scheduling, pod lifecycle, networking, storage, ClickHouse, Keeper, and ElasticQuery behavior.
5. Treat failure recovery as a tested product contract, not an operator assumption.
6. Keep test resources isolated by namespace, service ID, replica ID, and object-storage prefix.
7. Preserve logs, events, rendered manifests, and SQL evidence whenever an end-to-end run fails.

Test levels and current status

The main gap is not another list of manual checks. It is a reusable L3-L5 test harness that can provision the topology, execute actions, assert results, collect evidence, and clean up without human decisions.

Coverage ownership

The complete MCS Control Plane contains two execution domains. The management control plane and Runtime Operator currently ship from the same repository and may run in the same process, but they have different correctness boundaries. Tests must follow those boundaries:

• the management control plane proves that an authenticated user action becomes the correct desired-state object;
• the Runtime Operator proves that a desired-state object becomes the correct, healthy Kubernetes and data-plane resources;
• internal contract and end-to-end tests prove that the CR and status exchanged between these two parts of the Control Plane remain compatible.

Avoid proving Operator reconciliation through Console page tests, and avoid retesting form parsing in controller tests.

Management control plane key points

The management control-plane suite should stop at the desired-state boundary. It asserts the CR and directly owned bootstrap resources, but it does not claim that a ClickHouse StatefulSet successfully started.

Runtime Operator key points

The Runtime Operator suite starts from valid and invalid CRs. It must not depend on a human using the Console to create those fixtures.

Internal contracts and end-to-end points

The boundary between the two components needs a smaller set of high-value contract tests:

1. every management control-plane create/update action produces a CR accepted by the current CRD and Runtime Operator;
2. the Runtime Operator reports status and conditions the management control plane can project without special-case guessing;
3. field names, defaults, release identifiers, Secret references, and ownership rules remain compatible;
4. deletion requested through the management control plane reaches Runtime Operator finalization and removes the expected owned resources;
5. one end-to-end happy path proves user action → CR → reconciliation → Ready → SQL result;
6. one end-to-end failure path proves reconciliation error → CR condition → visible Control Plane error.

These tests detect integration drift without making every management control-plane test wait for pods or every Runtime Operator test start an HTTP server.

Automated today: unit and integration coverage

Go unit and service tests

The Go suite is organized around product modules rather than UI pages. It covers:

• API validation and defaulting;
• management control-plane service behavior and Kubernetes repository adapters;
• authentication, organization membership, cross-organization access, and administrator-only operations;
• ClickHouse service, organization, Keeper, and service-user controllers;
• ClickHouse and ElasticQuery resource generation;
• Runtime Operator status, reconciliation, ownership, and idempotency;
• release definitions and generated-resource examples;
• naming, Kubernetes apply helpers, pod helpers, and storage helpers;
• Console routes, authorization behavior, forms, SQL execution, monitoring, documentation, and query helpers.

Run the repository suite with:

make test

make test regenerates CRDs and deepcopy code, runs formatting and go vet, starts the Kubebuilder test control plane, executes all Go tests, and writes cover.out.

The package-level ownership is:

These tests intentionally mock or omit the scheduler, containers, DNS, networking, object storage, and real ClickHouse SQL. Those belong to the post-install flow, not to a unit-test assertion.

Envtest integration tests

Controller and Console packages use Kubebuilder envtest. These tests run against a real Kubernetes API server and etcd with the generated MCS CRDs. They are appropriate for:

• API validation and persistence;
• controller watches and reconciliation;
• status updates and ownership;
• repository read/write behavior;
• HTTP handlers that depend on Kubernetes objects.

Envtest does not provide a scheduler, kubelet, CNI, CSI, real pods, Services, or object storage. A passing envtest suite cannot prove workload readiness, anti-affinity, rolling-update safety, or ClickHouse query behavior.

Generated-resource tests

Release example tests compare generated Kubernetes objects with checked-in expected YAML. These tests catch unintended changes to StatefulSets, Deployments, Services, ConfigMaps, Secrets references, probes, resources, storage settings, and topology.

When an intentional resource change is made, review the complete diff before updating the expected output.

Helm validation

Validate both charts with:

make chart

This lints and renders the operator chart and bootstrap chart. CI also regenerates manifests and fails when generated chart artifacts differ from the committed files.

Current CI gates

The pull-request workflow currently runs:

• make lint;
• manifest, generated-code, and chart verification with a clean Git diff;
• the Go test and coverage workflow;
• release and image/chart publication only after the validation jobs pass.

These are necessary pull-request gates, but they are not a full live-cluster end-to-end test.

What automated tests do not prove

A passing make test does not prove any of the following:

• that a Pod can be scheduled or an image can be pulled;
• that Keeper elects a leader or retains quorum;
• that ClickHouse accepts SQL over a real Service;
• that CAIOS credentials and virtual-hosted addressing work;
• that a managed table disk is readable from ElasticQuery;
• that schema sync converges after a table is dropped and recreated;
• that a rolling update remains available during a fault;
• that the browser workflow is understandable to a user.

Those checks require the post-install acceptance flow below. Until that flow is automated, the engineer running it is responsible for recording the observed results.

Manual today: required live-cluster testing

The canonical local topology is described in Local installation. End-to-end runs must use the same Helm chart and values structure as deployment; ad-hoc manifests are not a substitute.

The scenarios in this section are product requirements, but they are not currently a complete automated CI suite. Until they are automated, an engineer must run them before relevant releases and retain the evidence described below.

Post-install acceptance walkthrough

Use this section after completing Local installation. It is the minimum product acceptance test: it proves identity and provisioning through the Console, real ClickHouse DDL and inserts, object-backed managed table disks, and ElasticQuery schema sync.

1. Confirm the installed runtime

Before opening the Console, verify that there are no pending or crashing workloads:

kubectl get nodes
kubectl get pods --all-namespaces
kubectl get mcskeepers.data.coreweave.com --all-namespaces
kubectl get mcsorganizations.data.coreweave.com --all-namespaces
kubectl get mcsservices.data.coreweave.com --all-namespaces

Pass criteria:

• every Kubernetes node is Ready;
• the Operator is Running;
• existing Keeper and Service CRs either report Ready or have an understood in-progress condition;
• no workload is in CrashLoopBackOff, ImagePullBackOff, or an unexpected Pending state.

2. Prove administrator and organization-user boundaries

Use the Console rather than applying CRs directly:

1. sign in as the bootstrap administrator;
2. create a Keeper fleet;
3. create an organization, assign that Keeper, and create its owner account;
4. sign out;
5. sign in as the organization owner;
6. create a three-node ClickHouse service with at least one ElasticQuery node;
7. wait until the service reports both ClickHouse and ElasticQuery ready;
8. confirm the organization user cannot access administrator-only organization or Keeper mutations.

This proves the management control plane and the handoff to the Runtime Operator. Record the organization ID, Keeper ID, Service ID, ClickHouse node IDs, and ElasticQuery node IDs.

3. Inspect the live topology

In SQL Console, select ClickHouse and any node, then run:

SELECT
    cluster,
    shard_num,
    replica_num,
    host_name,
    host_address,
    is_local
FROM system.clusters
WHERE cluster IN ('all_shards', 'all_replicas')
ORDER BY cluster, shard_num, replica_num;

Then prove both dynamic clusters can execute remote queries:

SELECT hostName() AS node
FROM cluster('all_shards', system.one)
ORDER BY node;

SELECT hostName() AS node
FROM clusterAllReplicas('all_replicas', system.one)
ORDER BY node;

Pass criteria: the reported membership matches the ClickHouse nodes shown by the Console, and each query returns every expected node exactly once.

4. Create an ElasticQuery-eligible table

Run the DDL once from a ClickHouse node:

CREATE DATABASE IF NOT EXISTS e2e ON CLUSTER all_replicas
ENGINE = Atomic;

CREATE TABLE e2e.events ON CLUSTER all_replicas
(
    id UInt64,
    event_time DateTime64(3),
    source LowCardinality(String),
    body String
)
ENGINE = MergeTree
ORDER BY (event_time, id)
SETTINGS table_disk = true;

Do not specify disk = .... In the managed ClickHouse image, table_disk = true is the user opt-in and MCS injects the inline object-storage disk.

Verify the resulting definition on every ClickHouse node:

SHOW CREATE TABLE e2e.events;

Pass criteria: the definition retains table_disk = true, contains the operator-managed inline object-storage disk, and uses a service/replica/table path rather than a shared unscoped prefix.

5. Insert distinct data through every ClickHouse node

Select each ClickHouse node explicitly in SQL Console. Run the statement below once per node, changing the numeric base to 1000, 2000, and 3000:

INSERT INTO e2e.events
SELECT
    1000 + number + 1,
    now64(3),
    getMacro('replica'),
    concat('event-', toString(number + 1))
FROM numbers(10);

On the same node, verify its local data:

SELECT
    hostName() AS node,
    count() AS rows,
    min(id) AS first_id,
    max(id) AS last_id,
    groupArrayDistinct(source) AS sources
FROM e2e.events
GROUP BY node;

Pass criteria:

• every ClickHouse node has exactly 10 local rows;
• ID ranges are 1001-1010, 2001-2010, and 3001-3010;
• source contains that node’s stable replica macro.

6. Verify ElasticQuery schema sync and reads

Wait for at least one schema-sync interval, select every ElasticQuery node in SQL Console, and open database e2e.

The table list must contain:

• one physical reader labelled Synced from <replica-id> for every ClickHouse node;
• one events table labelled All data.

Query the logical aggregate:

SELECT
    count() AS rows,
    uniqExact(source) AS source_nodes,
    min(id) AS first_id,
    max(id) AS last_id
FROM e2e.events;

Expected result:

rows = 30
source_nodes = 3
first_id = 1001
last_id = 3010

Also select each physical reader from the table list and run:

SELECT
    count() AS rows,
    groupArrayDistinct(source) AS sources
FROM e2e.`events__<replica-id>__<uuid-fragment>`;

Pass criteria: each physical reader returns 10 rows from only its source replica, and every ElasticQuery node returns the same aggregate result.

7. Prove the opt-in boundary and read-only boundary

Create a normal ClickHouse table without table_disk = true:

CREATE TABLE e2e.clickhouse_only ON CLUSTER all_replicas
(
    id UInt64,
    value String
)
ENGINE = MergeTree
ORDER BY id;

After a schema-sync interval, clickhouse_only must remain available on the ClickHouse nodes and must not appear in ElasticQuery.

On ElasticQuery, attempt a harmless write to the managed table:

INSERT INTO e2e.events
VALUES (999999, now64(3), 'elasticquery', 'must fail');

Pass criteria: the write is rejected as read-only and the aggregate row count remains 30.

8. Prove drop and recreate convergence

Capture the current table identity on a ClickHouse node:

SELECT uuid
FROM system.tables
WHERE database = 'e2e' AND name = 'events';

Drop the table:

DROP TABLE e2e.events ON CLUSTER all_replicas SYNC;

After schema sync, ElasticQuery must no longer expose the old physical readers or the events aggregate. Re-run the DDL from step 4 and confirm:

• ClickHouse assigns the recreated table its new identity;
• schema sync creates replacement readers;
• no stale reader remains visible;
• new inserts are visible through the recreated aggregate.

9. Record evidence and clean up

For a release or handoff, save:

• make test and Helm lint/render results;
• CR YAML with status and conditions;
• kubectl get pods --all-namespaces -o wide;
• topology query results;
• the three local ClickHouse results;
• ElasticQuery physical-reader and aggregate results;
• the read-only failure;
• the drop/recreate identity and cleanup result.

Clean up the SQL fixture when the environment is no longer needed:

DROP DATABASE IF EXISTS e2e ON CLUSTER all_replicas SYNC;

Service and Keeper deletion are separate lifecycle tests and should be performed through the Console when destructive cleanup is in scope.

Additional manual scenario coverage

Provisioning and identity

Verify the complete user path:

1. install the MCS Operator from zero;
2. log in as the platform administrator;
3. create a Keeper fleet and an organization;
4. log in as a user belonging to that organization;
5. create, inspect, update, and delete a ClickHouse service through the Console;
6. prove that users cannot read or mutate another organization’s resources;
7. verify credentials, endpoints, and public documentation access.

ClickHouse and ElasticQuery

For every supported ClickHouse release:

1. create a service with multiple ClickHouse nodes;
2. create eligible MergeTree-family tables with table_disk = true;
3. insert distinct rows through every ClickHouse node;
4. query each ClickHouse node and every ElasticQuery node;
5. verify all_shards and all_replicas cluster queries and ON CLUSTER DDL;
6. drop and recreate a table, then verify stale ElasticQuery readers are removed and the new table identity is mounted;
7. verify duplicate UUID handling and schema-sync failure reporting;
8. confirm ElasticQuery remains read-only and uses virtual-hosted object storage addressing;
9. confirm ordinary tables that do not opt into table-disk sharing are not mounted by ElasticQuery.

The assertions must check row identity and count, not only that a query returns HTTP 200.

Keeper topology and disruption

Exercise 3-, 5-, and 7-member Keeper fleets on enough schedulable Kubernetes nodes to satisfy hard anti-affinity.

For each topology:

• verify quorum and membership;
• continuously insert and read ClickHouse data;
• delete followers and the current leader;
• restart and roll Keeper members one at a time;
• inject a pod failure during a rolling update;
• drain or stop a Kubernetes node;
• verify recovery after quorum-safe and quorum-losing failures;
• confirm Keeper-only changes do not roll ClickHouse service pods;
• retain Keeper logs and membership output for every failure.

The expected availability during quorum loss must be explicit. The test must not report success merely because all pods eventually become Ready.

Scaling and lifecycle

Verify:

• ClickHouse scale-out through the supported replica range;
• ElasticQuery scale-up and scale-down;
• scheduler and anti-affinity behavior;
• deletion and recreation of services and Keeper fleets;
• namespace and credential lifecycle;
• cleanup of schema-sync readers and object-storage identities;
• operator restart while resources are reconciling.

Upgrade coverage

Upgrade testing is a separate release gate:

• operator upgrade with existing CRs;
• ClickHouse release upgrade without an unintended downgrade;
• Keeper upgrade without rolling ClickHouse workloads;
• schema-sync compatibility across supported versions;
• rollback behavior when a new operator cannot reconcile a resource.

Upgrade scenarios do not belong in every fast pull request, but must run before promoting a release.

Automation roadmap

Phase 1: build one reusable harness

Create a checked-in end-to-end runner rather than accumulating shell fragments. The runner must provide:

• Kind cluster creation with a configurable worker count;
• Helm installation using versioned test values;
• deterministic bootstrap Secrets and isolated object-storage prefixes;
• admin and organization-user HTTP clients;
• Kubernetes readiness checks with explicit timeouts;
• ClickHouse SQL execution against named nodes;
• assertions for rows, UUIDs, topology, status conditions, and pod revisions;
• fault actions such as pod deletion, rollout, node stop, and operator restart;
• automatic logs, events, manifests, Keeper state, and schema-sync collection;
• idempotent cleanup that never deletes outside the test prefix.

Each action should return structured evidence. A test must fail on a violated assertion instead of relying on an engineer reading Console output.

Phase 2: automate the L3 happy path

Turn the current from-zero runbook into one Kind smoke test:

1. create the cluster and install the charts;
2. create Keeper and an organization as the administrator;
3. create a service as an organization user;
4. wait for ClickHouse and ElasticQuery readiness;
5. create table-disk tables, insert per node, and verify ClickHouse and ElasticQuery results;
6. exercise drop/recreate schema sync;
7. delete resources and verify cleanup.

Once reliable, run it for every change to controllers, charts, storage, schema sync, or Console mutations. It can later become a required pull-request check.

Phase 3: automate L4 resilience

Reuse the same harness with a scenario matrix instead of a second test system:

• Keeper sizes 3, 5, and 7;
• leader and follower deletion;
• rolling update with concurrent inserts;
• pod failure during rollout;
• Kubernetes node disruption;
• operator restart during reconciliation;
• ClickHouse and ElasticQuery scaling;
• repeated table drop/recreate.

Run this matrix nightly. Keep a smaller representative fault case available for labeled pull requests.

Phase 4: automate L5 release qualification

Parameterize the harness with source and target versions. Release qualification must create data before an upgrade, perform the operator/Keeper/ClickHouse transition, and verify the same data and control-plane invariants afterward. The version matrix and rollback expectations belong in source control.

Target CI tiers

Pull request

Target: fast and deterministic.

• lint;
• generated artifact verification;
• Go unit and envtest suites;
• Helm lint and render;
• container build smoke test.

Kind smoke

Target: one live happy path for changes that affect controllers, charts, Console mutations, storage, or schema sync.

• install from zero;
• create Keeper, organization, and service through supported APIs/UI;
• run table-disk DDL, insert, ClickHouse query, and ElasticQuery query;
• delete all test resources cleanly.

This can run on labeled pull requests until its duration and reliability are appropriate for every pull request.

Nightly resilience

Target: failure discovery.

• 3/5/7 Keeper topology matrix;
• rolling updates with concurrent inserts;
• leader and pod deletion;
• node disruption;
• service and ElasticQuery scaling;
• repeated drop/recreate schema sync;
• operator restart and reconciliation recovery.

Release qualification

Target: supported-version confidence.

• full from-zero installation;
• supported ClickHouse and Keeper version matrix;
• upgrade and rollback paths;
• multi-architecture image smoke tests;
• retained test report and cluster artifacts.

Test data and cleanup

End-to-end data should be deterministic and large enough to expose correlation and lifecycle errors without becoming a performance benchmark. Every run must use a unique test ID in:

• Kubernetes names and labels;
• namespace names;
• service and replica IDs;
• object-storage prefixes;
• table names or comments when necessary.

Cleanup is part of the test. A successful run verifies that owned Kubernetes resources disappear and that recreated tables no longer expose stale readers. Destructive cleanup of a shared bucket is not permitted.

Failure artifacts

On failure, CI should retain:

• operator and workload logs;
• Kubernetes events;
• CR YAML including status and conditions;
• rendered Helm values and manifests;
• pod placement and rollout history;
• Keeper membership and quorum output;
• schema-sync logs;
• the SQL statements and result assertions used by the test.

Without these artifacts, an intermittent end-to-end failure is not diagnosable.

When a manual test becomes automated

A manual scenario is not considered automated merely because its commands were copied into a script. It becomes automated only when:

1. its runner and inputs are checked into the repository;
2. environment requirements and Secrets are declared;
3. setup has bounded readiness checks and failure timeouts;
4. success is decided by machine-readable assertions;
5. failure artifacts are collected automatically;
6. cleanup is isolated and repeatable;
7. it runs in a named CI tier on a documented trigger;
8. ownership exists for fixing a consistently failing test.

The next implementation priority is Phase 1 followed by the single L3 happy path. Browser coverage, schema-sync lifecycle, Keeper disruption, and upgrades should then be added to that same harness rather than implemented as unrelated one-off jobs.

New features must state which level proves them and add the smallest automated test that would have caught their most likely regression.