Operator observability

The MCS operator exposes Prometheus metrics for the control loop and the ClickHouse and Keeper resources it manages. In CoreWeave clusters, VMAgent discovers the operator VMPodScrape, writes the samples to VictoriaMetrics, and vmalert evaluates the platform alert rules.

Enable collection

metrics:
  enable: true
  secure: false
  port: 8080
victoriaMetrics:
  enable: true
  interval: 30s
  scrapeTimeout: 2s

The metrics endpoint is cluster-internal HTTP by default, matching the VMPodScrape pattern used by Polaris and the existing data-platform operators. Set metrics.secure: true only when the cluster requires authenticated HTTPS; the operator VMPodScrape then supplies the Kubernetes service-account token and TLS configuration automatically.

The chart also creates a VMPodScrape for every managed ClickHouse Writer and ElasticQuery Pod. ClickHouse exposes its native Prometheus endpoint on port 9363, including current metrics, events, and asynchronous metrics.

Signals

Controller-runtime provides reconciliation, workqueue, API client, process, and Go runtime metrics. MCS adds resource-state metrics under the cwmcs_operator namespace:

• cwmcs_operator_resource_collection_success
• cwmcs_operator_mcsservice_info
• cwmcs_operator_mcsservice_phase
• cwmcs_operator_mcsservice_condition
• cwmcs_operator_mcsservice_generation
• cwmcs_operator_mcsservice_observed_generation
• cwmcs_operator_mcsservice_clickhouse_replicas_desired
• cwmcs_operator_mcsservice_elasticquery_replicas_desired
• cwmcs_operator_mcskeeper_info
• cwmcs_operator_mcskeeper_phase
• cwmcs_operator_mcskeeper_condition
• cwmcs_operator_mcskeeper_generation
• cwmcs_operator_mcskeeper_observed_generation
• cwmcs_operator_mcskeeper_instances_desired
• cwmcs_operator_mcskeeper_instances_ready
• cwmcs_operator_mcskeeper_quorum_available

Resource metrics are rebuilt from the controller-runtime cache on every scrape. A process restart or custom-resource deletion therefore cannot leave a stale in-memory gauge behind. Labels contain Kubernetes resource identities and bounded status values; they do not contain user email addresses, request IDs, condition messages, or secrets.

The operator emits one structured JSON log object per line in production. Use cluster, namespace, controller, and resource identity fields to correlate logs with metrics. Never log credentials or object-storage secrets.

First response

Operator unavailable or metrics target down

1. Check the mcs-operator-controller-manager Deployment and pod events in data-mcs-operator.
2. Check the manager container logs and /healthz and /readyz probes.
3. For a scrape failure, check the VMPodScrape and metrics Pod endpoint endpoints. When metrics.secure is enabled, also check TLS configuration and token review/subject access review permissions.

Reconcile errors or stale observed generation

1. Inspect controller_runtime_reconcile_errors_total by controller.
2. Inspect the affected custom resource status and conditions.
3. Check manager logs for the same namespace and resource name.
4. Confirm dependent Secrets, storage endpoints, and Kubernetes resources are present before retrying or changing desired state.

MCSService degraded

1. Inspect the MCSService conditions and events.
2. Check ClickHouse writer and ElasticQuery pod readiness and logs.
3. Check Keeper health before restarting a writer.
4. Check storage credentials and endpoint connectivity.

Keeper quorum unavailable

1. Compare cwmcs_operator_mcskeeper_instances_ready with cwmcs_operator_mcskeeper_instances_desired.
2. Inspect Keeper pods, events, and logs.
3. Check object-storage credentials and connectivity.
4. Restore a majority of members before restarting dependent ClickHouse writers.

Dashboard

observability/grafana/mcs-operator-dashboard.yaml contains an importable Grafana dashboard. Select the regional VictoriaMetrics and Loki data sources, then filter by region, zone, and cluster. Publish a validated revision through the managed Grafana dashboard workflow.