Control Plane Connectivity

For: Platform operators running a self-hosted gateway connected to the Axemere Control Plane (Core Platform or higher).

Operations Overview | Approval Workflows | Quarantine | Risk Scoring | CP Connectivity | Telemetry

When a gateway is connected to the Axemere Control Plane (CP), it receives signed policy bundles, submits execution record hashes for the Merkle audit ledger, and keeps credentials and workloads in sync, all over gRPC with mutual TLS. This guide covers the connection lifecycle, offline fallback behavior, and how to diagnose connectivity issues.

Free Gateway (no CP): If you are running the gateway without a CP connection (no MVGC_CP_ADDR), this guide does not apply: your gateway operates entirely from local policy files.

Managed Gateway (Growth Pack / Dedicated): If you are using the Axemere-hosted gateway fleet, CP connectivity is handled automatically. You do not configure it directly.

Table of Contents


Connection Modes

ModeHow activatedConfig syncUse when
Free GatewayMVGC_CP_ADDR not setNone — local policy files onlyRunning standalone without console management (free tier)
Self-hosted with CPMVGC_CP_ADDR set + MVGC_GATEWAY_MODE=self-hosted (default)Idle check-in after MVGC_CHECKIN_IDLE_THRESHOLD of inactivityYour infrastructure, connected to Axemere's CP, requires Core Platform
ManagedMVGC_GATEWAY_MODE=managed (configured by Axemere)Streaming SubscribeUpdates gRPCAxemere-hosted gateway fleet (Growth Pack / Dedicated)

When MVGC_CP_ADDR is not set, the gateway runs as a Free Gateway with local policy files and no distributed features: no ledger, no remote bundles, no console-managed API keys or credentials. All CP-connected modes require MVGC_CP_ADDR.

Running your own CP is not a standard offering. Companies running self-hosted gateways connect to Axemere's CP, either through the console (Core Platform) or via the managed fleet. Private CP deployments are available through a separate enterprise arrangement.


Self-Hosted Gateway: Check-In Mode

The check-in is an idle keepalive: it fires after the gateway has been inactive for MVGC_CHECKIN_IDLE_THRESHOLD (default: 60s). While the gateway is actively processing requests, the CP connection is kept current through request activity itself; the idle check-in is the fallback that fires when traffic goes quiet.

This means your gateway always appears online in the console as long as it is processing requests or has been idle long enough to check in.

Control PlaneGatewayControl PlaneGatewayAfter MVGC_CHECKIN_IDLE_THRESHOLD (default 60s) of inactivityalt[policy_stale = true]alt[workloads_stale = true]alt[credentials_stale = true]alt[export_config_stale = true]CheckIn(node_id, org_id,policy_version, workloads_version,credentials_version, export_config_version) [mTLS]CheckInResponse(policy_stale, workloads_stale,credentials_stale, export_config_stale,org_status, suspended_reason)GetPolicyBundle(org_id, bundle_id)bundle YAMLVerify bundle signatureActivate and cache bundleGetWorkloads(org_id)workload listListNodeCredentials(org_id, node_id)resolved credential set (plaintext keys over mTLS)GetNodeExportDestinations(org_id, node_id)resolved export destination list

The gateway only fetches updated data when the version number has changed, minimizing bandwidth.

Configure the idle threshold:

export MVGC_CHECKIN_IDLE_THRESHOLD=60s   # default

Offline Fallback Behavior

When the control plane is unreachable, the gateway continues serving requests using its cached state:

MVGC_CP_ADDR configured, mTLS enrollment complete

CP unreachable (network failure or CP restart)

CP reachable again, check-in or stream resumes

CP reachable but returns errors

errors resolve

cp_status connected

cp_status offline, uses cached bundle + credentials

cp_status connected, hash submission may queue

What continues working offlineWhat degrades offline
Request evaluation (cached policy bundle)Policy updates delayed until reconnect
Credential lookup (cached credentials)New credentials not available
Workload enforcement (cached workloads)Workload changes not propagated
Record creation and local JSONL logHash submission queues (flushed on reconnect)

Hash submission queue: Record hashes are written to the pending_hash_submissions Postgres table when the CP is offline. The queue is durable: it survives gateway restarts. Entries are drained automatically when the CP reconnects. Monitor mvgc_pending_hash_submissions to track backlog depth during extended outages.

Enforcement posture: Depending on your org's configuration, the gateway may begin blocking AI requests after a grace period if the CP remains unreachable for an extended time. This posture (permissive vs. strict) is set by Axemere for your org. Contact support if you need to adjust it.


Enrollment and mTLS

Self-hosted gateways authenticate to the CP using mTLS. The private key is generated on the gateway and never transmitted. For full details on enrollment, certificate lifecycle, and revocation, see the mTLS Guide.

Quick reference:

Via environment variables:

# First startup — enrolls with the CP and stores cert in MVGC_KEY_DIR
export MVGC_BOOTSTRAP_TOKEN="<token from console>"
export MVGC_CP_ADDR="gcp.cp.axemere.ai:9090"
export MVGC_ORG_ID="<your org_id>"
export MVGC_KEY_DIR="/var/lib/mvgc/keys"
./mvgc-gateway

Or in mvgc.yaml (see Config File):

control_plane:
  addr:            "gcp.cp.axemere.ai:9090"
  bootstrap_token: "<token from console>"

gateway:
  org_id: "<your org_id>"

security:
  key_dir: "/var/lib/mvgc/keys"

Subsequent startups use the stored cert automatically. Certs are auto-renewed 14 days before expiry (90-day validity).


Policy Bundle Durable Cache

The last successfully fetched policy bundle is cached in the gateway_bundles Postgres table. This cache survives restarts:

  • If the CP is unreachable at startup, the gateway loads the cached bundle and operates normally.
  • The cache is updated after every successful GetPolicyBundle fetch.
  • To force a cache refresh: POST /v1/admin/policy/cache (deletes the cached bundle and triggers a fresh fetch on next check-in or stream reconnect).
# Force policy cache refresh
curl -s -X DELETE http://localhost:7080/v1/admin/policy/cache \
  -H "MVGC-Admin-Token: $MVGC_ADMIN_TOKEN"

Record Hash Submission

The gateway submits a SHA-256 hash of each execution record to the CP ledger via SubmitRecordHash. This is how records become part of the Merkle audit trail.

Control PlaneHash QueueGatewayControl PlaneHash QueueGatewayCP offlineCP comes back onlineEnqueue(record_hash, attribution)SubmitRecordHash(record_hash, org_id, ...)Receipt(ledger_seq)ledger_seq availableQueue record_hash (in memory)Flush queued hashes

Monitor the queue depth via the mvgc_pending_hash_submissions gauge. Alert if it stays elevated: it indicates the CP has been offline for an extended period.


Healthz Status Fields

GET /healthz (no auth required) exposes CP connectivity status:

curl -s http://localhost:7080/healthz | jq '{cp_status, bundle_id, last_hash_submit_at}'
{
  "cp_status": "connected",
  "bundle_id": "01955f3e-0000-7abc-8def-000000000001",
  "last_hash_submit_at": "2026-03-12T14:30:00Z"
}
cp_statusMeaning
"connected"Active CP connection; check-in or stream is healthy
"offline"CP unreachable; gateway using cached configuration
"unconfigured"MVGC_CP_ADDR is not set; embedded mode

Use last_hash_submit_at to detect a stalled hash submission pipeline. If this timestamp stops advancing while requests are being processed, investigate CP connectivity and queue depth.


Environment Variables

VariableDefaultDescription
MVGC_CP_ADDRControl plane gRPC address (host:port). Required for distributed mode.
MVGC_GATEWAY_MODEself-hostedself-hosted (check-in) or managed (streaming)
MVGC_ORG_IDOrganisation ID: embedded in gateway certs and requests
MVGC_BOOTSTRAP_TOKENSingle-use enrollment token from the console (first startup only)
MVGC_KEY_DIR./keysDirectory for storing the gateway cert, key, and CA bundle
MVGC_CHECKIN_IDLE_THRESHOLD60sIdle keepalive interval: gateway checks in after this duration of inactivity (self-hosted mode). Minimum 30s; values below 30s are clamped.

Troubleshooting

SymptomLikely causeFix
cp_status: "unconfigured"MVGC_CP_ADDR not setSet MVGC_CP_ADDR and restart
cp_status: "offline"Network blocked or cert validation failureCheck firewall on port 9090; inspect TLS errors in gateway logs
codes.PermissionDenied on gRPCCert revokedGenerate a new bootstrap token and re-enroll the node
Gateway stuck in "enrolling"Bootstrap token expiredGenerate a new token and restart
Stale policy after overlay updateCheck-in interval not elapsedDecrease MVGC_CHECKIN_IDLE_THRESHOLD or trigger via DELETE /v1/admin/policy/cache
mvgc_pending_hash_submissions growingCP offline for extended periodCheck CP connectivity; the queue flushes automatically on reconnect
Policy bundle loads from cache at startupCP offline at startup — expected behaviorEnsure the DB has a recent cached bundle; connect to CP to refresh

See Also