Telemetry and Observability

For: Platform operators and SREs monitoring Axemere Gateway in production.

Operations Overview | Approval Workflows | Quarantine | Risk Scoring | CP Connectivity | Telemetry

Axemere Gateway exposes four observability surfaces: a health endpoint, Prometheus metrics, a dashboard API, and SIEM event export. This guide covers all four, plus the anonymous usage telemetry that gateways send to Axemere.

Table of Contents

• Gateway Health Endpoint
• Prometheus Metrics
  • Metrics Reference
  • Key Alerts to Configure
  • Example Prometheus Scrape Config
• Dashboard API
• SIEM Event Export
  • Event Types
  • Webhook Export
  • Syslog Export
  • Event Payload Schema
• Anonymous Usage Telemetry
  • What is collected
  • What is not collected
  • Frequency and delivery
  • Disabling telemetry
• Environment Variables
• Troubleshooting
• See Also

Gateway Health Endpoint

GET /healthz: no authentication required.

curl -s http://localhost:7080/healthz | jq .

{
  "status": "ok",
  "node_id": "gw-node-prod-01",
  "connectors": [
    {"id": "openai", "version": "1.0.0", "healthy": true},
    {"id": "anthropic", "version": "1.0.0", "healthy": true}
  ],
  "cp_status": "connected",
  "bundle_id": "01955f3e-0000-7abc-8def-000000000001",
  "bundle_version": "2.1.0",
  "last_hash_submit_at": "2026-03-12T14:30:00Z"
}

Use /healthz as the liveness and readiness probe target in Kubernetes:

livenessProbe:
  httpGet:
    path: /healthz
    port: 7080
  initialDelaySeconds: 15
readinessProbe:
  httpGet:
    path: /healthz
    port: 7080
  initialDelaySeconds: 5

Prometheus Metrics

Prometheus metrics are available at GET /metrics when MVGC_METRICS_ENABLED=true (the default).

curl -s http://localhost:7080/metrics | grep ^mvgc_

mvgc_requests_total{action_type="ai.infer",decision="allow",org_id="org-001"} 1842
mvgc_request_duration_ms_bucket{action_type="ai.infer",decision="allow",le="100"} 1651
mvgc_risk_score{org_id="org-001"} 0.15
mvgc_approvals_pending 0
mvgc_pending_hash_submissions 0
mvgc_auth_failures_total{reason="invalid_key",org_id=""} 3

Metrics Reference

mvgc_auth_failures_total reasons:

Key Alerts to Configure

Example Prometheus Scrape Config

scrape_configs:
  - job_name: mvgc-gateway
    static_configs:
      - targets: ['gateway-host:7080']
    metrics_path: /metrics
    scrape_interval: 15s

Dashboard API

The dashboard API provides aggregated summaries without a separate analytics stack. Both endpoints require the admin token.

Summary: total requests, allow/deny counts:

curl -s "http://localhost:7080/v1/dashboard/summary?org_id=org-001&from=2026-03-01T00:00:00Z&to=2026-03-12T00:00:00Z" \
  -H "MVGC-Admin-Token: $MVGC_ADMIN_TOKEN" | jq .

{
  "org_id": "org-001",
  "from": "2026-03-01T00:00:00Z",
  "to": "2026-03-12T00:00:00Z",
  "total_requests": 14203,
  "total_allowed": 13801,
  "total_denied": 402,
  "total_tokens_in": 8450000,
  "total_tokens_out": 3200000,
  "total_cost_usd": "1842.50"
}

Decision breakdown: counts grouped by decision type:

curl -s "http://localhost:7080/v1/dashboard/decisions?org_id=org-001" \
  -H "MVGC-Admin-Token: $MVGC_ADMIN_TOKEN" | jq .

[
  {"decision": "allow",            "count": 13801},
  {"decision": "deny",             "count": 387},
  {"decision": "rate_limit",       "count": 12},
  {"decision": "require_approval", "count": 3}
]

SIEM Event Export

The gateway streams events to external security systems via webhook and/or syslog. Configure one or both; events fan out to all configured exporters.

Event Types

risk_alert events are emitted before any policy decision; they fire purely on score threshold. This lets your SIEM react to elevated risk patterns even when no quarantine or require_approval rule has fired.

Webhook Export

Events are POST'd as JSON to your configured URL. Retried up to 3 times with exponential backoff on failure:

export MVGC_EXPORT_WEBHOOK_URL=https://siem.example.com/ingest
export MVGC_EXPORT_WEBHOOK_TOKEN=<bearer-token>   # optional

The gateway sends:

POST https://siem.example.com/ingest
Content-Type: application/json
Authorization: Bearer <MVGC_EXPORT_WEBHOOK_TOKEN>

{ ...event payload... }

Syslog Export

Events are sent as structured syslog messages. Supports TCP and UDP:

export MVGC_EXPORT_SYSLOG_ADDR=tcp://siem.example.com:514
# or
export MVGC_EXPORT_SYSLOG_ADDR=udp://siem.example.com:514

Event Payload Schema

All events share a common envelope:

{
  "event_type": "execution",
  "event_time": "2026-03-12T14:30:00Z",
  "org_id": "org-example-001",
  "workload_id": "wl-prod-app-1",
  "payload": { ... event-specific fields ... }
}

execution event payload:

The full ExecutionRecord document, including:

• record_id, record_hash, decision, reason_codes
• action_type, target, caller_id, caller_ip
• Attribution: project_id, customer_id, account_id, labels
• Metering: tokens_in, tokens_out, cost_usd

risk_alert event payload:

{
  "risk_score": 0.85,
  "signals": ["rate_spike", "cost_anomaly"],
  "request_rate": "45.2",
  "cost_rate": "0.82",
  "org_id": "org-example-001",
  "workload_id": "wl-prod-app-1",
  "caller_id": "service-abc"
}

Anonymous Usage Telemetry

Axemere Gateway sends anonymous aggregate usage data to Axemere. This data helps us understand how the gateway is used across deployments so we can prioritize improvements.

No request content, prompt text, API keys, or user-identifiable information is ever included. Each gateway generates a random instance ID at startup that is not persisted across restarts, so telemetry reports cannot be correlated across gateway restarts.

What is collected

Reports use the schema mvgc.telemetry.v2 and contain aggregate counters for a time window (typically one hour). Each report covers:

Per-provider breakdown: for every AI provider the gateway routed traffic to during the window:

Top-level aggregates:

Example report shape:

{
  "schema": "mvgc.telemetry.v2",
  "instance_id": "3f2a1b4c-...",
  "version": "v0.53.0",
  "window_start": "2026-05-11T10:00:00Z",
  "window_end": "2026-05-11T11:00:00Z",
  "providers": {
    "anthropic": {
      "requests": 142,
      "tokens_in": 284000,
      "tokens_out": 71000,
      "models": {
        "claude-sonnet-4-6": { "requests": 142, "tokens_in": 284000, "tokens_out": 71000 }
      },
      "domains": { "api.anthropic.com": 142 },
      "connection_types": { "direct_api": 142 }
    }
  },
  "decisions": { "allow": 140, "deny": 2 },
  "action_types": { "ai.infer": 142 },
  "risk_alerts": 0
}

Token counts are only included for allowed requests; denied requests consume no tokens and are not included in provider token totals.

What is not collected

• Request or response content
• Prompt text or completions
• API keys or credentials of any kind
• Caller identity, user names, or email addresses
• IP addresses
• Individual request timing or latency data
• Any field from your policy configuration

Frequency and delivery

Telemetry is flushed once per hour by default. The Axemere ingest service may hint a different interval in the response; the gateway will never flush more frequently than the configured minimum (it can only be slowed down by the server, not sped up).

Telemetry is sent over HTTPS to gcp.telemetry.axemere.ai. There is no retry on delivery failure: if a flush fails, the counters for that window are dropped and the next window starts fresh. Failed submissions are logged at debug level:

time=2026-05-11T10:00:01Z level=DEBUG msg="telemetry POST failed" error="..."

Successful flushes are also logged at debug level. To see them, set MVGC_LOG_LEVEL=debug.

Self-hosted gateways connected to the CP send telemetry via the encrypted, authenticated gRPC channel to the CP instead of HTTP. The CP then forwards to the ingest service on your behalf.

Disabling telemetry

Self-hosted gateways connected to the CP can disable telemetry:

export MVGC_TELEMETRY_ENABLED=false

Note: Free gateways (self-hosted without CP connectivity) cannot disable anonymous telemetry; it is the only signal we have about the health of the Free Gateway population and is required to continue offering the free tier.

Environment Variables

Troubleshooting

See Also

• CP Connectivity — cp_status and hash submission queue
• Risk Scoring — mvgc_risk_score and mvgc_risk_signals
• Approval Workflows — mvgc_approvals_pending
• Quarantine — mvgc_quarantine_total
• Network Operations Guide — full ops reference
• Glossary — execution_record
• Glossary — caller_ip