Operations Overview

For: Platform operators and SREs running Axemere Gateway in production.

Axemere Gateway provides a set of operational controls for managing request flows, monitoring system health, and investigating incidents. This section covers the runtime concerns beyond initial setup.

Table of Contents

• Operations Areas
• Decision Types Quick Reference
• Operations Runbook Index
• See Also

Operations Areas

Decision Types Quick Reference

Operations Runbook Index

Elevated risk score alert:

1. Check mvgc_risk_signals in Prometheus to identify active signals.

2. Review recent requests in the dashboard API for the affected org.

3. If rate_spike: check for runaway agents or load test traffic.

4. If cost_anomaly: check for unexpectedly large model requests.

5. If the baseline has legitimately shifted, adjust thresholds. There are two independent levers:

   Detection sensitivity: how much deviation triggers a signal. Set in mvgc.yaml (or via env var):

   risk:
     rate_threshold: 4.0    # default 3.0; increase to reduce rate_spike sensitivity
     cost_threshold: 4.0    # default 3.0; increase to reduce cost_anomaly sensitivity
   

   Policy score threshold: what composite score triggers a decision:

   • CP-attached: Edit the risk rule threshold in the console policy editor.
   • Free Gateway: Edit the policy YAML in configs/policies/ (or your MVGC_POLICIES_DIR):

   risk:
     - id: risk.quarantine.high_score
       when:
         field: context.risk.score
         gt: "0.9"   # raise to reduce quarantine frequency; was 0.8
       effect:
         decision: quarantine
   

→ Risk Scoring

Approvals backlog (mvgc_approvals_pending > threshold):

1. List pending approvals: GET /v1/admin/approvals?status=pending
2. Review request bodies and context in the approval records.
3. Approve or deny; urgent bulk denials can be scripted against the admin API.
4. Consider loosening the triggering policy rule if false positives are frequent. → Approval Workflows

CP offline (cp_status: "offline"):

1. Check firewall rules on port 9090 to the CP address.
2. Verify the gateway certificate has not expired: inspect MVGC_KEY_DIR.
3. Check CP health independently (if self-hosting the CP).
4. Monitor mvgc_pending_hash_submissions; the queue drains automatically on reconnect. → CP Connectivity

Auth failure spike:

1. Check mvgc_auth_failures_total{reason="invalid_key"}: likely a rotated API key.
2. Check mvgc_auth_failures_total{reason="revoked"}: intentional revocation propagating.
3. Verify client applications are using the correct key after rotation. → Telemetry

See Also

• IT Setup Overview — full configuration reference and deployment options
• Network Operations Guide — policy DSL, credential and workload management
• Security Overview — mTLS, bundle signing, credential encryption
• Developer Integration — API reference for application developers