Self-Hosted Gateway Deployment and Operations

For: Infrastructure and IT teams installing, configuring, and operating Axemere Gateway.

Architecture Overview

Two deployment profiles are available:

• Embedded mode (default): a single gateway binary with a local Postgres instance. No additional services are required. This is the recommended starting point.
• Control Plane-connected mode: set MVGC_CP_ADDR to connect the gateway to the Axemere Control Plane for centralized policy management, Merkle anchoring, and key lifecycle management.

Prerequisites

• PostgreSQL 15 or later (self-hosted or managed)
• Docker, a Kubernetes cluster, or a Debian/Ubuntu/RHEL/Fedora Linux host
• A randomly generated MVGC_ADMIN_TOKEN for admin API authentication

Deployment Options

Choose the deployment method that best fits your environment:

Platform Deployment Guides

• PostgreSQL Setup -- shared database setup instructions for all platforms
• Linux: Debian and Ubuntu -- APT repository, .deb package, systemd
• Linux: RHEL, Fedora, and Amazon Linux -- RPM install, systemd
• macOS (Homebrew) -- Apple Silicon, brew services
• Docker and Podman -- multi-arch images, Compose, rootless Podman
• Kubernetes (Helm) -- Helm chart, rolling upgrades, PVC
• Windows (WSL2) -- WSL2 setup, Docker Desktop, .deb inside WSL
• Cloud Deployment -- OpenTofu modules for GCP, AWS, and Azure

PostgreSQL

All deployment methods require an external PostgreSQL 15+ instance. The gateway binary does not bundle or install a database server. Schema migrations run automatically on startup. No manual migration step is required.

See PostgreSQL Setup for step-by-step instructions covering Docker, bare-metal Linux, macOS, and cloud managed databases.

Configuration Reference

The gateway reads configuration from two sources, in priority order:

1. Environment variables -- always win when set; use for secrets and deployment-specific overrides (Docker, Kubernetes, systemd EnvironmentFile).
2. Config file (recommended) -- a single YAML file that documents every option with comments; ideal for bare-metal or disconnected deployments where managing many env vars is inconvenient.

Config File (mvgc.yaml) — Recommended

The gateway supports an optional YAML configuration file that documents every setting inline. It is an alternative to managing many individual environment variables, and is especially useful for bare-metal deployments or air-gapped environments.

# excerpt from mvgc.yaml
gateway:
  listen_addr: ":7080"
  node_id:     "node-prod-01"
  admin_token: "your-secret-token"   # or leave blank and set MVGC_ADMIN_TOKEN

database:
  # Format: postgres://<user>[:<password>]@<host>[:<port>]/<database>?sslmode=<disable|require>
  url: "postgres://mvgc_gateway:<password>@db:5432/mvgc_gateway?sslmode=require"

policies:
  dir: "/etc/mvgc/policies"

Any field left blank falls back to the built-in default. Secrets you prefer not to store in a file (admin token, database password) can remain as environment variables -- they override file values transparently.

File location resolution:

Obtaining the template:

• Docker / Podman -- download the template from the releases page and mount it as a volume (see Docker and Podman).
• Binary download -- download the template alongside the binary from the GitHub Releases page and place it next to the binary or set MVGC_CONFIG to its path.
• APT / deb / rpm packages -- the package installs a pre-populated template at /etc/mvgc/mvgc.yaml. Edit it in place; the systemd service automatically reads it via MVGC_CONFIG=/etc/mvgc/mvgc.yaml in /etc/default/mvgc-gateway.

Core Environment Variables

Credential Environment Variables

Credentials are declared in a credentials.yaml file inside the directory pointed to by MVGC_CREDENTIALS_DIR (default: configs/credentials). Each credential entry uses a secret_ref field to name the environment variable that holds the actual key:

# configs/credentials/credentials.yaml
credentials:
  - credential_id: cred-openai
    provider: openai
    mode: alias
    secret_ref: OPENAI_API_KEY

Set the referenced environment variables on the gateway server:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

These are set on the gateway server only. Applications never see these keys.

Connecting to the Control Plane

Connecting to the Axemere Control Plane requires a Control Plane subscription. Embedded mode is preserved when MVGC_CP_ADDR is not set.

Required to connect:

Advanced / optional:

Connecting to the Control Plane:

1. Sign in to the Cloud Console and open your organization.
2. Go to Gateways → + Enroll Gateway and click Generate Token.
3. Copy the bootstrap token (this is the value for MVGC_BOOTSTRAP_TOKEN) and your Organization ID from the Console.
4. Add these to your gateway configuration:

MVGC_CP_ADDR=us.cp.axemere.ai:9090     # hosted Control Plane gRPC endpoint
MVGC_ORG_ID=<your-org-id>             # Organization ID from the Cloud Console
MVGC_BOOTSTRAP_TOKEN=<your-token>     # one-time token from the Cloud Console

5. Start or restart the gateway; it registers with the Control Plane automatically and begins receiving policy and credential updates.

Performance Tuning Variables

Observability and Risk Variables

Node Identity and Key Setup

On first startup, the gateway generates an Ed25519 key pair and stores it under MVGC_KEY_DIR (default ./keys):

keys/
  node-local-dev.priv.pem   # Ed25519 private key (PKCS#8, mode 0600)
  node-local-dev.pub.pem    # Ed25519 public key (PKIX)

The key pair is tied to MVGC_NODE_ID and survives restarts. The private key is never logged or transmitted. Protect this directory as you would any other secret material.

Every execution record is signed with this key pair. The signed envelope looks like:

{
  "record_id": "...",
  "sig": {
    "alg": "ed25519",
    "kid": "kid_<base32-sha256>",
    "sig": "<base64url-signature>"
  }
}

The kid is a base32-encoded SHA-256 hash of the public key bytes, computed automatically on key generation.

Key Rotation

Self-hosted gateways connected to the Axemere Control Plane: node key lifecycle is managed automatically by Axemere. You do not need to rotate keys manually. Contact Axemere support if a key rotation is required.

The procedure below applies to standalone deployments (Free Gateway / embedded mode only).

Warning: Rotating the node key changes the kid embedded in all future execution records. Old records retain their original kid and remain verifiable with the old public key. Store old public keys if you need to verify historical records.

Procedure (standalone only):

1. Stop the gateway (or perform a rolling restart if using Kubernetes).
2. Back up the existing key files under MVGC_KEY_DIR.
3. Delete or rename <node-id>.priv.pem and <node-id>.pub.pem.
4. Restart the gateway; it auto-generates a new key pair on startup.

Signed Policy Bundles

When connected to the Axemere Control Plane, every effective policy bundle is cryptographically signed by the Control Plane before being delivered to gateways. Gateways automatically verify each bundle's signature using the signing key obtained from the Control Plane.

To enforce signature verification on all bundles received from the Control Plane:

1. Deploy the Control Plane with MVGC_BUNDLE_SIGN=true and MVGC_CP_POLICY_SIGNING_KEY set.
2. Confirm gateways are upgraded to a version that supports bundle signature verification.
3. Set MVGC_BUNDLE_VERIFY_SIGNATURES=true on all gateways.

Contact your Axemere representative for key generation, rotation, and revocation runbooks.

Health Check and Readiness

GET /healthz returns the gateway health status along with a list of registered connectors and their versions:

{
  "status": "ok",
  "version": "0.3.43",
  "node_id": "node-prod-01",
  "connectors": [
    {"id": "openai", "version": "1.0.0", "healthy": true},
    {"id": "anthropic", "version": "1.0.0", "healthy": true},
    {"id": "gemini", "version": "1.0.0", "healthy": true},
    {"id": "azure_openai", "version": "1.0.0", "healthy": true},
    {"id": "generic_http", "version": "1.0.0", "healthy": true}
  ],
  "cp_status": "unconfigured",
  "mitm_enabled": false
}

Startup grace period: During the first MVGC_HEALTHZ_STARTUP_GRACE seconds after the process starts (default 10s), /healthz returns {"status":"starting"} with HTTP 200. After the grace period expires, the full health response with "status":"ok" is returned. This prevents readiness probes from failing during initial subsystem initialization.

Use this endpoint for load balancer health checks and Kubernetes readiness probes. The connectors list confirms which provider connectors are registered and their versions, which is useful for verifying that all expected connectors are loaded after a deployment.

Smoke tests (optional, for deeper validation):

scripts/test/smoke.sh in the source repository runs automated checks against a running gateway. For a bare-metal production install, Tier 1 and the error-case tests in Tier 4 (S23--S25) run automatically:

Tier 2, Tier 3, and Tier 5 skips in the output are normal and do not indicate a problem with your install. They require a mock server, live provider credentials, or a Control Plane connection respectively, which are only present in development and CI environments.

Async Writer Backpressure

The gateway writes execution records asynchronously via a buffered queue (MVGC_RECORD_QUEUE_SIZE, default 1000). When the queue is full, the gateway returns HTTP 503 Service Unavailable instead of silently dropping the record. This ensures callers are aware that the audit trail may be incomplete and can retry.

Tuning guidance:

• Increase MVGC_RECORD_QUEUE_SIZE if you see 503 responses during sustained high traffic.
• Increase MVGC_RECORD_WORKERS (default 4) to drain the queue faster.
• Monitor the mvgc_record_write_errors_total Prometheus metric for persistent write failures that cause queue buildup.

Logging

The gateway emits structured JSON logs via Go's slog package to stdout. Each log line is a JSON object with at minimum the following fields:

Additional structured key-value fields are included depending on the event type (e.g. request_id, decision, connector_id, cost_usd, duration_ms).

Example log line:

{"time":"2026-03-11T08:00:00Z","level":"INFO","msg":"request executed","request_id":"req-001","decision":"allow","connector_id":"openai","cost_usd":"0.00012","duration_ms":342}

Sensitive fields never logged: credential values, API keys, raw delegation_token payloads, MVGC_ADMIN_TOKEN.

Log shipping: To forward logs to an aggregator (Datadog, Loki, CloudWatch, etc.), configure your container or systemd runtime to capture stdout and forward it. No additional gateway configuration is needed.

Log level: The gateway always logs at INFO and above. Log level is not currently configurable via environment variable.

Direct Export and SIEM Integration

In addition to stdout, the gateway can push execution events directly to external destinations without a log shipper:

These operate independently of stdout logging and can be combined, for example stdout for local collection and MVGC_EXPORT_WEBHOOK_URL for direct SIEM ingestion.

Upgrading

Schema Migrations

Schema migrations run automatically on startup via idempotent statements. No manual migration step is required.

Migrations are forward-only -- no rollback DDL is provided. Take a Postgres backup before upgrading.

Rolling Upgrade (Kubernetes)

1. Update image.tag in your Helm values file.
2. Run the Helm upgrade:

   helm upgrade mvgc-gateway deploy/gateway/k8s/helm \
     --namespace mvgc \
     --values deploy/gateway/k8s/helm/values-prod.yaml
   

3. Kubernetes performs a rolling restart. The first new pod runs migrations before the old pods are terminated.
4. Verify with:

   kubectl rollout status deployment/mvgc-gateway
   

See Kubernetes (Helm) for the full Helm deployment guide.

Bare-Metal / systemd Upgrade

sudo apt install mvgc-gateway   # or: sudo rpm -U mvgc-gateway-<version>.rpm
sudo systemctl restart mvgc-gateway

See Linux: Debian and Ubuntu or Linux: RHEL, Fedora, and Amazon Linux for platform-specific upgrade details.

Downgrade

Downgrade is not officially supported once migrations have run. Restore from your Postgres backup if needed.

Configuring SSL MITM Proxy

Per-platform CA trust store instructions are included in each platform deployment guide: Linux (Debian/Ubuntu) | Linux (RHEL/Fedora) | macOS | Docker/Podman | Kubernetes | Windows (WSL2) | Cloud

When MVGC_PROXY_MITM_ENABLED=true, the gateway acts as a transparent HTTPS proxy for traffic destined for configured AI provider domains. The gateway terminates TLS from the client using a dynamically generated leaf certificate signed by a local CA, applies the full execution pipeline (policy evaluation, attribution, budget enforcement, connector execution, record persistence), then re-establishes a new TLS connection to the upstream provider. Traffic to non-managed domains is forwarded as an opaque TCP tunnel with zero intervention.

Clients must trust the Axemere proxy CA certificate for TLS validation to succeed. Without this trust, clients will see certificate errors for all managed HTTPS domains.

Enabling MITM mode

Set the following environment variables before starting the gateway:

Example:

MVGC_PROXY_MITM_ENABLED=true
MVGC_MANAGED_DOMAINS=api.openai.com,*.anthropic.com,generativelanguage.googleapis.com

On startup, the gateway logs a structured warning when MITM mode is active:

level=WARN msg="MITM proxy mode enabled -- all HTTPS traffic to managed domains will be intercepted"
  managed_domains=["api.openai.com","*.anthropic.com"]
  ca_cert_url=http://<gateway-addr>/v1/proxy/ca.crt

The root CA key and cert are stored under MVGC_KEY_DIR (default ./keys) as proxy-ca.key (mode 0600) and proxy-ca.crt (mode 0644). They are generated automatically on first start if not present.

Retrieving the CA certificate

curl http://localhost:7080/v1/proxy/ca.crt > mvgc-proxy-ca.crt

This endpoint requires no authentication. It is only registered when MVGC_PROXY_MITM_ENABLED=true -- it returns 404 when MITM mode is disabled.

Trusting the CA certificate on client systems

Clients must install the CA cert into their trust store or pass it explicitly to their HTTP client library. See the per-platform guides linked above for OS-level trust store instructions.

curl (without system trust store):

curl --cacert mvgc-proxy-ca.crt --proxy http://localhost:7080 \
  https://api.openai.com/v1/chat/completions

Alternatively, set the CURL_CA_BUNDLE environment variable to the cert path so all curl invocations in your shell trust it:

export CURL_CA_BUNDLE=/path/to/mvgc-proxy-ca.crt

Python (requests / httpx):

export REQUESTS_CA_BUNDLE=/path/to/mvgc-proxy-ca.crt
# or set SSL_CERT_FILE for libraries that respect it
export SSL_CERT_FILE=/path/to/mvgc-proxy-ca.crt

Or pass it per-request:

import requests
response = requests.post(url, verify="/path/to/mvgc-proxy-ca.crt",
                         proxies={"https": "http://gateway:7080"})

Configuring HTTP clients to use the proxy

The gateway listens on port 7080 for proxy traffic. Most HTTP client libraries (curl, Python requests, Node.js node-fetch, Go's http.DefaultTransport) respect HTTPS_PROXY automatically:

export HTTPS_PROXY=http://localhost:7080

Replace localhost with your gateway address if running on a remote host.

App coverage: Most AI SDKs and CLI tools respect HTTPS_PROXY and are captured automatically. One known exception is ChatGPT Desktop (macOS), which uses a custom networking stack that bypasses HTTPS_PROXY. This is a limitation of the app, not the gateway. ChatGPT accessed via a web browser (chatgpt.com) is captured normally, as browsers respect system proxy settings.

Per-workload bypass

Policy YAML bundles can exclude specific domains from interception using the bypass_domains key. This is useful for domains that require certificate pinning, workload-identity mTLS, or direct connectivity for latency reasons.

Example -- see configs/examples/proxy-mitm.yaml for a full reference:

proxy:
  managed_domains:
    - api.openai.com
  bypass_domains:
    - internal-model.corp.example.com

Domains in bypass_domains always receive transparent TCP passthrough, even if they appear in the gateway-level MVGC_MANAGED_DOMAINS list.

Security considerations

• CA private key: The CA key ($MVGC_KEY_DIR/proxy-ca.key) must be protected with file mode 0600. The gateway enforces this on creation. Back up the key securely (encrypted storage) -- losing it does not affect existing client trust stores, but you will need to re-distribute a new CA cert if you regenerate the key.
• Key rotation: Rotating the CA key requires generating a new CA cert, re-distributing it to all clients, and re-installing it in every trust store. Plan rotations carefully.
• Leaf cert TTL: Default TTL is 24h. Reduce it (e.g., MVGC_PROXY_LEAF_CERT_TTL=1h) in high-security environments to limit the exposure window for any individual leaf cert.
• MITM warning: MITM mode logs a structured warning on every gateway restart. Review logs to confirm MITM mode is only active where intended.
• Certificate pinning: Clients that pin a specific certificate or public key for an upstream provider will reject the Axemere leaf certificate. Add such domains to bypass_domains in the policy YAML if pinning cannot be disabled.

Security Hardening

The following input-validation controls require no configuration changes; all are on by default.

SSRF Protection

The HTTP connector rejects outbound connections to RFC-1918 private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), IPv4 loopback (127.0.0.0/8), and link-local (169.254.0.0/16) before any packet is sent. This prevents a crafted target_host in an action request from routing connector traffic to internal services.

The check runs after DNS resolution: a hostname that resolves to a blocked private IP is rejected just the same as specifying the IP directly.

Operator override: If your deployment legitimately needs the gateway to reach a private endpoint (for example, an on-premise or locally-hosted AI model server), set MVGC_HTTP_CONNECTOR_TRUSTED_HOSTS to a comma-separated list of hostnames that should bypass the SSRF check:

MVGC_HTTP_CONNECTOR_TRUSTED_HOSTS=my-local-model.internal,gpu-server.corp.example.com

Only the exact hostnames listed are exempted; the block remains in effect for all other targets. Use this only for hosts you control and trust.

Request Body Size Limits

All JSON endpoints (including POST /v1/actions:execute and admin write endpoints) enforce a 10 MiB request body limit. Requests that exceed this limit receive HTTP 413.

Policy Bundle Path Traversal Prevention

Policy bundle files listed in the files: field of a bundle.yaml are validated to ensure they cannot reference paths outside the policy directory. A path such as ../secret.yaml is rejected at bundle load time with an error.

Operator note: All entries in bundle.yaml's files: list must be relative filenames without ../ components. Absolute paths are also rejected. If you see a load error mentioning "escapes policy directory", review your bundle file list for traversal sequences.

Webhook URL Validation

The SIEM webhook exporter (MVGC_EXPORT_WEBHOOK_URL) validates the configured URL at startup. The following are rejected:

• Non-HTTP(S) schemes (e.g., file://, ftp://)
• URLs with embedded credentials (e.g., https://user:pass@host/)

Only http:// and https:// URLs without embedded credentials are accepted.

Gateway Key Exposure in Reverse Proxy and Access Logs

Unlike the other controls in this section, this one requires action on infrastructure you operate; it is not something the gateway can enforce on your behalf.

Axemere gateway keys can be presented three ways: an Authorization: Bearer header, an X-Axemere-API-Key header, or embedded in the request path (/k/<key>/...). See Ways to present the key. The path form exists so that Authorization stays free to carry a customer's own upstream OAuth credentials unmodified (see the oauth credential mode); it is not the primary or preferred way to present the key.

When the path form is used, the gateway key appears in the URL of every proxied request. Axemere's own access logs already redact this path segment before writing; no customer action is required for logs the gateway itself produces.

That guarantee does not extend to infrastructure you operate. If you place your own reverse proxy, load balancer, ingress controller, or web server in front of or alongside the gateway, its access logs are outside Axemere's control and may capture the full request URI, including an embedded gateway key, unless you configure it otherwise.

Recommendation: configure any customer-operated reverse proxy, load balancer, or web server to redact or omit request URIs (or at minimum the path segment carrying credentials) from its own access logs. Most enterprise-grade gateways and proxies (Kong, Envoy, Traefik, NGINX, and equivalents) already support URI/path redaction in their logging configuration; use that existing capability rather than building something custom.

Lower-friction alternative: if you'd rather avoid this class of concern entirely, use the X-Axemere-API-Key header form instead of the path form. Headers are far less commonly captured by default proxy and load-balancer access-log configurations than full request URIs are.

Security Defaults and Hardened Behaviors

The following controls are active by default and require no configuration to enable.

Control Plane TLS

The gRPC connection to the Control Plane (MVGC_CP_ADDR) requires TLS. Ensure MVGC_CP_CA_CERT points to the CA certificate used to sign the Control Plane's TLS certificate. mTLS (mutual TLS) is enforced when both MVGC_NODE_CERT and MVGC_NODE_CERT_KEY are set alongside MVGC_CP_CA_CERT.

Admin API Per-IP Rate Limiting

Admin API endpoints (/v1/admin/...) enforce a per-source-IP rate limit: 20-request burst, refilling at 1 request per second. Requests that exceed the limit receive HTTP 429 with a Retry-After: 60 header.

Operator note: Monitoring systems that poll admin endpoints rapidly (e.g., health checks calling admin routes) should be adjusted to stay within the 1 req/sec sustained rate. The rate limit is per IP as seen by the gateway process (behind a load balancer, configure MVGC_PROXY_ENABLED=true so the real client IP is used).

Delegation Token Replay Protection

Each delegation token carries a JTI (JWT ID) that is tracked in memory after first verification. Presenting the same token a second time returns a 401 Unauthorized error with the reason replay.

Single-instance note: The JTI cache is in-process memory. In a multi-gateway deployment, replay protection applies only within the same process. For cross-instance replay protection, place a shared cache (Redis, Memcached) in front of the verification path; this is a known limitation.

Token rotation: Tokens are single-use. Issue a new delegation token for each client session rather than sharing a single long-lived token across parallel requests.

CORS

CORS is disabled by default. Browsers making cross-origin requests to the gateway API will receive no Access-Control-Allow-Origin header and the request will be blocked by the browser's same-origin policy.

To enable CORS (for example, when the gateway is accessed directly from a web application):

MVGC_CORS_ENABLED=true

Security note: Enabling CORS allows any origin (*). Only enable this when the gateway is explicitly serving browser-based clients and is protected by authentication at the application layer.

File and Directory Permissions

• Execution record log file (MVGC_RECORD_LOG_FILE) is created with mode 0600 (owner read/write only).
• The system directories /var/lib/mvgc and /var/log/mvgc are provisioned with mode 0700 (owner only).

Next steps