Troubleshooting Guide

For: Developers and operators diagnosing issues with Axemere Gateway.

Table of Contents

• HTTP Status Codes and What They Mean
  • 400 Bad Request
  • 401 Unauthorized
  • 202 Accepted — Approval Required
  • 403 Forbidden
  • 429 Too Many Requests — Rate Limited
  • 500 Internal Server Error
• Understanding Denial Reason Codes
• Health Check Steps
  • Gateway health
  • Control Plane connectivity
• Reading Gateway Logs
  • Finding a specific request
  • Log fields reference
• Reading the Decision Trace
• Common Scenarios
  • Request always returns 403 even with a valid API key
  • Workload not found
  • API key expired or revoked
  • Rate limit keeps firing unexpectedly
  • Streaming request hangs or returns empty
  • Execution record shows zero tokens
  • CP connection status shows "offline"
  • Direct/proxy-mode request to Gemini or Perplexity returns a provider-side 400/401
• See also

HTTP Status Codes and What They Mean

400 Bad Request

The request body is missing required fields or is malformed.

Common causes:

Check the error field in the JSON response body for the specific missing field.

401 Unauthorized

Authentication failed. In managed mode, every request requires a valid API key in the Authorization: Bearer <key> header.

Common causes:

Confirm the key hash:

# Compute SHA-256 of the key (what the gateway stores):
echo -n "$API_KEY" | sha256sum

Compare against what the CP has on record:

grpcurl -H "Authorization: Bearer $CP_ADMIN_TOKEN" \
  -d '{"org_id": "'"$ORG_ID"'"}' \
  $CP_GRPC_ADDR mvgc.v1.OrgService/ListAPIKeys | jq '.api_keys[] | {name, status, expires_at}'

202 Accepted — Approval Required

Not an error. A policy rule matched decision: require_approval. The response includes an approval_id. Poll for the decision:

curl -s "$GATEWAY_URL/v1/admin/approvals/$APPROVAL_ID" \
  -H "MVGC-Admin-Token: $ADMIN_TOKEN" | jq '{status, decision}'

Once approved, re-submit the original request. See the approval workflows guide.

403 Forbidden

The request was received and authenticated, but the policy engine denied it. The response body always includes a decision_trace with reason_codes; read it before investigating further.

Common reason codes:

See Reading the Decision Trace for how to extract these fields.

429 Too Many Requests — Rate Limited

A rate_limit policy rule fired. The response includes a Retry-After header (in seconds).

# Check the rate limit and wait
RETRY_AFTER=$(curl -sI -X POST "$GATEWAY_URL/v1/actions:execute" \
  -H "Authorization: Bearer $API_KEY" \
  ... | grep -i "retry-after" | awk '{print $2}')
sleep "$RETRY_AFTER"

If the rate limit fires more aggressively than expected, check the requests_per_minute value in your policy bundle:

curl -s "$GATEWAY_URL/v1/admin/policies" \
  -H "MVGC-Admin-Token: $ADMIN_TOKEN" | jq '.' | grep -A 5 "rate_limit"

500 Internal Server Error

Unexpected gateway error. Check gateway logs immediately:

# systemd
sudo journalctl -u mvgc-gateway --since "5 minutes ago" | grep '"level":"ERROR"'

# Docker Compose
docker compose logs gateway --since 5m | grep '"level":"ERROR"'

Look for the request_id from your client (or the X-Request-ID response header) to correlate the log entry.

Understanding Denial Reason Codes

Every 403 response from the gateway includes a decision_trace field:

{
  "decision": "deny",
  "reason": "target_not_allowed",
  "decision_trace": {
    "schema": "mvgc.decision_trace.v1",
    "decision": "deny",
    "reason_codes": ["target_not_allowed"],
    "evaluated_at": "2026-04-01T12:34:56Z",
    "attributes": {
      "policy_bundle_id": "default",
      "connector_id": "openai",
      "target_host": "api.openai.com",
      "workload_id": "wl-my-app"
    }
  }
}

The reason_codes array maps to policy evaluation layers:

Health Check Steps

Gateway health

curl -s "$GATEWAY_URL/healthz" | jq .

Expected response:

{
  "status": "ok",
  "version": "0.3.69",
  "node_id": "node-abc123",
  "cp_status": "connected",
  "connectors": [
    {"id": "openai", "version": "1.0.0"},
    {"id": "anthropic", "version": "1.0.0"},
    {"id": "gemini", "version": "1.0.0"},
    {"id": "azure_openai", "version": "1.0.0"},
    {"id": "cohere", "version": "1.0.0"},
    {"id": "generic_http", "version": "1.0.0"}
  ]
}

Diagnose specific fields:

Control Plane connectivity

For managed deployments, verify gRPC connectivity:

grpcurl -plaintext $CP_GRPC_ADDR list mvgc.v1.RegistryService

If this fails, check:

1. CP_GRPC_ADDR is reachable (firewall, VPC peering, DNS)
2. MVGC_CP_TOKEN on the gateway is valid
3. The gateway process is running: check sudo systemctl status mvgc-gateway or docker compose ps

Reading Gateway Logs

The gateway emits structured JSON logs (slog format). Each log line is a JSON object.

Finding a specific request

Every executed request generates a log entry tagged with request_id. If you passed a request_id in the body, search for it directly. If not, use the record_id from the response:

# systemd
sudo journalctl -u mvgc-gateway --since "10 minutes ago" | grep '"record_id":"rec-abc123"'

# Docker Compose
docker compose logs gateway --since 10m | grep '"record_id":"rec-abc123"'

# Log file
grep '"record_id":"rec-abc123"' /var/log/mvgc/gateway.log

Log fields reference

Reading the Decision Trace

Every response from POST /v1/actions:execute includes a decision_trace object, regardless of whether the decision was allow or deny. For allowed requests:

curl -s -X POST "$GATEWAY_URL/v1/actions:execute" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{...}' | jq '{
    decision: .decision,
    reason: .reason,
    trace: .decision_trace
  }'

For execution records (after the fact):

curl -s "$GATEWAY_URL/v1/records/$RECORD_ID" \
  -H "Authorization: Bearer $API_KEY" | jq '.decision_trace'

The attributes map in decision_trace contains the values that were active at evaluation time; useful for verifying that the correct policy bundle, credential, and connector were selected.

Common Scenarios

Request always returns 403 even with a valid API key

1. Check the decision trace: the reason_codes array pinpoints the failing layer.
2. Verify workload registration: curl -s "$GATEWAY_URL/v1/admin/workloads" -H "MVGC-Admin-Token: $ADMIN_TOKEN" | jq '.workloads[] | .workload_id'
3. Verify target allowlist: confirm api.openai.com (or your target) appears in the targets policy layer.
4. Check org_id: the org_id in the request body must match the org of the API key. Managed gateway enforces this strictly.

Workload not found

# Register the workload:
curl -s -X PUT "$GATEWAY_URL/v1/admin/workloads" \
  -H "MVGC-Admin-Token: $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"workload_id": "wl-my-app", "org_id": "'"$ORG_ID"'", "name": "My App"}'

# Confirm:
curl -s "$GATEWAY_URL/v1/admin/workloads" \
  -H "MVGC-Admin-Token: $ADMIN_TOKEN" | jq '.workloads[] | select(.workload_id == "wl-my-app")'

API key expired or revoked

Revocation is permanent; you must create a new key:

grpcurl -H "Authorization: Bearer $CP_ADMIN_TOKEN" \
  -d '{"org_id": "'"$ORG_ID"'", "name": "replacement-key", "expires_at": "2027-01-01T00:00:00Z"}' \
  $CP_GRPC_ADDR mvgc.v1.OrgService/CreateAPIKey

Rate limit keeps firing unexpectedly

The rate limiter uses a token bucket per (org_id, workload_id) pair. If you have multiple applications sharing a workload, their combined rate counts against the single bucket. Solutions:

1. Register separate workloads per application.
2. Increase requests_per_minute in the policy bundle.
3. Add caller_id-based sub-limits in the policy DSL.

Streaming request hangs or returns empty

For SSE streaming ("stream": true in action.params):

1. Verify the provider supports streaming for the requested model.
2. Confirm the client reads SSE chunks as they arrive (do not buffer the full response before parsing).
3. Check that Content-Type: text/event-stream appears in the response headers:

   curl -v -X POST "$GATEWAY_URL/v1/actions:execute" ... 2>&1 | grep "content-type"
   

4. Streaming is exempt from MVGC_CONNECTOR_TIMEOUT; increasing that variable will not help streaming 502s. Streaming responses run until the client disconnects; the gateway does not impose an upper time limit. If you are seeing 502 {"error":"upstream request failed"} mid-stream, the most common causes are:
   • The upstream provider closed the connection (check provider status / rate limits).
   • A reverse proxy or load balancer in front of the gateway has its own timeout set (e.g. Cloud Run request timeout, nginx proxy_read_timeout). Increase that value.

Execution record shows zero tokens

The gateway reads token counts from the upstream provider's response body. If the connector does not recognize the response format, it falls back to zero. Verify:

1. The correct connector is selected (check decision_trace.attributes.connector_id).
2. The model name is spelled correctly (the connector parses usage.prompt_tokens / usage.completion_tokens for OpenAI-compatible responses).

CP connection status shows "offline"

The OfflineDetector marks the CP offline after 3 consecutive gRPC errors. It does not automatically recover until the gateway restarts or a successful call completes.

1. Verify MVGC_CP_ADDR is gcp.cp.axemere.ai:9090 and port 9090 outbound is not blocked
2. Check gateway logs for gRPC error details: look for "offline" in the log output
3. If the CP is reachable, restart the gateway to reset the detector:
   • systemd: sudo systemctl restart mvgc-gateway
   • Docker Compose: docker compose restart gateway

Direct/proxy-mode request to Gemini or Perplexity returns a provider-side 400/401

In transparent proxy mode, the gateway does no path translation; whatever path your client sends after the /proxy/{provider}/ prefix is forwarded verbatim upstream. If that path or auth header doesn't match the provider's actual convention, the provider itself rejects the request with a 400 or 401 that gives no hint the real problem is "wrong endpoint." This looks identical to a credential problem, so check the path/header combo before assuming the stored credential is bad:

See Gemini SDK and Perplexity for full base-URL examples of each.

Confirm what actually went out: gateways on v0.65.7+ echo an X-MVGC-Upstream-Path response header showing the exact path the request was dispatched to upstream; compare it against the provider's documented path to confirm a mismatch instead of guessing. This header is currently available on dev deployments only; it has not yet been promoted to staging or production.

See also

• Admin API Reference
• Developer Integration Guide: request schema, response codes
• FAQ: policy, billing, and proxy questions
• Approval workflows
• Operations guide: log rotation, metrics, SIEM export