NVIDIA NIM — Self-Hosted Containers

For: Platform operators integrating a self-hosted NVIDIA NIM deployment running on their own GPU infrastructure.

Providers | Custom Connectors | Provider Pricing | NVIDIA NIM — Self-Hosted Containers

"NVIDIA NIM" is not one thing. It covers three distinct offerings, and only two of them are standard, built-in Axemere providers. This page is about the third: the one that isn't.

The three NIM tiers
Why self-hosted NIM isn't a standard provider
Setting it up as a custom connector
A concrete shape: self-hosted gateway + self-hosted NIM
Cost accounting is yours to own
See also

The three NIM tiers

Tier	What it is	Where it runs	Axemere support
Free Endpoint	NVIDIA-hosted, rate-limited, unbilled	NVIDIA's infrastructure	Standard provider — `nvidia-nim`
Partner Endpoint	NVIDIA-hosted API surface, backed by third-party inference providers (Together AI, OpenRouter, Deepinfra, and others) on the backend	NVIDIA's infrastructure	Standard provider — `nvidia-nim`
Self-hosted NIM containers	You download the NIM container and run it yourself, licensed under NVIDIA AI Enterprise	Your GPU infrastructure	Not a standard provider — see below

The first two tiers are already fully supported. nvidia-nim is a built-in openai_compat provider (see Providers → All Built-in Providers) and routes through integrate.api.nvidia.com like any other provider — enable it, add a credential, and go. Nothing on this page applies to those two tiers.

The third tier is what this page covers.

Why self-hosted NIM isn't a standard provider

A standard provider (like the nvidia-nim endpoint above) has one thing a self-hosted NIM container never will: a price NVIDIA sets and Axemere can look up. Self-hosted NIM breaks that assumption at the root:

There's no upstream endpoint to route to. The container runs on infrastructure you own: a different host per customer, sometimes a different host per deployment. There's nothing for Axemere to hard-code.
There's no NVIDIA-set price to look up. NVIDIA AI Enterprise licenses self-hosted NIM per-GPU (published at $1/GPU-hour or $4,500/GPU-year, via build.nvidia.com/models). That's a licensing cost between you and NVIDIA; it never touches Axemere, and it isn't a per-token price the gateway's cost model can express. See Cost accounting is yours to own below.

Both gaps are exactly what the Custom Connectors mechanism exists for: an upstream you own, at an address only you know, with a price only you can set. Self-hosted NIM isn't a special case Axemere doesn't support; it's the standard case that custom connectors were built for.

Don't confuse this with BYOK. BYOK means the caller supplies a provider API key per-request and Axemere never stores it. Self-hosted NIM is about who runs the compute: the inference server itself lives on your infrastructure, not NVIDIA's. You still register a normal credential for it (see below); what's different is the base_url it points to.

Setting it up as a custom connector

Register the container as a custom connector, with compat_type: openai_compat and base_url pointing at your NIM container's address (e.g. http://nim-host.internal:8000/v1).
Set auth_mode to match however you've secured the container: passthrough if the container is open on a private network with no auth of its own, or bearer/header with a credential_id if you've put an API key or token in front of it.
Set a price on it. Open Provider Pricing, click Add Override, and select your connector from the dropdown, the same flow used for any custom connector. This is the org_connector_pricing entry: a per-org, per-connector price you set, because there's no NVIDIA list price to inherit.

Managed gateway users do steps 1–2 from console.axemere.ai/providers → Add Custom Connector instead of editing YAML. See Custom Connectors for the full field reference either way.

A concrete shape: self-hosted gateway + self-hosted NIM

The most common deployment isn't "managed gateway talks to a NIM container somewhere out on the internet"; it's both pieces living inside the same infrastructure you control:

A gateway relay deployed inside your own cluster or VPC, routing to a NIM container on GPU nodes in that same network, using a private base_url that's never reachable from outside your infrastructure. This keeps model traffic entirely inside your perimeter: no request or response body ever leaves your network, which is often the point of running NIM self-hosted in the first place.

Register the connector exactly as described above; only the base_url changes to reflect an internal address instead of a public one.

Cost accounting is yours to own

Axemere's cost model is per-token: cost_usd = (tokens_in × input_price) + (tokens_out × output_price), computed from the Provider Pricing override you set on the connector. That model has no field for GPU-hours or a flat annual license fee: it cannot represent NVIDIA AI Enterprise's $1/GPU-hour or $4,500/GPU-year licensing cost, and it never will without a genuinely different pricing dimension.

In practice this means:

Axemere gives you request routing, authentication, and (if you set a per-token override) a consistent view of request volume in Records and Analytics, the same visibility every other connector gets.
Axemere does not track your NVIDIA AI Enterprise licensing cost. That's a bill NVIDIA sends you directly, independent of anything routed through the gateway, and reconciling GPU-hour spend against your license is on you.

If you want request-volume visibility even though it won't reflect true GPU-hour cost, set a per-token override anyway: $0 is a valid input if you'd rather track raw request/token counts than force an inaccurate per-token approximation of a per-GPU-hour bill.