LiteLLM

What CostGraph scrapes from the LiteLLM proxy and how its token metrics are priced

LiteLLM reports input and output tokens as separate counters from its Prometheus callback, labelled with the virtual key, team, and end user of every request. See the upstream Prometheus metrics reference.

What CostGraph scrapes

Namespace	litellm
Pod selector	app.kubernetes.io/name=litellm
Port and path	4001 /metrics/
Metrics	names starting litellm_

Port 4001 is LiteLLM’s separate metrics server. The proxy’s own /metrics on port 4000 rejects a scrape without a virtual key, so turn the metrics server on wherever the proxy runs.

The metrics server has no authentication. Keep it off public ingress.

On Kubernetes

Enable it in the litellm-helm chart:

values.yaml

metricsServer:
  enabled: true
  port: 4001

On a virtual machine

Start the proxy with a metrics port:

litellm --config config.yaml --port 4000 --prometheus_metrics_port 4001

The CostGraph agent on the machine reads port 4001 on the loopback address. The metrics port listens on every interface by default, so block 4001 at your firewall. Connect the gateway with A virtual machine under Where it runs. See Virtual machine.

Turn on the metrics

Add the Prometheus callback to the proxy config:

values.yaml

proxy_config:
  litellm_settings:
    callbacks: ["prometheus"]
    enable_end_user_cost_tracking_prometheus_only: true
    prometheus_exclude_labels: ["client_ip", "user_agent", "user_email"]

enable_end_user_cost_tracking_prometheus_only puts the end user on the metrics, which is how CostGraph attributes usage to your customers. The excluded labels are not used by CostGraph and would otherwise leave your cluster with the scrape.

How the metrics map

Each becomes costgraph_gen_ai_usage_total, with the direction set by which metric the value came from:

LiteLLM metric	costgraph_usage_kind
litellm_input_tokens_metric	input
litellm_output_tokens_metric	output
litellm_input_cached_tokens_metric	cached_input
litellm_input_cache_creation_tokens_metric	cache_write

LiteLLM label	CostGraph label
model	costgraph_model
api_provider	costgraph_host
end_user	costgraph_sub_account

The unit is always tokens. model is the upstream model LiteLLM called, not the alias from your proxy config, so each call is priced at the rate of the provider that served it.

LiteLLM’s input count already includes cached and cache-write tokens, for every provider. CostGraph takes those out of input before pricing, so each token is billed once at its own rate.

Request latency, queue time and time to first token feed the serving performance view. LiteLLM reports time to first token for streaming requests only.

A response served from LiteLLM’s own response cache still reports its tokens, so CostGraph prices it as if the provider had served it.

Attribute usage to your customers

Pass your customer’s id as the user field on each request. LiteLLM records it as the end user:

curl https://litellm.example.com/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "user": "acme-corp", "messages": [{"role": "user", "content": "Hello"}]}'

A request with no user is reported as end_user="None" and stays unattributed.

Avoid counting usage twice

LiteLLM can also be connected as a billing integration, which reads its spend logs once a day. Connect one or the other for the same proxy, not both.

The same applies to the providers behind the proxy. If LiteLLM routes to OpenAI or Anthropic and you also connect that provider’s bill, the requests that went through LiteLLM appear in both.

AI gateways

Connecting, pricing, and sending usage yourself.