> ## Documentation Index
> Fetch the complete documentation index at: https://docs.costgraph.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LiteLLM

> What CostGraph scrapes from the LiteLLM proxy and how its token metrics are priced

LiteLLM reports input and output tokens as separate counters from its
Prometheus callback, labelled with the virtual key, team, and end user of every
request. See the upstream
[Prometheus metrics reference](https://docs.litellm.ai/docs/proxy/prometheus).

## What CostGraph scrapes

| | |
| - | - |
| Namespace | `litellm` |
| Pod selector | `app.kubernetes.io/name=litellm` |
| Port and path | `4001` `/metrics/` |
| Metrics | names starting `litellm_` |

Port `4001` is LiteLLM's separate metrics server. The proxy's own `/metrics` on
port `4000` rejects a scrape without a virtual key, so turn the metrics server
on wherever the proxy runs.

The metrics server has no authentication. Keep it off public ingress.

### On Kubernetes

Enable it in the `litellm-helm` chart:

```yaml values.yaml theme={null}
metricsServer:
  enabled: true
  port: 4001
```

### On a virtual machine

Start the proxy with a metrics port:

```bash theme={null}
litellm --config config.yaml --port 4000 --prometheus_metrics_port 4001
```

The [CostGraph agent](/costgraph/agent) on the machine reads port `4001` on the
loopback address. The metrics port listens on every interface by default, so
block `4001` at your firewall.
Connect the gateway with **A virtual machine** under **Where it runs**. See
[Virtual machine](/costgraph/integrations/ai-gateways#virtual-machine).

## Turn on the metrics

Add the Prometheus callback to the proxy config:

```yaml values.yaml theme={null}
proxy_config:
  litellm_settings:
    callbacks: ["prometheus"]
    enable_end_user_cost_tracking_prometheus_only: true
    prometheus_exclude_labels: ["client_ip", "user_agent", "user_email"]
```

`enable_end_user_cost_tracking_prometheus_only` puts the end user on the
metrics, which is how CostGraph attributes usage to your customers. The
excluded labels are not used by CostGraph and would otherwise leave your
cluster with the scrape.

## How the metrics map

Each becomes
[`costgraph_gen_ai_usage_total`](/costgraph/integrations/ai-gateways#send-from-anything-else),
with the direction set by which metric the value came from:

| LiteLLM metric | `costgraph_usage_kind` |
| - | - |
| `litellm_input_tokens_metric` | `input` |
| `litellm_output_tokens_metric` | `output` |
| `litellm_input_cached_tokens_metric` | `cached_input` |
| `litellm_input_cache_creation_tokens_metric` | `cache_write` |

| LiteLLM label | CostGraph label |
| - | - |
| `model` | `costgraph_model` |
| `api_provider` | `costgraph_host` |
| `end_user` | `costgraph_sub_account` |

The unit is always `tokens`. `model` is the upstream model LiteLLM called, not
the alias from your proxy config, so each call is priced at the rate of the
provider that served it.

LiteLLM's input count already includes cached and cache-write tokens, for every
provider. CostGraph takes those out of input before pricing, so each token is
billed once at its own rate.

Request latency, queue time and time to first token feed the serving
performance view. LiteLLM reports time to first token for streaming requests
only.

<Note>
  A response served from LiteLLM's own response cache still reports its tokens,
  so CostGraph prices it as if the provider had served it.
</Note>

## Attribute usage to your customers

Pass your customer's id as the `user` field on each request. LiteLLM records it
as the end user:

```bash theme={null}
curl https://litellm.example.com/chat/completions \
  -H "Authorization: Bearer $LITELLM_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "user": "acme-corp", "messages": [{"role": "user", "content": "Hello"}]}'
```

A request with no `user` is reported as `end_user="None"` and stays
unattributed.

## Avoid counting usage twice

LiteLLM can also be connected as a [billing integration](/costgraph/integrations/litellm),
which reads its spend logs once a day. Connect one or the other for the same
proxy, not both.

The same applies to the providers behind the proxy. If LiteLLM routes to OpenAI
or Anthropic and you also connect that provider's bill, the requests that went
through LiteLLM appear in both.

<Card title="AI gateways" icon="chart-line" href="/costgraph/integrations/ai-gateways">
  Connecting, pricing, and sending usage yourself.
</Card>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.