> ## Documentation Index > Fetch the complete documentation index at: https://docs.costgraph.ai/llms.txt > Use this file to discover all available pages before exploring further. # Request header attribution on macOS > Attribute request bytes on a Mac to the X-CostGraph headers your services send, through the agent's local attribution proxy On a Mac, the [CostGraph Agent](/costgraph/agent) can read the `X-CostGraph-` headers your services send and attribute each request's bytes to their values. A Mac reports the same `attribution_` tags as a Linux host. Each request is attributed to the process that sent it. The feature is off by default. Header names become tag keys by the same rules on every host. See [How header names become keys](/costgraph/network/usage#how-header-names-become-keys). ## Decide whether you need this Most Macs do not need request header attribution. The agent already reports each Mac's network flows without any setup: the destination, the bytes in each direction, and the process that opened the connection. For Linux hosts and Kubernetes, header attribution works with no proxy. Turn this feature on only when services that run on the Mac send `X-CostGraph-*` headers and you need their bytes split by those values. It has a cost: * HTTPS requests are intercepted, so clients must trust the agent's CA. * Programs that pin certificates fail through the proxy. * Go programs and other clients that verify through the keychain fail until the CA is in the keychain. * Only requests that go through the proxy are attributed. If you accept these costs, the rest of this page sets it up. ## How it works A Linux host reads headers with eBPF. A Mac uses the **attribution proxy**, a local forwarding proxy that the agent runs on a loopback port. Requests that clients send through the proxy are attributed. Requests that do not use it are not. For an HTTPS request, the proxy terminates TLS with a certificate authority (CA) that the agent creates on the Mac. It then opens its own TLS connection to the destination. The client must trust that CA, as described in [Trust the CA](#trust-the-ca). The proxy keeps only the values of `X-CostGraph-*` request headers. It does not keep or send any other header, path, query string, or body. The CA private key stays on the Mac. The agent matches each request to the process that opened the connection. Requests to Apple services and certificate authorities, such as `apple.com`, `icloud.com`, and `letsencrypt.org`, pass through the proxy without interception. ## Requirements * The agent is installed on the Mac with Homebrew and registers with CostGraph. See the macOS tab in [Installation](/costgraph/agent#installation). * `metrics_only` is not `true`. Attribution needs a registered agent. * Network flow tracing is on. `network_trace` defaults to `true` in the configuration file. * Clients send their requests through the proxy and trust its CA. * For HTTPS, clients offer application-layer protocol negotiation (ALPN) in their TLS handshake. curl and browsers do. ## Enable attribution Add an `http_attribution` block to the agent configuration file at `$(brew --prefix)/etc/costgraph_agent/config.yaml`. The following example listens on a fixed port. The agent exports the CA files that clients read to `/usr/local/share/costgraph/` unless you set other paths. ```yaml theme={null} http_attribution: enabled: true listen_port: 8888 ``` Restart the service to apply the change. ```bash theme={null} sudo brew services restart costgraph-agent ``` The agent refuses to start when `enabled: true` is combined with `metrics_only: true`. It also refuses a CA path that is not absolute, and an `upstream` other than `auto`, `none`, or a `host:port`. Turns attribution on. When you set it back to `false`, the agent removes its CA, the exported CA files, and any keychain trust it added. It also restores the system proxy. Loopback port the proxy listens on at `127.0.0.1`. `0` picks a free port each time the agent starts. Set a fixed port when you configure clients to use the proxy yourself. Points the Mac's web and secure web proxy settings at the attribution proxy. See [Use the system proxy](#use-the-system-proxy). Where the proxy sends requests. `auto` forwards through the web proxy the Mac already has configured, or connects directly when there is none. `none` connects directly. A `host:port` value forwards through that HTTP proxy. An upstream proxy that also intercepts TLS breaks HTTPS through the attribution proxy. Adds the agent's CA to the System keychain. See [Trust the CA](#trust-the-ca). Absolute path where the agent writes the public CA certificate. The file is readable by every user. Defaults to `/usr/local/share/costgraph/attribution-ca.pem`. Absolute path where the agent writes the macOS system root certificates followed by the agent's CA. The file is readable by every user. Defaults to `/usr/local/share/costgraph/attribution-bundle.pem`. ## Point clients at the proxy A client uses the proxy when its own proxy setting names it. Many clients read the `HTTPS_PROXY` and `HTTP_PROXY` environment variables. ```bash theme={null} export HTTPS_PROXY=http://127.0.0.1:8888 export HTTP_PROXY=http://127.0.0.1:8888 ``` ### Use the system proxy With `system_proxy: true`, the agent sets the web and secure web proxy of the Mac's active network service to the attribution proxy after the proxy is healthy. Clients that follow the macOS proxy settings then use it without further configuration. Every client that follows the system proxy goes through interception. A client that does not trust the agent's CA fails every HTTPS request. Turn on `system_proxy` only on a Mac where those clients trust the CA. The agent records the previous settings and restores them exactly when it stops, when the proxy goes down, and after a crash. Restoring also clears a proxy host and port that the agent set. If the proxy keeps failing, the agent restarts it with a growing delay and then retries every five minutes. The system proxy stays restored until the proxy is healthy again. The agent does not apply the system proxy in these cases: * The network service uses an automatic proxy configuration (PAC) file. * The network service already uses a different proxy and `upstream` is `none`. With `auto`, the existing proxy becomes the upstream instead. ## Trust the CA The CA matters only for HTTPS. Requests over plain HTTP are not intercepted, so they are attributed with no CA at all. For HTTPS, a client that does not trust the CA fails with a certificate error. macOS does not let the agent trust its CA by itself, because an administrator trust change needs interactive approval. Give clients the CA in one of these ways: * Point clients at the exported files (`ca_cert_path` and `ca_bundle_path`, by default under `/usr/local/share/costgraph/`). This works for clients that read a certificate file. * Set `trust_system_keychain: true` only on a Mac where the trust change can be approved, such as one managed by mobile device management (MDM). If macOS refuses the change, attribution stays off. * Deliver the certificate from `ca_cert_path` to the System keychain with your MDM. Clients that verify through the keychain need this. The agent writes both files each time it starts. The CA is valid for one year. The agent replaces it when it starts within 30 days of expiry. When you turn attribution off, the agent deletes both files. If the macOS system roots at `/etc/ssl/cert.pem` are missing or empty, the agent does not write the bundle and attribution stays off. Set the variable each client reads. | Variable | Points at | Effect | | - | - | - | | `SSL_CERT_FILE` | The bundle | Replaces the default certificate list. | | `REQUESTS_CA_BUNDLE` | The bundle | Replaces the default certificate list. | | `CURL_CA_BUNDLE` | The bundle | Replaces the default certificate list. | | `NODE_EXTRA_CA_CERTS` | The CA certificate | Adds the CA to the default list. | The first three variables replace the client's default list. The bundle includes the system roots, so the client still verifies servers that the proxy does not intercept. ```bash theme={null} export SSL_CERT_FILE=/usr/local/share/costgraph/attribution-bundle.pem export REQUESTS_CA_BUNDLE=/usr/local/share/costgraph/attribution-bundle.pem export CURL_CA_BUNDLE=/usr/local/share/costgraph/attribution-bundle.pem export NODE_EXTRA_CA_CERTS=/usr/local/share/costgraph/attribution-ca.pem ``` ### Client support The following results were measured through the attribution proxy. | Client | Trust setting | Result | | - | - | - | | curl from macOS (`/usr/bin/curl`) | System keychain | Trusted and attributed once the CA is in the System keychain. | | curl from Homebrew or another build that uses its own certificate list | `CURL_CA_BUNDLE` | Trusted and attributed. | | `nscurl` | System keychain | Trusted and attributed once the CA is in the System keychain. | | Python 3.12 or later, `urllib` | `SSL_CERT_FILE` | Trusted and attributed. | | Python `requests` | `REQUESTS_CA_BUNDLE` or `CURL_CA_BUNDLE` | Trusted and attributed. `requests` ignores `SSL_CERT_FILE`. | | Python `urllib3` | `SSL_CERT_FILE` | Trusted and attributed. | | Node.js | `NODE_EXTRA_CA_CERTS` | Trusted and attributed. | | Clients that send no ALPN, such as the macOS system Python 3.9 | None | Passed through unchanged and not attributed. | | Go programs, such as `gh`, `terraform`, and `kubectl` | System keychain | TLS fails unless the CA is trusted in the System keychain. | | Safari, Xcode, and apps that use `URLSession` | System keychain | TLS fails unless the CA is trusted in the System keychain. | Go programs on macOS ignore `SSL_CERT_FILE` and verify through the keychain. Keep Go tools off the proxy on a Mac where the keychain does not trust the CA. Python 3.13 and later verify certificates strictly, and the agent's CA meets those checks. ## Verify attribution Check that the proxy is up. The agent serves its local metrics on port 9101. ```bash theme={null} curl -s http://localhost:9101/metrics | grep costgraph_attribution_proxy_state ``` A value of `1` means the proxy is up. Then send a tagged request through the proxy. ```bash theme={null} curl --proxy http://127.0.0.1:8888 \ --cacert /usr/local/share/costgraph/attribution-ca.pem \ -H "X-CostGraph-Tenant: acme" \ -o /dev/null -w '%{http_code}\n' https://github.com/ ``` The command prints `200`. The proxy counts the request in `costgraph_attribution_proxy_records_total{outcome="received"}` on the same local endpoint. ## Read the metrics The agent ships two request series to CostGraph. They are not served on the local `/metrics` endpoint. | Series | Value | | - | - | | `costgraph_flowtrace_http_req_bytes_total` | Bytes the client sent. | | `costgraph_flowtrace_http_resp_bytes_total` | Bytes the client received. | Each sample holds the bytes counted since the previous collection. Both series carry these labels: * `attribution_` for each `X-CostGraph-` header on the request * `server_address`, the host the request was sent to * `name` and `effective_name`, the process that sent the request * `src_ip`, `dst_ip`, and `dst_port` of the outbound connection The tags are exact. The byte counts are estimates built from the decoded request and response, which includes the request line, headers, and body. Measured against curl, response bytes are within about 0.01 percent. Request bytes leave out headers that the proxy adds when it forwards the request. HTTP/2 sizes are decoded sizes, not compressed bytes on the wire. The agent also reports the health of the proxy. | Metric | Meaning | | - | - | | `costgraph_attribution_proxy_state` | `0` stopped, `1` up, or `2` degraded after repeated failures. | | `costgraph_attribution_proxy_restarts_total` | Number of proxy restarts. | | `costgraph_attribution_proxy_records_total` | Request records by `outcome`, which is `received`, `dropped_full`, or `invalid`. | | `costgraph_flowtrace_http_spans_dropped_total` | Requests not fully attributed, by `reason`. | The `reason` label on dropped requests has three values: * `unowned` means no process was found for the request. It is published without `name`. * `cap-exceeded` means a limit was reached. See [Limits](#limits). * `malformed` means the request record was invalid and was dropped. ## Limits * **8 tag keys per request** and **256 bytes per value**, as on Linux. * **64 server addresses per process.** Requests to further hosts are published without `server_address`. * **256 distinct tag sets per process** in each collection. Requests with further tag sets are published without tags. * Interception covers only clients that offer ALPN. Other HTTPS clients pass through unattributed. * Clients that do not send requests through the proxy are not attributed. * Tart virtual machine guests on the Mac are not covered. ## Troubleshoot attribution When attribution cannot start, the agent keeps running without it and logs a line that starts with `http attribution is off`, followed by the reason. Read it in `$(brew --prefix)/var/log/costgraph_agent.log` or `$(brew --prefix)/var/log/costgraph_agent.error.log`. | Reason in the log | Fix | | - | - | | `trust the interception CA` | macOS refused the keychain trust change. Set `trust_system_keychain` to `false` and use the exported files, or approve the change through MDM. | | `write the CA bundle` | The system roots at `/etc/ssl/cert.pem` are missing or empty. Restore them. The attribution proxy stays off until the file is valid. | | `state dir` | The `attribution` directory in the agent's state directory is not owned by the user the agent runs as. The state directory is `/Library/Application Support/costgraph-agent` by default. Make root its owner. | | `uses an automatic proxy configuration` | The network service uses a PAC file. Turn off the automatic proxy configuration, or set `system_proxy` to `false`. | | `already uses the proxy` | Another proxy is set and `upstream` is `none`. Set `upstream` to `auto` to forward through it, or set `system_proxy` to `false`. | With `network_trace: false`, the agent instead logs that HTTP attribution needs network trace and stays off. Set `network_trace` to `true`. The following problems appear in the client rather than in the agent log. | Symptom | Cause | Fix | | - | - | - | | HTTPS requests fail with a certificate error. | The client does not trust the agent's CA. | Set the variable the client reads, as described in [Trust the CA](#trust-the-ca). | | Requests succeed but are not attributed. | The client sends no ALPN, so the proxy passes its connection through. | Use a client that offers ALPN. | | A Go tool such as `gh` or `terraform` fails TLS. | Go programs on macOS verify only through the keychain. | Trust the CA in the System keychain, or keep the tool off the proxy. | This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.