> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Honeycomb

> Learn how to export LLM Gateway traces and metrics to Honeycomb using OpenTelemetry integration.

This guide provides instructions for integrating [Honeycomb](https://www.honeycomb.io) with the TrueFoundry AI Gateway to export OpenTelemetry traces and metrics.

## What is Honeycomb?

Honeycomb is an observability platform designed for debugging and understanding complex distributed systems. It ingests OpenTelemetry data natively over OTLP (HTTP and gRPC) and provides a query builder, trace viewer, and dashboards for analysing LLM gateway behaviour in production.

### Key Features of Honeycomb

* **[Trace Query Builder](https://docs.honeycomb.io/investigate/query/)**: Slice and filter spans by any attribute, including TrueFoundry-emitted fields like `tfy.span_type` and `gen_ai.request.model`.
* **[Metrics Explorer](https://docs.honeycomb.io/investigate/metrics/)**: Visualise OTLP metrics over time with aggregations and groupings across models, users, and environments.
* **[Environments and Datasets](https://docs.honeycomb.io/get-started/environments/)**: Organise telemetry into named datasets per service or team, with the `x-honeycomb-dataset` header controlling routing.

## Prerequisites

Before integrating Honeycomb with TrueFoundry, ensure you have:

1. **TrueFoundry Account**: Create a [TrueFoundry account](https://www.truefoundry.com/register) and follow the instructions in our [Gateway Quick Start Guide](https://docs.truefoundry.com/gateway/quick-start).
2. **Honeycomb Account**: Sign up at [honeycomb.io](https://ui.honeycomb.io/signup) and create an ingest API key with **Send Events** permission.

## Understanding the Honeycomb Endpoint

Honeycomb exposes two regional base URLs:

| Region       | Base URL                       |
| ------------ | ------------------------------ |
| US (default) | `https://api.honeycomb.io`     |
| EU           | `https://api.eu1.honeycomb.io` |

TrueFoundry does **not** auto-append signal paths, so you must supply the full path including `/v1/traces` or `/v1/metrics`. For example, a US traces endpoint is `https://api.honeycomb.io/v1/traces`.

Your account region is shown in the Honeycomb UI URL — `ui.honeycomb.io` indicates US; `ui.eu1.honeycomb.io` indicates EU.

## Integration Steps

<Steps>
  <Step title="Create a Honeycomb Ingest API Key">
    1. Log into your Honeycomb account and navigate to **Team Settings** → **API Keys**.
    2. Click **Create API Key** and select the **Send Events** permission scope.
    3. Copy the generated API key — you will need it in the next step.

    <Tip>
      Honeycomb routes traces into a dataset matching your `service.name` attribute. TrueFoundry sets this to `tfy-llm-gateway` automatically, so your traces will appear under a dataset of that name unless you override it via the `x-honeycomb-dataset` header.
    </Tip>
  </Step>

  <Step title="Configure OTEL Export in TrueFoundry">
    1. Go to **AI Gateway** → **Controls** → **Settings** in the TrueFoundry dashboard.
    2. Scroll down to the **OTEL Config** section and click the edit (✏️) button.

    <Frame>
      <img src="https://mintcdn.com/truefoundry/m9STfp4rwJMcbGcj/images/honeycomb-otel-config-section.png?fit=max&auto=format&n=m9STfp4rwJMcbGcj&q=85&s=4d8d682fe69c7b7fcfbdaa6d1305c3b1" alt="TrueFoundry AI Gateway Settings page showing the OTEL Config section with Traces Endpoint and Headers" width="2880" height="1620" data-path="images/honeycomb-otel-config-section.png" />
    </Frame>

    3. Enable the **Otel Traces Exporter Configuration** toggle and fill in:

    | Field            | Value                                                                                      |
    | ---------------- | ------------------------------------------------------------------------------------------ |
    | **Toggle**       | Enabled                                                                                    |
    | **Protocol**     | HTTP Configuration                                                                         |
    | **Endpoint**     | `https://api.honeycomb.io/v1/traces` (US) or `https://api.eu1.honeycomb.io/v1/traces` (EU) |
    | **Encoding**     | Proto                                                                                      |
    | **Header Key**   | `x-honeycomb-team`                                                                         |
    | **Header Value** | `<your-honeycomb-api-key>`                                                                 |

    <Frame>
      <img src="https://mintcdn.com/truefoundry/m9STfp4rwJMcbGcj/images/honeycomb-traces-exporter-config.png?fit=max&auto=format&n=m9STfp4rwJMcbGcj&q=85&s=6296cf6af9bbac7e1bb5887d261ce8b8" alt="TrueFoundry OTEL Traces Exporter Configuration showing Honeycomb endpoint and x-honeycomb-team header" width="1476" height="1358" data-path="images/honeycomb-traces-exporter-config.png" />
    </Frame>

    4. Enable the **Otel Metrics Exporter Configuration** toggle and fill in:

    | Field            | Value                                                                                        |
    | ---------------- | -------------------------------------------------------------------------------------------- |
    | **Toggle**       | Enabled                                                                                      |
    | **Protocol**     | HTTP Configuration                                                                           |
    | **Endpoint**     | `https://api.honeycomb.io/v1/metrics` (US) or `https://api.eu1.honeycomb.io/v1/metrics` (EU) |
    | **Encoding**     | Proto                                                                                        |
    | **Header Key**   | `x-honeycomb-team`                                                                           |
    | **Header Value** | `<your-honeycomb-api-key>`                                                                   |

    5. Click **Save** to apply the configuration.

    <Note>
      To route metrics into a specific Honeycomb dataset rather than the default `unknown_metrics`, add a second header: key `x-honeycomb-dataset`, value set to your chosen dataset name. Click **+ Add Headers** in TrueFoundry to add multiple header pairs.
    </Note>
  </Step>

  <Step title="Verify the Integration">
    1. Make a request through the TrueFoundry AI Gateway.
    2. Log into your Honeycomb account and open the **Traces** tab on the `tfy-llm-gateway` dataset.
    3. You will see a list of traces with **Root Service** set to `tfy-llm-gateway`. Span names follow the pattern `MCPGateway: resources/list`, `ChatCompletion`, and so on depending on the operation type.

    <Frame>
      <img src="https://mintcdn.com/truefoundry/m9STfp4rwJMcbGcj/images/honeycomb-traces-list.png?fit=max&auto=format&n=m9STfp4rwJMcbGcj&q=85&s=036dd86b9782e26732573055d5804e9a" alt="Honeycomb Traces tab showing tfy-llm-gateway traces with Root Service, Root Name, Duration, and Trace ID columns" width="2568" height="1554" data-path="images/honeycomb-traces-list.png" />
    </Frame>

    4. Click any Trace ID to expand it and inspect individual spans. You will see the full span waterfall including nested child spans with latency breakdowns.

    <Frame>
      <img src="https://mintcdn.com/truefoundry/m9STfp4rwJMcbGcj/images/honeycomb-trace-waterfall.png?fit=max&auto=format&n=m9STfp4rwJMcbGcj&q=85&s=fada8980c0aaae07cde225df055d1a66" alt="Honeycomb expanded trace view showing span waterfall for a tfy-llm-gateway MCPGateway operation with nested spans" width="2570" height="1560" data-path="images/honeycomb-trace-waterfall.png" />
    </Frame>

    5. From the dataset **Overview** tab you can also monitor aggregate Trace Volume, Span Volume, and Error Volume over time to confirm data is flowing continuously.

    <Frame>
      <img src="https://mintcdn.com/truefoundry/m9STfp4rwJMcbGcj/images/honeycomb-dataset-overview.png?fit=max&auto=format&n=m9STfp4rwJMcbGcj&q=85&s=23486c44032863df934b803a51745518" alt="Honeycomb dataset overview showing Total Spans, Total Errors, Trace Volume, Span Volume, and Error Volume charts for tfy-llm-gateway" width="2568" height="1554" data-path="images/honeycomb-dataset-overview.png" />
    </Frame>
  </Step>
</Steps>

## Configuration Reference

| Configuration                       | Value                                     |
| ----------------------------------- | ----------------------------------------- |
| **Traces Endpoint (US)**            | `https://api.honeycomb.io/v1/traces`      |
| **Traces Endpoint (EU)**            | `https://api.eu1.honeycomb.io/v1/traces`  |
| **Metrics Endpoint (US)**           | `https://api.honeycomb.io/v1/metrics`     |
| **Metrics Endpoint (EU)**           | `https://api.eu1.honeycomb.io/v1/metrics` |
| **Protocol**                        | HTTP                                      |
| **Encoding**                        | Proto                                     |
| **Auth Header Key**                 | `x-honeycomb-team`                        |
| **Auth Header Value**               | Your Honeycomb ingest API key             |
| **Dataset Header Key (optional)**   | `x-honeycomb-dataset`                     |
| **Dataset Header Value (optional)** | Your chosen Honeycomb dataset name        |