> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hexr.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability Stack

> OpenTelemetry Collector, Prometheus, Grafana, and Loki — full observability for every agent, tool call, and LLM interaction.

## Components

| Component            | Role                                               |
| -------------------- | -------------------------------------------------- |
| **OTel Collector**   | Receives traces, metrics, and logs from all agents |
| **Prometheus**       | Time-series metrics storage and alerting           |
| **Grafana**          | Dashboards and visualization                       |
| **Loki**             | Log aggregation                                    |
| **Tempo** (optional) | Distributed trace storage                          |

***

## Data Flow

<Tabs>
  <Tab title="Agent Pods">
    Every agent pod sends telemetry (traces, metrics, logs) to the OTel Collector:

    * Agent 1 → OTel Collector
    * Agent 2 → OTel Collector
    * Agent 3 → OTel Collector
  </Tab>

  <Tab title="Platform Services">
    Platform services also emit telemetry:

    * Credential Injector → OTel Collector
    * Gateway → OTel Collector
    * Vault → OTel Collector
  </Tab>

  <Tab title="Storage & Visualization">
    The OTel Collector routes telemetry to the appropriate backends:

    | Backend        | Data Type                                           |
    | -------------- | :-------------------------------------------------- |
    | **Prometheus** | Metrics (counters, histograms, gauges)              |
    | **Loki**       | Logs (structured, labeled)                          |
    | **Tempo**      | Traces (distributed spans)                          |
    | **Grafana**    | Visualization — queries Prometheus, Loki, and Tempo |
  </Tab>
</Tabs>

```
Agent Pods + Platform Services → OTel Collector → Prometheus (metrics)
                                                  → Loki (logs)
                                                  → Tempo (traces)
                                                  → Grafana (dashboards)
```

***

## Auto-Instrumented Spans

The Hexr SDK automatically generates OpenTelemetry spans:

| Span Name             | Triggered By                          |
| --------------------- | ------------------------------------- |
| `hexr.agent.invoke`   | `@hexr_agent` decorated function call |
| `hexr.tool.call`      | `hexr_tool()` call                    |
| `hexr.llm.call`       | `hexr_llm()` call                     |
| `hexr.a2a.send`       | `A2AClient.send()`                    |
| `hexr.vault.get`      | `vault.get()`                         |
| `hexr.sandbox.exec`   | `sandbox.exec()`                      |
| `hexr.browser.browse` | `browser.browse()`                    |
| `hexr.guard.scan`     | LLM Guard scan (automatic)            |
| `hexr.cred.exchange`  | Credential exchange                   |

***

## Metrics

### Agent Metrics

| Metric                         | Type      | Description              |
| ------------------------------ | --------- | ------------------------ |
| `hexr_agent_invocations_total` | Counter   | Total agent invocations  |
| `hexr_agent_duration_seconds`  | Histogram | Agent execution duration |
| `hexr_agent_errors_total`      | Counter   | Agent errors by type     |

### Tool Metrics

| Metric                       | Type      | Description                   |
| ---------------------------- | --------- | ----------------------------- |
| `hexr_tool_calls_total`      | Counter   | Total tool calls by service   |
| `hexr_tool_duration_seconds` | Histogram | Tool call duration            |
| `hexr_tool_errors_total`     | Counter   | Tool errors by service        |
| `hexr_cred_cache_hits_total` | Counter   | Cache hits by tier (L1/L2/L3) |

### LLM Metrics (GenAI Semantic Conventions)

| Metric                             | Type      | Description                        |
| ---------------------------------- | --------- | ---------------------------------- |
| `gen_ai_client_token_usage`        | Histogram | Tokens by direction (input/output) |
| `gen_ai_client_operation_duration` | Histogram | LLM call duration                  |
| `hexr_llm_cost_dollars`            | Counter   | Estimated cost by model + tenant   |

***

## Grafana Dashboards

The platform ships with pre-built Grafana dashboards:

| Dashboard          | Shows                                            |
| ------------------ | ------------------------------------------------ |
| **Agent Overview** | All agents, invocations, errors, latency         |
| **Tool Usage**     | Tool calls by service, cache hit rates, latency  |
| **LLM Costs**      | Token usage, cost per tenant, model distribution |
| **Security**       | Guard blocks, credential exchanges, audit events |
| **Infrastructure** | Pod health, SPIRE status, Envoy metrics          |

Access Grafana at the configured endpoint (Hexr Cloud: available via dashboard).

***

## Configuration

OTel Collector is configured via `otel-collector-config.yaml`:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
```