> ## Documentation Index > Fetch the complete documentation index at: https://docs.hexr.dev/llms.txt > Use this file to discover all available pages before exploring further. # Observability Stack > Full OpenTelemetry pipeline with distributed tracing, metrics, and dashboards. Every agent operation is instrumented automatically. ## Architecture All telemetry flows through the OpenTelemetry Collector, then routes to dedicated backends: | Source | What It Emits | Protocol | | ------------------------------------------------------- | ------------------------------------------------------ | ------------------- | | **Python SDK** (`hexr_llm`, `hexr_tool`, `@hexr_agent`) | Agent spans, LLM metrics, tool invocations | OTLP gRPC → `:4317` | | **Envoy Proxies** | mTLS metrics, connection counts, TLS handshake latency | OTLP | | **A2A Sidecars** | Task lifecycle, message throughput, SSE connections | OTLP | | **Platform Services** (Vault, Gateway, CI) | Operation rates, latency, error counts | OTLP | | Component | Role | Port | | ------------------ | ------------------------------------------ | -------------------------- | | **OTel Collector** | Aggregation point for all telemetry | `:4317` gRPC, `:4318` HTTP | | **Jaeger** | Distributed trace storage and UI | `:16686` | | **Prometheus** | Metrics storage, 11+ scrape targets | `:9090` | | **Grafana** | Dashboards — 42 panels across 2 dashboards | `:3000` | ``` Python SDK ──┐ Envoy Proxies ──┤──► OTel Collector (:4317) ──┬──► Jaeger (traces) A2A Sidecars ──┤ └──► Prometheus (metrics) Platform Svcs ──┘ │ Grafana (dashboards) ``` *** ## Automatic Instrumentation The Hexr SDK instruments everything without extra code: ```python theme={null} @hexr_agent(name="analyst", tenant="acme") def analyze(topic: str): # Span: hexr.agent.invoke (auto) s3 = hexr_tool("aws_s3") # Span: hexr.tool.invoke {service: aws_s3} # Span: hexr.cache.lookup {tier: L1|L2|L3} # Span: hexr.credential.exchange (if cache miss) client = hexr_llm(openai.OpenAI()) response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": f"Analyze {topic}"}] ) # Span: hexr.llm.chat {model: gpt-4o, tokens_in: 42, tokens_out: 256} secret = hexr.vault.get("openai/api-key") # Span: hexr.vault.get {path: openai/api-key} return response ``` **Zero configuration.** All OTel providers are set up by `@hexr_agent` at decoration time. *** ## Trace Spans | Span Name | Attributes | Source | | -------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- | | `hexr.agent.invoke` | `agent_name`, `tenant`, `framework`, `status` | `@hexr_agent` decorator | | `hexr.tool.invoke` | `service`, `region`, `cache_tier` | `hexr_tool()` | | `hexr.cache.lookup` | `tier` (L1/L2/L3), `hit`, `duration_ms` | Credential cache | | `hexr.credential.exchange` | `provider`, `service`, `spiffe_id` | Credential Injector client | | `hexr.llm.chat` | `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | `hexr_llm()` proxy | | `hexr.vault.get` | `path`, `tenant` | `hexr.vault` module | | `hexr.gateway.call` | `tool_name`, `arguments` | `hexr.gateway` module | | `hexr.a2a.client.send` | `target_agent`, `task_id`, `task_state` | `A2AClient` | | `hexr.a2a.bridge.execute` | `source_agent`, `task_id` | A2A Bridge | | `hexr.sandbox.exec` | `language`, `timeout`, `exit_code` | `hexr.sandbox` | | `hexr.browser.browse` | `url`, `actions_count` | `hexr.browser` | | `hexr.guard.scan` | `scan_type` (prompt/output), `is_valid` | `hexr.guard` | #### LLM Guard Span Attributes When LLM Guard blocks a prompt or response, additional attributes are set on the parent `hexr.llm.chat` span: | Attribute | Type | Description | | ---------------------------- | -------- | ----------------------------------------------- | | `hexr.guard.prompt_blocked` | `bool` | `true` if the input prompt was blocked | | `hexr.guard.scanners` | `string` | Scanner results that triggered the prompt block | | `hexr.guard.output_blocked` | `bool` | `true` if the LLM response was blocked | | `hexr.guard.output_scanners` | `string` | Scanner results that triggered the output block | Blocked requests set the span status to `ERROR` with `"Blocked by LLM Guard"` or `"Output blocked by LLM Guard"`. *** ## Metrics ### Agent Metrics | Metric | Type | Description | | ------------------------ | ------------- | ------------------------------ | | `hexr.agent.invocations` | Counter | Total agent invocations | | `hexr.agent.active` | UpDownCounter | Currently active invocations | | `hexr.agent.duration` | Histogram | Invocation duration in seconds | ### Tool & Credential Metrics | Metric | Type | Description | | ---------------------------- | --------- | ----------------------------- | | `hexr.tool.invocations` | Counter | Total tool calls by service | | `hexr.tool.duration` | Histogram | Tool call duration | | `hexr.cache.hits` | Counter | Cache hits by tier (L1/L2/L3) | | `hexr.cache.misses` | Counter | Cache misses | | `hexr.cache.lookup.duration` | Histogram | Cache lookup latency | | `hexr.credential.exchanges` | Counter | Full credential exchanges | | `hexr.credential.failures` | Counter | Failed exchanges | ### LLM Metrics | Metric | Type | Description | | ------------------------ | --------- | ------------------- | | `hexr.llm.calls` | Counter | Total LLM API calls | | `hexr.llm.call_errors` | Counter | Failed LLM calls | | `hexr.llm.call.duration` | Histogram | LLM call latency | | `hexr.llm.input_tokens` | Counter | Total input tokens | | `hexr.llm.output_tokens` | Counter | Total output tokens | ### A2A Metrics | Metric | Type | Description | | ---------------------------- | --------- | -------------------- | | `hexr.a2a.sends` | Counter | Messages sent | | `hexr.a2a.send_failures` | Counter | Failed sends | | `hexr.a2a.send.duration` | Histogram | Send latency | | `hexr.a2a.bridge.executions` | Counter | Bridge handler calls | ### LLM Guard Metrics | Metric | Type | Description | | ---------------------------------- | --------- | ------------------------------------------------------- | | `hexr_guard_scans_total` | Counter | Total scans by direction (`input`/`output`) and scanner | | `hexr_guard_blocks_total` | Counter | Total blocks by direction and scanner | | `hexr_guard_scan_duration_seconds` | Histogram | Scan latency by direction | *** ## Grafana Dashboards Hexr ships with two pre-built Grafana dashboards: ### Platform Overview (23 panels) Covers system-wide health: * Agent pod status and container health * Credential exchange rates and cache hit ratios * mTLS connection counts and TLS handshake latency * SPIRE entry counts and SVID rotation rates * OTel Collector throughput (traces/sec, metrics/sec) * Vault operation rates and latency * Gateway tool invocation rates ### A2A Communication (19 panels) Covers inter-agent messaging: * Task lifecycle (submitted → working → completed/failed) * Message throughput per agent pair * Task duration histograms * SSE streaming connection counts * Valkey task store operations * Error rates by task state transition * Cross-namespace communication patterns *** ## GenAI Semantic Conventions `hexr_llm()` follows the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/): | Attribute | Example Value | | -------------------------------- | ---------------------------------------------------------- | | `gen_ai.system` | `openai`, `anthropic`, `google_genai`, `cohere`, `mistral` | | `gen_ai.request.model` | `gpt-4o`, `claude-3-opus`, `gemini-pro` | | `gen_ai.response.model` | `gpt-4o-2024-08-06` | | `gen_ai.usage.input_tokens` | `1200` | | `gen_ai.usage.output_tokens` | `800` | | `gen_ai.response.id` | `chatcmpl-abc123` | | `gen_ai.response.finish_reasons` | `["stop"]` | This means your Hexr traces are compatible with any OTel-native LLM observability tool. *** ## Prometheus Scrape Targets | Target | Labels | Metrics | | ------------------- | ----------------------- | ---------------------------------- | | A2A Sidecars | `namespace=tenant-*` | Task lifecycle, message throughput | | Credential Injector | `namespace=hexr-system` | Exchange rates, OPA decisions | | Gateway | `namespace=hexr-system` | Tool calls, import counts | | Vault | `namespace=hexr-system` | Secret operations, encryption | | OTel Collector | `namespace=hexr-system` | Collector health, pipeline stats |