Skip to main content

Overview

When you call hexr_tool("aws_s3"), Hexr returns an authenticated boto3 S3 client — without any API keys in your code. Behind the scenes, a three-tier caching system makes this near-instantaneous.

Three-Tier Cache Architecture

L1: In-Memory (Process-Local)

~0.001ms latency. Credentials live in the Python process’s memory (ContextVar-based). TTL = credential expiry minus 10 minutes. Dies when the process dies.

L2: Valkey (Distributed)

~1-3ms latency. Shared across all pods in the cluster (3-node HA). Key format: cred:{spiffe-id}:{service}:{region}. If agent A already fetched S3 credentials, agent B can use the cached result.

L3: Credential Exchange (Full Round-Trip)

~50-200ms latency. Agent → Envoy (mTLS) → Credential Injector → OPA policy check → Cloud STS AssumeRoleWithWebIdentity. Issues temporary credentials (15-60 min TTL).
On a cache hit, the call returns in microseconds (L1) or low milliseconds (L2). The full 50-200ms exchange only happens on first access or after credential expiry.

Exchange Flow

What happens when both L1 and L2 cache miss — the full credential exchange:

hexr_tool('aws_s3') called

Your agent calls hexr_tool("aws_s3"). SDK checks L1 (memory) → miss. Checks L2 (Valkey) → miss.

Request sent via Envoy

Agent sends POST /exchange {service: "aws_s3"} to the Envoy sidecar (localhost). Envoy adds the X.509-SVID as client certificate and forwards over mTLS to the Credential Injector in hexr-system.

JWT-SVID verification

The Credential Injector verifies the agent’s JWT-SVID via the SPIRE Workload API. This confirms the caller’s SPIFFE identity is legitimate.

OPA policy check

CI queries OPA: {spiffe_id, service: "aws_s3", tenant: "acme-corp"}. OPA evaluates the Rego policy and returns ALLOW or DENY.

Cloud STS exchange

CI calls AssumeRoleWithWebIdentity on AWS STS, presenting the JWT-SVID as the web identity token. AWS trusts Hexr’s OIDC endpoint and returns temporary credentials.

Credentials cached + client returned

{AccessKeyId, SecretAccessKey, SessionToken} with 15-min TTL flows back through Envoy to the agent. Stored in L1 + L2 cache. SDK creates and returns an authenticated boto3.client('s3').

Supported Cloud Providers

AWS

Exchange: JWT-SVID → STS AssumeRoleWithWebIdentityServices: S3, EC2, DynamoDB, SQS, Lambda, Bedrock, and any AWS SDK service.Credential TTL: 15 minutes (configurable up to 12 hours)

GCP

Exchange: JWT-SVID → Workload Identity Federation → Service Account tokenServices: BigQuery, Cloud Storage, Vertex AI, Pub/Sub, and any Google Cloud API.Credential TTL: 1 hour

Azure

Exchange: JWT-SVID → Federated Token → Managed Identity tokenServices: Blob Storage, Cosmos DB, Azure OpenAI, and any Azure SDK service.Credential TTL: 1 hour

Multi-Cloud in One Agent

An agent can use tools from multiple clouds simultaneously:

OPA Policy Enforcement

Before any credential exchange, OPA validates the request:
Policies are distributed via Kubernetes ConfigMaps and reload within 30 seconds.

Proactive Refresh

A background daemon proactively refreshes credentials before they expire:
This means agents never see credential expiry errors during normal operation.

Observability

Every cache lookup and exchange emits OpenTelemetry spans:
Grafana dashboards show cache hit rates, exchange latencies, and credential refresh patterns in real-time.