Architecture: Collect, Store, Analyze

Hyperparam is two products on an open architecture. HypAware is the improvement loop for all your agents: it records AI sessions as open files (Apache Iceberg tables) on your machine, or in HypAware Cloud for a whole team. Hyperparam reads them on your laptop. HypAware is open source, and so are the libraries Hyperparam is built on. The open-source pieces are collectively published as HypStack.

If you only want to use Hyperparam against existing files or buckets, you can skip this page. If you are setting up AI observability for an organization and asking "where do these logs go and who owns them," this is the architecture we recommend.

Why a New Stack

Every company already has a data stack. Datadog and Splunk for ops. Snowflake, Databricks, BigQuery, Redshift for analytics. None of it was designed for AI workloads.

Two things have changed at once:

1. A new class of applications. Codex, Claude Code, Cursor, Claude Desktop, Copilot, plus every product team adding an LLM to their own app. These run on the client (terminal, IDE, desktop, browser), not behind your load balancer.
2. A new shape of data. A single agent turn is a structured document with prompt, retrieved context, tool calls, tool responses, completion, and token counts. A real session is hundreds of those. That is megabytes of structured text per user per hour, not 200-byte access log lines.

Observability vendors are billed per GB ingested, default to short retention, and are structured for metrics rather than text. Warehouses are great at SQL but bad at nested LLM payloads, and the compute is rented. AI traces are the worst of both worlds for both products, and the typical answer is to add yet another vendor (LLM observability, eval, agent tracing, RAG monitoring, prompt management), each a copy of your data behind their API.

Our answer is one stack you can own end to end.

How It Works

Collect with HypAware (OpenTelemetry)

HypAware is an OpenTelemetry collector designed for AI workloads, deployable across your fleet via your existing MDM (Jamf, Kandji, Intune). Underneath it is pure OTel, so anything that already speaks OTLP works.

What it captures:

• Developer laptops and IDE plugins
• Agent runs and MCP tool calls
• Chatbots and customer-facing LLM features
• Production services that wrap an LLM
• Prompts, retrieved context, tool input and output, evaluations, token usage

HypAware keeps the data on each machine, or syncs it to HypAware Cloud for a whole team. See hypaware.ai/docs for setup.

If you only need Claude Code traces on a single developer machine, install HypAware with npm i -g hypaware, then run hyp setup. See Claude Code OpenTelemetry Setup for the single-machine flow.

Analyze with Hyperparam

Hyperparam is a local client that reads Iceberg and Parquet directly from your laptop, with no cluster, SQL endpoint, or ingestion step. It runs in the browser or as Hyperparam Desktop, both built on hyparquet, icebird, and squirreling, and can open millions of traces in a single tab.

The differentiator is joins across sources. Agent traces become useful when you can join them with the code the agent was editing, the issue it was assigned, or the eval results from yesterday's prompt change. Hyperparam can join Iceberg with GitHub repos, other buckets, other Iceberg tables, and existing warehouse exports in one workspace.

Because Hyperparam runs as JavaScript on your own machine, not as a hosted query engine you send SQL to:

• Open massive trace datasets instantly, without downloading them or renting a cluster to scan them.
• Your data stays private: prompts, traces, and credentials never leave your machine, and no vendor keeps a copy.
• Nothing to operate: no query endpoint, ingestion pipeline, or auth proxy to stand up, so even an agent can read its own traces directly.

See Data Sources for connecting Hyperparam to S3, GCS, Azure, Iceberg, and Hugging Face.

Open files in between

The data between HypAware and Hyperparam is just files: Parquet for raw traces, Apache Iceberg when you want a managed table on top. Both are open formats that Snowflake, Databricks, DuckDB, Spark, and Trino read natively, so a customer who wants their AI traces in their existing warehouse can point at the same files with no export job. Iceberg adds schema evolution, time travel, and compaction without adding a service to run. Storage runs at object-storage prices with infinite retention if you want it, and a catalog of your choice (Polaris, Lakekeeper, Glue, Nessie) if you want one.

HypAware Cloud stores the same open files for a whole team, with each organization's data kept separate.

Closing the Loop

The point of this architecture is not a dashboard. It is a feedback loop between your agents and your knowledge base.

The workflow inside Hyperparam is three steps:

1. Explore. Open a multi-million-row trace dataset in a single tab on your laptop. Use AI-assisted search to find the conversation, retry, or tool call you care about. Join with GitHub or a warehouse to correlate with code or product context.
2. Surface. Generate label columns at scale ("did this conversation succeed", "was this tool call necessary", "did the user retry"), then filter and cluster down to the 1% that is actually broken: token-burn hotspots, retry loops, tool failures, rabbit-holes.
3. Improve. A prompt or tool change is a hypothesis you can validate against the traces you already have. The change goes back into the knowledge base your team already maintains as markdown (CLAUDE.md, prompts, skills, standards). The agents pick it up. The loop repeats.

See How to Debug Wasted Tool Calls in LLM Logs for a worked example of the explore, surface, improve cycle.

What This Enables

• Full-fidelity AI observability: prompts, tools, retrieval, evals, all queryable in one place
• Infinite retention at object-storage prices, not per-GB ingest pricing
• Agents that read their own traces directly from object storage, with no SQL endpoint to provision
• Compliance that does not fight you: your IAM, your VPC, your region, no vendor copy of sensitive prompts

What You Bring, What We Bring

This is a pattern, not a SaaS contract. Most of it is open source you can deploy yourself.

You bring:

• Your apps, agents, and services
• The machines your agents run on, where HypAware records them
• Your existing warehouse, if you want to query traces from it
• Your security and compliance posture

The open-source pieces:

• HypAware for collection
• Iceberg for storage, plus a catalog of your choice
• The hyparam JS libraries that Hyperparam is built on, for analysis

For organizations rolling this out at scale, HypAware Cloud runs the loop for a whole team, with storage included. We also offer paid help with OTel and MDM instrumentation patterns, opinionated schemas for AI workloads, collector and Iceberg deployment, and analysis-client support. Contact us to get started.

Cost Shape

For a sense of why this matters financially:

Two orders of magnitude on storage, with compute moved to the client.

Who Owns What

Next Steps

• HypAware Team and Fleet Setup: enroll a team or push unattended installs over MDM
• Claude Code OpenTelemetry Setup: record Claude Code on one machine with HypAware
• Data Sources: connect Hyperparam to S3, GCS, Azure, and Iceberg
• Quick Start: load your first log file
• Exporting Chat Logs: pull traces out of Claude Code, ChatGPT, Langfuse, LangSmith, Phoenix, Datadog, and more