Classifying Prompt Patterns in LLM Logs

Use AI-generated columns to categorize unstructured fields in your logs and understand what your system is actually doing.

Overview

LLM logs often contain free-text system prompts that are hard to analyze at scale. In this example, we load a large conversation dataset (Orca), use Hyperparam's AI agent to classify each system prompt into a category, and review the distribution of prompt types across the dataset.

Demo showing prompt classification in Hyperparam

Steps

  1. Load the log dataset

    Open OpenOrca/partial-train/0000

  2. Examine the system prompts

    Review the system_prompt column. These are the instructions each conversation was given, but as free text they're hard to analyze in aggregate.

  3. Classify with AI

    In chat, request: "create a new column that categorizes the system prompt"

    The agent analyzes each system prompt and generates categories (e.g., "Education/Tutor", "General Assistant", "Information retrieval/QA"). A new system_prompt_category column appears.

  4. Review the distribution

    Scroll through rows to verify category assignments. Sort or create a view to see which prompt patterns dominate your dataset.

  5. Export the results

    Export the dataset with the new system_prompt_category column included. Export processes the full 100k+ dataset.

Expected Results

  • Categorical column: system_prompt_category classifying each system prompt
  • Pattern visibility: See which prompt types dominate and which are rare
  • Actionable metadata: Filter to specific prompt categories, compare behavior across categories, or identify prompt patterns that correlate with failures

Other Use Cases

Classifying Prompt Patterns in LLM Logs - Hyperparam