# Complete Workflow: Extract, Filter, Export

Canonical: https://hyperparam.app/docs/usecases/patient-data-workflow

Extract structured fields from unstructured text, filter by criteria, and export a refined dataset. The same pattern works on agent traces, chat logs, customer-support transcripts, or any other unstructured text dataset.

## Overview

This walkthrough uses a 150,000+ row parquet file of unstructured medical records as a concrete example. Use LLM-based extraction to create structured columns from the free-text `note` field, filter the dataset by age and diagnosis criteria, and export a subset with selected columns. Substitute your own log file (Claude Code transcripts, chat histories, support tickets) and the workflow is identical: extract fields, build a SQL view, export.

![Demo showing patient data extraction and filtering in Hyperparam](https://s3.hyperparam.app/video/Complete_Workflow.gif)

## Steps

1. **Load the dataset**

   Open [Asclepius-Synthetic-Clinical-Notes/0000](https://hyperparam.app/files?key=https%3A%2F%2Fhuggingface.co%2Fdatasets%2Faisc-team-a1%2FAsclepius-Synthetic-Clinical-Notes%2Fresolve%2Frefs%252Fconvert%252Fparquet%2Fdefault%2Ftrain%2F0000.parquet)

2. **Extract structured fields using chat**

   Open any cell from the 'note' column containing patient information

   > **Note:** We can view full unstructured text data for an individual chart by scrolling down

   Use chat to request extraction: "extract age, diagnosis, symptoms, comorbidities, treatments, outcome in separate columns from 'note' column"

   > **Note:** Hyperparam will create 6 new columns and populate them with the extractions.

   Columns appear as: `age`, `diagnosis`, `symptoms`, `comorbidities`, `treatments`, `outcome`

   > **Note:** Scroll down and you will see Hyperparam filling out all rows

3. **Create a SQL view for the target cohort**

   In chat, request: "create a view of patients over 50 with a respiratory diagnosis and fever symptoms, keeping only subject_id, age, diagnosis, symptoms, comorbidities, treatments, outcome"

   Hyperparam calls `add_view` with a query like:

   ```sql
   SELECT subject_id, age, diagnosis, symptoms, comorbidities, treatments, outcome
   FROM "0000.parquet"
   WHERE age > 50
     AND diagnosis ILIKE '%respiratory%'
     AND symptoms ILIKE '%fever%'
   ```

   > **Note:** Views replace the old filter UI. They are composable, support JOINs across multiple sources, and can project specific columns. Views appear in the workspace file browser alongside sources.

4. **Export the view**

   Open the view tab, click the ⋮ button, and choose "Write Table"

   Select export options: columns, max rows, file name (e.g., `filtered_patients.parquet`)

   Export runs the view's SQL over the full dataset.

## Expected Results

- **Extracted columns**: Structured fields parsed from unstructured patient text
- **Final export**: Exported file includes only patients with matching criteria and only selected columns. 

---

## Other Use Cases

- [Dataset Discovery](https://hyperparam.app/docs/usecases/dataset-discovery): Use natural language to find public datasets
- [Classifying Prompt Patterns](https://hyperparam.app/docs/usecases/data-transformation): Categorize unstructured prompts to see your real traffic mix
- [Quality Filtering](https://hyperparam.app/docs/usecases/quality-filtering): Score and filter low-quality, sycophantic responses in chat logs
- [Deep Research](https://hyperparam.app/docs/usecases/summary-comparison): Multi-step AI workflow for comparing model outputs