✦ Overview
In this live Data Science Lab session, Hadley Wickham (creator of the tidyverse and Chief Scientist at Posit) demonstrates a compelling workflow for data cleaning that revolves around three files kept in sync: a data cleaning R script, a data dictionary in YAML format, and the cleaned data as a Parquet file. Using Claude Code with the MCP REPL tool inside Positron, Hadley iteratively cleans the NYC Elevators dataset — normalizing whitespace, fixing date types, replacing sentinel values, investigating geocoding anomalies — while the data dictionary captures everything learned along the way. The session is equal parts practical tutorial, philosophical discussion about AI-assisted data work, and an entertaining investigation into a mysterious elevator apparently located in the middle of Central Park.
1 The Three-File Workflow
Hadley's central thesis: if you're going to use AI agents for data analysis, you need to write down what you know about the data so agents can be as helpful as possible. Of course, human colleagues benefit from this too, but he acknowledges "some unfortunate quirk of human psychology" makes it more motivating to write documentation for machines than for people.
The workflow centers on three files kept in sync through Git:
- Cleaning script (
clean.R) — the R code that transforms raw data into clean output - Data dictionary (
data-dict.yaml) — a YAML file documenting what every column means, its type, constraints, examples, and known issues - Cleaned data (
elevators.parquet) — the final output in Parquet format
Claude Code edits all three simultaneously: when you ask it to fix a data quality issue, it updates the cleaning script, records the change in the dictionary, re-runs the script to produce a new Parquet file, and you can verify everything through git diff.
2 MCP REPL — Sandboxed R/Python for AI Agents
The tooling stack: Positron (Posit's IDE), Claude Code, and MCP REPL — an open-source MCP server developed by Hadley's colleague Tamas at Posit. MCP REPL gives AI agents a persistent R or Python session so they can run data analysis code directly.
Critical Safety Feature: Sandboxing
MCP REPL runs all code in an operating system-level sandbox:
- Cannot connect to anything over the network
- Cannot access files outside the current working directory
- Uses OS-level isolation — no way for bad R code to escape and "do naughty things"
This means you can let an LLM run arbitrary R code with confidence that it can't cause damage outside your project directory. MCP REPL works on Windows, macOS, and Linux.
Hadley notes this is just one of his experiments with different tools — he's also familiar with Posit Assistant and doesn't want anyone to read too much into his specific tool choice.
3 Starting Point — CSV to Parquet
The demo uses the NYC Elevators dataset (from Emil Hvitfeldt's package) — a dataset of every elevator, escalator, dumbwaiter, and related conveyance device registered in New York City.
Hadley starts by asking Claude to write a minimal cleaning script that reads the CSV and saves a Parquet file. Claude initially goes overboard, writing extensive cleaning code that wasn't requested — "wow, it's really gone all out there and it's done a bunch of stuff that I didn't want." He tells it to ramp it back to just the basics: read CSV, write Parquet.
4 Why Parquet Over CSV
Hadley makes a strong case for Parquet files as "basically a CSV file but better in every single possible way":
- Smaller — excellent compression
- Faster — columnar storage enables efficient reads
- Type-safe — no ambiguity about types or missing values
- Universal — readable from every programming language (R, Python, SQL, Julia, etc.)
- Powers enterprise tools — Snowflake, Databricks, DuckDB all use Parquet natively
Downsides
- Not human-readable — can't open in a text editor or view diffs easily on GitHub
- No label support — Parquet files can't store the labeled values that statisticians love (Hadley envisions an R package that applies labels from the data dictionary)
Positron's built-in Parquet file viewer mitigates the readability issue significantly.
5 The Data Dictionary YAML Spec
Hadley introduces his data-dict.yaml specification — a lightweight YAML format for documenting datasets. He asks Claude to read the spec from his GitHub repo, then generate an initial dictionary for the elevators dataset.
Structure
The YAML format describes:
- Tables — one entry per table (this dataset has one:
elevators) - Fields — each column gets: name, type (string, integer, date, enum), constraints (primary key, not-null), a human-readable description, and example values
- Enums — categorical columns list all possible values with descriptions (e.g., device_status: "A" = Active, "R" = Retired)
- Domain terminology — a glossary for domain-specific terms
- Relationships — foreign key references between tables
Claude generates the initial dictionary by inspecting the data (tabulating values, checking types) and using its general knowledge about elevators. The result isn't perfect — it's a starting point that gets refined iteratively.
6 Enriching with README Context
Hadley enables git init and makes an initial commit, then feeds Claude the original dataset's README file which contains additional context — for example, "ZIP code is frequently listed as a nine-digit ZIP+4" and "seven test rows carry the literal street name 'DUMMY RECORD' and should be filtered out."
Claude updates the data dictionary with this contextual information, enriching entries with notes about data quality issues and known quirks. The Git diff shows exactly what changed — new annotations about inconsistent whitespace, dirty ZIP codes, and test records that need removal.
This demonstrates the iterative nature: start with what the LLM can infer, then layer in domain knowledge from READMEs, documentation, or your own experience.
7 Iterative Cleaning — Removing Dummy Records
The first cleaning task: remove the seven dummy/test records. Hadley simply tells Claude: "Remove the dummy records, updating clean.R, data-dict, and the Parquet file."
Claude updates all three files atomically:
- Adds a filter line to
clean.Rthat strips rows where street_name is "DUMMY RECORD" - Removes the dummy record note from the data dictionary (since they're now cleaned out)
- Re-runs the script to produce a new Parquet file with 7 fewer rows
8 Whitespace Normalization & Git Commits
The dictionary notes that street_name has inconsistent whitespace — multiple internal spaces and untrimmed edges. Hadley types a characteristically casual prompt: "normalize white space and screen" (with a typo in "street_name" — no shift key, no underscore). Claude figures it out perfectly.
Claude adds gsub to replace multiple spaces with single spaces and trimws to strip leading/trailing whitespace. Hadley manually removes an over-verbose comment from the dictionary — "I don't think you need to record that sort of information in the data dictionary when it's recorded in your clean.R."
Git Integration
Hadley then asks Claude to commit. Claude checks git status, looks at the git log, and produces the commit message "Add normalized street name whitespace." This prompts a discussion about whether Claude should auto-commit — Hadley says "you can do that" but doesn't always enable it.
9 ZIP Code Cleanup
The ZIP code column reveals several issues. Hadley asks Claude to analyze the sizes (character lengths) of ZIP codes. The result: 82% are nine-digit ZIP+4 codes, and the column is stored as a number rather than a string.
Two fixes applied:
- Convert to string — ZIP codes aren't numbers; you'll never do math on them. Storing as numeric is a classic data quality smell.
- Replace zeros with proper missing values (
NA) — zero is used as a placeholder, not a real ZIP code
The dictionary is updated to reflect: "usually a nine-digit ZIP+4, ~80% are nine digits." New example values show the actual string format.
10 Date Column Parsing
Examining the Parquet file in Positron's viewer reveals that date columns are stored incorrectly: lastInspectDate is a number, dbLastPerInspectDate is a string, and dvApprovalDate is also a string. None are actual dates.
Hadley's prompt: "Make sure you parse all the date columns as dates." Claude writes a for loop in base R to convert each column — not how Hadley would have written it ("I don't know if this is the way I would have written this code") but functional. His pragmatic take: "I kind of don't really care because I didn't have to write that code."
This is a good example of the data dictionary being wrong — it claimed these were dates, but the actual data didn't match. The goal is to create a process where mistakes are discovered and fixed incrementally.
11 Adding Ranges & Data Validation
Hadley asks Claude to add ranges (min/max values) to all numeric and date columns in the dictionary. This is useful for at-a-glance validation — you can immediately spot if a latitude value is 999 or a date is from 1899.
Data Dictionaries vs Data Contracts
A community question sparks an insightful discussion. Hadley initially saw data dictionaries and data contracts/validation (like Pointblank) as very distinct, but as he's worked on this format, he realizes they converge. The dictionary specifies types, ranges, enums, and constraints — which is exactly what a validation tool needs.
His vision: a tool where you take a data dictionary and a Parquet file, compare them, and discover any inconsistencies automatically. And critically, he wants it to work beyond just R — for Python, SQL, any language that can read YAML and Parquet.
12 Eliminating Placeholder Values
Hadley asks Claude to find and eliminate all placeholder/sentinel values across the entire dataset. This is a common pain point in data collected by non-data-scientists who don't have a proper concept of missing values.
Claude discovers at least three different placeholder schemes across various columns:
0— used as "no value" in numeric columns10000— used as a sentinel in speed-related columns** - ----— literal string placeholder in text columns
All are replaced with proper NA values. The host, Libby Heeren, notes this is "a really excellent use of Claude — tell me what all these disparate missing values are in all these different variables. And fix it for me."
13 The Central Park Elevator Mystery
The session takes a delightful turn when Hadley asks Claude to plot all elevator locations on a map. The initial plot reveals obvious outliers — some coordinates land in Pennsylvania rather than New York City. After filtering to a NYC bounding box, the map looks much better, but one point sits right in the middle of Central Park.
What follows is a collaborative investigation between Hadley, the live audience in Discord, and Claude:
- The device is a freight elevator registered at "1000 Fifth Avenue"
- Discord participants identify this as likely the American Museum of Natural History (not the Met, as initially guessed)
- The coordinates appear to point to the Great Lawn area — clearly a geocoding error
- Someone finds a Gizmodo article about an artificial cave beneath Central Park with maintenance elevators, spawning an "elevator conspiracy" in the chat
Hadley tries a Leaflet interactive map to investigate further, and the audience helps narrow down whether it's a genuine subterranean elevator or just bad geocoding. The consensus: it's most likely a geocoding error for the Museum of Natural History.
14 Productivity & Workflow Reflections
A community member asks the direct question: "How do you know this is actually faster than just writing the code?"
Hadley's honest answer:
- He could have written some of this code fairly quickly himself
- For someone less familiar with R, the time savings would be more dramatic
- The real value isn't just speed — it's the workflow of keeping three files in sync, which is powerful regardless of whether AI writes the code
- The session was much slower than normal because he was explaining everything and answering questions
- "It's not like ten times faster. It's not faster for everything. But you also don't have to use it when it's slower."
On Model Choice & Cost
Hadley uses Opus 4.7 with 1 million context through his Claude subscription. His take on cost: "I have little time and a lot of money, so I optimize for that." He suspects cheaper models would do almost as well for this type of work but hasn't investigated.
Why YAML Over JSON
When asked why not use JSON Schema, Hadley explains: the data dictionary is meant for humans to edit too, and YAML is dramatically easier to read and write by hand compared to JSON.
🎯 Key Takeaways
🔑 Key Takeaways
- Three-file workflow is the core pattern — cleaning script + data dictionary YAML + Parquet output, all versioned in Git. Claude updates all three atomically per change.
- Data dictionaries serve humans and machines — documenting column types, constraints, examples, and known issues helps both human collaborators and AI agents work effectively with your data.
- Parquet is strictly superior to CSV for cleaned data — smaller, faster, type-safe, universally readable. The only downside is you can't diff it in Git or view it in a text editor.
- MCP REPL provides safe AI-driven data analysis — OS-level sandboxing means the AI can run arbitrary R/Python code without network access or file system escape.
- Start simple, iterate — begin with a minimal CSV-to-Parquet conversion, let the LLM generate an initial dictionary, then refine through targeted cleaning passes.
- Git is essential to the workflow — every change produces a reviewable diff across all three files, making it easy to verify Claude's work and roll back mistakes.
- Data dictionaries and data contracts converge — the same YAML spec that documents your data can evolve into a validation/contract tool that checks data quality automatically.
- LLMs excel at finding heterogeneous placeholder values — discovering that zeros, 10000, and "** - ----" all mean "missing" across different columns is tedious for humans but trivial for an AI scanning the full dataset.
- Don't over-invest in LLM-generated code style — if the code works and is readable enough to verify, the implementation style doesn't matter much.
- The YAML format is language-agnostic by design — Hadley explicitly wants the data dictionary spec to work across R, Python, SQL, and any other language, not just the tidyverse ecosystem.
- Spatial data needs human verification — LLMs can generate maps and bounding boxes, but trusting geocoding results without visual inspection leads to elevators in Central Park.
- Honest productivity assessment — AI-assisted data cleaning isn't 10x faster for experts, but it's meaningfully faster for exploration, and the three-file sync workflow is valuable regardless of who writes the code.
🔗 Resources & Links
- MCP REPL — open-source MCP server for persistent R/Python sessions with OS-level sandboxing
- data-dict.yaml spec — Hadley's YAML format specification for data dictionaries
- Parquet in R4DS — Parquet files chapter from R for Data Science
- NYC Elevators Dataset — the dataset used in this demo
- Pointblank — data validation package for R (discussed in context of data contracts)
- Arrow R Book — comprehensive guide to Apache Arrow in R
- Monaspace Font Family — the ligature font Hadley uses in Positron
- YAML Multiline Strings Reference — handy reference for YAML formatting