Data dictionaries for humans, agents and jumping frogs

Apparently, you can rent a frog and enter a jumping competition. You might find yourself up against teams with years or even decades of experience selecting and preparing their frogs. Welcome to the Calaveras County Jumping Frog Jubilee.

In Astley et al. (2013), researchers studied bullfrog jumps at the event, comparing frogs rented by fairgoers with those entered by experienced teams.¹ Their measurements provide the example for the Data Dict quickstart. It is certainly a memorable way to get into data documentation.

Meet the data

How far can a bullfrog jump? Even with the data in hand, there is a detail worth checking: what counts as a jump? The collection notes explain that the competition scored the straight-line distance across three successive jumps. The researchers measured individual jumps from video and excluded short, continuous movements called “skitters”. A dictionary should distinguish individual-jump distances from three-jump measures and document which movements were excluded. Someone analysing the data, including an AI agent, needs that context.

The context includes how the data was obtained. The measurements come from a live-animal competition, and the collection notes describe how participants prompt the frogs to jump. For me, that raises animal-welfare concerns alongside the curiosity of the example.² That ethical context is worth keeping visible too.

A data dictionary can record what each row represents, how values were measured and how tables fit together. Data Dict provides a format for writing this down alongside rules the data should satisfy. Both live in one file, data-dict.yaml. Its command-line tool checks the rules against the data and turns the dictionary into readable documentation.

The open-source project was initiated by Hadley Wickham and is supported by Posit. The format is designed for teams working across R, Python and SQL.

Take a look

To see how a dictionary handles several related tables, take a short detour from frogs to otters. The sea-otter example is already rendered, so you can browse it without installing anything. One table describes the animals; another records the occasions on which measurements were collected. An otter caught twice can have two measurement records, so counting those records would overcount animals. The dictionary explains how the tables connect, gives weight in kilograms and flags an unexplained length measurement.

Compare it with the YAML source, or browse the example gallery. The dictionary can be edited and versioned as text, while the rendered page can be shared with colleagues who do not need to work with YAML.

Give it a jump

To try the frog example, install the CLI, then clone the repository and create a draft:

git clone https://github.com/hadley/frog-jumping
cd frog-jumping
data-dict draft frogs.parquet

This creates data-dict.yaml with inferred column types and TODOs for missing explanations. No agent is required. Use the collection notes to complete what you can, then check and render it:

data-dict validate-data data-dict.yaml
data-dict render-spec data-dict.yaml

The first command checks the data against the declared rules; the second produces a self-contained HTML page, like the otter example. The quickstart explains both. The checks act as tests for your data: depending on the rules you declare, they can flag missing values, duplicate identifiers or references to records that do not exist.

You can also require an end date to fall on or after its start date, using SQL-, R- or Python-style syntax. These are supported subsets interpreted by Data Dict, rather than arbitrary code in those languages. Those checks can be run again when new data arrives. But a distance could pass a range check without revealing whether it describes one jump or three. Writing useful rules still requires understanding what was measured.

Let an agent help

The agent-assisted route starts with the same data and notes. Ask an agent to document them using the Data Dict CLI; the bundled instructions guide it through drafting and consulting the notes. Unresolved questions can remain as todos. You still need to review the descriptions and rules against what you know about the data.

The dictionary is useful to agents reading the data too. Posit’s querychat, which lets people ask questions about data in natural language, already accepts Data Dict files. Descriptions, relationships and terminology give the model context for writing queries.

Keep an eye on it

Data Dict is still an early project. The CLI currently validates Parquet files, with SQL sources expected in the future. There is also an open proposal to generate sample data from a dictionary. That could be handy for testing, although satisfying relationships between tables makes it a substantial task.

There are related approaches with different scopes. Within a dbt project, documentation describes models and columns and shows dependencies between models, while data tests check declared assumptions. The Open Data Contract Standard goes beyond schema descriptions to include responsibilities and service-level agreements. The starting point explored here is more focused: a dictionary to describe a dataset, check expectations and share context with people and agents.

For a first experiment, though, the question is smaller: will the next person opening this dataset have an easier time understanding it? That person might be a colleague, an agent, or you six months from now. If a few descriptions and checks can save them some head-scratching, that is a useful jump forward.