Apparently, you can rent a frog and enter a jumping competition. You might find yourself competing against teams with decades of experience choosing and preparing theirs. Welcome to the Calaveras County Jumping Frog Jubilee.
In Astley et al. (2013), researchers studied bullfrog jumps at the event, comparing frogs rented by fairgoers with those entered by experienced teams. Their measurements now provide the example for the Data Dict quickstart. That is a pretty good way to get me interested in data documentation.
Meet the data
Say someone sends you the frog dataset. Before comparing jumps, you need to know what was measured. One leap? Three in a row? Did every jump count? The collection notes explain that the competition judged the straight-line distance across three successive jumps, while the researchers measured individual jumps from video. They also excluded certain short, continuous movements called “skitters”.
data-dict gives information like this a home alongside the data. A data-dict.yaml describes tables, columns, units and relationships. A command-line tool checks declared rules and turns the dictionary into a readable website. The project is open source and supported by Posit.
Take a look
You can see the result without installing anything. Open the sea-otter dictionary, a separate example linked from the quickstart, and compare it with its YAML source. Start with otters and measurements. One describes animals, the other capture or collection events. An otter can appear in several events, connected through otter_no. Look at weight and you learn that it is measured in kilograms. Elsewhere, an unidentified length measurement is documented as still unexplained. That is useful documentation: you can find your way around the data, including the parts nobody has fully figured out. There are more examples in the gallery.
The presentation matters. A feature can exist and still be cumbersome enough that nobody uses it. I have often appreciated the attention to usability in Hadley Wickham’s tools and the wider tidyverse, reflected in its design principles. Data Dict’s short route from a text file to something people can browse appeals to me for the same reason.
Give it a jump
After installing the CLI, clone the frog repository and create a draft:
git clone https://github.com/hadley/frog-jumping
cd frog-jumping
data-dict draft frogs.parquet
Now open data-dict.yaml. Review the proposed descriptions and fill in the TODOs using the collection notes. Then check and render it:
data-dict validate-data data-dict.yaml
data-dict render-spec data-dict.yaml
This is the workflow from the quickstart: draft, review, check, share. The result is a self-contained HTML page for your frog dictionary, like the otter example above.
The checks can flag missing values, duplicate identifiers and broken links between tables. You can also write your own rules in familiar SQL-, R- or Python-style syntax. For example, an end date should not precede its start date. Supported expressions run through Data Dict’s own engine, so these are subsets of the languages rather than arbitrary R or Python code.
Let an agent help
The quickstart also offers an agent-assisted route. Point an agent at the data and collection notes, and ask it to use the Data Dict CLI to document the dataset. The bundled instructions guide it through drafting, consulting the notes and raising unresolved questions.
I like that division of work. The agent can help turn the data and existing notes into structured metadata and leave unresolved points as todos. The CLI then does the less glamorous but important part: it deterministically checks whether the data matches the documented expectations and can render the dictionary as readable documentation. What remains unclear stays visible until someone who understands the data can resolve it.
Keep an eye on it
Data Dict is still an early project. The current specification supports Parquet sources, with SQL sources described as a future direction. There is also an open proposal for generating sample data from a dictionary. That could be handy for testing, although satisfying relationships between tables makes it a substantial task.
There is already useful work in this space, including the Open Data Contract Standard and dbt’s documentation and tests. For a team with established tooling, I would ask how a new dictionary can reuse what is already maintained.
For a first experiment, though, the question is smaller: will the next person opening this dataset have an easier time understanding it? That person might be a colleague, an agent, or you six months from now. If a few descriptions and checks can save them some head-scratching, that is a useful jump forward.