Model cards and datasheets: documenting AI models and datasets
Model cards and datasheets are documentation practices that accompany AI models and datasets. They describe intended uses, performance, composition, and limitations. These frameworks aim to increase transparency and responsible deployment of AI technology.
Why documentation matters
Machine learning models and datasets are increasingly used in high-impact areas such as law enforcement, medicine, education, and employment. Without proper documentation, they may be applied in contexts for which they were not intended, leading to harmful outcomes.
Documentation frameworks such as model cards and datasheets provide structured information about a model's capabilities and a dataset's origins. They help developers, reviewers, and end users make informed decisions.
These frameworks are part of broader efforts in AI safety and responsible AI, aiming to reduce unintended biases and increase accountability.
Model cards: describing a model
The concept of model cards was proposed in the paper Model Cards for Model Reporting. Model cards are short documents that provide benchmarked evaluation in a variety of conditions, including across cultural, demographic, or phenotypic groups and intersectional groups, and disclose the context in which the model is intended to be used.
On the Hugging Face Hub, model cards are README.md files with YAML metadata. According to the Hugging Face model cards documentation, they should describe the model, its intended uses and potential limitations including biases, training parameters, datasets used, and evaluation results.
The metadata supports discoverability, reproducibility, and sharing. Users can filter models, display licenses, and link datasets. The metadata can be added via a UI or by editing the YAML section directly.
Datasheets: documenting a dataset
The datasheet proposal parallels electronics industry datasheets. The Datasheets for Datasets paper suggests that every dataset be accompanied by a datasheet documenting its motivation, composition, collection process, recommended uses, and other information.
On the Hugging Face Hub, dataset cards serve this purpose. The Hugging Face dataset cards documentation explains that each dataset may be documented by a README.md file, including metadata on language, license, size, tags, and task categories to inform responsible use.
Dataset cards also include fields to force a dataset modality or associate a library. They help users understand the contents and give context for how the dataset should be used.
Practice in model hubs and industry
Google DeepMind publishes model cards for its AI models. The Google DeepMind model cards page describes them as simple, structured overviews of how an advanced AI model was designed and evaluated.
The page lists model cards for Gemini, generative models, robotics, and Gemma families, with published or updated dates for each model. This provides a consistent format for users to find evaluation details across model lines.
Case study: documenting web-scale corpora
A case study on the Colossal Clean Crawled Corpus (C4) provides some of the first documentation for a large webtext corpus. The paper Documenting Large Webtext Corpora investigates where the data came from and finds significant text from unexpected sources such as patents and US military websites.
The study also finds machine-generated text and evaluation examples from other benchmark NLP datasets. It shows that blocklist filtering disproportionately removes text from and about minority individuals, and offers recommendations for creating and documenting web-scale datasets.
| Aspect | Model Card | Datasheet |
|---|---|---|
| Purpose | Clarify intended use and minimize misuse | Document motivation, composition, and recommended uses |
| Main contents | Intended uses, limitations, training params, datasets, evaluation results | Motivation, composition, collection process, recommended uses |
| Evaluation focus | Benchmarked across cultural, demographic, and intersectional groups | Not the primary focus; composition and collection matter |
| Format on Hub | README.md with YAML metadata | README.md with YAML metadata |
| User benefit | Supports discoverability, reproducibility, and sharing | Facilitates communication between dataset creators and consumers |
Frequently asked questions
What is a model card?
A model card is a short document that accompanies a trained machine learning model, providing benchmarked evaluation across demographic groups, intended use context, and performance details.
What is a datasheet for a dataset?
A datasheet documents a dataset's motivation, composition, collection process, recommended uses, and other information, analogous to datasheets in electronics.
How do model cards and datasheets differ?
Model cards describe a model's performance and intended uses, while datasheets describe a dataset's origins and composition. Both aim to increase transparency and accountability.
How are these documents implemented on the Hugging Face Hub?
On the Hub, model cards and dataset cards are README.md files with YAML metadata. They support discoverability, filtering, and display of licenses and datasets.