Update: this proposal has been moved to stage 2. Continue discussion here

Summary

This proposal adds support for "data collections," or content collections using simple data storage formats (namely JSON and YAML). This also introduces a pluggable API to support any data format in the future.

Background & Motivation

Content collections are restricted to supporting .md and .mdx files. This is limiting for other forms of data you may need to store, namely raw data formats like JSON, YAML, and TOML. Common use cases include storing blog or documentation authors separate from the content of each post.

The content collections API was built generically to support this future, choosing format-agnostic naming like data instead of frontmatter and body instead of rawContent. Because of this, expanding support to new data formats without API changes is possible to achieve.

Goals

• Introduce JSON, YAML, and TOML collection support, configurable and queryable with similar APIs to content collections.
• Determine where data collections are stored. We may introduce a new src/data/ directory distinct from src/content/, or simply allow data collections within src/content/.
• Introduce an integration API to 1) add new data formats / file extensions to content collections, and 2) implement how each format's collection entry info (data and body) is retrieved.

Non-goals

• Pluggable APIs for content authoring formats like Markdoc and Org-mode. We recognize the need for support beyond Markdown and MDX, so this will be considered in a future proposal.
• A mechanism to "reference" collections from other collections. It's clear that some data collections are related to others. For example, you may want to connect an "author" property in a blog post to an "authors" collection entry. This will be explored in a future proposal.

Example

Using the built-in JSON format

Say you have a collection of blog post authors you would like to store as JSON. You can create a new collection under src/content/ (or src/data/, pending conversation) like so:

src/content/
  authors/
    ben.json
    fred.json
    matthew.json

This collection can be configured with a schema like any other content collection:

// src/content/config.ts
import { defineCollection, z } from 'astro:content';

const authors = defineCollection({
	schema: z.object({
		name: z.string(),
		twitter: z.string().url(),
	})
});

export const collections = { authors };

It can also be queried like any other collection, this example using getCollection('authors'):

---
import { getCollection } from 'astro:content';
const authors = await getCollection('authors');
---
<ul>
{authors.map(author => (
	<li>
		<a href={author.data.twitter}>{author.data.name}</a>
	</li>
)}
</ul>

Each entry will have the same type signature as a content collection entry, with the slug property and render() function removed:

type DataEntry = {
	id: 'file-1.json' | 'file-2.json'...
	collection: 'authors'
	data: ConfiguredSchema<'authors'>
	body: string;
}

type ContentEntry = {
	id: 'file-1.md' | 'file-2.jmd'...
	slug: 'file-1' | 'file-2'
	collection: 'blog'
	data: ConfiguredSchema<'blog'>
	body: string;
	render(): Promise<{ Content: AstroComponentFactory }>;
}

Configuring and using a custom format

Say you want to store these authors as TOML instead. Assuming TOML were not built-in, you could configure the TOML collection type using the addContentEntryType integration hook:

// tomlIntegration.ts
import tomlParser from 'toml';

export const toml: AstroIntegration = () => ({
		'astro:config:setup'({ addContentEntryType }) {
			addContentEntryType({
				extensions: ['.toml'],
				getEntryInfo({ contents }) {
					const data = tomlParser.parse(contents);
					return {
						// The unparsed data object that can be passed to a Zod schema.
						data,
						// The body of the data file. You can pass the file contents unchanged, or strip unrelated metadata (ex. strip frontmatter from Markdown body)
						body: contents,
					}
				}
			});
		}
	}
});

// astro.config.mjs
import { toml } from './tomlIntegration'

export default {
	integrations: [toml()],
}

Now, you are free to use .toml files in your collections, even mixed alongside files of another type for smoother migration:

src/content/
	authors/
		ben.toml
		fred.toml
		matthew.json