Course documents

The immutable, hash-verified JSON format a course revision is serialized into. This is the portable data contract built by Goal 1 of the roadmap. Published revisions are now the production authority and live in the persistent filesystem store on the VPS; the document format remains independent of the storage provider.

The contract

• A revision is an immutable set of documents, written manifest-last: manifest.json, one topics/<topicId>.json per topic, one collections/<collectionId>.json per collection, and indexes/search.json, indexes/review.json, indexes/graph.json, indexes/identity.json. Nothing overwrites an existing revision's files; a change publishes a new revision. Enforced by buildContentDocuments and serializeCourseRevision in src/course-os/documents/serialize.ts, with paths from src/course-os/documents/paths.ts.

• Documents are keyed by id, never by slug. A topic's filename is topics/<id>.json even when its authored slug differs — the committed fixture course has a topic with id: "intro" and slug: "introduction", and a dedicated test (fixtures.test.ts, "names the topic document by id while the slug lives inside it") asserts the document is named for the id. The slug still exists, inside the topic body, because it drives the human-readable URL segment (courseTopicHref in src/course-os/services/routes.ts); it is just not what a reader used to locate the document.

• Publication metadata is supplied by the caller, never invented. RevisionStamp (revisionId, sequence, publishedAt, and optionally parentRevisionId / restoresRevisionId) is a plain argument to serializeCourseRevision. Nothing in src/course-os/documents/serialize.ts reads a clock, a counter, or a random source, which is what makes two runs over the same course and stamp byte-identical and therefore hash-comparable.

• Canonical JSON is the codebase's single canonicalizer. canonicalJson in src/course-os/documents/canonicalJson.ts sorts every object's keys before stringifying and leaves array order untouched — order inside a topic or collection is content, not encoding. src/course-os/services/localData.ts re-exports this implementation for the Study Hub backup fingerprint rather than carrying its own; the two used to disagree, and Goal 1 collapsed them onto one. Every sort that feeds a document body — the manifest's file list, collectAssetReferences's asset list — uses compareOrdinal from the same module, never localeCompare: localeCompare consults the platform's ICU data, so the same array can sort differently between a developer machine and CI, and these orderings land inside a SHA-256.

• A round trip does not preserve authored key order inside block props. Because canonicalJson sorts keys, a document's bytes depend only on content, never on the order an author wrote it in. The round-trip suite (src/course-os/documents/roundTrip.test.ts, "keeps block props canonically identical") therefore compares canonical JSON on both sides, not raw text — a raw JSON.stringify comparison fails on the key-order sort alone, even when nothing was lost. An earlier draft of that assertion compared raw JSON.stringify and failed for all five courses; the assertion was wrong, not the serializer, and the fix was to compare canonical JSON instead of loosening what is proven.

• The identity index is self-contained and ships as its own document. RevisionIdentityIndex (src/course-os/documents/identity.ts, built by buildIdentityIndex) carries live entries, tombstones stamped with the removing revision's sequence, previously-existing tombstones left at their original sequence, and fully resolved (not chained) historical and permanent path aliases — a reader never needs an intermediate revision to interpret it, which matters because intermediate revisions may be garbage collected. Block and review-item keys carry their container id (identityKey("block", containerId, blockId)) because a block id is unique only inside its container, not inside a course: 2IRR00 uses processes and smells as a block id in both a topic and the quizzes collection, so without the container segment the two would collide. The index lives in indexes/identity.json (parseIdentityIndexDocument in src/course-os/documents/parse.ts), not inside the manifest: the manifest is fetched on every cold course read, while the identity index — an entry per block and per review item — is read only when personal state reconciles against a revision change. The manifest lists the identity document in files with its hash, so revision integrity is unchanged. Publication reads that parent identity document and passes it as previousIdentity; a missing or invalid parent index fails the publish instead of silently resetting retained tombstones.

• Assets are declared by block definitions, not inferred. A block type opts in to assetReferences(props) (src/course-os/registry/blocks.ts), and collectAssetReferences (src/course-os/documents/assets.ts) walks every container's blocks and asks each block's registered definition what it references. The undeclared-asset guard, findUndeclaredAssetPaths, scans every string prop that looks like a site-relative path and flags any that is neither declared nor a course route. It is deliberately a route pattern (courseRoutePattern, built from UNIVERSAL_SECTION_IDS) and not a bare /<courseId>/ prefix test, because 2IC30 stores real images at /2ic30/img/... — a prefix rule would ignore exactly the assets this guard exists to find. If the guard fires on your change, either declare the path through the block's assetReferences hook or confirm it is genuinely a page route, not an asset.

• A revision references assets by path, not by bytes. AssetReference carries a path and the blocks that reference it — no byte content, no hash. Deployed /public files remain the actual source; this revision format does not copy them. Whether publication should start copying them, and what happens when a live revision outlives a deployed file, is Goal 3's decision (decision 8), not settled here.

• The schema contract and the expand-before-contract rule. COURSE_SCHEMA_CONTRACT (src/course-os/versioning/contracts.ts) separates supportedForRead from published; assertExpandBeforeContract fails unless the published version is itself readable, so a release always learns to read a schema before it publishes it. Reading applies migrateCourseDocument (src/course-os/documents/migrate.ts) forward from a document's own schemaVersion toward the target using a registered chain of migrators. Three things a future reader will otherwise get wrong:

• Every parser rejects unknown fields (assertKnownFields, src/course-os/documents/parse.ts), so an additive field is unreadable by the previous deployment. Adding a field is a schema change that needs a version bump and a migrator, even when it feels purely additive.
• requiredPlugins was added to CourseManifest without a version bump only because no revision had ever been published — there was no stored document a rolled-back deployment could fail to read. That window closed the moment a revision is actually published. Do not point at this as precedent for adding a field without a version bump.

• The migration engine exists (COURSE_DOCUMENT_MIGRATORS, migrateCourseDocument) but is not wired into any read path, which is safe only while exactly one schema version is readable. A contract test in migrate.test.ts reads the live COURSE_SCHEMA_CONTRACT and COURSE_DOCUMENT_MIGRATORS and fails the moment supportedForRead holds more than one version while the migrator list is still empty — it is a real tripwire, not a comment, because widening supportedForRead to [1, 2] today, with no migrator added, fails it.

• Block props are parsed against the block registry in exactly one place: loadCourse. src/course-os/documents/parse.ts is deliberately shape-only for the manifest, topic, collection, and index documents — it validates structure and known fields but never imports blockRegistry to interpret a block's props against its type. An earlier implementation added that import and was reverted. Keep parsing and block-type interpretation apart: if you need to validate a block's props against its registered type, that validation belongs in loadCourse (src/course-os/services/content/loadCourse.ts), not in the document parser.

• Review and identity indexes now share a container-scoped item identity. Review entries carry containerId and containerKind; reviewKey is <courseId>:<containerId>:<blockId>:<itemId> and maps 1:1, after dropping the course segment, to identityKey("review-item", containerId, blockId, itemId) in indexes/identity.json. Personal state resolves through that identity document for live, aliased, tombstoned, and restored content. legacyReviewKey(courseId, blockId, itemId) remains only for the one-time stored-event migration and the reader's safe single-match fallback. containerId and containerKind are required: every live revision republished them during the Goal 6 migration, so the expand-before-contract window is closed and a review entry missing either field is a parse failure.

• The manifest is the document every read fetches first. It carries the full course manifest, the asset list, and the file list with hashes. That is why manifestBytes is tracked separately from documentBytes in src/course-os/baseline/content-footprint.generated.json: a cold course read pays the manifest's bytes on every load, so its share of the total matters on its own, not folded into a total. The identity index was moved out of the manifest for exactly this reason and is tracked as identityBytes. See buildRevisionManifestBody in src/course-os/documents/serialize.ts.

Storage layout

Serialized documents carry logical paths (topics/<id>.json, indexes/search.json, …). When a revision is stored remotely (Goal 2 reads, Goal 3 writes), the physical layout is:

courses/<course-id>/
  latest.json                                  # published pointer readers fetch by URL
  latest/<sequence>.json                       # the pointer a writer reads; immutable
  revisions/<revision-id>/manifest.json        # one commit record per revision
  objects/<sha256>.json                        # content documents, content-addressed, shared

• Nothing decides anything by reading a mutable key back. A public Blob key is served through a CDN that keeps a copy for at least the sixty seconds a public object's cache-control is clamped to, and ignores a query-string cache key, Cache-Control: no-cache, and ?cache=0 alike; only a write purges it. head reports the current ETag immediately, so a read can pair a fresh validator with a body that predates the last write — and the compare-and-swap that followed would pass its precondition and write the older document back over the newer one. Every mutable document therefore keeps its authority in a sequence of immutable keys (latest/<sequence>.json, catalog/courses/<sequence>.json, global-indexes/v1/<shard>/<sequence>.json, plugins/registry/<sequence>.json), resolved with list — an API call, which sees a new key at once. The published .json beside each sequence is a mirror for readers, who can tolerate a stale minute. The sequence number is the concurrency control: a writer creates n + 1, and one that loses the race is refused instead of overwriting the winner. src/course-os/publication/sequencedDocument.ts is the mechanism.

• Content documents are content-addressed. A document is stored once per distinct byte content at courseObjectPath(courseId, sha256) (src/course-os/documents/paths.ts) and shared across revisions. The manifest's files[] entries already carry each document's sha256, so a reader maps logical path → hash → object key with no extra schema.

• Only the manifest is written per revision. Publication uploads the objects that do not already exist, then the manifest, then conditionally moves the pointer. An unchanged topic costs zero writes on the next publish; this is what keeps publish cost proportional to the edit, not to the course (see the publish-cost model spec).
• GC liveness: an object is live iff any retained revision manifest references its hash. A revision is deletable only by the reviewed cleanup workflow; deleting a revision may orphan objects, which become collectable when no other manifest references them.

Rollout boundary

Owner publication and direct editing exist for explicitly allowlisted Blob courses. Registered production courses remain compiled until their migration gate moves them to the Blob repository.

Reading a revision

latest.json names the revision, sequence, course base URL, and manifest SHA-256. The hash lives in the pointer because the revision manifest cannot hash itself. A reader validates the course id and trusted HTTPS origin, fetches revisions/<id>/manifest.json, verifies its exact bytes, then maps each logical file through files[].sha256 to objects/<sha256>.json.

Identifier allowlists decide which courses may leave compiled truth; origin allowlists decide which network locations may be fetched. They are independent. Page reads fetch the manifest and content documents, not the four index fragments. verifyFetchedObjects proves fetched bytes match the manifest; verifyRevisionIntegrity proves the reconstructed revision is internally complete. Search and review read their own hash-verified fragments where a cross-course aggregate is needed.

Every stored JSON document, including the pointer, enters through readCourseDocument, the sole schema migration boundary. Pointer and content fetches have deadlines; timeout is an ordinary read failure. A fallback without a manifest hash is absent, not partially trusted. COURSE_MIGRATION_STATE in src/course-os/repository/migrationState.ts is the single expression of where each course is read from — compiled, dual, or blob — and it is build-time state on purpose, so a Blob outage can never change a course's source at the moment its fallback is needed. An environment entry for a course the map governs is ignored, and COURSE_BLOB_DISABLED_IDS is refused for a blob-state course because no compiled data remains to fall back to. All four production courses are blob; compiled registration remains supported for a course that has never been published.

The mutable pointer keeps a 60-second application cache window, while immutable revision objects stay force-cached. When Blob credentials are available, the server fills that cache from an origin read that bypasses Blob CDN staleness. Successful direct-editor publication immediately expires the pointer tag and the course route tree. Open Blob-backed pages also compare revision identity every three seconds and retry their route refresh until the invalidated route cache serves the published revision; compiled course paths remain statically generated.

A minimal workflow

import { serializeCourseRevision } from "@/course-os/documents/serialize";
import { courseCatalog } from "@/courses";

const course = courseCatalog.get("2wbb0");
const stamp = { revisionId: "r1", sequence: 1, publishedAt: "2026-08-03T00:00:00.000Z" };

const revision = await serializeCourseRevision(course, stamp);
// revision.documents is manifest-last: every topic, collection, and index document
// precedes `manifest.json`, and every entry carries its own `bytes` and `sha256`.

To go the other direction — parse documents back into a CourseBundle and verify a revision's manifest against its own file hashes — see src/course-os/documents/deserialize.ts and verifyRevisionIntegrity.

Verification

• npm run check runs the full round-trip suite under src/course-os/documents against every registered course plus the committed schema-1 fixture.
• npm run baseline regenerates the committed fixture at src/course-os/documents/fixtures/revision-v1 from the current serializer. CI's git diff --exit-code gate on that path (see .github/workflows/ci.yml) is what makes "regenerating is a no-op" a checked claim rather than a hope — without it, the fixture tests would only verify the generator against itself.
• Measured evidence from the run that shipped this format is recorded in docs/superpowers/reports/2026-08-03-goal-1-serialization-evidence.md.