@seedprotocol/archive

@seedprotocol/archive packages a published item as a BagIt bag: its Html, its files and its attested values, with SHA-256 and SHA-512 checksum manifests and Dublin Core and PREMIS metadata, so an archive can validate and ingest it without Seed code. It also checks bags, offline or against an archive anchor on EAS, and reads and writes the path manifests that lay a bag out on Arweave.

npm install @seedprotocol/archive

It depends on @seedprotocol/arweave, @seedprotocol/eas and @seedprotocol/query at its own exact version, and lists @seedprotocol/sdk as an optional peer at the same version, so npm install @seedprotocol/archive doesn't install the SDK. @seedprotocol/publish depends on it for anchorArchive.

Export a published item

exportArchive builds the bag for an item's published state, in memory.

function exportArchive(
  item: ExportableItem | ArchiveSeedRef,
  options?: ExportArchiveOptions,
): Promise<ExportArchiveResult>

item is an Item, as Item.create returns, or, for a reproducible bag only, an ArchiveSeedRef: { seedUid, modelName, schemaName? }. The declared ExportableItem type doesn't accept the IItem<any> that Item.find returns (its getService() type differs), so cast those: exportArchive(post as ExportableItem), with ExportableItem imported as a type from @seedprotocol/archive.

An ordinary export first waits for the writes pending in the outbox on the item and the items its publish covers, as publish() does, then reads this device's rows and files.

It resolves to { bag, omitted, arweaveTransactions }:

What the bag contains

Json storage seeds and the content of related items aren't exported.

Every time in the bag says whose clock it came from, in each PREMIS event's detail, as attestedAtClock next to each attestedAt in data/item.json, and in Seed-Bag-Times: chain-block is the attestation's block time as EAS reports it; local-record is the time stored on the exporting device, which for an attestation that device published may be its own clock; export-device is the exporting device's clock, for an ordinary bag's own events.

Errors

It throws ArchiveExportError, with code:

Example

On Node, writeBagDirectory writes the bag to a directory, which must be empty or not exist yet. In a browser, zip bag.files or write them to OPFS.

import { exportArchive, type ExportableItem } from '@seedprotocol/archive'
import { writeBagDirectory } from '@seedprotocol/archive/node'

const { bag, omitted } = await exportArchive(post as ExportableItem, { sourceOrganization: 'The Daily Example' })
await writeBagDirectory(bag, './post-bag')
for (const { property, reason } of omitted) console.warn(`${property}: ${reason}`)

Reproducible bags

With reproducible: true, every client builds the same bytes for a seed at one time, so the same tagmanifest-sha256.txt hash. This is the bag to anchor, sign or publish. It reads EAS and Arweave and no rows, so a device that holds nothing for the seed can name it by UID; only the schema has to be loaded.

import { exportArchive } from '@seedprotocol/archive'

const { bag, arweaveTransactions } = await exportArchive(
  { seedUid: '0xa313944ea62a86bfbb9db2d9be984d7ad8641eead88d5769500de5f6c3bdb3eb', modelName: 'Post', schemaName: 'blog' },
  { reproducible: true, asOf: 1793799143 },
)

The bag depends on these inputs and no others:

• The time it covers, asOf, recorded as Seed-Bag-As-Of in bag-info.txt and asOf in data/item.json. The bag holds the newest Version live then and the property values current then, so a later patch, Version, retraction or revocation doesn't change it. Without asOf it covers the latest state: the time of the last attestation or revocation on the newest live Version's properties and its storage seeds, so the bag follows the seed.
• The seed's attestation and its Versions, as EAS reports them. Only the seed attester's Versions and the Version attester's property attestations count, since anyone can attest with any refUID.
• Each property's attestation current at asOf by sync's rule: the newest live one, and no value when all were revoked.
• The bytes of the Arweave transactions those attestations name. Files are always read from Arweave, because a local copy can differ from what was uploaded, and are typed by their bytes (application/octet-stream when the bytes don't show a type).
• The model name and each attested property's key and data type. Not the schema's name: a schema named differently gives the same bag. A property added later changes nothing; renaming an attested property or changing its type does.
• The chain and the format version, Seed-Bag-Format (1).

The export time, device clocks, local file names, the configured gateway, unpublished edits and sourceOrganization never reach the bytes. Every time is chain-block, and PREMIS names files on Arweave as ar://<tx>. History stops at the current Version and leaves revocations out: check revocation on EAS, not in the bag.

asOf says what to read, not that EAS has it all. Soon after a publish, pass the attestation UIDs the publish made as expectedAttestations; without them a bag built while the indexer is part way through the publish is valid and incomplete. A not_synced refusal means retry later. To read what EAS held for a seed at a time without building a bag, use query's readSeedAtTime.

Verify a bag

verifyArchive checks a bag with no rows and no network. Each check is reported on its own, with no overall verdict.

function verifyArchive(bag: Bag, options?: { expectedTagmanifestSha256?: string }): ArchiveVerification

status is 'passed', 'failed' or 'not_checked'. fixity alone says the bag is intact, not that it's the bag you expect: anyone who changes a file can rewrite the manifests too. Only anchor, against a hash from outside the bag (an archive anchor, a signature, a rebuild), ties the bag to its source. A bag whose payload was changed without touching the tag manifest still passes anchor while fixity fails.

import { readBagInfo, verifyArchive } from '@seedprotocol/archive'

const { fixity, anchor } = verifyArchive(bag, { expectedTagmanifestSha256: anchorHash })
const asOf = readBagInfo(bag).get('Seed-Bag-As-Of')

readBagInfo(bag) returns bag-info.txt as a Map of label to value, with the first value of a repeated label.

The commands in VERIFY.txt do the same check without code: shasum -a 256 -c tagmanifest-sha256.txt and shasum -a 256 -c manifest-sha256.txt in the bag's directory, then shasum -a 256 tagmanifest-sha256.txt to compare with an outside hash. verifyInstructions({ seedUid, versionUid, explorerUrl?, asOf? }) returns that text.

verifyArchiveAgainstChain

verifyArchiveAgainstChain(bag, item) runs verifyArchive's checks and adds rebuilt: it makes a reproducible export of item (an Item or an ArchiveSeedRef) at the bag's Seed-Bag-As-Of and compares the two, listing in differingFiles the paths that differ or exist in only one bag. It takes no outside hash, so anchor is not_checked. rebuilt is not_checked, with the reason in detail, for an ordinary bag, a bag of another seed, another Seed-Bag-Format, or an export that refuses.

import { verifyArchiveAgainstChain } from '@seedprotocol/archive'

const { fixity, rebuilt, differingFiles } = await verifyArchiveAgainstChain(bag, {
  seedUid: '0xa313944ea62a86bfbb9db2d9be984d7ad8641eead88d5769500de5f6c3bdb3eb',
  modelName: 'Post',
})

Check a published bag

An archive anchor is an attestation on a Version that records a reproducible bag's asOf, its path manifest's Arweave transaction and its tagmanifest hash. Archive anchors covers writing one with anchorArchive. The functions here read anchors from @seedprotocol/archive/verify, which pulls in no Html parser or Seed code, so a reader's page can load it on demand:

import type { SeedRecord } from '@seedprotocol/query'

async function checkArchive(record: SeedRecord) {
  const { verifyPublishedArchive } = await import('@seedprotocol/archive/verify')
  return verifyPublishedArchive({
    versionUid: record.versionUid,
    easEndpoint: 'https://optimism-sepolia.easscan.org/graphql',
    gateways: ['https://arweave.net'],
  })
}

A reader counts an anchor only from the Version's attester, because anyone can attest with any refUID. These functions ignore anchors from anyone else.

verifyPublishedArchive

From a Version UID alone, it finds the Version's newest live anchor, reads the bag its path manifest lays out from a gateway, and checks it. Each check is reported on its own, with no overall verdict.

The result also has archiveAnchor (the anchor checked: uid, attester, timeCreated, coveredUntil, manifestTx, tagmanifestSha256), size ({ payloadBytes, payloadFiles } from Payload-Oxum) and recorded (the seed, Version, and per-property attestation and transaction from data/item.json, which is what a passed record vouches for).

It doesn't throw for what it finds: a service that can't be reached leaves its checks not_checked with the reason in detail, so an outage never reads as a forgery. It throws only when signal aborts.

findArchiveAnchors

function findArchiveAnchors(options: {
  versionUid: string
  easEndpoint: string
  fetch?: typeof fetch
  signal?: AbortSignal
}): Promise<ArchiveAnchor[]>

The live anchors on a Version from its attester, newest first, without downloading a bag, for deciding whether to offer the check. Empty when the indexer hasn't reported the Version. Unlike verifyPublishedArchive, it throws when EAS can't be read.

decodeArchiveAnchorData(data) decodes an anchor attestation's data (three ABI words) to { coveredUntil, manifestTx, tagmanifestSha256 }, or returns undefined for anything else. ARCHIVE_ANCHOR_SCHEMA_UID (0xf7f6e2f226f29542fa8bd1763d042d04f47a0a9c3647753f68cc113e26f1739e) and VERSION_SCHEMA_UID are the two schema UIDs it reads; the anchor schema is uint64 covered_until,bytes32 manifest_tx,bytes32 tagmanifest_sha256 and its UID is the same on every chain. @seedprotocol/eas has the same constants, a raw anchor read and the ID converters for code that already uses it.

A bag on Arweave

A path manifest (arweave/paths 0.2.0) lays a bag out on Arweave: each file is its own transaction, and the path manifest maps bag paths to them, so a gateway serves the whole bag under one ID. anchorArchive uploads one for you; these functions are what it and verifyPublishedArchive use.

import { buildPathManifest, parsePathManifest, readBagFromManifest, verifyArchive } from '@seedprotocol/archive'

// Writing: transactions for the bag's other files come from your own uploads
const { bytes } = buildPathManifest(bag, { ...arweaveTransactions, ...uploaded })

// Reading, from the path manifest's bytes alone
const fetchTransaction = async (id: string) => {
  const response = await fetch(`https://arweave.net/raw/${id}`)
  return response.ok ? new Uint8Array(await response.arrayBuffer()) : undefined
}
const copy = await readBagFromManifest(parsePathManifest(manifestBytes), fetchTransaction)
const { fixity, anchor } = verifyArchive(copy, { expectedTagmanifestSha256: anchorHash })

buildPathManifest(bag, transactions) returns { manifest, bytes }. transactions maps every bag path to its transaction ID; a missing one throws ArchiveError invalid_manifest (VERIFY.txt has no transaction). data/index.html is the path manifest's index. A gateway serves the index at the path manifest's root, so each data/assets/ file is also listed as assets/<file> for the page's relative links. A bag with several Html properties has no index. bytes depend only on the input; upload them with the content type PATH_MANIFEST_CONTENT_TYPE (application/x.arweave-manifest+json).

parsePathManifest(data) takes a Uint8Array or string and checks every path and transaction ID, since the path manifest comes from Arweave. It throws ArchiveError: invalid_manifest for anything that isn't arweave/paths 0.2.0, isn't JSON or has a malformed ID, and invalid_path for a path such as ../x.

readBagFromManifest(manifest, fetchTransaction, { only? }) fetches each path and returns the bag without the assets/ aliases. fetchTransaction resolves to the transaction's bytes, or undefined, which throws missing_file. only limits it to the paths it accepts, such as the tag files ((path) => !path.startsWith('data/')).

The path manifest says where the files are and doesn't vouch for them. Only verifyArchive against an outside hash does.

Build bags directly

The building blocks under exportArchive work on any content.

import { createBag, validateBag } from '@seedprotocol/archive'
import { readBagDirectory, writeBagDirectory } from '@seedprotocol/archive/node'

const bag = createBag({
  payload: [{ path: 'index.html', bytes: new TextEncoder().encode(html) }],
  info: [['Source-Organization', 'The Daily Example']],
})
await writeBagDirectory(bag, './bag')
const { valid, errors } = validateBag(await readBagDirectory('./bag'))

createBag({ payload, tagFiles?, info?, baggingDate?, softwareAgent? }) builds a BagIt 1.0 bag: bagit.txt, bag-info.txt, SHA-256 and SHA-512 payload and tag manifests. payload paths are relative to data/; tagFiles paths (such as metadata/dc.xml) are relative to the bag's base and go in the tag manifests. It adds Bagging-Date (today, UTC), Payload-Oxum and Bag-Software-Agent (DEFAULT_SOFTWARE_AGENT) unless info has them.

It throws ArchiveError for a path that won't survive another file system: empty, . or .. segments, a backslash, control characters, %, text not in Unicode NFC, or two paths that differ only in letter case. % is refused because bagit-python 1.9.0 doesn't decode the %25 RFC 8493 writes in manifests. assertBagPath(path) runs the same check. ArchiveError's code is 'invalid_path', 'duplicate_path', 'invalid_bag_info', 'invalid_manifest' or 'missing_file'.

validateBag(bag) returns { valid, errors }, one error per problem naming the file. It checks bagit.txt, every payload and tag manifest entry, unlisted payload files and Payload-Oxum, and reports a manifest for an algorithm other than SHA-256 and SHA-512 as an error. For an independent check, run the Library of Congress validator on the directory:

pipx run --spec bagit==1.9.0 bagit.py --validate ./bag

writeBagDirectory(bag, directory) throws a plain Error (… is not empty) for a directory that has files, so a bag never mixes with files its manifests don't list. readBagDirectory(directory) reads every file under it as a bag.

Also exported: getBagFile(bag, path), encodeManifestPath and decodeManifestPath (RFC 8493 percent-encoding), digestHex(algorithm, bytes), BAG_ALGORITHMS (['sha256', 'sha512']) and BAGIT_VERSION ('1.0').

Archive documents and metadata

parseArchiveDocument(html) reads the body and descriptive fields of Html marked up as an h-entry. The body comes from .h-entry .e-content, else the first [itemprop=articleBody], else a lone top-level <article> without its direct <header> and <footer>, else the whole input; method says which ('h-entry', 'articleBody', 'article', 'fragment'). It returns content (sliced from the input, not re-serialized), name, summary, published, authors, featured and url. Implied properties and value-class patterns aren't applied.

rewriteHtmlUrls(html, replace) replaces whole src, href and poster values for which the function replace returns a new URL, splicing them into the original string so every other byte stays; srcset isn't rewritten. findHtmlUrls(html) lists those values once each in document order, and rewriteHtmlAttributes(html, rewrite) is the general form for any attribute.

dublinCoreXml(record) writes an oai_dc:dc record from the 15 DCMI elements, each a string or array. premisXml({ objects, events, agents }) writes PREMIS 3.0 with event and agent types from the Library of Congress preservation vocabularies, listed in PREMIS_EVENT_TYPES and PREMIS_AGENT_TYPES.

Related

• Archive anchors: anchorArchive, its cost and the anchor schema
• Permanence: what a published record does and doesn't guarantee
• Arweave and gateways
• @seedprotocol/eas: raw anchor reads