Arweave and gateways

Seed Protocol keeps files, images and Html bodies on Arweave and points at them from EAS attestations by Arweave transaction ID. Arweave has no testnet, so these uploads are on Arweave mainnet, public and permanent, while the attestations are on OP Sepolia (what "permanent" means today).

For the API that implements all this, see @seedprotocol/arweave. For how bytes get onto Arweave, see Uploads and ANS-104 data items.

Gateways

Arweave nodes store and replicate data, but apps don't usually talk to them. They talk to a gateway: an HTTP server that indexes Arweave transactions, caches their data and serves it. A gateway gives you:

Any gateway can serve any transaction, so the content doesn't depend on one operator. What differs between gateways is what they've indexed so far. A just-uploaded transaction, and especially an ANS-104 data item, may be on the gateway it went through well before the others have it.

AR.IO

AR.IO is a network of independently run Arweave gateways built on the open-source ar-io-node. The Seed Protocol team runs its own private AR.io gateway (Seed's gateway). The SDK has no AR.IO-specific code: it treats *.ar-io.net hosts as known gateways and otherwise speaks the plain gateway HTTP API above to any host.

How the SDK reads

The SDK, @seedprotocol/query and @seedprotocol/feed read Arweave through @seedprotocol/arweave, which keeps one primary gateway and an ordered fallback list:

primary:   arweave.net
fallbacks: zigza.xyz, ar.seedprotocol.io

A read of a file's bytes goes like this:

1. Preferred gateways first. If you set gateway.preferredReadBaseUrls (typically the gateway or proxy your app uploads through), /raw/{id} is tried there first, because a new upload reaches that gateway before the public ones.
2. Probe. If the primary hasn't been checked recently, the SDK sends GET /info to it and then to each fallback in order, with a 5-second timeout, and switches to the first that answers with JSON. The choice is cached for 3 minutes (1 minute when nothing answered).
3. Circuit breaker. A host that fails is skipped for 60 seconds, then tried once more before it's trusted again.
4. Fallback reads. Raw reads that fail on one gateway move on to the next, with a 10-second timeout each.

Presence checks (getTransactionStatus) send GET /{id} down the same list without the probe, 10 seconds per gateway and 20 seconds in all. A 404 counts as an answer and isn't retried on the next gateway, so a fresh upload can read as missing on a gateway that hasn't indexed it yet. See fetchTransactionStatusAcrossGateways.

When you read through a Gateway Hyper sidecar or an app-server proxy, /raw reads try that host first and then the public gateway and the default list, without the probe. Presence checks ask only that host and the preferred gateways, because a data item it just accepted isn't on the public gateways yet. See Gateway Hyper.

To change the list, set the primary with arweaveDomain and the fallbacks with ARWEAVE_READ_GATEWAYS, which replaces the default list. That variable is read from process.env only, so a browser build sees it only if the bundler defines it (environment variables). Seed's gateway, ar.seedprotocol.io, is last by default; an app that wants it first sets arweaveDomain: 'ar.seedprotocol.io', which also becomes the host in the Html links it publishes (Links in published content). See Client and configuration.

Links in published content

Attestations point at Arweave content by transaction ID (storage_transaction_id), which every gateway serves, so they don't depend on any one host. Html is the exception: the links inside an Html body are URLs, and a URL names a gateway. Publish chooses what those links look like when it uploads the Html, and every Seed reader moves them onto its own gateways when it shows the Html.

What publish writes

Before a publish uploads an Html value, it replaces each seed:property/ reference, and each data: image under the default htmlEmbeddedDataUriPolicy: 'materialize', with a link to the uploaded file. It also rewrites every Arweave link already in the Html into the same form, so a link a reader re-hosted onto a sidecar, a proxy or another gateway, and that an editor saved back, goes up on the public gateway. The htmlArweaveRefs publish option picks the form, in the initPublish config or per publish in createPublish:

The host is in the uploaded bytes for good. Publish saves the rewritten Html to the local file too, so the local copy matches the upload; Html already published isn't rewritten. The public gateway is never a local sidecar or an app proxy: before client init resolves endpoints, publish uses the read host unless it's a loopback address or has a path prefix, such as an app proxy, and otherwise the host environment variable or arweave.net. An empty arweaveDomain means the default.

Viewers outside Seed show the Html as stored, so its images depend on the host publish wrote, with none of the SDK's fallbacks. To keep small images free of any host there, htmlEmbeddedDataUriPolicy: 'preserve' leaves data: images inline, at the cost of a larger Html upload and no separate Image item for each (publish options).

How readers re-host links

Seed readers rewrite each Arweave link in Html onto their first read gateway before showing it, with rehostArweaveLinksInHtml. An Arweave link is the URL in a src, href or poster attribute, or a srcset candidate, that is ar://<id>, /raw/<id> on any host, /<id> on a known gateway (a read base URL at a host's root, KNOWN_ARWEAVE_GATEWAYS, *.arweave.net, *.ar-io.net), or <base>/<id> under a read base URL with a path prefix, such as an app proxy. The path form is kept, ar://<id> becomes /raw/<id>, and all other markup stays as stored. Html published with links on another gateway, such as ar.seedprotocol.io, is re-hosted the same way, so it doesn't need republishing.

The first read gateway is a preferred read gateway if you set one, else the Gateway Hyper sidecar or app proxy, else the current primary gateway. A sidecar or proxy serves only this reader, which is why re-hosted Html must not be stored or uploaded and why publish moves links back onto the public gateway. Only SeedHtml retries a failed load; the other readers pick one host per link.

Transaction IDs and content hashes

Seed points at content by Arweave ID, recorded in a storage_transaction_id attestation. There are two kinds of ID:

• An ANS-104 data item ID, which every publish records, is the SHA-256 hash of the data item's signature, which covers its owner, anchor, tags and data. A bundler later packs data items into an L1 transaction of its own.
• An L1 transaction ID, for files uploaded on the L1 path by publish 0.7.8 or earlier, is the SHA-256 hash of the transaction's signature. The signature covers the transaction's fields, including its tags and the Merkle root of its data (data_root).

Either way, the bytes behind an ID can't change without breaking the signature. But an ID isn't a content address: the same file uploaded twice gets two IDs, and you can't compute an ID from the bytes alone.

For that, every Seed upload also carries a Content-SHA-256 tag: the lowercase hex SHA-256 digest of the bytes, with one exception on older L1 uploads:

Tags Seed writes

Each upload gets these tags, in this order, and nothing else unless you add it:

Seed doesn't add an App-Name or any Seed-specific tag by default, so you can't find Seed uploads on Arweave by tag unless you add one yourself, for example arweaveUploadTags: [{ name: 'App-Name', value: 'MyApp' }]. The way to find an item's files is through its attestations. Bundles made by a bundler carry the bundler's own tags; Seed doesn't set those.

To read the tags of an upload, query a gateway's GraphQL API:

query ($id: ID!) {
  transaction(id: $id) {
    owner { address }
    tags { name value }
  }
}

Data items signed with an Ethereum key show an 0x… owner address.

Verifying content

The SDK checks what it downloads. Before downloading files into the browser's storage, it looks up their Content-SHA-256 tags with batched GraphQL requests (getTransactionTagsByIds), and compares each file's bytes with its tag. A gateway that serves bytes that don't match is skipped for the next one; if no gateway serves matching bytes, the file isn't saved and a later run tries again. Only verified digests are stored in the local metadata. A file whose tag the gateway doesn't return yet (a fresh upload that isn't indexed) is downloaded without the check. Both digests in the table above are accepted, through the SDK's exported contentSha256TagMatches(bytes, tagValue).

To check an upload yourself, outside the SDK:

import { createHash } from 'node:crypto'

const GATEWAY = 'https://arweave.net'

/** Returns the algorithm whose digest matches the Content-SHA-256 tag, or null if neither does. */
export async function verifyUpload(txId: string): Promise<'sha256' | 'sha3-256' | null> {
  const res = await fetch(`${GATEWAY}/graphql`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      query: 'query($id: ID!) { transaction(id: $id) { tags { name value } } }',
      variables: { id: txId },
    }),
  })
  const { data } = (await res.json()) as {
    data: { transaction: { tags: { name: string; value: string }[] } | null }
  }
  const expected = data.transaction?.tags.find((t) => t.name === 'Content-SHA-256')?.value
  if (!expected) return null

  const bytes = new Uint8Array(await (await fetch(`${GATEWAY}/raw/${txId}`)).arrayBuffer())

  // SHA3-256: L1 uploads from 0.7.2 and earlier.
  for (const algorithm of ['sha256', 'sha3-256'] as const) {
    if (createHash(algorithm).update(bytes).digest('hex') === expected) return algorithm
  }
  return null
}

This catches a gateway serving the wrong or truncated bytes. It doesn't protect against a gateway that lies about both the tags and the data, since both came from the same gateway. For that, compare across gateways you trust, or verify the transaction's signature and data root with an Arweave client.

Related

• @seedprotocol/arweave: gateway resolution, probing and read helpers
• Uploads and ANS-104 data items: how bytes get there
• Gateway Hyper: reading through a sidecar or proxy
• Permanence: what's stored where
• Client and configuration: the SDK's gateway settings