Import an RSS feed into your models

Moving a blog onto Seed usually starts with content that already exists somewhere else, often as an RSS feed. @seedprotocol/mapping connects the fields of that outside content to your model's properties: title to title, content:encoded to body, the author's name to an Author item. You then apply the mapping to each entry and create items from the result. This guide imports an RSS feed into the blog schema's Post model, first in a script and then with a UI where users adjust the mapping themselves.

You'll need:

• The blog schema from Your first schema, with Post and Author
• A running SDK client: in a browser app as in Add Seed to your app, or in Node as in Using Seed in Node and Electron
• For the UI, React 18 or 19

Install

npm install @seedprotocol/mapping

The main entry has no React dependency, so it works in scripts and on servers. The FieldMapper component comes from @seedprotocol/mapping/react and needs react and react-dom. The package doesn't depend on any other Seed package: it produces plain values, and you create the items with the SDK.

A Vite app that calls rssXmlToSources in the browser, as the UI below does, needs seedVitePlugin({ includeNodePolyfills: true }), which polyfills the timers module the feed parser reads. In a browser has the config.

Describe the target model

Mapping works against a list of TargetProperty objects: a property name, its data type, and for relations the related model. Build them from your schema file so they stay in step with it. Note that the schema file calls the related model model, while a TargetProperty calls it ref.

import type { PropertyBag, TargetProperty } from '@seedprotocol/mapping'
import schema from '../seed.schema.json' with { type: 'json' }

type SchemaProperty = { type: string; model?: string; refValueType?: string; required?: boolean }

export function targetsForModel(modelName: keyof typeof schema.models): TargetProperty[] {
  const properties: Record<string, SchemaProperty> = schema.models[modelName].properties
  return Object.entries(properties).map(([name, p]) => ({
    name,
    dataType: p.type,
    ref: p.model,
    refValueType: p.refValueType,
    required: p.required,
  }))
}

export const postTargets = targetsForModel('Post')

required only matters to the FieldMapper UI, which flags required properties nothing maps to.

Parse the feed into sources

rssXmlToSources parses RSS or Atom XML and turns each entry into a list of sources, one per field. Fetching the feed is up to you; the package never makes network requests. In a browser, most feeds don't send CORS headers, so fetch them through your server.

import { rssXmlToSources } from '@seedprotocol/mapping'

const xml = await (await fetch('https://blog.example.com/rss.xml')).text()
const { channel, items } = await rssXmlToSources(xml)

for (const source of items[0]!) console.log(source.id, '=', source.value)

For a typical RSS 2.0 item that prints sources like these:

IDs are rss- plus the field name the parser produced, so the same field gets the same ID in every entry of every feed. Fields an entry doesn't have produce no source.

Map sources onto properties

A mapping is a list of edges, each from a source ID to a property name. autoMap makes a first guess from field names:

import { autoMap } from '@seedprotocol/mapping'
import { postTargets } from './postMapping.js'

const mappings = autoMap(items[0]!, postTargets)
// [
//   { sourceId: 'rss-content:encoded', propertyName: 'body' },
//   { sourceId: 'rss-creator', propertyName: 'author', resolve: 'lookup' },
//   { sourceId: 'rss-title', propertyName: 'title' },
// ]

It matched the title and body, and connected the author's name to the author relation with resolve: 'lookup', a step you complete below. It left summary, publishedAt and tags alone because their names don't match any field, so add those edges yourself:

import type { FieldMapping } from '@seedprotocol/mapping'

const extraEdges: FieldMapping[] = [
  { sourceId: 'rss-contentSnippet', propertyName: 'summary' },
  { sourceId: 'rss-isoDate', propertyName: 'publishedAt' },
  { sourceId: 'rss-categories', propertyName: 'tags' },
]
for (const edge of extraEdges) {
  if (!mappings.some((m) => m.propertyName === edge.propertyName)) mappings.push(edge)
}

Each property takes at most one source, but one source can feed several properties. autoMap sends rss-link to every property named like a URL (importUrl, canonicalUrl, link), for example.

Point relations at existing items

A relation needs the ID of an item, not a name. A lookup edge stores a table from the source's text to item IDs, so "Jane Doe" becomes the seedLocalId of the matching Author. Names match without regard to case or extra spaces. autoMap creates the edge with an empty table; you fill it:

import { Item } from '@seedprotocol/sdk'
import type { LookupEntry } from '@seedprotocol/mapping'

const authorEntries: LookupEntry[] = []
for (const name of ['Jane Doe', 'Sam Roe']) {
  const values = { modelName: 'Author', name }
  const author = await Item.create(values)
  authorEntries.push({ value: name, refs: [author.seedLocalId] })
}

for (const edge of mappings) {
  if (edge.propertyName === 'author') edge.lookup = { entries: authorEntries }
}

Here the authors are created on the spot. In a real import, look up existing Author items first, and create only the missing ones. The package never fetches or creates items for you.

Apply the mapping

applyMappingAsync(sources, mappings, targets) turns one entry's sources into a property bag, an object keyed by property name, ready for Item.create. Values are converted to the target's type on the way: Number to a number, Json parsed, and Boolean true for "true", "1" or "yes" ("yes" in any case, "true" only in lower case). coerceValue lists every type.

A Date target gets a timestamp in milliseconds, but the SDK stores dates as ISO 8601 strings, and Item.create rejects the number with ItemCreateValidationError (publishedAt: Expected string). Convert it before creating items:

/** Mapped dates are milliseconds; the SDK stores ISO 8601 strings. */
export function toItemValues(bag: PropertyBag): Record<string, unknown> {
  const values: Record<string, unknown> = { ...bag }
  if (typeof values.publishedAt === 'number') {
    values.publishedAt = new Date(values.publishedAt).toISOString()
  }
  return values
}

A List target gets the source's text unchanged. For rss-categories that's a JSON array string, which Item.create accepts and stores as an array.

Then create one Post per entry:

import { Item, ItemCreateValidationError } from '@seedprotocol/sdk'
import { applyMappingAsync } from '@seedprotocol/mapping'
import { postTargets, toItemValues } from './postMapping.js'

for (const sources of items) {
  const { properties, errors } = await applyMappingAsync(sources, mappings, postTargets)
  if (errors.length > 0) {
    console.warn('Skipped fields:', errors.map((e) => `${e.propertyName}: ${e.message}`))
  }
  const values = { modelName: 'Post', ...toItemValues(properties) }
  try {
    await Item.create(values)
  } catch (err) {
    // One bad entry (a title over 255 characters, say) shouldn't stop the import
    if (!(err instanceof ItemCreateValidationError)) throw err
    console.warn('Skipped entry:', err.problems.map((p) => `${p.property}: ${p.message}`))
  }
}

A copy edge whose source the entry doesn't have is skipped. A lookup, extract or file edge whose source is missing adds Source not found: <sourceId> to errors. A lookup with no entry for the source's text (an author you didn't list) leaves the property out and adds No lookup entry for "<name>" to errors; a source with an empty value, as Markdown frontmatter can have, is no lookup and adds no error. On an edge with split, which looks up each name in a joint byline, each part with no entry adds No lookup entry for "<part>" (split from "<value>"), and the property still gets the other parts' IDs (Relation lookups).

The items are local drafts, like any other new item. Publish them when you're ready; see Publish and revoke items.

Edges that need your app: transforms

An edge's resolve field names its transform. Two transforms need work only your app can do, and autoMap sets them:

• resolve: 'extract' when a property is HTML but the entry only has a link: someone has to fetch the page and extract the article.
• resolve: 'file' when an image or audio URL (an enclosure, say) maps onto an Image or File property: someone has to download it.

applyMapping, the synchronous version, skips those edges. applyMappingAsync calls a resolve callback for each one and uses what it returns:

const { properties, errors } = await applyMappingAsync(sources, mappings, targets, {
  resolve: async ({ job, url }) => {
    if (job === 'extract' && url) return extractArticleHtml(url)
    throw new Error(`Unsupported transform: ${job}`)
  },
})

Without a callback, those edges end up in errors as No resolve callback for extract. Edges with resolve: 'derive' or 'assemble' are stored on the mapping for your app to apply, and neither apply function touches them.

Save the mapping for next time

A mapping depends on the feed, not the entry, so you can store it and reuse it for every later import from the same feed. Wrap it in a MappingDocument and keep it as JSON:

import type { MappingDocument } from '@seedprotocol/mapping'
import { writeFile } from 'node:fs/promises'

const doc: MappingDocument = { version: 1, sourceKind: 'rss', mappings }
await writeFile('post-mapping.json', JSON.stringify(doc, null, 2))

The lookup tables are saved on their edges, so authors you matched once stay matched.

Let users edit the mapping

FieldMapper is a React component for editing a mapping. It shows one row per property, with the source chosen for it, the transform (copy, extract, file, lookup) and a preview of the value. It's controlled: you pass mappings and onChange, and keep the state.

import { useEffect, useState, type ReactNode } from 'react'
import { Item } from '@seedprotocol/sdk'
import {
  rssXmlToSources,
  autoMap,
  applyMappingAsync,
  lookupKey,
  type FieldMapping,
  type SourceNode,
} from '@seedprotocol/mapping'
import { FieldMapper } from '@seedprotocol/mapping/react'
import { postTargets, toItemValues } from './postMapping.js'

export function MapFeed({ xml }: { xml: string }) {
  const [items, setItems] = useState<SourceNode[][]>([])
  const [mappings, setMappings] = useState<FieldMapping[]>([])

  useEffect(() => {
    rssXmlToSources(xml).then(({ items }) => {
      setItems(items)
      if (items[0]) setMappings(autoMap(items[0], postTargets))
    })
  }, [xml])

  async function importAll() {
    for (const sources of items) {
      const { properties } = await applyMappingAsync(sources, mappings, postTargets)
      const values = { modelName: 'Post', ...toItemValues(properties) }
      await Item.create(values)
    }
  }

  if (!items[0]) return <p>Loading feed…</p>

  return (
    <>
      <FieldMapper
        sources={items[0]}
        targets={postTargets}
        mappings={mappings}
        onChange={setMappings}
        slots={{ preview: 'row', lookups: 'row' }}
        renderLookup={(row) => (
          <AuthorPicker
            name={row.sampleValue}
            onPick={(seedLocalId) => {
              // onLookupChange replaces the whole table, so keep the other names
              const key = lookupKey(row.sampleValue)
              const others = (row.mapping?.lookup?.entries ?? []).filter(
                (entry) => lookupKey(entry.value) !== key,
              )
              row.onLookupChange([...others, { value: row.sampleValue, refs: [seedLocalId] }])
            }}
          />
        )}
      />
      <button type="button" onClick={importAll}>
        Import {items.length} posts
      </button>
    </>
  )
}

// Your own component: lists Author items and calls onPick with one's seedLocalId
declare function AuthorPicker(props: { name: string; onPick: (seedLocalId: string) => void }): ReactNode

sources is one sample entry; the mapping you build applies to all of them. renderLookup draws your picker on relation rows. The package has no idea what an Author is, so it gives you the row's sample text (row.sampleValue, here the author's name) and row.onLookupChange to save entries onto the edge. onLookupChange replaces the edge's whole table, so the example keeps the entries for other names. It compares names by lookupKey, as matching does, so picking an author for "mira chen" replaces a "Mira Chen" entry instead of leaving it first in the table, where it would still win.

The other props are in the FieldMapper reference: slots to turn parts of the UI on or off, rowKey for one row per edge instead of per property, theme for styling, and renderRowAccessory, renderRowExpansion and transformOptions for your own controls on rows. For a custom UI, useFieldMapper gives you the same rows, coverage and setters without the markup.

Some older @seedprotocol/mapping/react props still work but are deprecated; don't use them in new code. FieldMapper UI lists them with their replacements.

The document-wide lookups table is gone, and a saved mapping's lookups are ignored without a warning, so those values stop matching. Move each entry onto its edge's lookup.entries with withLookupEntries.

Complete example

A Node script that imports a feed into Post items, reusing the helpers in src/postMapping.ts from above. Compile it, then run the output with node, as Using Seed in Node and Electron describes. It needs no loader hook. The relative filesDir puts the database in .seed/data under the working directory.

import { client, Item, ItemCreateValidationError } from '@seedprotocol/sdk'
import {
  rssXmlToSources,
  autoMap,
  applyMappingAsync,
  type FieldMapping,
  type LookupEntry,
  type MappingDocument,
} from '@seedprotocol/mapping'
import { readFile, writeFile } from 'node:fs/promises'
import { postTargets, toItemValues } from './postMapping.js'

const schema = JSON.parse(await readFile('seed.schema.json', 'utf8'))
await client.init({ config: { filesDir: 'data', schema } })

// 1. Fetch and parse the feed
const xml = await (await fetch('https://blog.example.com/rss.xml')).text()
const { items } = await rssXmlToSources(xml)

// 2. Authors the posts will point to, keyed by the name the feed uses
const authorEntries: LookupEntry[] = []
for (const name of ['Jane Doe', 'Sam Roe']) {
  const values = { modelName: 'Author', name }
  const author = await Item.create(values)
  authorEntries.push({ value: name, refs: [author.seedLocalId] })
}

// 3. Auto-map the first item, then fill the gaps by hand
const mappings: FieldMapping[] = autoMap(items[0]!, postTargets)
const extraEdges: FieldMapping[] = [
  { sourceId: 'rss-contentSnippet', propertyName: 'summary' },
  { sourceId: 'rss-isoDate', propertyName: 'publishedAt' },
  { sourceId: 'rss-categories', propertyName: 'tags' },
]
for (const edge of extraEdges) {
  if (!mappings.some((m) => m.propertyName === edge.propertyName)) mappings.push(edge)
}
for (const edge of mappings) {
  if (edge.propertyName === 'author') edge.lookup = { entries: authorEntries }
}

// 4. Save the mapping so the next import reuses it
const doc: MappingDocument = { version: 1, sourceKind: 'rss', mappings }
await writeFile('post-mapping.json', JSON.stringify(doc, null, 2))

// 5. Apply it to every item and create Posts
for (const sources of items) {
  const { properties, errors } = await applyMappingAsync(sources, doc.mappings, postTargets)
  if (errors.length > 0) {
    console.warn('Skipped fields:', errors.map((e) => `${e.propertyName}: ${e.message}`))
  }
  const values = { modelName: 'Post', ...toItemValues(properties) }
  try {
    const post = await Item.create(values)
    console.log('Imported', post.seedLocalId)
  } catch (err) {
    if (!(err instanceof ItemCreateValidationError)) throw err
    console.warn('Skipped entry:', err.message)
  }
}

await client.stop()

Importing Markdown

The same flow works for Markdown files with YAML frontmatter. markdownToSources(markdown) returns one source per frontmatter field (fm-title, fm-summary, …), one for the whole body (section-full) and one per # or ## section:

import { markdownToSources, autoMap, applyMapping } from '@seedprotocol/mapping'
import { postTargets } from './postMapping.js'

const sources = markdownToSources(markdown)
const mappings = autoMap(sources, postTargets)
// fm-title → title, fm-summary → summary, section-full → body
const values = applyMapping(sources, mappings, postTargets)

The body is copied as Markdown. Convert it to HTML yourself before you store it in an Html property. Frontmatter must be flat: nested YAML isn't supported. A list such as tags: [news, seed] arrives as the text "news, seed", which Item.create rejects for a List property, so split it into an array first. An empty field (summary: with no value) still produces a source, and mapping it sets the property to the empty value ('' for text); to leave the property out, drop empty sources first with sources.filter((s) => s.value !== '').

Troubleshooting

• A field you can see in the feed has no source. Print the sources of one entry and look for the parser's name for it (see Parse the feed into sources). Fields with nested elements become JSON strings, and empty fields are dropped.
• autoMap put the plain description into body. When an entry has no content:encoded, autoMap maps rss-content (the description) onto the HTML property. Remove that edge if you'd rather leave body empty.
• Item.create rejects publishedAt with Expected string. The mapped date is a millisecond timestamp. Convert it with toItemValues as above.
• FieldMapper looks wrong next to your styles. The default theme injects its own dark CSS. Override its --sfm-* CSS variables, or use theme="structural" or theme="none" and style the fm-* classes and data-state attributes yourself. See Styling.

Related

• Serving a feed: the other direction, published items out as RSS
• Modeling relations: what author points to and why
• Publish and revoke items: publishing what you imported
• Reference: @seedprotocol/mapping, Mapping helpers, FieldMapper UI