Helpers

Everything on this page comes from the main entry, @seedprotocol/mapping, and runs in Node and in the browser without React. A Vite production build that uses rssXmlToSources needs one alias; see In a browser. The Overview explains the types these functions take and return.

import {
  rssXmlToSources,
  markdownToSources,
  autoMap,
  applyMapping,
  applyMappingAsync,
  coerceValue,
} from '@seedprotocol/mapping'

rssXmlToSources

Parses RSS or Atom XML and turns every entry into a list of sources.

function rssXmlToSources(xml: string): Promise<RssXmlSourcesResult>

type RssXmlSourcesResult = {
  channel: { title?: string; link?: string; description?: string }
  items: SourceNode[][]
}

Parameters

Returns

The channel's title, link and description, and one SourceNode[] per entry, in document order. It rejects when the XML can't be parsed.

Parsing uses rss-parser with the same extra fields as @seedprotocol/feed: content:encoded, featureImage, feature_image, seedUid, SeedUid, media:content and media:thumbnail. Each top-level field the parser produces becomes one source:

The rules behind the IDs:

• IDs are rss- plus the parser's field name, so the same field has the same ID in every entry and every feed. Print one entry's sources to find a field's ID.
• Values are trimmed, and empty fields produce no source. Arrays become JSON strings. An object becomes its text content (_ or value) when it has one, otherwise a JSON string.
• Media fields become one source per URL. Enclosures, media:content and media:thumbnail share one counter, so with an enclosure and one media:content you get rss-enclosure-0 and rss-mediaContent-1. Their value is the URL and their meta holds url, contentType, length and class (a UrlMediaClass from the MIME type or the file extension).
• A featureImage or feature_image value that starts with http gets meta.url and meta.class too.

Example

import { rssXmlToSources } from '@seedprotocol/mapping'

const xml = await (await fetch('https://blog.example.com/rss.xml')).text()
const { channel, items } = await rssXmlToSources(xml)

console.log(channel.title, items.length, 'entries')
for (const source of items[0] ?? []) console.log(source.id, '=', source.value.slice(0, 60))

In a browser

Most feeds don't send CORS headers, so fetch them through your own server.

The XML parser inside rss-parser reads setImmediate from Node's timers module. In a Vite app, turn on the plugin's includeNodePolyfills, which covers timers:

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import { seedVitePlugin } from '@seedprotocol/vite'

export default defineConfig({
  plugins: [react(), ...seedVitePlugin({ includeNodePolyfills: true })],
})

Without it, Vite replaces timers with an empty module and logs Module "timers" has been externalized for browser compatibility, so the parser's setImmediate is undefined. Either way, vite build logs the same warning for http and https: rss-parser uses them to fetch URLs, and rssXmlToSources only parses the string you pass.

The rest of the main entry, rssItemToSources included, builds without the polyfills.

rssItemToSources

Turns one already-parsed entry into sources, with the same rules as rssXmlToSources. Use it when you parse the feed yourself, for example with @seedprotocol/feed's parser.

function rssItemToSources(item: Record<string, unknown>): SourceNode[]

Keys named items or starting with $ are skipped.

markdownToSources

Turns a Markdown document with optional YAML frontmatter into sources.

function markdownToSources(markdown: string): SourceNode[]

Sources come in this order:

Limits of the frontmatter parser:

• Flat keys only. Nested YAML isn't supported; indented lines are ignored.
• Lists in brackets (tags: [a, b]) become one comma-separated string, "a, b", not a JSON array.
• true and false become "true" and "false". null and empty values become "".
• ### and deeper headings don't start a section; they stay in the section above them.

Bodies are copied as Markdown. Convert them to HTML yourself before you store them in an Html property.

An empty frontmatter field (summary: with no value) still produces a source with an empty value, and section-full is produced even for a document with no body. Mapping one sets the property to the empty value converted with coerceValue: '' for text types, 0 for Number, false for Boolean. To leave those properties out instead, drop empty sources before you map: sources.filter((s) => s.value !== '').

Example

import { markdownToSources } from '@seedprotocol/mapping'

const markdown = `---
title: "Hello"
tags: [intro, news]
---
# Intro
First paragraph.
`

const sources = markdownToSources(markdown)
// fm-title = "Hello", fm-tags = "intro, news", section-full = the body, section-0 = "First paragraph."

parseMarkdownWithFrontmatter and parseMarkdownSections

The two steps markdownToSources is built from, if you need the parts.

function parseMarkdownWithFrontmatter(markdownContent: string): {
  frontmatter: { id: string; key: string; value: string; valueType: 'string' | 'boolean' | 'array' }[]
  body: string
}

function parseMarkdownSections(md: string): {
  id: string
  heading: string
  level: number
  content: string
}[]

Without a frontmatter block, frontmatter is empty and body is the whole input.

autoMap

Guesses a first set of edges from field names.

function autoMap(sources: SourceNode[], targets: TargetProperty[], previous?: FieldMapping[]): FieldMapping[]

Parameters

Returns

A new FieldMapping[], at most one edge per property. It never fills in lookup tables itself: a lookup edge's entries and split come from previous or stay empty.

Name matching ignores case, spaces, :, _ and -, and also accepts one name containing the other (image matches featureImage). The passes run in this order, except that 3 to 5 run together, one source at a time in source order:

1. Whole document. A Markdown section-full source goes to the first property whose name is exactly (in any case) text, content, body, fulltext or markdown.
2. Full HTML. content:encoded, or a content source that contains <, goes to the first Html property or the first property named like html, content, body or text.
3. Links. A source named link or url goes to every free property named like importUrl, canonicalUrl, url, link or permalink, as a plain copy.
4. Media URLs. A URL source classified as an image goes to the first Image property (or one named like featureImage, image, cover or coverImage) with resolve: 'file'. Audio and video URLs go to a File property, or one named like audio, video, file, enclosure, attachment or media, with resolve: 'file'.
5. Aliases and names. Known RSS aliases are tried first: title → title, name, headline; description/summary → summary, description, excerpt; creator/author → author, authors, creator, byline; and so on. Then any property whose name matches the source's. A match on a relation target gets resolve: 'lookup', with no entries unless previous has some for that property. A raw URL is never copied onto an Html, Text, Image or File property.
6. Extract fallback. If an Html-like property is still free and there's a link, url or permalink source with a URL, it gets that source with resolve: 'extract'.

Properties whose names don't match anything are left unmapped. In an RSS import that usually includes the summary (contentSnippet doesn't match summary), the date (isoDate doesn't match publishedAt) and the tags, so add those edges yourself.

Example

import { autoMap, type FieldMapping, type SourceNode, type TargetProperty } from '@seedprotocol/mapping'

declare const sample: SourceNode[] // one entry from rssXmlToSources

const targets: TargetProperty[] = [
  { name: 'title', dataType: 'Text' },
  { name: 'body', dataType: 'Html' },
  { name: 'author', dataType: 'Relation', ref: 'Author' },
]

const mappings: FieldMapping[] = autoMap(sample, targets)
// [
//   { sourceId: 'rss-content:encoded', propertyName: 'body' },
//   { sourceId: 'rss-creator', propertyName: 'author', resolve: 'lookup' },
//   { sourceId: 'rss-title', propertyName: 'title' },
// ]

applyMapping

Applies edges to one entry synchronously and returns the property bag. It never throws for a missing source or property; it skips the edge.

function applyMapping(
  sources: SourceNode[],
  mappings: FieldMapping[],
  targets: TargetProperty[],
  options?: { onUnmatched?: (miss: UnmatchedLookup) => void },
): PropertyBag

type UnmatchedLookup = {
  sourceId: string
  propertyName: string
  value: string
  splitFrom?: string
}

Parameters

Returns

A PropertyBag. For each edge:

• No resolve: the source's value, even when it's empty, converted with coerceValue. Edges onto relation targets are skipped: a name is never copied into a relation.
• resolve: 'lookup': the matching entry's refs. A List gets all of them, a Relation the first. With split, the refs of the parts that have entries, in the value's order without repeats (see Relation lookups). No match leaves the property out and is reported to onUnmatched. An empty source value is no lookup: no property and no report.
• extract, file, derive, assemble: skipped. Use applyMappingAsync for extract and file.

Use applyMapping for previews, or when your mappings have no extract or file edges.

applyMappingAsync

Applies edges to one entry, runs extract and file edges through your callback, and reports what failed.

function applyMappingAsync(
  sources: SourceNode[],
  mappings: FieldMapping[],
  targets: TargetProperty[],
  options?: ApplyMappingAsyncOptions,
): Promise<ApplyMappingAsyncResult>

type ApplyMappingAsyncOptions = {
  resolve?: ResolveCallback
}

Parameters

Returns

{ properties, errors }. Copy and lookup edges set the same values as in applyMapping. The cases that set nothing:

Edges run one after another, in order. The result is partial: one failed edge doesn't stop the others.

Example

import { applyMappingAsync, type FieldMapping, type SourceNode, type TargetProperty } from '@seedprotocol/mapping'

declare const sources: SourceNode[]
declare const mappings: FieldMapping[]
declare const targets: TargetProperty[]

const { properties, errors } = await applyMappingAsync(sources, mappings, targets, {
  resolve: async ({ job, url }) => {
    if (job === 'extract' && url) {
      const html = await (await fetch(url)).text()
      return html // extract the article body here
    }
    throw new Error(`Unsupported transform: ${job}`)
  },
})

for (const e of errors) console.warn(`${e.propertyName}: ${e.message}`)

Resolve callbacks

applyMappingAsync calls options.resolve with a ResolveContext for every extract or file edge and stores what it returns.

The return value is converted for the target:

• A string goes through coerceValue, so an Html target keeps it as is.
• null or undefined becomes coerceValue(''): '' for text types, 0 for Number, false for Boolean.
• Anything else is kept as is for Image, File, Relation, List and Json targets, so you can return an item ID, an object or a Blob for a file. For other types it's turned into a string first.

The package never fetches anything itself. For a file edge, download the URL and store it the way your app stores files (for example with the SDK), then return what the Image or File property should hold.

coerceValue

Converts a source string to a target data type. Both apply functions use it for every copied value.

function coerceValue(rawValue: string, dataType: string): unknown

Relation lookups

A lookup edge stores a table from source text to item IDs on the edge (lookup.entries). The apply functions read it; these helpers let you read and write it yourself.

resolveLookup looks for the whole value first, in the edge's entries by lookup key. That entry wins, so a joint byline can have its own entry with several refs. Without one, an edge with split breaks the value on its separators and looks up each part the same way. 'and' matches in any case and only with whitespace around it, so "Anderson" stays whole; '&' and ',' match with or without spaces. The result's refs are the matched parts' refs in the value's order, without repeats; unmatched lists the parts with no entry; split is true when the value broke into more than one part. A value with no separator, or an edge without split, is looked up whole and, when it misses, lands in unmatched as is.

import {
  DEFAULT_LOOKUP_SEPARATORS,
  resolveLookup,
  withLookupEntries,
  withLookupSplit,
  type FieldMapping,
} from '@seedprotocol/mapping'

const edge: FieldMapping = withLookupSplit(
  withLookupEntries(
    { sourceId: 'rss-creator', propertyName: 'authors', resolve: 'lookup' },
    [
      { value: 'Mira Chen', refs: ['mira-local-id'] },
      { value: 'Sam Ortiz', refs: ['sam-local-id'] },
    ],
  ),
  DEFAULT_LOOKUP_SEPARATORS,
)

resolveLookup(edge, 'sam ortiz & Mira Chen')
// { refs: ['sam-local-id', 'mira-local-id'], unmatched: [], split: true }
resolveLookup(edge, 'Mira Chen and Nobody')
// { refs: ['mira-local-id'], unmatched: ['Nobody'], split: true }

To apply the entries your own app collects, replace them on the edge:

import { withLookupEntries, type FieldMapping, type LookupEntry } from '@seedprotocol/mapping'

declare const mappings: FieldMapping[]
declare const authorEntries: LookupEntry[] // e.g. [{ value: 'Jane Doe', refs: [author.seedLocalId] }]

const next = mappings.map((edge) =>
  edge.resolve === 'lookup' && edge.propertyName === 'author' ? withLookupEntries(edge, authorEntries) : edge,
)

Extract and file candidates

An edge's extract or file transform can also be expressed as an extra source, which is how the deprecated wires layout shows it. These helpers convert between the two forms.

Always save edges in the normalized form (origin ID plus resolve). The default FieldMapper layout works with origin sources only, so you don't need these unless you build your own picker.

URL classification

function classifyUrl(input: { url: string; contentType?: string | null }): UrlMediaClass
function looksLikeUrl(value: string): boolean

classifyUrl returns 'image', 'audio', 'video' or 'html' from the MIME type when one is given and recognized, otherwise from the path's file extension, and 'unknown' when neither says. It does no I/O. looksLikeUrl is true for strings that, trimmed and in any case, start with http://, https://, blob: or data:. The type ClassifyUrlInput is exported for the argument.

Edge helpers

Small pure functions for editing edges without losing what's stored on them. FieldMapper uses them; reach for them if you build your own editor.

Saving and loading a mapping

There is no serialize or parse function: a MappingDocument is plain JSON. Write it with JSON.stringify and read it back with JSON.parse. The package doesn't validate what you load, so check the shape yourself:

import type { MappingDocument } from '@seedprotocol/mapping'
import { readFile, writeFile } from 'node:fs/promises'

export async function saveMapping(path: string, doc: MappingDocument): Promise<void> {
  await writeFile(path, JSON.stringify(doc, null, 2))
}

export async function loadMapping(path: string): Promise<MappingDocument> {
  const doc = JSON.parse(await readFile(path, 'utf8')) as Partial<MappingDocument>
  if (doc.version !== 1 || !Array.isArray(doc.mappings)) {
    throw new Error(`${path} is not a mapping document`)
  }
  return doc as MappingDocument
}

Lookup tables are stored on their edges, so matched authors stay matched. Entries saved by 0.7.12 or earlier as { value, ref } still apply, read as refs: [ref].

A document saved by 0.7.13 or earlier can have a top-level lookups table (property, then source text, then one ID or a list). The package doesn't read it and nothing warns: the table's values count as lookup misses. Move it onto the edges before you apply the document:

import { lookupEntriesFromMapping, withLookupEntries, type MappingDocument } from '@seedprotocol/mapping'

type OlderDocument = MappingDocument & {
  lookups?: Record<string, Record<string, string | string[]>>
}

export function moveLookupsOntoEdges({ lookups = {}, ...doc }: OlderDocument): MappingDocument {
  const mappings = doc.mappings.map((edge) => {
    const table = lookups[edge.propertyName]
    if (edge.resolve !== 'lookup' || !table) return edge
    const moved = Object.entries(table).map(([value, refs]) => ({ value, refs: [refs].flat() }))
    return withLookupEntries(edge, [...lookupEntriesFromMapping(edge), ...moved])
  })
  return { ...doc, mappings }
}

Entries already on the edge stay first, and the first match wins. Table rows for a property with no lookup edge are dropped.

Related

• Overview: the types and how the pieces fit
• FieldMapper UI: letting users edit the edges
• Import an RSS feed into your models
• @seedprotocol/feed: the other direction, published items out as a feed