Arcform

Arcform runs analytical data pipelines from a single arcform.yaml — steps that produce assets, wired by the data dependencies between them, executed against DuckDB by one static binary (arc).

Why it matters

A real analytical workflow is a chain: fetch a couple of files, load them into DuckDB, model them with SQL, export the result. People wire this together with a Makefile or a pile of shell scripts — and nothing in that pile knows what depends on what, so a one-line change means re-running everything.

Arcform treats the data outputs as the graph. It reads each SQL step with sqlparser-rs to discover which tables the step reads and writes, so the dependency graph is built from lineage rather than hand-maintained ordering. Preconditions skip steps whose inputs are still fresh; a step's SQL is content-hashed so unchanged work isn't repeated. The manifest is the whole pipeline — the same one runs on your laptop and on a schedule.

Meridian's own dataset publishing is converging on arcform: the goal is that every published dataset carries its arcform.yaml as its provenance record — the pipeline you can read and rerun. (Today's publish scripts are moving over incrementally.)

What running a Protocol costs you

Arcform runs your steps on your machine, as you. There is no sandbox and no container, and that is deliberate. Running a Protocol you did not write is equivalent to running a shell script you did not read.

A step's command: runs as sh -c <command>. So does a command: on a precondition or a hook, and a precondition's runs while Arcform works out what is stale, even when the step it guards is then skipped. A sql: step is passed to the DuckDB CLI, which reads and writes whatever the SQL directs. An op: step runs a catalog operator inside arc, or a script arc embeds: no shell, and no confinement either. Each of these inherits your environment, and arc registry run fetches a Protocol and runs it by the same path.

So read a Protocol's arcform.yaml before you run one someone sent you. Its command:, sql: and op: lines are what will run as you.

Installation

curl -fsSL https://install.meridian.online/arc | bash

This installs a release build for macOS (Apple silicon and Intel) or Linux (x86_64 and aarch64, glibc 2.35 or later) and links arc into ~/.local/bin. When no duckdb is on your PATH, or the one there cannot read SQL into tokens, it then installs DuckDB's v2.0.0 release build into ~/.duckdb/cli/2.0.0 and says what to put on PATH; set ARC_SKIP_DUCKDB=1 to leave DuckDB alone. To build from source instead:

CXXFLAGS='-DDUCKDB_VERSION="v2.0.0"' cargo install arcform --locked --features mcp,bundled-duckdb

To pin a release, pass its version to either:

curl -fsSL https://install.meridian.online/arc | bash -s -- vX.Y.Z
CXXFLAGS='-DDUCKDB_VERSION="v2.0.0"' cargo install arcform --locked --version X.Y.Z --features mcp,bundled-duckdb

The CXXFLAGS setting makes the bundled DuckDB report its release version; without it a source build reports a development version.

Both builds carry DuckDB v2.0.0 inside arc. Arcform still runs a Protocol's SQL steps with the DuckDB CLI, so duckdb must be on your PATH at run time: v2.0.0 recommended, >=2.0, <3 accepted. It must be DuckDB's release build, or another build that includes DuckDB's autocomplete extension, which Homebrew's 2.0.0 lacks at the time of writing; curl https://install.duckdb.org | DUCKDB_VERSION=2.0.0 sh installs the release build. A 1.x duckdb is refused, and Arcform v0.2.1 is the release for DuckDB 1.5. The FineType tools need finetype 0.6.54 or later, and the Python operators need uv.

Commands

arc init <name>            # scaffold a new project (arcform.yaml, models/, sources/)
arc run                    # run the pipeline in the current directory
arc run --force            # re-run every step, ignoring staleness
arc run --param KEY=VALUE  # override a runtime parameter (repeatable)
arc table describe <table> # row count, columns and sample rows of a table, as JSON
arc sql query '<sql>'      # read-only SELECT against the Protocol's database, as JSON
arc mcp                    # serve arc to an AI agent over the Model Context Protocol

arc registry list/show/fetch/run is scaffolding for a future curated pipeline catalogue; the transport isn't wired to a live index yet.

Using arc from an agent

arc mcp serves arc to an AI agent over the Model Context Protocol. Through it an agent can create a Protocol, record SQL steps and operations into it, run it, describe its tables and query them read-only, and every change it makes to arcform.yaml is a version in arc's history. It also serves Finetype's infer, profile, taxonomy, validate and generate, each run through the finetype binary. Both install routes above include it. To add it to Claude Code:

claude mcp add arc -- arc mcp

An agent started from a desktop app may not see your shell's PATH: give the absolute path of arc (which arc prints it) and a PATH that holds duckdb, finetype and uv. The README has the configuration for other clients.

Arcform's next release adds six agent skills that teach an agent the analyst's workflow through these tools, with an arc skills install command and a plugin for Claude Code, Codex, Gemini CLI and Cursor. Arcform v0.2.2, the latest release on 2026-10-27, does not have them yet.

A worked example: publishing a dataset

The repo's reference pipeline, examples/brewtrend, ranks trending Homebrew packages — fetch, load, model, publish a Parquet. Every step is either a command: (raw shell) or a sql: (a .sql file run against DuckDB). Abridged:

# examples/brewtrend/arcform.yaml
name: brewtrend
engine: duckdb
engine_version: ">=1.2"
db: trends.db

params:
  trend_threshold:
    default: "10"

steps:
  # Fetch: pull Homebrew analytics (cached 24h via a precondition)
  - name: fetch_cask_30d
    command: "mkdir -p data/cask && curl -fsSL https://formulae.brew.sh/api/analytics/cask-install/homebrew-cask/30d.json -o data/cask/30d.json"
    produces: [cask_30d]
    preconditions:
      - modified_after: { path: data/cask/30d.json, period: 24h }
  # … five more fetch_* steps (cask/formula catalogues + 90d analytics) …

  # Load: raw JSON → DuckDB tables
  - name: load
    sql: models/load_data.sql
    depends_on: [cask_json, formula_json, cask_30d, cask_90d, formula_30d, formula_90d]

  # Transform: compute the trend metric (reads trend_threshold)
  - name: trending
    sql: models/trending.sql

  # Publish: final ranking → data/ranking.parquet
  - name: rank
    sql: models/full_ranking.sql

The rank step is a plain sql: file that builds the ranking table and exports it:

-- models/full_ranking.sql (tail)
CREATE OR REPLACE TABLE ranking AS SELECT … ;

-- Export the ranking as the pipeline's portable output artifact.
COPY ranking TO 'data/ranking.parquet';

Arcform reads that SQL and learns, on its own, that rank writes the ranking table and depends on what the upstream steps produced — no manual DAG wiring. Run it:

cd examples/brewtrend
arc run

This fetches the source data into data/, loads it into trends.db, computes the ranking, and writes data/ranking.parquet — the pipeline's portable output. Re-running within 24h skips the cached fetches and just recomputes; arc run --force re-runs everything; arc run --param trend_threshold=50 raises the bar for what counts as trending.

The example also ships a Frictionless datapackage.json describing that output — field names, types, and provenance. Per Meridian's convention, a Data Package describes the data; it is not the runnable artifact. arc run executes arcform.yaml — there is no arc run datapackage.json.

Links

• GitHub Repository
• examples/brewtrend — the full runnable pipeline
• Changelog

See also

Dovetail decides how data loads and relates · Finetype types the columns arcform pipelines produce