Development
mise install # rust, just, uv, python, mdbook, cmake, ninja
git submodule update --init --recursive # DuckDB's source, for `just ext`
just check # fmt, clippy, unit and property tests, and that the C headers match the FFI
just headers # regenerate the C headers from the Rust FFI
just ext # the extension, built here from DuckDB v1.5.6 -> target/ext/release/
just ext-nightly [WHEEL] # a DuckDB 2.0 dev wheel (default: newest), built in a container -> target/ext/nightly/
just contract [EXT] # storage contract on disposable PostgreSQL (PG_IMAGE=postgres:11..18); with EXT, the end-to-end test too
just compat # the contract on every supported PostgreSQL
just e2e [release|nightly] # build, then contract + DuckDB end-to-end
just sqllogic [PATTERN] [JOBS] # DuckDB's own SQL tests, each on a pgdb:// database (after `just ext`)
just bench [city|hits] # benchmarks (see Performance)
just bench-db # pgdb:// against a local DuckDB file (see Databases)
just site # this site from local builds -> .tmp/site/
Every build, test, benchmark and release command is a recipe in the
justfile; just --list
shows them. CONTRIBUTING.md
has the house style: code carries the design, comments only where they state
an invariant or DuckDB/PostgreSQL behaviour that is not derivable, plain
structs and free functions over stateful objects, and no globals beyond
pgvfs_stats’ counters and the environment knobs. Docker is needed for the contract, the nightly build and the
benchmark. just contract starts a primary, a streaming standby and a TLS-only
server with a throwaway CA, and tests against all three: exact reads, concurrent
writers, same-path publish conflicts, cooperative reaps, function refresh, listing
ranges, readers with SELECT only and on the standby, TLS verification, lost
connections, interrupted writes, readers during writes, and offsets past 4 GiB.
For pgdb:// (tests/db_contract.rs, run twice: with the default write
batches and with tiny ones): a randomized writer checked against an in-memory
model at every version, readers keeping their version, one writer and recovery
from a dead one, and reaping. With an extension, it also runs the DuckDB
end-to-end test through it: a second process’s writes, a SELECT-only role’s
refused writes, volumes routed to two databases by their secrets, and for
pgdb:// (extension/test/pgdb.py) keys and constraints, WAL replay, refused
second writers, refresh, the cache, pinned versions, a killed writer process,
the standby and a SELECT-only role.
DuckDB’s own SQL tests on pgdb://. just sqllogic runs DuckDB’s
test/sql through test/configs/pgvfs.json, which attaches a pgdb://
database per test on a disposable PostgreSQL and uses it, as DuckDB’s
encryption config does with an encrypted file (and with its skip list: tests
that use the memory catalog by name). Tests that load a database file of
their own are skipped by that mechanism. Skips need a reason, and the list
churns with DuckDB upgrades.
Layout
| Path | |
|---|---|
schema.sql, functions.sql | the storage layout, and the functions writers keep current |
src/store.rs | reads, the COPY writer, the advisory-locked install, reaping |
src/pg.rs | connection pool and TLS |
src/lib.rs | the C interface used by the extension, the open cache, pgvfs_stats() |
dbschema.sql, src/db.rs | pgdb://: its tables, the versioned writer, reads, refresh, reaping |
tantivy/ | tantivy splits (a workspace crate, linked into the same staticlib): build, bundle, search |
extension/ | the C++ DuckDB FileSystem adapters (pgdb_filesystem.cpp: pgdb://, its cache and refresher), the tantivy SQL functions, the container build, the end-to-end tests |
tests/ | the storage contracts, against real PostgreSQL; headers.rs checks pgvfs.h against the FFI |
test/ | SQL tests for make test, and configs/pgvfs.json for DuckDB’s own tests on pgdb:// |
cbindgen.toml, tantivy/cbindgen.toml | how the two C headers in extension/src/include/ are generated from the Rust FFI |
deny.toml | supply-chain policy for cargo deny: permissive licences, crates.io only, advisories |
bench/ | benchmark harness (lake.py), datasets (city.py, hits.py), the IVF check (ivf.py), pgdb.py |
tools/ | the Pages site generator (site.py) |
justfile | every build, test, benchmark and release command |
docs/ | this site (mdbook) |
The storage is Rust. The extension is a thin C++ layer because DuckDB’s stable C API can use filesystems but cannot register one.
The C ABI. extension/src/include/pgvfs.h and tantivy.h are generated
from the extern "C" functions in Rust (cbindgen) and checked in. A signature
that differs between Rust and C++ still compiles on both sides and crashes at
run time, so just check fails when a header is out of date: run just headers
and commit the result. Document an FFI function in Rust; its comment becomes the
header’s.
Properties. tantivy/src/search/properties.rs runs proptest over random
queries, options and exclusions on four splits: the answer is the same on one
thread and on many; a page is a slice of the ranking; excluded documents are the
ranking without them (and never crowd out top_k); collapse keeps the best hit
of each group; bytes and JSON that are not a bitmap or a query are refused, not
a panic. PROPTEST_CASES=3000 cargo test -p duckdb-tantivy properties runs them
longer. A failure prints a minimal case and saves its seed in
tantivy/proptest-regressions/: commit that file with the fix.
Supply chain. CI runs cargo deny check licenses bans sources on every
change and the RustSec advisories weekly. Dependencies must be permissively
licensed and come from crates.io: no git dependencies, so nothing unreleased is
linked into the extension. An accepted advisory goes in deny.toml with its
reason.
Building the extension
A C++ DuckDB extension must be statically linked against the exact DuckDB
build that loads it, as DuckDB’s own extensions are (Python loads DuckDB with
RTLD_LOCAL, so the host’s symbols are out of reach).
- Stable (
just ext): DuckDB’s extension template compiles the pinnedduckdbsubmodule (v1.5.6) and links the extension, with the Rust crate built by cargo through Corrosion (CMakeLists.txt,extension_config.cmake). The first build compiles DuckDB once, about 7 minutes; later builds are incremental.just extfails if the submodule is any other release, and writestarget/ext/release/with the wheel (DUCKDB_PY) and the version (DUCKDB_VERSION) thatbenchand the release recipes read.make testrunstest/sql/. TheDistributionworkflow runs this build for Linux (amd64, arm64, glibc and musl), macOS (Intel and Apple Silicon) and Windows on tags, weekly and by hand. - Nightly (
just ext-nightly): each 2.0 dev wheel is cut by a run of DuckDB’sMainCI on its exact commit, which keeps the static libraries for 90 days. The recipe downloads them and the source tree, checks the digest, and links the extension in a container (extension/Containerfile,extension/build.sh) without compiling DuckDB, in seconds. The download needs a GitHub token (gh auth loginorGH_TOKEN).
CI and releases
- Community releases: DuckDB builds and signs pgvfs for
INSTALL pgvfs FROM community. Updates are submitted to duckdb/community-extensions, pinning the release commit indescription.yml. Our weekly unsigned builds remain available for 2.0 dev versions. - Each push (
ci.yml):just check, then the stable build (DuckDB compiles once per commit; the cache falls back to an older commit’s), the contract on PostgreSQL 18 and the end-to-end test. - Tags, weekly and by hand (
distribution.yml): the multi-platform build with DuckDB’s extension pipeline (Linux amd64 and arm64, glibc and musl; macOS Intel and Apple Silicon; Windows). - Weekly, Monday (
weekly.yml, or by hand): the contract on PostgreSQL 11; end-to-end tests for the stable and newest 2.0 dev builds; each build published as the pre-releaseduckdb-<version>(every stable build and the last 4 dev builds are kept);just bench cityappended to thebenchpre-release’shistory.jsonl. - Pages (
pages.yml, weekly and on docs changes): this site, assembled from the releases bytools/site.py, is also theINSTALL ... FROMrepository.