Engines

Tablemere is an Apache Iceberg REST catalog plus S3-compatible storage. Your engine talks to both directly; the recipes below are what tablemere connect --catalog lake --engine … prints, with the credential and the bucket filled in. Catalog: https://catalog.tablemere.eu. Storage: https://s3.tablemere.eu.

DuckDB (1.5.5 or newer) — read and write

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs;
CREATE SECRET tablemere (TYPE ICEBERG, CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>',
                         OAUTH2_SERVER_URI 'https://catalog.tablemere.eu/v1/oauth/tokens');
CREATE SECRET tablemere_s3 (TYPE S3, KEY_ID '<client_id>', SECRET '<client_secret>',
                            ENDPOINT 's3.tablemere.eu', URL_STYLE 'path', USE_SSL true, REGION 'us-east-1');
ATTACH 'w-<warehouse_uuid>' AS lake (TYPE ICEBERG, ENDPOINT 'https://catalog.tablemere.eu', SECRET tablemere);
SELECT * FROM lake.<namespace>.<table> LIMIT 10;

The tablemere_s3 secret is not needed to read or write tables through the catalog (vended credentials cover that); it lets read_blob('s3://w-…/**') and friends inspect the bucket.

Importing Parquet

-- a new table, from a local file or a URL
CREATE TABLE lake.demo.cities AS SELECT * FROM read_parquet('cities.parquet');
-- into a table that exists (created by the API or an engine)
INSERT INTO lake.demo.cities SELECT * FROM read_parquet('more-cities.parquet');
COPY lake.demo.cities FROM 'more-cities.parquet' (FORMAT PARQUET);
-- namespaces are schemas
CREATE SCHEMA lake.staging;

Measured against the service: a million rows (18 MB of Parquet) in about 2.3 s from a laptop, three data files, one commit; format-version 2.

PyIceberg (0.12) — read and write

from pyiceberg.catalog.rest import RestCatalog
catalog = RestCatalog(name='tablemere', uri='https://catalog.tablemere.eu',
                      warehouse='s3://w-<warehouse_uuid>/', credential='<client_id>:<client_secret>',
                      **{'s3.endpoint': 'https://s3.tablemere.eu', 's3.path-style-access': 'true'})
table = catalog.load_table(('<namespace>', '<table>'))

# writing: create from an Arrow schema, then append
import pyarrow.parquet as pq
arrow = pq.read_table('cities.parquet')
tbl = catalog.create_table(('demo', 'cities'), schema=arrow.schema)
tbl.append(arrow)

PyIceberg writes one data file per append and a complete snapshot summary; DuckDB splits large writes into several files. Both are read identically by either engine.

Spark (3.5 + Iceberg 1.11) — read and write, including format-version 3

spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.lake.type=rest
spark.sql.catalog.lake.uri=https://catalog.tablemere.eu
spark.sql.catalog.lake.warehouse=s3://w-<warehouse_uuid>/
spark.sql.catalog.lake.credential=<client_id>:<client_secret>
spark.sql.catalog.lake.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.lake.s3.endpoint=https://s3.tablemere.eu
spark.sql.catalog.lake.s3.path-style-access=true
spark.sql.catalog.lake.header.X-Iceberg-Access-Delegation=vended-credentials

Pass these as --conf flags or in spark-defaults.conf; the catalog is then lake in SQL (SELECT * FROM lake.demo.cities).

Snowflake

The catalog integration connects (catalog only). The data path is not available yet; it needs work on both sides and is on the roadmap.

Gotchas, all of them measured

  • DuckDB: ATTACH the bare bucket name. ATTACH 'w-<uuid>' is read-write. ATTACH 's3://w-<uuid>/' attaches READ-ONLY and every INSERT fails with "attached in read-only mode". PyIceberg and Spark want the s3://w-<uuid>/ form for warehouse, so copying one recipe's identifier into the other engine produces a false "DuckDB cannot write Iceberg".
  • DuckDB 1.5.5 or newer. 1.5.0 writes manifest lists the catalog's maintenance cannot read, so tables it wrote were neither compacted nor expired by the scheduler. 1.5.5 writes the spec-compliant schema.
  • Format version 2, unless you have Spark. Format-version 3 tables are read-only for DuckDB ("Not implemented Error: Insert into Iceberg V3 tables") and PyIceberg 0.12. Spark 3.5 + Iceberg 1.11 is the only writer verified. tablemere table create defaults to 2 and warns when it hands you a 3.
  • The catalog only accepts its own tokens. Engines mint them from the catalog credential (catalog_credential in the recipe) at https://catalog.tablemere.eu/v1/oauth/tokens (client_credentials; client_id = access key, client_secret = secret key) and refresh them themselves; they expire after 900 s. Tablemere API keys (al_live_…) and tokens from POST /v1/tokens are 401 at the catalog by design.
  • PyIceberg asks for vended credentials on every request. That is the intended path: the catalog returns per-table storage credentials which override any static s3.* key you pass. If a cross-bucket copy fails with ACCESS_DENIED, pass "header.X-Iceberg-Access-Delegation": "none" to use your own keys instead.
  • Table buckets accept only Iceberg files. Data files must be .parquet/.orc/.avro/.lance and metadata must look like Iceberg metadata, under <namespace>/<table>/(data|metadata)/; anything else is refused on write. Put photos, documents and other blobs in the warehouse's blob bucket b-<uuid> (same credential, S3 path-style, region us-east-1; tablemere connect prints a boto3 example) and index them in a table.
  • CTAS makes every column optional. CREATE TABLE … AS SELECT from DuckDB cannot express identifier fields, required columns or partitioning. When you need them, tablemere table create --column name:type[:required][:identifier] first, then INSERT.
  • Maintenance is on and not disableable. Compaction (128 MB target files), snapshot expiry (20 snapshots, 7 days) and orphan cleanup run on every catalog. A client that hard-codes vN.metadata.json file names will 404 after a maintenance commit; list the prefix or go through the catalog.
  • Rotation. tablemere catalog rotate replaces the catalog credential: the old key and every catalog token from it are refused at once; storage sessions already vended run out within 900 s. Fetch a fresh recipe afterwards.

Direct S3 access from a script

tablemere credentials --catalog lake --namespace demo --table cities (POST /v1/credentials) returns storage credentials scoped to that table's object prefix, with their expiry, for a boto3 or aws-cli session that needs the files themselves rather than the table.