Lifecycle rules on the blob bucket: expire files by age, abort stale uploads

Measured 2026-09-21, 05:50–06:10 UTC, against the live service (api.tablemere.com, s3.tablemere.com, al-lake1) with a fresh QA account and against the SeaweedFS 4.47 source (tag 4.47-tm.1); the brief was platform/STATE.md "Additions to the brief" item (A). Built the same morning: one bucket policy per blob bucket, three API routes, a CLI group, a guard. This document says what a tenant can do, how, when it takes effect, and the one rule that never bends: table buckets never carry lifecycle rules.

tablemere catalog lifecycle set   garden --prefix camera/ --expire-days 90
tablemere catalog lifecycle set   garden --abort-multipart-days 7          # a second rule; the first is kept
tablemere catalog lifecycle get   garden
tablemere catalog lifecycle clear garden

What a tenant can do

On the catalog's blob bucket (<handle>--<catalog>--blobs; b-<uuid> on catalogs from before 2026-09-20; the API's bucket field names it), the holder of the catalog credential at write level can set S3 lifecycle rules. Every rule below was PUT and read back identically, and the ones marked honoured were traced to the code path that executes them:

Not implemented at the store and staying so: x-amz-expiration on HEAD/GET (you cannot ask an object when it will go), AWS's rejection of Days: 0, midnight rounding, transitions.

Who: the catalog credential at write level sets, reads and clears rules (the API needs write to set and clear, read to read). Uploaders cannot (their policy is s3:PutObject under a prefix; measured 403 on GET, PUT and DELETE while their PutObject was 200) and the read-only identity cannot (same mechanism; a read holder must not delete by rule). Rules apply to every object under the prefix whoever wrote it: a 90-day rule on camera/ removes what the camera's uploader dropped there.

How: the API, the CLI, or boto3

The API takes a small shape and writes the XML for you, as the tenant (the tenant's own key pair; the control plane's admin key is not used), then reads it back:

PUT    /v1/warehouses/{id}/lifecycle   {"rules": [{"id"?, "prefix"?, "expire_days" | "expire_date",
                                                  "abort_incomplete_multipart_days"?, "enabled"?}, …]}   write level
GET    /v1/warehouses/{id}/lifecycle   [?include_xml=true]                                                read level
DELETE /v1/warehouses/{id}/lifecycle                                                                       write level

PUT replaces the whole configuration (S3 semantics). It refuses, before the store sees anything: expire_days: 0 or negative (400 invalid_value, the remedy says why), anything named transition or storage_class (400 not_supported), unknown fields (400 with accepted), both expiry forms on one rule, duplicate ids, an empty list (that is DELETE), more than 1,000 rules, more than 1 MiB of XML (413 too_large; the store's own EntityTooLarge maps to the same). Ids default to <prefix without slashes>-<n>. The prefix grammar is the uploader's: letters, digits, . _ - / = @ :, no leading slash, no * ? $. GET parses the stored document back, boto3-set rules included: tags, size filters and other elements appear on the rule and expressible: false says PUT here could not re-create it. Audit events: lifecycle.set, lifecycle.clear.

The CLI merges: set reads the current rules, replaces the one with the same id (default id: the prefix without slashes, camera; bucket for the whole bucket) and re-puts the rest, so two set calls for two prefixes leave two rules. It refuses to rewrite a configuration carrying rules it cannot express (tag or size filters set with boto3) unless --replace-all. get --xml prints the stored document; --json everywhere.

With boto3 and the catalog credential (catalog_credential from tablemere connect), for tag and size filters or an existing S3 workflow — exactly what was measured:

import boto3
from botocore.config import Config
s3 = boto3.client("s3", endpoint_url="https://s3.tablemere.com", region_name="us-east-1",
                  aws_access_key_id=CLIENT_ID, aws_secret_access_key=CLIENT_SECRET,  # catalog_credential from `tablemere connect`
                  config=Config(s3={"addressing_style": "path"}))
s3.put_bucket_lifecycle_configuration(
    Bucket="<handle>--<catalog>--blobs",
    LifecycleConfiguration={"Rules": [
        {"ID": "camera-90d", "Status": "Enabled", "Filter": {"Prefix": "camera/"}, "Expiration": {"Days": 90}},
        {"ID": "abort-stale-uploads", "Status": "Enabled", "Filter": {"Prefix": ""},
         "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7}},
    ]})
print(s3.get_bucket_lifecycle_configuration(Bucket="<handle>--<catalog>--blobs")["Rules"])
# s3.delete_bucket_lifecycle(Bucket=...) removes every rule

When it takes effect

The store runs the s3_lifecycle worker job once a day and about five minutes after every restart of the store (al-lake1: enabled: true, detection_interval_minutes: 1440; 11 successful runs of 8–32 ms since 2026-09-19, one within ~5 min of every al-weed start). One pass compiles every bucket's rules, replays the filer's write log for Days and multipart rules and walks the buckets with Date rules; an object is deleted at the first pass after it is due, so up to 24 h after Days elapses. A day is 24 h from the object's last write. Deletion is a plain delete on an unversioned bucket: permanent, no versions, no undelete; an object overwritten since the rule saw it is left alone. A failed delete stalls the pass and is retried at the next.

The storage meter (GET /v1/usage, the catalog row's storage) shows the reduction at its next hourly sample, or immediately with POST /v1/warehouses/{id}/measure / tablemere catalog measure <catalog> (measured: blob_objects 3 → 1 on the next measure after the pass).

Never on the table bucket

Iceberg data and metadata are retained by the catalog's own maintenance — snapshot expiry (168 h / 20 snapshots) and orphan cleanup (files no snapshot references for 24 h) — never by key age: a lifecycle rule on a table bucket would delete Parquet and metadata.json files the catalog still references and corrupt every table under it. The store's handler does not distinguish table buckets (the admin identity could set a rule there, and the live worker would execute it), so the guard is ours, in three layers:

1. No tenant path reaches it. The tenant's user policy carries the five data-path actions; PutUserPolicy refuses s3:PutLifecycleConfiguration / s3:GetLifecycleConfiguration as "not a valid action" (measured 400); the table-bucket policy names s3tables:* and the data path. ?lifecycle on the table bucket is 403 for the tenant (measured, GET and PUT).
2. The control plane refuses to grant it. s3_put_bucket_policy raises (lifecycle_guard) on any document naming a lifecycle action — s3:*Lifecycle*, s3:*, s3:Put* — for a bucket that is not a blob bucket by shape or that the database knows as a table bucket (a catalog named blobs has the table bucket <owner>--blobs). Unit-tested against every policy document the fake store receives.
3. The audit. GET /internal/lifecycle-audit (bootstrap secret) reads ?lifecycle and ?policy on every table bucket as the admin and lists anything but 404 / no lifecycle grant; the nightly sweep is to call it and alert on ok: false.

The mechanism, for the record

A tenant could not set a rule before this: ?lifecycle resolves to s3:GetLifecycleConfiguration / s3:PutLifecycleConfiguration (DELETE uses the Put action) and no user policy can carry them. What opens them is a plain bucket policy on the blob bucket naming the identity as an IAM-user ARN — plain-bucket principals are ARNs (arn:aws:iam::<account>:user/<name>), so arn:aws:iam::*:user/t-<uuid> matches and a bare name (the S3 Tables normalisation CLAUDE.md records) stays 403; a bucket-policy Allow bypasses the IAM check. PutBucketPolicy is an S3 call: no IAM write, no propagation deadline. The control plane writes this one statement at catalog creation (create_tenant), when a legacy catalog gets its blob bucket (ensure_blob_bucket), before every lifecycle request (idempotent, one GET), and once for every existing blob bucket at the first startup after the deploy (the blob_lifecycle_grant_v1 migration in the background worker: 301 rows, milliseconds each, retried at the next start if anything failed):

{"Version": "2012-10-17", "Statement": [{
  "Sid": "TenantLifecycle", "Effect": "Allow",
  "Principal": {"AWS": ["arn:aws:iam::*:user/t-<warehouse_uuid>"]},
  "Action": ["s3:GetLifecycleConfiguration", "s3:PutLifecycleConfiguration"],
  "Resource": ["arn:aws:s3:::<handle>--<catalog>--blobs"]}]}

The public-read policy of a public catalog lives on the table bucket, so the two documents never share a bucket; the grant is merged by Sid all the same. Deleting the catalog deletes the blob bucket with its policy and its rules (measured: both QA buckets gone, no bucket on the host carried a rule afterwards).

Sources read: weed/s3api/s3api_bucket_handlers.go (validation: 1 MiB, MalformedXML, 501 for an enabled Transition), s3api_bucket_lifecycle_config.go (storage on the bucket entry), weed/worker/tasks/s3_lifecycle/ and weed/s3api/s3lifecycle/ (the daily pass, evaluate.go ExpirationDays <= 0 -> none, due_at.go 24 h days), s3api_internal_lifecycle.go (the delete), weed/iam/helpers.go (baseS3ActionMap, no Lifecycle), auth_credentials.go (buildPrincipalARN, the bucket-policy bypass), s3_action_resolver.go. A per-write TTL fast path exists in the store (Seaweed-X-Amz-Lifecycle-Ttl-Fast-Path, weed shell only, irreversible once stamped); it is off on every Tablemere bucket and not for tenants.

Live smoke, after the deploy

# a QA catalog and its write-level key (tablemere login / an org admin key in TABLEMERE_API_KEY)
tablemere catalog create --name qa_lifecycle
tablemere catalog lifecycle get qa_lifecycle                       # None configured: 0 rules
tablemere catalog lifecycle set qa_lifecycle --prefix camera/ --expire-days 90 --abort-multipart-days 7
tablemere catalog lifecycle get qa_lifecycle --xml                 # the stored document
# the same through boto3 with the catalog credential from `tablemere connect --json`: put/get as above; 200/200
# the table bucket stays closed: as the tenant, GET ?lifecycle on <handle>--qa-lifecycle must be 403
# the guard's audit: on the host, curl -sS -H "X-Bootstrap-Secret: $AL_BOOTSTRAP_SECRET" 127.0.0.1:8080/internal/lifecycle-audit  -> ok: true
tablemere catalog lifecycle clear qa_lifecycle                     # Cleared 1 lifecycle rule
tablemere catalog delete qa_lifecycle --confirm qa_lifecycle