Notebook Access to the Lake#

Status: decided: option (d), “the policy file decides per table” (plan2 D11, 2026-10-30). Sub-decisions 13a–13f below are leanings until each is implemented. Origin: plan0 §9, open question 13. Source doc: dagster_aws_architecture.docx (notebook and governance sections). Related: plan2 D10 (governance), plan0 open questions 8 (lake layout), 11 (asset metadata) and 12 (policy source).

1. The problem#

The platform has three ways to read lake data. Only one of them can enforce fine-grained policy:

Path

Who uses it

What the store can enforce

Pipelines (Dagster run containers)

A service identity

Bucket and prefix, read or write

Query engines: Trino + OPA/Ranger on-prem, Athena + Lake Formation on AWS

Superset, JupySQL trino:// / athena://

Table, column and row, per user

Direct file readers: DuckDB, Polars, pandas in a notebook

Analysts and engineers

Bucket and prefix only

Column- and row-level policy exists only in query engines. A Parquet file holds every column of its rows, so any identity allowed to GetObject a table’s files can read all of its columns and rows. D10’s parity policy “analyst cannot see ssn” therefore holds only on the engine path.

AWS has the same problem. The source doc says “credentials come from the task role, no keys in the notebook”. That hides the issue: a task role with s3:GetObject on the lake bypasses Lake Formation exactly as a SeaweedFS key does. Lake Formation applies its filters only through engines it trusts (Athena, EMR, Glue, Redshift Spectrum), and DuckDB in a notebook is not one of them. So this is a design choice for both sides, not an on-prem gap.

Two details make direct reads riskier than they first appear:

  • Delta time travel. If a table once held ssn and a later version drops the column, the old Parquet files stay readable until VACUUM removes them. A prefix grant gives access to every past version of a table.

  • Shared keys defeat audit. If every notebook uses one key, the D10 audit layer can only report “the notebook key read X”, never which user did.

2. Options considered#

(a) One shared read-only key, limited to a “published” zone#

Notebooks get one read-only identity scoped to published data (e.g. marts/, reports/). Anything with column or row policy lives outside that zone and is reachable only through Trino or Athena.

  • Pros: simple; keeps DuckDB speed and the duckdb-kernel experience; no extra RAM-heavy services.

  • Cons: no per-user audit or revocation. Safety depends on publication discipline, which has to be enforced.

(b) Per-user keys mapped from the login identity#

Each user gets their own store identity, scoped by OIDC group. On AWS this is a Cognito identity pool or STS web-identity credentials. On-prem it means per-user SeaweedFS identities (to verify: whether SeaweedFS 4.48 can issue these from OIDC/STS, or we generate them ourselves).

  • Pros: per-user audit and revocation.

  • Cons: still only bucket and prefix level, so it doesn’t replace engine enforcement. It is (a) plus accountability, not an alternative to (a). It also needs identity plumbing on both sides.

(c) Engine only: notebooks get no lake credentials#

All notebook SQL goes through Trino (on-prem) or Athena (AWS) under the user’s own identity (Trino OAuth2/JWT passed through from JupyterHub; Athena with per-user credentials). DuckDB is used only for local files and query results.

  • Pros: one enforcement point; column and row policies hold everywhere; per-user audit through the Trino event listener or CloudTrail; on-prem behaves like Lake Formation.

  • Cons:

    • Trino has to be running whenever a notebook is in use. Its coordinator needs several GB, which every deployment pays for, including small ones.

    • It gives up the source doc’s main pitch: DuckDB querying the lake directly, fast, with no extra service.

    • Large pulls into dataframes are slower.

    • Athena bills per TB scanned.

(d) The policy file decides per table: chosen#

Use (a) where it is safe, use (c) where policy requires it, and let the open question 12 policy file decide which applies to each table:

  • A table whose rules are table-level only (“role X may or may not read it”) becomes a prefix (or bucket) grant. Those roles can read it directly with DuckDB.

  • A table with any column or row rule is engine-only. It sits outside every direct-read grant and is reachable only through Trino or Athena.

Roles:

Role

Direct read

Through the engine

data_engineering

Whole lake; they already have pipeline-level access, so this adds no exposure

Everything

analyst (and narrower groups)

Published tables only

Restricted tables, with column and row policy applied

“Published” is a rule we check, not a habit:

  • Asset check: fails if a published table has a column tagged sensitive (open question 11 metadata) or appears in any column or row rule.

  • D10 “no PII in marts” check (Presidio): covers the same ground from the data side.

  • VACUUM retention for published tables, so a dropped sensitive column eventually stops being readable through time travel.

Per-user identities (b) are added later for audit, once OIDC exists: Phase 1 Cognito, and JupyterHub login in Phase 5.

3. Sub-decisions#

#

Question

Options

Leaning

13a

Physical layout of the direct-read zone

Separate buckets (lake restricted, lake-published for analysts) vs prefixes in one bucket

Decided (D12): one bucket, per-table prefix grants. Prefix-scoped identities were verified on SeaweedFS 4.48 (docs/lake_convention.md §8), and prefix grants are standard on S3.

13b

How credentials reach the notebook

Baked into the image (no) / env vars injected by the spawner from a secret / OpenBao or Secrets Manager (Phase 7) / AWS STS

Spawner-injected env vars, read by DuckDB’s credential_chain secret; move to OpenBao in Phase 7

13c

Notebook writes

None / per-user scratch / sandbox bucket

No lake write access. Each user gets sandbox/<user>/. Promoting work into the lake goes through Dagster via git, so a notebook can never corrupt a Delta log.

13d

On-prem spawner and limits

DockerSpawner / SystemdSpawner

DockerSpawner on ods-core, with per-user mem_limit/cpu_limit, jupyterhub-idle-culler, and DuckDB memory_limit set explicitly below the container limit. The hub needs the Docker socket, which is root-equivalent (the same caveat as the Dagster daemon, D9).

13e

AWS notebook host

FargateSpawner (source doc; maintenance to verify) / EC2 + DockerSpawner / SageMaker / none

None for the MVP. The on-prem hub reaches AWS data through Athena (pyathena). The data-gravity rule agrees: DuckDB reading S3 from on-prem pays egress (~$0.09/GB [est.]) on every scan, while Athena returns only results. The swap point is the query engine, not where the notebook runs. An AWS-hosted hub stays an optional demo.

13f

Timing

Phase 5 vs 6

Phase 5 starts with single-user JupyterLab and the engineer role only; no restricted data exists until Phase 6 (faker ssn). Option (d) arrives with D10’s governance parity demos in Phase 6.

4. Parity checks#

Each check runs on both SeaweedFS and S3:

  1. An analyst identity’s direct read of a restricted table returns 403.

  2. The same identity’s direct read of a published table succeeds.

  3. Trino (on-prem) and Athena (AWS) hide ssn and filter out enterprise rows for the same user.

  4. An earlier version of a published table contains no sensitive columns, or has been vacuumed.

  5. A notebook write to the lake is refused; a write to sandbox/<user>/ succeeds.

5. To verify before implementing#

  • SeaweedFS 4.48: OIDC/STS-issued credentials. (Prefix-level access was verified 2026-10-30.)

  • FargateSpawner maintenance status (only matters if an AWS-hosted hub is ever built).

  • Lake Formation’s options for third-party engines (credential vending without filters), to confirm that DuckDB on AWS really is table-level only.

  • DuckDB’s handling of container memory limits (we plan to set memory_limit explicitly either way).