Working Alongside Databricks and Snowflake#

Recapture (some) of your Databricks/Snowflake budget by bringing (some) of your workflows back on-prem.

Status: proposed (2026-11-02), the plan asked for by plan0 open question 24. Nothing here is built yet; the decisions in §11 need approval. Related:

  • plan2 D10/D11 (governance, access policy), D12 (lake convention), D13 (Delta, Iceberg exit path), D14 (asset metadata)

  • plan0 open questions 17 (SQLAlchemy), 18 (dev/prod environments), 21 (Delta Sharing)

  • Feature comparison, Cost comparison

1. The idea#

Most teams that run Databricks or Snowflake aren’t going to leave, and this plan doesn’t ask them to. It assumes they keep their platform and move some workloads: the ones where managed compute costs the most for the least benefit. Data flows both ways, so moving a workload doesn’t cut it off from the rest of the estate.

Three rules:

  1. Workloads move, not the whole platform. Every move is reversible: the same tables stay readable from the commercial platform.

  2. Share tables, don’t copy them, where the formats allow it. Copies are the fallback, and they are explicit, scheduled assets with lineage like any other.

  3. Measure the money. A move counts only if the commercial bill for that workload goes down by more than the new cost on our side (§9).

2. What moves well, and what doesn’t#

Workload

Move?

Why

Dev and test runs (dbt builds, pipeline debugging, CI)

Yes, first

High volume, low stakes, no SLA. On a commercial platform each run uses warehouse or cluster time; here it runs on hardware already paid for.

Steady daily batch on small-to-medium data

Yes

Predictable, fits one node (DuckDB, Spark local mode). Ongoing managed compute for a job that would fit on one machine is the classic overspend.

Interactive notebooks and ad-hoc SQL

Yes

Warm clusters and warehouses that sit idle between queries are expensive. JupyterLab and DuckDB/Trino on-prem are always on for free.

Heavy, occasional jobs (backfills, large joins)

Maybe

Burst to AWS (Glue/Fargate, Phase 1/4) or leave them where they are. Don’t buy hardware for the peak.

BI serving to many concurrent users

Usually not

Snowflake’s concurrency scaling and caching are hard to beat; keep them. Superset on-prem suits internal dashboards with few users.

Workloads tied to platform features (Snowpark Container Services, Databricks Model Serving, Delta Live Tables, Cortex/Mosaic AI)

No, at first

No like-for-like analog yet; see the AI placeholder (plan0 OQ14) and the feature comparison.

Data the commercial side’s governance must control (regulated, row/column masked)

Only with care

See §8; such tables must not leave the platform without the same policy.

3. The boundary: S3 is the exchange zone#

Databricks and Snowflake run in the cloud; the on-prem lake (SeaweedFS) isn’t reachable from them and shouldn’t be. The hybrid design already has a place where both sides meet: the AWS side of the lake (S3). Data crosses the boundary there and nowhere else.

 Databricks / Snowflake  ◄──reads/writes──►  S3 (AWS side, D12 layout)  ◄──replica / dlt──►  on-prem lake

  • Outbound (us → them): tables we publish land in S3, either built there on the AWS target, or copied there by a selective version of the existing replica asset (Phase 2 copies the whole lake; publishing needs a per-table list, driven by D14 metadata, §8).

  • Inbound (them → us): we read their tables from S3 where the format and the catalog let us (§4–5), otherwise we load them with dlt into raw/ (D6) like any other source.

  • Cost: reading S3 from on-prem is billed as AWS egress, so inbound tables should be read incrementally (Delta’s log makes that cheap) and cached on-prem, never re-read in full per run.

This depends on Phase 1 (AWS), which is on hold. §10 shows what can be built and tested before then.

4. Databricks#

Delta is native to Databricks (D13), so this is the easy side.

Direction

Mechanism

Notes

Their tables → us

Read the Delta files in S3 directly with delta-rs / DuckDB delta_scan, using short-lived credentials from Unity Catalog credential vending

Databricks documents external read and write for external Delta tables and managed Iceberg tables, and read-only access for managed Delta tables (managed Delta writes from outside are in beta). Tables with row filters, column masks or Delta Sharing are excluded from vending (Databricks docs). The principal needs EXTERNAL USE SCHEMA.

DuckDB’s unity_catalog extension

Attaches a Unity Catalog (Databricks or OSS) as a DuckDB database (DuckDB docs). Marked experimental, so useful for notebooks, not yet for production assets.

Our tables → them

Register our S3 Delta tables as external tables in Unity Catalog

No copy: Databricks reads the same files. We stay the only writer, and they read.

Both ways

Delta Sharing (plan0 OQ21)

Databricks shares to open recipients with bearer tokens or OIDC (Databricks docs). On our side, the open-source Delta Sharing server and Python client.

Compute

dagster-databricks (Pipes)

Our Phase 4 pattern (one script, Pipes, run by different clients) extends to Databricks: the same customer_ltv.py could run on a Databricks job cluster. Useful while a workload is being moved.

Single writer rule: a table has exactly one writing platform at a time. Delta’s conditional-write commits (D3) are safe between writers on one store, but two platforms with different catalogs and governance writing one table is an operational trap. Moving a workload means moving the writer.

5. Snowflake#

Snowflake’s own tables are proprietary; interop goes through Iceberg and through Snowflake’s ability to read Delta files.

Direction

Mechanism

Notes

Our tables → them

Delta Direct: a catalog integration with CATALOG_SOURCE = OBJECT_STORE, TABLE_FORMAT = DELTA, then an Iceberg table over our Delta files in S3

Registers our Parquet files without copying them, so we keep writing; refreshes on an interval or manually. Needs only read access to storage (Snowflake docs). Snowflake documents that it can’t create these from Delta definitions in other catalogs (Unity, Glue), so it reads files, not our catalog.

Their tables → us

Snowflake-managed Iceberg tables read through the Horizon Catalog Iceberg REST API

Snowflake lists DuckDB, PyIceberg, Spark and Trino as supported engines; external writes are in public preview; API calls are billed in credits (billing scheduled for 2026) (Snowflake docs).

Native (non-Iceberg) Snowflake tables: dlt via SQLAlchemy (sql_database, plan0 OQ17), or COPY INTO an S3 stage as Parquet

These are copies and use Snowflake credits to read. Prefer converting the hand-off tables to Iceberg on the Snowflake side.

Compute

dagster-snowflake resource

For assets that stay in Snowflake but belong to our graph (§6).

Where Iceberg comes in. D13 keeps Iceberg as the exit path, not a second write format. Snowflake interop needs an Iceberg reader on our side (DuckDB’s iceberg extension and PyIceberg are both readers that need no catalog service of our own; they talk to Snowflake’s). That’s compatible with D13: we read Iceberg, we write Delta. No change to D13 is proposed.

Under evaluation: Delta UniForm (plan0 open question 21). UniForm writes Iceberg metadata next to the Delta log, so a Snowflake account could read an install’s tables as ordinary Iceberg tables instead of through Delta Direct. Iceberg engines read them; they can’t write them. The catch is the writer: UniForm comes from Delta on Spark (3.1+) or Databricks, and delta-rs, which writes most of the platform’s tables, doesn’t produce it. A metadata converter (Apache XTable), offered as an optional per-table step, is the option for delta-rs tables. Until a spike settles it, Delta Direct (above) stays the plan for Snowflake.

6. One graph across the boundary#

Dagster is the piece that makes “some workloads here, some there” manageable: assets on either side are nodes in one asset graph, with one lineage view.

  • Asset keys don’t change when a workload moves. marts/customer_ltv is the same key whether it’s built by our Spark, by a Databricks job (Pipes) or by Snowflake. Only the materializing code changes. Downstream assets don’t notice.

  • External assets for tables we read but don’t build: declared with their owner and platform (ods/platform: databricks|snowflake in D14 metadata), observed by a sensor that watches the Delta log or the Iceberg snapshot, so freshness and lineage work across the boundary.

  • dbt: one dbt project per platform is the realistic setup. Our dbt-duckdb project (Phase 3) and a team’s dbt-snowflake or dbt-databricks project meet at sources: their marts are our sources and vice versa. A model moving from their project to ours goes through the dialect checklist in §7.

7. Migration guide (outline)#

The guide itself is written per workload when the first real move happens. Its checklist:

  1. Pick and baseline: choose the workload from §2 and record its commercial cost per run and per month (§9).

  2. Dialect: Snowflake SQL and Databricks SQL → DuckDB. Common friction points: QUALIFY (DuckDB has it), FLATTEN/LATERAL VIEW EXPLODE (→ unnest), semi-structured VARIANT access (→ DuckDB JSON or STRUCT), date functions, and identifier case. sqlglot can transpile between these dialects as a first pass; every translated model needs its tests.

  3. Notebooks: Databricks notebooks export as .ipynb or source; dbutils calls, widgets and %sql magics need replacing (JupySQL covers %sql). Snowpark DataFrame code → pandas/Polars/PySpark against the lake (D5: any of them).

  4. Data: decide whether the workload’s inputs are shared in place (§4–5) or loaded. Its output keeps its asset key (§6) and moves its writer (single writer rule).

  5. Parity check: run old and new side by side; compare row counts and checksums with the same kind of test as our on-prem/AWS parity suite.

  6. Cut over: switch the writer; leave the commercial job disabled, not deleted, for one cycle.

  7. Measure (§9) and record the result.

8. Identity and governance across both#

  • What may leave: publishing a table outward (§3) is gated by D14 metadata. sensitivity: internal and public can be published; confidential and above need an explicit, reviewed entry in the D11 policy file naming the target platform. The selective publisher refuses anything else.

  • Policy follows the table, by hand at first: Unity Catalog grants and Snowflake roles are separate systems. The first version keeps our policy file as the source of truth for tables we publish, and documents the equivalent grants on their side. Automated grant sync is a later option, not a promise.

  • Masked tables don’t cross: Databricks excludes tables with row filters or column masks from credential vending, and that’s the right default for us too. A masked table crosses only as a separately built, already-masked mart.

  • Identity: service principals on both sides, short-lived credentials (UC vending, Snowflake OAuth or key-pair), no long-lived keys in notebooks. SSO across platforms waits on the platform’s own OIDC work (Phase 7).

9. Measuring the money recaptured#

The pitch is a number, so the plan has to produce one.

  • Their side: Databricks system.billing.usage and Snowflake ACCOUNT_USAGE metering views give compute spend per job or warehouse. Tag moved workloads (job tags, query tags) so their before/after cost is separable.

  • Our side: the cost per run on AWS when bursting (Phase 1 metrics), plus an agreed share of on-prem hardware and power. On-prem compute that already exists counts at its marginal cost, which should be stated rather than called free.

  • Report: a Dagster asset that collects both into a marts/workload_costs table per workload per month. The cost comparison page switches from symbols to measured numbers as this fills in.

10. Phasing#

Step

What

Needs AWS (Phase 1)?

I1

Unity Catalog OSS and the open-source Delta Sharing server in a test fixture, like SeaweedFS and moto today: register our Delta tables, read them back through UC and through a share, in the parity suite. Phase 6 already plans Unity Catalog OSS as the on-prem physical catalog, so this is a head start on it, not extra work

No

I2

Iceberg reader assets (DuckDB iceberg / PyIceberg) against a local REST catalog standing in for Horizon; external-asset sensors on Delta log and Iceberg snapshots

No

I3

Selective publisher (D14-gated) to the S3 exchange zone; Delta Direct and UC external-table registration documented and tried against real accounts

Yes

I4

One real workload moved end to end with the §7 checklist, and the §9 cost report

Yes, plus a Databricks or Snowflake account

I1 and I2 can start now. Neither touches billable resources.

11. Decisions needed#

  1. S3 is the only exchange zone (§3): commercial platforms never reach the on-prem lake directly.

  2. Single writer per table (§4): moving a workload moves its writer.

  3. Read Iceberg, write Delta (§5): Iceberg readers are added for Snowflake interop; D13 stands unchanged.

  4. D14 gates publishing (§8): only internal/public by default; more sensitive tables need a policy entry.

  5. Start with I1 and I2 (§10) while AWS is on hold.

12. To verify#

  • Whether Snowflake can be a Delta Sharing recipient natively; I found no documentation that it is. If not, Snowflake gets our tables through Delta Direct only.

  • Whether delta-rs can write UniForm (Iceberg) metadata. Delta’s docs describe UniForm metadata as generated by the Delta writer after each commit (Delta docs); without delta-rs support, our tables reach Iceberg-only readers through Delta Direct, not UniForm.

  • Unity Catalog OSS feature parity with Databricks’ Unity Catalog for external tables and credential vending, before I1 relies on it.

  • Delta Direct behaviour with partitioned tables and with our schema_mode="overwrite" rewrites (dbt plugin, Phase 3).