Lakehouse Fast-Path

The Lakehouse fast-path ingests high-cardinality immutable rows — frames, telemetry samples, event-stream rows — without materialising them into engine RAM. Such rows live in Hive-partitioned Parquet and are read in place through virtual labels; the graph itself holds only typed schema, adjacency, and a catalog pointer.

Use owned labels (bulk_create_*) when:

• The class is mutable (charting, entity-resolution merges, derived state).
• The class is low-cardinality (≤ ~50K rows total — players, plays, devices, agents).
• The rows fit comfortably in memory.

Use the fast-path (virtual labels) when:

• The class is immutable observation rows — anything that arrived once and never changes.
• The class is high-cardinality — frames per game, samples per device, events per stream.
• Materialising those rows would grow engine RAM faster than disk should justify.

Step 1 — Classify your node classes#

For each class in your schema, apply the mechanical decision rule (see World Graph for the full R1–R3 boundary):

1. Is the class mutable? Yes → keep as Owned (bulk_create_*).
2. Is the class an immutable observation row? Yes → make it a Virtual label.
3. Edges are always Owned regardless of endpoint classification.

A worked example from a sports-tracking workload:

If a class produces an ambiguous classification — mutable AND high-cardinality AND read-by-traversal, or immutable AND low-cardinality AND written-multiple-times — the R1–R3 rules cannot resolve it in isolation. Treat that as a stop condition: surface the class for review, pick one axis as the dominant, and document the tradeoff.

Step 2 — Author the lake:// mount config#

The substrate addresses Lake partitions through the lake:// URI scheme. A registration uses bare brace-delimited placeholders (one per partition column):

lake://<mount>/<table>/{var}[/{var}]…/<file-glob>.parquet

The brace-delimited placeholder binds to a partition column at registration; the planner discovers the actual key/value pairs at scan time from the directory layout. Combining placeholders with literal key=value segments in the registration pattern (e.g. {season}=2024) is rejected at DDL time — that mixes registration syntax with filter syntax.

CREATE NODE LABEL is the canonical reference for the VIRTUAL FROM PARTITION DDL and the lake:// URI-template grammar; this guide applies it in a worked pipeline.

For the worked-example schema above:

The mount (nfl, sensors) is configured at workspace open time and binds the URI's authority to a backing storage location (a local directory, an S3 bucket, a GCS bucket, an Iceberg catalog endpoint). Template variables in braces are recognised as Hive-partitioned columns and used by the engine for partition pruning at query time.

The full Virtual Labels Over Parquet cookbook walks through a runnable example end-to-end.

Step 3 — Register the virtual label#

Two paths, same effect.

Via DDL#

CREATE NODE LABEL Frame (
  entity_id STRING,
  ts        TIMESTAMP,
  x         DOUBLE,
  y         DOUBLE,
  speed     DOUBLE
) VIRTUAL FROM PARTITION 'lake://nfl/tracks/{season}/{week}/{game_key}.parquet';

The DDL parser validates the typed schema against the Parquet files' schema. A VirtualLabelEntry { label, partition_pattern, schema_ref, resolver_kind } row is committed to the catalog manifest at <workspace>/canonical/manifest_<epoch>.json. The manifest commit is atomic (write-tmp + fsync + atomic_rename with two-file protocol; F_FULLFSYNC on macOS, fdatasync on Linux).

Via Python FFI#

from arcflow import ArcFlow
 
db = ArcFlow("/path/to/workspace")
epoch = db.register_virtual_partition(
    label="Frame",
    partition="lake://nfl/tracks/{season}/{week}/{game_key}.parquet",
)

The C ABI counterpart is arcflow_register_virtual_partition(session, label, partition) -> i64.

Step 4 — Stop ingesting Virtual classes through bulk_create_*#

Once a class is registered as Virtual, its rows live in the Lakehouse partitions. The bulk-ingest path no longer applies to that class. New observation rows arrive as new partitions in the Lake; the manifest version advances; the graph picks up the new partition on its next manifest read.

If your existing pipeline still calls bulk_create_nodes against a class you've moved to Virtual, the path becomes a no-op classification error at the schema layer — exactly the wrong thing was attempted. Remove the bulk_create_* calls; replace them with whatever writes the Parquet files.

Step 5 — Verify the workspace is on the fast-path#

Two checks confirm the fast-path is active.

Catalog inspection — list every virtual label registered against the workspace:

CALL db.constraints() YIELD name, kind, target
WHERE kind = 'VIRTUAL_LABEL'
RETURN name, target;

Each row is a label/partition-pattern pair. The target is the lake:// URI.

Manifest reading — every committed epoch's manifest survives on disk:

ls <workspace>/canonical/manifest_*.json
cat <workspace>/canonical/manifest_$(cat <workspace>/canonical/CURRENT).json | jq '.virtual_labels'

The virtual_labels array enumerates every Virtual class with its partition pattern and resolver kind. Atomic-commit guarantees the manifest is never half-written; the CURRENT pointer is the two-rename target.

Step 6 — Query against virtual labels#

The intent at the query surface is that virtual labels are indistinguishable from Owned labels:

MATCH (f:Frame {entity_id: 'Unit-01'})
WHERE f.ts >= datetime('2026-03-14T08:00:00')
  AND f.ts <  datetime('2026-03-14T09:00:00')
RETURN f.ts, f.x, f.y, f.speed
ORDER BY f.ts;

The planner-side rewriter for MATCH (:VirtualLabel ...) patterns decomposes the pattern into a manifest-pruned, predicate-pushed Parquet scan. count(:Frame) resolves to a parquet footer scan; WHERE f.x > 50 RETURN f resolves to a column-pruned predicate-pushed scan; COMPUTE-declared derived properties evaluate at row-decode time with predicates pushable through the planner.

Downstream consumers can also read the partitions directly through any Arrow / Parquet / Iceberg tool — the partitions remain Lakehouse-shaped and the catalog metadata is open.

A note on overlay tables — correcting a Virtual row#

Virtual rows are immutable by contract. Corrections happen via overlay tables: an Owned class that the Query Engine joins at read time. Pattern:

-- The Virtual class
CREATE NODE LABEL Frame (...) VIRTUAL FROM PARTITION '...';
 
-- The Owned overlay class — small, mutable
CREATE NODE LABEL FrameCorrection (
  frame_id    STRING,    -- the entity_id of the Frame being corrected
  field_name  STRING,
  new_value   ANY,
  authored_at TIMESTAMP,
  authored_by STRING
);

A reader query joins both: take the Frame row from the Parquet partition; if a FrameCorrection exists for that frame_id + field_name, the overlay wins. This is the discipline :CAUSED_BY edges layer on top of — see Causal Edges for the full pattern.

Worked example — project-merlin#

The project-merlin NFL stress harness applies this exact pattern at scale: 22 entities × 5 plays × 1M frames per game × hundreds of games. The worked example covers:

• The schema split (Frame as Virtual; Player + Play as Owned).
• The partition layout authored against the merlin-nfl-2025/canonical/ Iceberg-shaped tree.
• The mount config + registration sequence.
• The verification that the catalog scan opens in well under 100 ms over 280+ partitions.

For any similar shape — high-cardinality observation rows alongside a small mutable entity model — that's the closest worked precedent.

See also#

• Virtual Labels Over Parquet — the runnable cookbook recipe.
• World Graph — the conceptual layer and R1–R3 boundary.
• Perception Lake — the sibling immutable-observation layer.
• World Graph Substrate — the engine-architecture deep-dive.
• Causal Edges — the discipline for overlay-table corrections.
• CREATE NODE LABEL — the VIRTUAL FROM PARTITION DDL reference.