Phase 01
Maintain the index
secondary_index_{column}
Quanton keeps a Record-Level or Secondary Index on the frequently-joined columns in the Hudi metadata table, refreshed on write. It lives alongside the table, so it's there for every future merge — no rebuild per query.
Phase 02
Look up the source keys
index.lookup(source_keys)
At merge time the source-side key values are looked up in the index. The result is the exact list of files, row positions, and partitions that can match — computed before the target scan runs.
Phase 03
Read only what matches
read matched files + row groups
The file and row-position list goes to the Arrow parquet reader. It reads only the matching files, and only the row groups that hold matches — and because rows never fan out to find their partner, the whole-table shuffle is gone.
You don't change the query. Turn the index-aware path on once, and Quanton applies it whenever a usable index is present:
```sql
MERGE INTO customers t
USING source_updates s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET t.amount = s.amount
WHEN NOT MATCHED THEN INSERT *
```
```properties
spark.quanton.sql.optimizer.dynamicPruning.enabled=true
```
The important part is that the decision is made **per query, at plan time**. Quanton doesn't blindly use the index. It reads the column statistics for the join key to estimate how many target files the source keys could actually touch, and then picks the cheaper plan:
- If plain DPP already narrows the target to very few files, the index earns nothing, so Quanton just lets DPP run.
- When the keys are spread across most of the table and DPP would prune almost nothing — the common case for late-arriving updates — Quanton switches to the index lookup.
Same SQL either way. The plan adapts to the data in front of it, run by run, instead of committing to one strategy up front.
## Full scan vs. index lookup
Updating ~1M keys scattered randomly across a 10 TB table (~20,000 files of 512 MB):
Scan + shuffle
100% shuffled
DPP prunes almost nothing on random keys, so the whole target is scanned and shuffled just to locate the matches
Index-aware
0shuffle
the low-shuffle columnar merge writes the matched files in place — zero data shuffled, however many files match
The work now scales with the size of your change, not the size of your table. That's what `MERGE` was supposed to do all along.
## Beyond keys — expression and vector indexes
A secondary index is just one shape of "map a value to where it lives." The same machinery covers more:
- **Expression indexes** index a *computed* value like `lower(email)`, `date(ts)`, or `substr(sku, 1, 4)`, so a merge or filter on that expression prunes the same way a raw-column index does.
- **Vector indexes** index embeddings of unstructured data like text, images, and audio, so you can find the nearest rows to a query vector without scanning every record. It's the same lookup pattern, now inside the same table.
Different index, same principle — don't scan to find something you can look up.
## Takeaway
A join or a merge is, underneath, a search for the target rows that match your source. Scanning and shuffling the entire table to run that search is the expensive way to do it. An index answers the same question by pointing at the exact file and row position, so Quanton reads only the files that match, only the row groups inside them that hold matches, and skips the target shuffle. The cost of changing your data finally scales with how much you're changing, not with how much you've stored.