Can you do RAG without a vector database?

Yes, if the vectors live in a lakehouse table alongside the text and the relational data. Retrieval becomes a SQL query that ranks by cosine similarity, and because the result is an ordinary row set you can filter and join it against your other tables in the same statement. A dedicated vector store earns its place when you need approximate-nearest-neighbour indexes at a scale where scoring every row per query stops being viable.

How do you read PDFs into a Spark DataFrame?

Register the quanton_unstructured Python data source and call spark.read.format('quanton_unstructured').load('s3://bucket/prefix/'). It returns one row per file with the extracted text, pre-split chunks, a BLOB-style content column, and a parse_status, routing each file to a parser by extension with a magic-byte check as fallback. It needs Spark 4.0 or newer, because the Python Data Source API is what lets a pure-Python connector plug into spark.read.format.

Can Apache Hudi store vector embeddings?

Yes. Hudi 1.2 supports Lance base files, so an array embedding column round-trips inside a normal copy-on-write Hudi table and upserts in place on the record key. Setting the base file format to LANCE is the only change to an otherwise ordinary Hudi write, and the Lance jar ships in the Quanton image alongside the Hudi bundle.

Why is my RAG retrieval worse than expected?

Check your chunk size against the embedding model's context window. A 2000-character chunk is roughly 500 tokens, and all-MiniLM-L6-v2 caps at 256, so sentence-transformers silently truncates half of every chunk before embedding it — no error, no warning. Keep model context at or above chunkSize/4 tokens, or shrink the chunks.

Should structured files be parsed as documents for RAG?

No. Running a CSV or Parquet export through a text extractor produces garbage that then gets chunked and embedded. The reader flags them STRUCTURED_DATA and leaves them to the native readers, so you keep them as relational tables and join them to the chunks at query time — which is where most of the useful context actually comes from.

RAG on Your Lakehouse: Context Served Across Structured and Unstructured Data in SQL

What you’ll build

Object storage at any company holds two kinds of data. Structured — CSV, JSON, Parquet (the fact tables, exports, CDC feeds and reference data). Unstructured — PDF, DOCX, TXT, images, video (contracts, tickets, reports, recordings), often on a NAS reached through an object-storage client rather than in a bucket directly.

The standard RAG setup puts the vectors somewhere else. Documents get chunked, embedded, and loaded into a vector index — pgvector in Postgres, or a dedicated store like Milvus, Qdrant or Chroma — and similarity search runs against that. Which one you pick matters less than the fact that it’s a different system from where the fact tables live. So a question that needs both “what does the contract say” and “which of our contracts are still active in this region” has to be stitched together across two of them.

This guide keeps both in one place: read the documents and the structured files with the same engine, land them as tables in the lakehouse, and answer with a SQL query that ranks by vector similarity and filters and joins against your relational data in the same statement.

The examples use CUAD, a Hugging Face dataset of real commercial contracts where lawyers have already marked which clauses appear in which document — the PDFs are the unstructured side, their labels the structured side.

One engine reads documents and tables out of object storage and curates datasets in the lakehouse that serve both retrieval and training.

Four steps:

object storage                  contract PDFs + an annotations CSV
s3:// · gs:// · abfss://        (or a NAS mount through an object-storage client)
   │  spark.read.format("quanton_unstructured")
   ▼
contract_documents              Hudi COW · Lance base files · text, chunks, content
   │  explode chunks → embed
   ▼
contract_chunks                 Hudi COW · Lance base files · array<float> embeddings
   │  rank by similarity, filter and join in SQL
   ▼
contract_annotations            Hudi COW · Parquet · the clause labels, joined at query time

Everything below runs on Quanton.

Setup

quanton_unstructured ships inside the Quanton image, as do Hudi and Lance and the Spark configs they need, so the session is just:

spark = SparkSession.builder.appName("rag_demo").getOrCreate()

The embedding model is yours to choose — different corpora and languages suit different models, and you may already be standardised on one. Install whichever you use:

pip install sentence-transformers

This guide uses BAAI/bge-base-en-v1.5: it runs on CPU, it’s Apache-2.0, and its 512-token window comfortably fits the default chunk size. See choosing a model below for what to look at.

Step 1 — Read the documents

import quanton_unstructured
quanton_unstructured.register(spark)

docs = spark.read.format("quanton_unstructured").load("s3://bucket/contracts/")

One row per file, with the parser chosen per file — PDF, HTML, Office, plain text. A file that can’t be parsed doesn’t kill the job: it lands as a row carrying a status and the reason it failed, so you can query for the failures and deal with them.

Column	What it holds
`uri`	full path, and a natural record key for upserts
`size`, `modification_time`	as listed in object storage
`content_type`	detected from the file’s magic bytes, so a mislabelled extension doesn’t route it wrong
`content`	the file itself — raw bytes inline when it’s small, a `path`/`offset`/`length` pointer when it’s large
`text`	extracted text
`chunks`	`array<string>`, pre-split and ready to embed
`parse_status`	`SUCCESS`, `FAILED`, or `STRUCTURED_DATA` for CSV/JSON/Parquet, which are left to the native readers instead of being run through a text extractor
`parse_error`	the reason, when parsing failed
`parser_used`, `chunker_used`	which parser and which splitter actually ran

Step 2 — Land it in Hudi on Lance base files

Hudi 1.2 can write Lance base files, so the text, the chunk array and later the embedding vector sit together in one copy-on-write Hudi table:

(docs.where("parse_status = 'SUCCESS'")
     .write.format("hudi")
     .option("hoodie.table.name", "contract_documents")
     .option("hoodie.datasource.write.recordkey.field", "uri")
     .option("hoodie.table.base.file.format", "LANCE")
     .mode("overwrite")
     .save(path))

The annotations CSV goes in as its own Hudi table on Parquet base files — same lakehouse, ordinary relational columns.

uri as the record key is what makes this incremental. Re-run the job after a batch of contracts is refreshed and only the changed files are rewritten, upserted in place rather than appended as duplicates — so “was the sales agreement updated after the new pricing sheet landed?” is a question you can answer from the table, and downstream jobs can read just what changed instead of rescanning the corpus.

It also gives you a second chance at the documents that failed. Filter parse_status = 'FAILED', look at parse_error, fix whatever it was — a missing extra for that file type, a corrupt scan, a size limit — and re-run. The good rows stay put; only the repaired ones change.

Step 3 — Embed the chunks

The reader already split the text. Explode it to one row per chunk and embed:

chunks = (
    docs.select("uri", "file_name", F.posexplode("chunks").alias("pos", "chunk"))
        .withColumn("chunk_id", F.concat_ws("#", "uri", "pos"))
)

def embed_partition(iterator):
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer("BAAI/bge-base-en-v1.5", device="cpu")
    for pdf in iterator:
        vecs = model.encode(pdf["chunk"].tolist(), batch_size=64,
                            normalize_embeddings=True)
        pdf["embedding"] = [v.tolist() for v in vecs]
        yield pdf

embedded = chunks.repartition(8).mapInPandas(embed_partition, schema=EMBED_SCHEMA)

mapInPandas loads the model once per worker rather than once per row. normalize_embeddings=True scales every vector to unit length, which is what lets the similarity query later be a plain dot product.

Write the result back as another Hudi table on Lance base files, keyed on chunk_id — uri#pos, so each chunk has a stable identity across re-runs. Re-embed a document and its chunks upsert in place; the id also carries you back to the source file, which matters when a retrieved chunk needs to be traced to the contract it came from.

Choosing an embedding model

Three things to check, in rough order of how often they bite:

Context window against your chunk size. The default chunkSize is 2000 characters, roughly 500 tokens of English prose. A model with a shorter window silently truncates — all-MiniLM-L6-v2, the default in most tutorials, caps at 256 tokens, so it embeds about half of each chunk and returns a perfectly well-formed vector for it. No error, no warning. Keep model context ≥ chunkSize / 4 tokens, or shrink the chunks.

Domain and language. General-purpose models do fine on general prose. Legal, medical or code corpora, and anything non-English, are worth benchmarking before committing.

Where it runs. sentence-transformers models run locally on CPU or GPU, which keeps the corpus inside your VPC. Hosted embeddings — OpenAI, Voyage, Cohere and others — are a call away and often stronger, at the cost of sending your documents to a third party and paying per token. Either works here: the embedding step is your code, and the connector’s job ends once the chunks are on the table.

Step 4 — Retrieve with SQL

Because the vectors are a column in a table, retrieval is a query:

WITH scored AS (
    SELECT c.file_name, c.pos, c.chunk,
           aggregate(zip_with(c.embedding, q.v, (a, b) -> a * b),
                     0.0F, (acc, x) -> acc + x) AS cosine
    FROM chunks c CROSS JOIN q
)
SELECT file_name, pos, cosine, substring(chunk, 1, 140) AS excerpt
FROM scored ORDER BY cosine DESC LIMIT 5

zip_with multiplies the two vectors elementwise and aggregate sums them — a dot product, which equals cosine because both sides are unit length. The q view holds the single query vector.

That scores every chunk, which is fine at this size and won’t be at tens of millions. Approximate-nearest-neighbour search over the same tables is its own topic — a guide on that is coming.

Serving context needs more than similarity

The interesting questions don’t stop at “which chunks look like this sentence”. They look like “what do our active contracts in Germany say about termination” — and active and Germany are not in the embedding. They’re columns in the structured tables sitting next to the chunks.

So the real query ranks by similarity and filters and joins in the same statement:

SELECT c.chunk, m.renewal_date, a.clause_type
FROM chunks c
JOIN contract_metadata m ON m.file_name = c.file_name
LEFT JOIN annotations   a ON a.file_name = c.file_name
WHERE m.status = 'active' AND m.jurisdiction = 'DE'
ORDER BY cosine DESC
LIMIT 5

The WHERE runs before the ranking, so you rank the candidates that qualify rather than ranking everything and discarding most of it afterwards.

Some questions aren’t retrieval at all. “Which clause types do we actually have coverage for?” is a GROUP BY over the labels and the chunks together — no nearest-neighbour search involved:

+---------------------------+-------------------+------------------+
|clause_type                |annotated_contracts|retrievable_chunks|
+---------------------------+-------------------+------------------+
|Anti-Assignment            |...                |...               |
|Termination For Convenience|...                |...               |
|Exclusivity                |...                |...               |
|Non-Compete                |...                |...               |
+---------------------------+-------------------+------------------+

Labels that arrived as a CSV, chunks that came out of PDFs, one GROUP BY across both.

That’s the case for assembling context where the data already is. Building the context an LLM sees is a data engineering job — joins, filters, aggregates, window functions over documents and tables together — and the lakehouse is where both already live, which is why Quanton does that assembly rather than a separate system stitching results together after the fact.

Serving it to an application is a different problem with different requirements: millisecond lookups, high concurrency, transactional reads. That’s what Lakegres is for — a Postgres-compatible endpoint over the same Hudi and Iceberg tables, with vector search and intelligent indexing on top of them. Point your existing Postgres clients at it and the context you assembled here is what the application queries, with no copy drifting out of sync.

Where to go next

The same tables feed model training. quanton_llm_training exports a validated fine-tuning dataset straight out of them, joining the chunks to the expert labels so each training example is grounded in something a human marked — covered in From Lakehouse Tables to a Fine-Tuning Dataset.

The connector is a pure-Python package and every piece is swappable: the parsers (pypdfium2, selectolax, markitdown), the splitter (langchain-text-splitters) and the embedding model (BGE via sentence-transformers). All permissively licensed — PyMuPDF was rejected for being AGPL.