Chunking a PDF for retrieval, measured against a fixed window
Almost every RAG pipeline starts by cutting a PDF into 1,000-character
pieces. That is a default, not a decision. This page measures what the default costs,
on three public documents, against /v1/pdf/chunks, which ends a chunk where
the document ends a section.
Measured 2026-11-02 on v0.21.1. Both splitters get the same target of 1,000 characters and no overlap. The corpus and the script are below, so the whole table can be regenerated.
Results
| document, pages | splitter | chunks | starts mid-sentence | ends mid-thought | carries section path |
|---|---|---|---|---|---|
| Attention Is All You Need (arXiv 1706.03762)pages 1-15 of 15 | fixed window, 1,000 chars | 70 | 44.3% | 70.0% | n/a |
| docsmith /v1/pdf/chunks | 55 | 10.9% | 36.4% | 98.2% | |
| BERT (arXiv 1810.04805)pages 1-16 of 16 | fixed window, 1,000 chars | 101 | 56.4% | 87.1% | n/a |
| docsmith /v1/pdf/chunks | 79 | 7.6% | 45.6% | 100.0% | |
| requests 2.32 manual (Sphinx)pages 1-25 of 125 | fixed window, 1,000 chars | 48 | 4.2% | 37.5% | n/a |
| docsmith /v1/pdf/chunks | 57 | 12.3% | 50.9% | 100.0% |
A chunk "starts mid-sentence" when its first character is a lowercase letter, and "ends mid-thought" when it ends without terminal punctuation. Those are proxies, not judgements: they count seams a retriever has to paper over. "Section path" is the headings a chunk sits under; a character-count splitter cannot know it, so the column is n/a there. The run also counts page furniture - running heads and feet, meaning lines that repeat on at least 60% of the document's pages, counted from the PDF itself rather than from docsmith's own rules. Neither splitter leaked any of it on this corpus, so that column is not in the table: a column of zeros is not evidence.
Where this loses
On the third document, a Sphinx-generated manual, docsmith cuts more
seams than the fixed window does, not fewer. Code samples and prose interleave into
very long blocks there, and a block longer than target_chars has to be
split somewhere: docsmith prefers a sentence end, falls back to a space, and lands
inside a sentence more often than a splitter that never tried to respect a sentence
at all. The section path still covers every chunk in that document, which is the
part the fixed window cannot offer at any setting.
That is the honest shape of the trade-off: structure costs something when the document's own structure is a staircase of code blocks. It is published here rather than left out because a benchmark that only shows the good documents is an advertisement.
Text that survives
Better boundaries are worthless if content goes missing, so the same run counts
the document's distinct words (three characters or more, hyphenation across line
breaks undone) and asks how many appear in what the API returns:
97.8% on Attention Is All You Need, 96.7% on BERT, 99.6% on requests 2.32 manual. The remainder is not lost text: a word hyphenated across a page break
comes back joined, a heading travels in heading_path metadata rather than
inside the chunk, and page numbers are dropped on purpose.
What a chunk looks like
curl -sS -X POST "$BASE/v1/pdf/chunks" \ -H "X-API-Key: $DOCSMITH_KEY" \ -F "file=@report.pdf" -F "target_chars=1000" | jq '.chunks[2]'
{
"id": "72d6355a3a6a2f1b",
"text": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. ...",
"heading_path": ["Attention Is All You Need", "Abstract"],
"page_start": 1,
"page_end": 1,
"characters": 724
}
That is a real chunk of the first document in the table, not an illustration: the id is derived from the text, so converting the same PDF again returns the same id and a re-ingest updates rows instead of duplicating them.
Corpus
| document | pages read | pages in file |
|---|---|---|
| Attention Is All You Need (arXiv 1706.03762) https://arxiv.org/pdf/1706.03762v7 | 1-15 | 15 |
| BERT (arXiv 1810.04805) https://arxiv.org/pdf/1810.04805 | 1-16 | 16 |
| requests 2.32 manual (Sphinx) https://requests.readthedocs.io/_/downloads/en/latest/pdf/ | 1-25 | 125 |
All three are public files, fetched from the URL shown, and the
baseline reads them with pdftotext -layout. The corpus is two papers and
a manual, chosen for being public and ordinary, and the table above is that run and
not a selection from several.
Reproduce it
The script that produced the table is served from this host, so the numbers are not a claim about a private harness:
curl -sSO https://103-82-194-130.nip.io/bench/bench_chunks.py curl -sSO https://103-82-194-130.nip.io/bench/bench_corpus.json # a free plan key is 200 requests a month, no account: https://103-82-194-130.nip.io/try export DOCSMITH_KEY=sk_live_... python3 bench_chunks.py --corpus bench_corpus.json --pdf-dir ./pdfs \ --base https://103-82-194-130.nip.io
The whole script is 400 lines of Python with no dependencies beyond
pdftotext, and it makes three requests, one per document. The fixed-window
side is LangChain's RecursiveCharacterTextSplitter algorithm transcribed
into that script, not a crude character slice: it tries paragraph breaks, then line
breaks, then spaces. Set TARGET and OVERLAP at the top to
measure a different chunk size. The docsmith side posts to the same handler this
service runs in production.
Try it on your own document
Convert the first five pages of your own PDF with no account and no key, and read the headings and chunk boundaries it recovers before deciding whether it is worth $9.99 a month. If your document is the kind that breaks it, that is worth knowing before you pay for it.