How BookTrace Was Built — and What It Cannot Tell You

Last updated: 2026-07-08

What this index is

BookTrace is a finding-aid over 15,442 books catalogued in the Internet Archive's genealogy and americana collections. It is not a full-text search of Internet Archive; it is a filtered, curated slice built for genealogists working to Board for Certification of Genealogists (BCG) standards and the Genealogical Proof Standard.

The tool searches two layers simultaneously: (1) the extracted names and places from the OCR text of 7,179 books whose text could be read, indexed via natural language processing (spaCy en_core_web_sm), and (2) the catalogue metadata (title, subject, description) of all 15,442 books whether or not their text was readable.

Total extracted entities: 39,258,145, consisting of 28,549,829 PERSON spans, 10,018,095 place spans (GPE), and 690,221 other locations (LOC).

How books were selected

Books entered the index through two automated pipelines, run against Internet Archive's public API, between May and June 2026. The first pipeline queried the genealogy collection with keyword and subject filters for genealogically-relevant material. The second expanded to the americana collection, applying a relevance scoring function to identify county and local histories, published genealogies, vital records abstracts, city directories, and biographies.

A book scoring above threshold entered the catalogue. If it had OCR text available, the pipeline attempted to download and process it. If it had no OCR — because it is a microfilm scan of handwritten records, a scan without a text layer, or an item Internet Archive has flagged as access-restricted — the book still entered the catalogue at Layer 2 (metadata), but produced no entities.

A small number of harvested items (467 of 15,909) are excluded from BookTrace entirely — non-English, extremely short, or flagged during initial filtering. These never appear in any BookTrace result, including catalogue-only results.

The 53.5% text-searchable gap

Of the 15,442 catalogued books, only 7,179 (46.5%) have their full text in the Layer 1 index. The remaining 8,263 (53.5%) fall into three categories:

Status	Count	What this means
No OCR	6,298	The book is a microfilm scan, a handwritten manuscript, or lacks a text layer for a technical reason. OCR cannot process these regardless of tool.
OCR degraded	1,177	OCR was attempted but text quality was below the reliability threshold (score < 0.25 on a 0–1 scale). Indexing degraded text produces more false extractions than valid ones.
Download error	788	The OCR file was unreachable — access-restricted, network failure, or 404. Some are retryable in future runs.

This is not a failure of the pipeline — it is a fact of the corpus. Genealogy sources are heavily hand-written or microfilmed. Excluding these books from Layer 2 would hide their existence from researchers; including them in Layer 1 would produce a false signal that the tool has read them. BookTrace shows both their catalogue presence and their un-searched status.

What "extracted entity" means, and does not mean

An entity is a text span that the spaCy statistical model labeled as a PERSON or a place. This is statistical classification, not verified fact:

The model produces false positives visible in the raw data: surname="LIBRARY", surname="Reunions", given_names="liam" (from an OCR fragment of "William"), entity_text="RESPECTFULLY" labeled as a place.
The model produces false negatives: names in inverted format ("Lake, James"), names preceded by titles ("Mrs. John Lake" → the model labels "John Lake" as the person), and names in unusual script or spacing.
The married-women gap is systematic and severe. A book referring to a woman as "Mrs. John Smith" cannot be found by her own given name. Her surname search hits her husband's surname; her given name is invisible. Researchers documenting women in pre-20th-century sources must know this.

Every result BookTrace returns should be read as: "the string [term] was detected as a name or place by an automated NER model in this book's OCR text." It is a signal for the researcher to open the book at approximately that page and read the actual passage. It is not a substitute for reading.

How the surname–given-name split works, and why it is fragile

For each PERSON entity the pipeline stored, it split the raw text span on the last space to produce surname and given_names:

"James E. Lake" → surname Lake, given_names James E.
"William Lake" → surname Lake, given_names William
"Lake" (single word) → surname Lake, given_names NULL

This is positional, not linguistic. It handles the majority of Western name formats correctly, but breaks on:

Inverted names: "Lake, James" → surname James, given_names Lake,
Titles: "Mrs. John Lake" → surname Lake, given_names Mrs. John
Suffixes: "John Lake Jr." → surname Jr., given_names John Lake
Single-token names: OCR fragments and initials get stored as surnames.

BookTrace's given-name filter is therefore best-effort. A search for surname "Smith" and given name "William" will miss records where the pipeline stored "William" as a surname (single-token) or where it stored "Mrs. William" as the given names (title contamination). Search surname-only first.

What "approximately page 47" means

Every entity in the index has an estimated_page, computed as:

int(character_offset_in_OCR / total_characters * imagecount)

where imagecount is the book's page count from the Internet Archive metadata. This estimate is accurate to roughly ±3–10 real pages on typical books; more accurate for books with clean OCR and uniform page density, less accurate for books with unnumbered plates, index pages, or heavy hand-written insertions.

Page numbers in BookTrace results are ALWAYS shown with a ~ prefix. The symbol is not decorative — it is the tool telling you the exact page number was not recorded and cannot be recovered from this index. Open the book at the estimated page, then scroll a few pages in either direction.

What BookTrace cannot show you

No source sentences. The database stores only the extracted name and its approximate page. It does not store the sentence the name appeared in. BookTrace cannot show you "…the passage where 'Hanks' is mentioned…" the way most search interfaces can. This is a known limitation of the pipeline design and would require re-processing all 7,179 books to fix (a named future enhancement). To read the passage, follow the Internet Archive link and navigate to the approximate page.

No sentence-level co-occurrence. When BookTrace says "surname Hanks and place Kentucky both appear on page 47," it means both were extracted independently from that page's text. It does NOT mean they appeared in the same sentence — or even the same paragraph. Two entities on the same estimated page are within a few dozen lines of each other, no more. For sentence-level proof, open the book.

What a negative result means, and does not mean

BookTrace distinguishes three states when a search returns zero matches:

True negative: zero matches in text AND zero matches in catalogue. This is the strongest negative signal BookTrace can produce. It means: across the 15,442 books in this index, neither the extracted text of the 7,179-book Layer 1 nor the metadata of any book contained the search terms. Copy the citation summary for your research notes.
Partial negative — not text-searchable: zero matches in text, but the catalogue layer returned results. Some books have title/subject/description matches but their text is not indexed (no OCR, degraded, or download error). The term may appear in those books' actual text; BookTrace cannot confirm.
Partial negative — narrow scope: the researcher's filters (date range, record type, source type) may have excluded books that would otherwise match. The "include unknown date" toggle default is ON precisely to avoid hiding 10.0% of books silently, but any deliberate filter is a source of hidden negatives that BCG documentation must acknowledge.

Even a true negative in BookTrace is not a true negative for Internet Archive as a whole. BookTrace catalogues only genealogy-relevant and americana-relevant books that scored above a relevance threshold in processing. It is not a search of all 40 million Internet Archive books.

Publication vs. scan date

Some books in this index have "dates" that reflect the year they were scanned by Internet Archive, not the year they were published — you will see values like 2024, 2025, and 2026 in books that are clearly older. The field is labeled "Publication/scan year (as recorded by IA)" rather than "published" for this reason. Filter with awareness that the boundary may be imprecise. 1,544 books (10.0%) have no recorded date at all.

Contributing institutions

The top institutions contributing books to this index include the Allen County Public Library Genealogy Center, Harvard University, the Library of Congress, the New York Public Library, and the University of Michigan. "Contributor" as displayed on each result card indicates the digitizing institution — not the publisher, author, or subject.

What comes next

Named future enhancements (not implemented in v1) include: source sentence storage for snippet display, sentence-level co-occurrence proximity, given name variant expansion beyond static abbreviations, per-field keyword search, and expansion of the corpus to additional genealogy-relevant sub-collections. Progress on any of these will be noted here.

← Back to BookTrace search