Skip to main content
Eric Kouassi

02/NLP · Data Engineeringproduction

Distributed Semantic Search

Oxide AI, 2021 to 2023

Semantic search and retrieval-based question answering across an enterprise document corpus, with the distributed data pipelines needed to keep it current.

Architecture diagram for Distributed Semantic Search: sources feed a pipeline of Ingest, Embed, Index, Rank, Answer, producing a grounded answer.

Problem

Precision degrades as a corpus grows. Retrieval that works over thousands of documents returns plausible noise over millions unless ranking and the ingestion path are engineered together.

Result

Processed more than 5 million documents on 100 GB to 500 GB pipelines, improving accuracy and performance by 20 to 35 percent.

Pipeline stages

  1. 01

    Ingest

    Distributed jobs parse and clean documents across mixed formats. Batch rather than streaming, because the job that sets the requirement is reprocessing everything, not adding one file.

  2. 02

    Embed

    Embeddings are computed across the corpus in the same distributed pass, so a change of model is one long job rather than a migration.

  3. 03

    Index

    The vector index and the lexical index are built together over the same corpus at the same version, which is what lets the two retrieval paths be compared rather than guessed between.

  4. 04

    Rank

    Keyword and embedding results are fused. Part numbers, acronyms, and exact strings survive here; in embedding space alone they get averaged into something close but wrong.

  5. 05

    Answer

    The ranked passages ground the generated answer, so what comes back is bounded by what the corpus actually contains.

Decisions

  • Hybrid retrieval rather than embeddings alone

    Most of the accuracy gain came from here. Embeddings are good at meaning and bad at exact tokens, and an enterprise corpus is full of exact tokens that carry the whole question. Fusing a lexical index back in recovers the cases where the right document was findable by name and unfindable by similarity.

  • Distributed batch ingestion

    The requirement is not indexing a new document. It is reprocessing millions of them when the embedding model changes, and that job has to finish in hours rather than days or the model is effectively frozen.

    Cost
    Freshness. A new document is searchable on the next run, not on arrival, so real-time updates were traded away for the ability to rebuild the whole index.

Stack

  • Python
  • Spark
  • ETL
  • Vector databases
  • AWS
All systems