Document Ingestion — RAG Knowledge Base

Problem

An agent with RAG is only as good as its knowledge base — and that base almost never lives in one place: internal PDFs, web pages, and data coming out of third-party APIs. Without an organized ingestion pipeline, every new source ends up as a loose script and the context the agent receives is inconsistent or goes stale.

Solution

An ingestion pipeline that connects multiple sources (PDF documents, web content, APIs), normalizes the text, and splits it into chunks with controlled overlap so ideas don't get cut in half. Embeddings get indexed in Pinecone or in the AWS stack (OpenSearch / Bedrock Knowledge Bases) depending on the case, while PostgreSQL stores structured metadata for each document — source, version, processing status — so context can be reprocessed or invalidated without rebuilding the whole index. The result is the knowledge base my RAG agents query against.

Stack and architecture

• Chunking with size and overlap configurable per source type, so paragraphs or code blocks don't get split mid-way.
• Pinecone as the primary vector store, with the option to index into AWS (OpenSearch Serverless / Bedrock Knowledge Bases) when the project already lives in that ecosystem.
• PostgreSQL for structured metadata: which document came from where, when it was processed, and its status — kept separate from the vector store so sources can be audited and reprocessed without touching the embeddings.
• LangChain for source-specific loaders and the embedding step, with text normalization before chunking.
• Designed to run as a scheduled or event-triggered job, so the agent's knowledge stays current without manual intervention.

Challenges

Balancing chunk size: pieces that are too small lose context, pieces that are too large dilute relevance in similarity search. The other challenge was clearly separating what lives in the vector store (embeddings for semantic search) from what lives in PostgreSQL (structured metadata and version control), so reprocessing one source doesn't mean rebuilding the whole index from scratch.