Building a Production ready RAG Pipeline: TF-IDF, HNSW, LSH, CAG, guardrails and More

less than 1 minute read

Published:

TL;DR

This post outlines a potentially effective approach to user queries by implementing a Retrieval-Augmented Generation (RAG) strategy, and 10-guardrail safety system.. The proposed solutions involve utilizing Cache-Augmented Generation alongside Context Engineering, Semantic Search, Embeddings, Chunking, Page Indexing, a Web Chat User Interface, and large language models such as Olama and Gemeni. Additionally, it incorporates Hugging Face’s Chain and the MCP Server for Claude Desktop.

Standard RAG Pipeline

  • Raw Documents → /data/
  • Agentic Chunking + TF-IDF (semantic boundaries + vocabulary scoring)
  • Sentence Transformers — BGE model, dim=384, normalize
  • ChromaDB + HNSW + LSH — O(log n) ANN with the layer graph visualised
  • CAG (Redis) + Context Engine (7 steps) + LLM (Gemini/Ollama)

References are available at:

You May Also Enjoy

Configuring Wifi in ESP32 WORM using code

1 minute read

Published:

Recently, I have been delving into a specific use case that involves consuming a voice REST endpoint using the ESP32 microcontroller. This task requires not only utilizing the capabilities of the ESP32 but also ensuring that the device is connected to a Wi-Fi network for seamless communication with the endpoint.

Data mocking using Faker

2 minute read

Published:

Ideally, test data is of priority and the project teams always face an issue in getting the relevant and realistic test data for pre-production activities. More issues(refresh of data; data manipulations etc.,) arise, when programs consume data from a shared environment. Sometimes, requirements of data varies and a new set of data should be replicated through external tools and technologies. Many commercial data mocking/stubbing tools are available in the market, but as a open source lover, I recommend using Faker library.