Meta-RAG — Adaptive Model-Routing Gateway

The problem

Sending every query to the most capable model and the most expensive retrieval strategy is wasteful — most questions do not need it — but routing by hand needs a definition of “hard” that nobody has written down.

Why it needed solving

RAG systems typically apply one retrieval strategy uniformly. A definitional lookup and a multi-hop reasoning question have completely different needs, and paying the multi-hop cost for every query is how inference budgets disappear. The interesting problem is learning the routing decision rather than hard-coding it.

The approach

Fine-tune a complexity classifier on LLM-labeled queries, use it to drive a three-tier dispatcher across retrieval strategies and model tiers, then close the loop: an LLM judge scores each strategy in shadow mode and a Bayesian optimizer tunes the routing thresholds against an accuracy SLA.

Architecture

• FastAPI gateway — single entry point for incoming queries.
• Complexity classifier — a fine-tuned DeBERTa-v3-base model scoring each query.
• 3-tier dispatcher — routes to naive, parent-document or HyDE retrieval, and to models of differing cost.
• Retrieval-ingestion pipeline — chunking, embedding and vector indexing in Qdrant over a Wikipedia-sourced corpus.
• Shadow evaluation harness — an LLM-as-judge scoring faithfulness and relevancy per strategy, off the serving path.
• Bayesian optimizer — Optuna tuning routing thresholds against an accuracy SLA.
• Redis cache — response caching in front of the dispatcher.
• Streamlit dashboard — operational view of routing decisions and cost.

Implementation

• Generated a synthetic, LLM-labeled query dataset spanning complexity levels, since no off-the-shelf labeled set existed.
• Fine-tuned DeBERTa-v3-base on that dataset to produce the routing signal.
• Implemented three retrieval strategies — naive, parent-document and HyDE — behind one interface so the dispatcher can swap between them freely.
• Built the ingestion pipeline over a Wikipedia-sourced corpus with chunking, embedding and Qdrant indexing.
• Ran the evaluation harness in shadow mode so judge scoring never added latency to live responses.
• Used Optuna to search routing thresholds against an accuracy SLA rather than tuning them by intuition.
• Added a Redis response cache and a Streamlit dashboard for routing and cost monitoring.

AI/ML components

Challenges & how they were handled

What comes next

• Extend the tier set beyond three routing levels.
• Replace the Wikipedia corpus with a domain corpus to test transfer of the complexity signal.