Self-correcting RAG code assistant

Problem

Answering questions about code with a small local model, without sending anything to an external API. Small models hallucinate readily, so retrieval quality and answer checking matter more than model size.

Approach

Retrieval. MiniLM sentence embeddings, fine-tuned on the CoSQA code-search dataset, stored in a Qdrant vector database.
Generation with self-correction. Phi-3 running locally through Ollama drafts an answer, critiques it against the retrieved code, and revises it when the critique fails. The loop is orchestrated with LangChain.
Evaluation. An LLM-as-judge harness scores correctness and faithfulness to the retrieved sources. Every run is tracked in MLflow.

Findings

I benchmarked the pipeline against agentic variants in which a 3B model chooses its own tools. The simpler self-correcting pipeline won clearly: 83% against 33% correctness. At this scale the agents often ran out of steps before producing an answer.

The project is now growing into a full agentic coding assistant, with tools for semantic search, file reading and grep. The original pipeline stays in place as the baseline any new version has to beat.

Bar chart of correctness and faithfulness per agent. The Phi-3 self-correcting baseline scores 83% correctness; tool-calling agents on 3B models score between 8% and 33%. — LLM-as-judge benchmark. The self-correcting pipeline reaches 83% correctness; the best 3B tool-calling agent reaches 33%, as its loops often hit the step limit before answering.

Bar chart of percentage improvement over the base embedding model. MRR@10 improves by about 4.6%, Recall@10 by about 1%, nDCG@10 by about 3.6%. — Fine-tuning MiniLM embeddings on CoSQA improves retrieval on all three metrics. MRR@10 rises from 0.85 to 0.89.