RAG Question Generator

01 · The problemGreat at Science. Terrible at Bahasa Melayu.

The Question Generator lets a Malaysian teacher pick a syllabus, grade, subject, chapter and topic, then generates up to 10 multiple-choice questions and pushes them straight into Quiz Builder. It is one of the first AI tools a new teacher tries, so it sets their first impression of the product.

For fact-based subjects (Science, Maths, History, Islamic Studies) the output was good. For Bahasa Melayu and English, two of the most-taught subjects in the country, it was embarrassing:

02 · Root causeThe model wasn’t the only problem. Our assumption was.

The pipeline passed the teacher’s dropdown selections straight into the prompt as literal text from the national curriculum document (the DSKP). That quietly assumed the curriculum document describes what students are tested on.

• For Science, that holds. The curriculum’s content standard for Form 1 Science, “Cell as the basic unit of life”, is literally Chapter 2 of the textbook. Feed it to the model and you get questions about cells.
• For language subjects, it doesn’t. The BM curriculum is organised by skill modules (listening & speaking, reading, writing, literature, grammar). The textbook is organised by themed units (“Health brings happiness”, “Value your life”), each weaving every skill together. The two share no chapter titles and there is no one-to-one mapping.

So the model latched onto skill descriptors (“students can understand information in a text”) instead of the grammar and vocabulary teachers actually drill. A model several generations old made it worse.

03 · StrategyTwo tracks: a fast fix and a durable one

I wrote up the diagnosis for leadership with two strategies to run in parallel:

The recommendation to leadership: ship Track 1 once it cleared an agreed quality bar, and treat Track 2 as the durable fix, but audit the question bank first. RAG is only as good as what it retrieves.

04 · The evaluationWhen two evaluations disagree, reconcile them

We ran two evaluations to choose the production model: a Mathematics run and a much larger Bahasa Melayu run. They picked different winners. Maths favoured the most accurate model; BM favoured the most reliable one.

Instead of picking a favourite, I dug into why. The Maths run was small and ran on a broken pipeline: off-syllabus context and short batches for every model. The BM run was roughly eight times larger and its pipeline worked. So I merged them on one balanced scale, weighting the BM evidence about 65/35, and scored five dimensions:

Combined model scorecard (vendors anonymised, scores out of 5, rounded)

Dimension	Model A · fast	Model B · quality	Model C · reasoning
Accuracy25%	3.8	4.6	4.2
Question quality20%	3.8	4.3	4.0
Reliability25%	5.0	3.4	4.2
Latency15%	5.0	4.8	2.3
Structural integrity15%	4.3	4.8	4.8
Weighted total	4.4	4.3	4.0

The result was a tiered setup rather than a single winner:

• Primary: the fast model for real-time, teacher-facing generation. Valid output on every run and the lowest latency. Its one weakness was a prompt bug, not a model limitation.
• Quality fallback: the most accurate model for harder topics and retries, once its timeouts were fixed with smaller batches and automatic retry.
• Offline only: the slow reasoning model, for batch authoring with human review.

05 · Build & shipGrounding generation in real questions

Phase 2 was deliberately boring: no UI changes, no new vendor, no vector search. The question bank already carried curriculum tags, so structured filters beat embeddings on cost and explainability.

1. Audit the bank first. A one-time, idempotent script scores every legacy question from 0 to 100. Questions scoring 75 or above become active (retrievable). The rest are excluded from retrieval but never deleted. Duplicates keep only their best version, and anything with missing metadata goes to review instead of being guessed.

2. Retrieve with a fallback ladder. It must never block a teacher:

1. Teacher clicks GenerateSubject, grade, topic, language, count
2. Retrieve examplesActive questions only, exact subject + grade + topic + language, ranked by quality, diversified by Bloom level and stem type. Target: p95 under 500 ms.
3. At least 5 examples?Yes: inject 15–20 into the prompt.
4. Widen to subject + gradeStill under 5, or the DB is unreachable? Skip examples and use the Phase 1 path.
5. Generate, then validateParse JSON, run the launch gates, and discard any question more than 80% similar to an example (no copying).
6. Questions land in Quiz Builder

3. Change what the model sees. For Science, Maths and History the curriculum text still goes in. For BM and English it’s removed, and the retrieved examples replace it: the exact layer that confused the model. Other rules: formal Bahasa Malaysia, level-appropriate English for each grade, and examples always in the teacher’s chosen language.

4. Instrument it. Every request logs examples injected, whether the filter widened, fallback use and retrieval time, with an alert if any subject’s fallback rate stays high for a week. A rising fallback rate means that subject’s bank needs content, which is a roadmap signal, not just an ops alert.

06 · Key decisionsThe calls I made

1. D1

   Run the fast fix and the structural fix in parallel.

   The model upgrade protected trust within days while RAG was scoped properly. Waiting for the perfect fix would have cost credibility every day.

2. D2

   Audit the bank before building retrieval.

   Retrieval amplifies whatever it finds. A bad bank would have made RAG confidently wrong.

3. D3

   Structured filters, not vector search, for v1.

   The bank was already curriculum-tagged. Filters are cheaper, faster to ship and easy to explain when a teacher asks "why this question?"

4. D4

   Remove curriculum text for language subjects.

   It was the root cause. Real examples show the model what a good BM question looks like better than any skill descriptor.

5. D5

   Degrade gracefully, never block.

   Every failure path falls back to the previous generation path. A slightly weaker question beats a spinner that never ends.

6. D6

   Decide on operations when quality ties.

   With quality within 0.1, reliability and latency are what teachers feel. So we shipped a primary and a fallback.

07 · ResultsWhat moved

• 53%of all assessment quizzes now come from the RAG generator
• 57case acceptance matrix I signed off for distractor quality
• 1M+curriculum-tagged questions in the grounding bank
• ~98%accuracy after the model upgrade (internal test)

I owned distractor-quality sign-off: every wrong option had to be plausible and relevant, never random. That’s what separates a question a teacher can use from one they have to rewrite.

08 · Lessons learnedWhat I’d tell the next player

• The bug was an assumption, not a model. “The curriculum describes what’s tested” held for Science and broke for languages. Understanding how teachers actually teach found the root cause faster than prompt tweaking.
• Evaluations disagree when they test different things. Reconcile the evidence (sample size, pipeline health) before you trust a leaderboard.
• Make quality model-independent. Grounding plus validators means we can switch to whichever model is most cost-efficient next quarter without re-litigating quality.