CS Chatbot for a Performance Marketing Company

Hybrid RAG ON/OFF & o1 Routing Architecture

Role	AI Product Manager / Workflow Architect
Context	Performance Marketing Agency CS Automation
Stage	Evaluation & Production Architecture
Time	2024.11

Summary	Redesigned a customer-service chatbot by routing inquiry types across RAG ON/OFF paths via an o1 router.
Scope	Lifted end-to-end answer accuracy from 27% to 85% through evaluation-driven routing.
Outcome	Answer accuracy 27% → 85%
Tags	Product, AI Workflow, Evaluation System, RAG & Routing

Problem

Capability thesis	Diagnosed why a naive all-RAG chatbot failed in production (27% accuracy) and engineered an evaluation-driven hybrid routing architecture that reached 85% accuracy.
TL;DR	Instead of endlessly tweaking a single prompt, decomposed customer inquiries by failure mode, split execution into RAG ON vs. RAG OFF pipelines, and placed a reasoning model (o1) strictly as a deterministic router—raising answer accuracy from 27% to 85%.
What	An AI customer-support workflow for a performance marketing agency that classifies incoming client messages by risk and evidence requirements before generating a response.
Why now	Agencies handle high volumes of repetitive yet context-sensitive client questions; naive single-pipeline LLM bots hallucinate policy details or inject irrelevant retrieved chunks into simple greeting/triage turns.
Problem	The baseline chatbot forced every user message through vector retrieval (RAG ON) and a single generation prompt, resulting in a 27% accuracy rate due to retrieval noise on non-factual queries and weak reasoning on multi-condition policy questions.

Role & ownership

Role	AI Product / Workflow Designer
Team	Cross-functional AI implementation unit
Ownership	Owned failure-case taxonomy, evaluation dataset & scoring rubric, RAG ON/OFF split architecture, and o1 router prompt/schema design.
Constraints	Strict requirement to prevent hallucinated contract/refund promises, plus latency and token-cost limits on reasoning models.

Validation loops

Supported

Loop 1 — Separating RAG ON vs. RAG OFF Execution Paths

Separating inquiries by whether they genuinely require document evidence (RAG ON vs. RAG OFF) will eliminate context-pollution errors.

Discover

Why	Understand why the baseline chatbot failed on 73% of test cases.
Actions	Audited failure logs and categorized every wrong answer by root cause (retrieval noise vs. missing policy vs. reasoning failure).
Outputs	Found that queries needing no external document were being derailed by irrelevant RAG chunks.

Define

Why	Establish clean execution boundaries.
Actions	Defined two distinct pipelines: RAG ON (strictly grounded in policy docs) and RAG OFF (conversational triage and general guidance).
Outputs	Dual-path response specification and evaluation rubric.

Run

Why	Verify accuracy gains when each query is sent to its proper path.
Actions	Benchmarked the separated pipelines against the baseline test suite.
Outputs	Confirmed substantial accuracy recovery once unnecessary RAG injection was removed.

Review

Why	Identify the remaining bottleneck for full automation.
Actions	Analyzed edge cases in path selection.
Outputs	Determined that a high-precision router was needed to select the right path automatically in production.

Supported

Loop 2 — Automating Path Selection with a Constrained o1 Router

Using a reasoning model (o1) strictly to output routing decisions—rather than generating final prose—will maximize routing accuracy while preserving tone control.

Discover

Why	Define the exact signals that determine which execution path an inquiry belongs to.
Actions	Mapped inquiry intent, policy-dependency, and risk level into explicit routing criteria.
Outputs	Structured routing decision tree.

Design & Build

Why	Prevent the router from hallucinating or altering response style.
Actions	Configured the o1 router to emit only a deterministic route classification, handing final response generation to path-specific prompts.
Outputs	Hybrid o1 Router + RAG ON/OFF production workflow.

Run

Why	Measure end-to-end accuracy against the evaluation benchmark.
Actions	Ran the full test suite through the automated router + dual-path generator.
Outputs	Achieved 85% end-to-end answer accuracy (up from 27% baseline).

Review

Why	Document architectural rules for ongoing maintenance.
Actions	Codified the evaluation sheet and routing rules for operations handoff.
Outputs	Production-ready CS routing architecture and evaluation playbook.

Design decisions

Problem zoom	Realized the bottleneck was not 'prompt wording'—it was mixing queries that need strict document grounding with queries that are harmed by noisy retrieval context.
Options	(1) Keep single RAG pipeline and tune chunk sizes/prompts, (2) Use o1 to generate every full response directly (too slow and hard to constrain tone), (3) Split RAG ON/OFF paths and use o1 strictly as a structured router.
Trade-off	Adding a routing step introduced a small classification hop, but restricting o1 to output only a routing token kept latency controlled while dramatically eliminating retrieval pollution.
Final rules	(1) Never run RAG on queries that do not require document evidence; (2) Router outputs only the path decision and required constraints, never the final customer-facing prose; (3) Score every architecture change against the fixed evaluation set.
Tools	OpenAI API (o1 / GPT-4o), RAG Vector Pipeline, Structured Evaluation Sheets
Frameworks	Evaluation-Driven Development (EDD), Intent & Risk Routing, RAG Pollution Control

Outcomes

Metrics	End-to-end answer accuracy increased from 27% (baseline single-pipeline bot) to 85% (hybrid RAG ON/OFF + o1 router).
Qualitative outcome	Eliminated retrieval-induced hallucinations on general inquiries and established a repeatable evaluation harness for future prompt/model upgrades.

Learnings

Learnings	In production LLM systems, architectural decomposition and evaluation rubrics beat monolithic prompt engineering every time.
If again	Add a confidence-threshold escalation branch that routes ambiguous high-risk billing disputes directly to a human manager with a pre-filled summary.

Evidence

Before	Monolithic RAG-for-everything pipeline suffering from context pollution and 27% answer accuracy.
After	Deterministic o1 router dispatching between RAG ON and RAG OFF execution paths, achieving 85% verified accuracy.