Daily RAG Quality Monitoring with OpenAI, Data Tables & Telegram

Daily RAG Quality Monitoring that alerts you when your support bot degrades

This n8n workflow automatically runs every day (or on demand) to evaluate your RAG-style support bot using OpenAI for embeddings, answer generation, and judging—then saves results in n8n Data Tables and sends a Telegram alert if retrieval accuracy or answer quality drops.

What this workflow does

• Runs daily with configurable thresholds: Sets monitoring limits such as top-K retrieval size, minimum quality scores, and an allowed pass-rate drop.
• Builds a test retrieval environment: Loads sample help center articles, generates OpenAI embeddings, and indexes them into an in-memory vector store.
• Evaluates a golden test set: For each golden question, the workflow retrieves relevant context from the vector store and generates an answer with an OpenAI chat model constrained to the retrieved text (groundedness expectation).
• Judges response quality with a second model: Uses a separate OpenAI chat model to score groundedness, correctness, and whether the bot abstains appropriately for out-of-scope questions.
• Aggregates run metrics: Produces pass rate, retrieval hit rate, average scores, abstention rate, and failing cases.
• Tracks regressions & notifies: Stores the run summary in an n8n Data Table, compares it to the previous run, and sends a Telegram message when status is marked as degraded.

Use cases

• Keep a RAG support bot reliable after document updates, prompt changes, or model upgrades.
• Detect retrieval regressions early by monitoring hit rate and pass-rate drop over time.
• Provide SaaS operators and automation engineers with a daily quality dashboard and instant Telegram alerts.

Technical details

• Integrations: OpenAI (embeddings + answer model + judging model) and Telegram (alerts).
• n8n nodes used: if, set, code, no op, telegram, and data table (n8n Data Tables for run-level metrics).
• Setup: Configure OpenAI credentials for embeddings and both chat models, add Telegram credentials/chat ID, and ensure Data Tables are available in your instance.