{"product_id":"daily-rag-quality-monitoring-with-openai-data-tables-telegram","title":"Daily RAG Quality Monitoring with OpenAI, Data Tables \u0026 Telegram","description":"\u003ch3\u003eDaily RAG Quality Monitoring that alerts you when your support bot degrades\u003c\/h3\u003e\n\u003cp\u003eThis n8n workflow automatically runs every day (or on demand) to evaluate your RAG-style support bot using OpenAI for embeddings, answer generation, and judging—then saves results in n8n Data Tables and sends a Telegram alert if retrieval accuracy or answer quality drops.\u003c\/p\u003e\n\n\u003ch3\u003eWhat this workflow does\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cb\u003eRuns daily with configurable thresholds:\u003c\/b\u003e Sets monitoring limits such as top-K retrieval size, minimum quality scores, and an allowed pass-rate drop.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eBuilds a test retrieval environment:\u003c\/b\u003e Loads sample help center articles, generates OpenAI embeddings, and indexes them into an in-memory vector store.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eEvaluates a golden test set:\u003c\/b\u003e For each golden question, the workflow retrieves relevant context from the vector store and generates an answer with an OpenAI chat model constrained to the retrieved text (groundedness expectation).\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eJudges response quality with a second model:\u003c\/b\u003e Uses a separate OpenAI chat model to score groundedness, correctness, and whether the bot abstains appropriately for out-of-scope questions.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eAggregates run metrics:\u003c\/b\u003e Produces pass rate, retrieval hit rate, average scores, abstention rate, and failing cases.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eTracks regressions \u0026amp; notifies:\u003c\/b\u003e Stores the run summary in an n8n Data Table, compares it to the previous run, and sends a \u003cb\u003eTelegram\u003c\/b\u003e message when status is marked as degraded.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eUse cases\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eKeep a \u003cb\u003eRAG support bot\u003c\/b\u003e reliable after document updates, prompt changes, or model upgrades.\u003c\/li\u003e\n  \u003cli\u003eDetect retrieval regressions early by monitoring hit rate and pass-rate drop over time.\u003c\/li\u003e\n  \u003cli\u003eProvide SaaS operators and automation engineers with a daily quality dashboard and instant Telegram alerts.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eTechnical details\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cb\u003eIntegrations:\u003c\/b\u003e OpenAI (embeddings + answer model + judging model) and Telegram (alerts).\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003en8n nodes used:\u003c\/b\u003e if, set, code, no op, telegram, and data table (n8n Data Tables for run-level metrics).\u003c\/li\u003e\n  \u003cli\u003e\n\u003cb\u003eSetup:\u003c\/b\u003e Configure OpenAI credentials for embeddings and both chat models, add Telegram credentials\/chat ID, and ensure Data Tables are available in your instance.\u003c\/li\u003e\n\u003c\/ul\u003e","brand":"N8N Commerce","offers":[{"title":"Default Title","offer_id":46153933947059,"sku":"N8N-20326","price":55.99,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0749\/6279\/6723\/files\/PDEpF2F-6Sz8fjExZrixa_Vf8pNRdP.png?v=1790931729","url":"https:\/\/buyflowscripts.com\/products\/daily-rag-quality-monitoring-with-openai-data-tables-telegram","provider":"N8N Commerce","version":"1.0","type":"link"}