n8n Webhook RAG Evaluator: Ollama + PostgreSQL Scores

Evaluate your RAG outputs with an n8n Webhook—Ollama scoring + PostgreSQL storage

This n8n workflow exposes a POST webhook that automatically evaluates Retrieval-Augmented Generation (RAG) responses using a local Ollama model. It scores groundedness, retrieval relevance, answer relevance, hallucination, and citation quality—then saves the full evaluation in PostgreSQL and returns the stored results as JSON.

What this workflow does

• Receives a webhook request containing a JSON payload (question, answer, retrieved context chunks, optional citations, and metadata).
• Validates the request for required fields, size limits, and unique chunk IDs; returns 400 if validation fails.
• Calls Ollama for evaluation by sending the question, answer, citations, and retrieved context to score:
  • Groundedness
  • Retrieval relevance
  • Answer relevance
  • Hallucination detection
  • Citation quality
• Retries once with a stricter prompt if Ollama’s response doesn’t match the required JSON schema.
• Aggregates results into an overall score and grade, deduplicates unsupported claims, and generates issues and recommendations.
• Stores evaluations in PostgreSQL (including evaluation_id, scores, issues, and recommendations) and returns the stored evaluation.
• Returns 503 if Ollama is unavailable or if the PostgreSQL insert fails.

Use cases

• Quality assurance for a SaaS RAG pipeline (before shipping an answer to users).
• Automated hallucination and citation quality checks for internal knowledge assistants.
• Batch evaluation of generated responses during development and iteration of prompts/retrievers.

Technical details

• Webhook endpoint for receiving evaluation inputs
• Nodes used: if, code, webhook, postgres, http request, respond to webhook
• Ollama integration: expects a running instance reachable at http://ollama:11434 using an evaluation model (SOURCE indicates “an evaluation model a”).