n8n CI Webhook: Gemini Prompt Grading & Regression Checks

Run Gemini prompt tests in your CI—automatically grade outputs, catch regressions, and fail the build when quality drops

This n8n CI Webhook: Gemini Prompt Grading & Regression Checks exposes a secured webhook your CI pipeline can call to execute a prompt test suite against Google Gemini. It grades each output against your rules, detects regressions versus the previous run, and returns an HTTP status code (pass or fail) to gate deployments.

What this workflow does

• Receives a CI POST request to a webhook secured with header authentication.
• Validates and expands the payload into individual test cases (including optional repeated runs per case) by filling the prompt with each input.
• Executes the prompt-under-test for each case using Google Gemini (PaLM).
• Grading pass with a separate Gemini pass that strictly evaluates each output against the provided rules, returning structured results (pass/fail, severity, broken rules).
• Aggregates into a scorecard, flags unstable inputs across repeated runs, and compares per-input verdicts against the last stored run to detect regressions.
• Returns a build gate response: HTTP 200 with the scorecard when the pass-rate threshold is met and no regressions are found; otherwise HTTP 422 with a failure report plus a Gemini-generated explanation and rewritten prompt.

Use cases

• Prevent prompt regressions by failing CI when Gemini outputs no longer match grading rules.
• Continuously validate prompt changes before deploying a SaaS feature that relies on Gemini.
• Detect flaky/unstable prompt behavior via repeated runs per input and severity-based broken rule reporting.

Technical details

• n8n nodes/workflow: if, code, webhook, sticky note, split in batches, respond to webhook.
• Integrations: Google Gemini (PaLM) for both the model-under-test and grading/rewrite steps.
• Setup requirement: activate the workflow so n8n static data persists between runs for regression comparison.