Monitor AI Crawler Policy Signals: llms.txt & HTTP Checks

Monitor AI crawler policy signals (llms.txt + HTTP checks) and get email alerts for changes

This n8n workflow watches selected websites for AI crawler policy updates—optionally including llms.txt presence and content hashes—and sends one SMTP email when baselines or changes are detected. It keeps persistent snapshots in an n8n Data Table so you can track evolution over time.

What this workflow does

• Runs on demand or hourly: A manual trigger or hourly tick (gated by policyCheckHour in Config) loads per-site snapshots once daily from your Data Table.
• Fetches public policy files safely: It retrieves rules with a descriptive User-Agent, using a 20-second timeout, paced requests, and up to three transient attempts (preserving successful responses during retries).
• Parses and matches crawler rules correctly: It parses user-agent groups case-insensitively, selects the most specific match, merges equally specific groups, falls back to *, strips comments, handles BOM and CRLF, ignores empty restrictions, and uses the longest matching root rule with Allow winning ties.
• Compares status + canonical paths: It checks bot status and canonical path lists. A robots.txt 404 means allow all.
• Optional llms.txt monitoring: It reports llms.txt presence and verifies content via a hash.
• Sends email only when needed: One SMTP email is sent if required; otherwise, no change sends no email. New baselines can be silent via Config.
• Persists successful snapshots: It upserts updated site records by siteKey, including robotsHash, policyJson, llmsTxt, and checkedAt.

Use cases

• SaaS operators tracking whether customers’ sites update AI crawling guidance.
• Automation engineers auditing robots.txt behavior and canonical allow/disallow paths for specific bots.
• Teams monitoring llms.txt adoption or changes in published policy content.

Technical details

• n8n requirement: n8n 2.8.4+
• Data source: n8n Data Tables (columns: siteKey, robotsHash, policyJson, llmsTxt, checkedAt)
• Email: SMTP credential integration (sent via n8n’s SMTP node)
• Core nodes mentioned: if, set, code, wait, slack, data table
• Setup guidance: Use SETUP.md; run the workflow twice manually before enabling the daily schedule.