{"product_id":"audit-ai-search-bot-access-with-robots-txt-gmail-reports-n8n","title":"Audit AI \u0026 Search Bot Access with Robots.txt + Gmail Reports (n8n)","description":"\u003ch3\u003eAudit AI \u0026amp; Search Bot Access to Your Pages—Automatically, Monthly, With Gmail Reports\u003c\/h3\u003e\n\u003cp\u003eThis n8n workflow checks whether popular AI and search crawlers can access a specific URL by comparing \u003cem\u003erobots.txt\u003c\/em\u003e rules against real HTTP responses, then emails you a clear verdict report using \u003cstrong\u003eGmail\u003c\/strong\u003e—on a monthly schedule.\u003c\/p\u003e\n\n\u003ch3\u003eWhat this workflow does\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cstrong\u003eRuns monthly\u003c\/strong\u003e using a scheduled trigger.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eConfigures the audit\u003c\/strong\u003e by setting a \u003cem\u003etestUrl\u003c\/em\u003e, a list of bot \u003cem\u003eUser-Agent\u003c\/em\u003e strings to test, a \u003cem\u003ebrowser user-agent control\u003c\/em\u003e, and a \u003cem\u003esecondsBetweenRequests\u003c\/em\u003e delay to reduce rate limiting.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eFetches robots.txt\u003c\/strong\u003e from the target site and evaluates whether the page is allowed for each bot using longest-match and wildcard fallback rules.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003ePerforms real page requests\u003c\/strong\u003e once per bot using that bot’s User-Agent, pacing the requests according to your delay setting.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eCollects control signals\u003c\/strong\u003e by making one additional request using a normal browser User-Agent, then compares bot HTTP status and response size to the control.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eClassifies results\u003c\/strong\u003e for each bot as \u003cstrong\u003eALLOWED\u003c\/strong\u003e, \u003cstrong\u003eBLOCKED\u003c\/strong\u003e, \u003cstrong\u003ePARTIAL\u003c\/strong\u003e, \u003cstrong\u003eCONTRADICTION\u003c\/strong\u003e, \u003cstrong\u003eRATE_LIMITED\u003c\/strong\u003e, or \u003cstrong\u003eUNKNOWN\u003c\/strong\u003e, and aggregates a summary.\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eSends the full audit report\u003c\/strong\u003e to your configured recipient via the Gmail node.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eUse cases\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003eVerify that AI crawler access matches your intended robots.txt policy after site changes.\u003c\/li\u003e\n  \u003cli\u003eSpot crawlers that are \u003cem\u003eblocked despite robots.txt allowing\u003c\/em\u003e (or vice versa) by catching contradictions.\u003c\/li\u003e\n  \u003cli\u003eMonitor for bot-specific rate limiting by observing RATE_LIMITED classifications over time.\u003c\/li\u003e\n\u003c\/ul\u003e\n\n\u003ch3\u003eTechnical details\u003c\/h3\u003e\n\u003cul\u003e\n  \u003cli\u003e\n\u003cstrong\u003eScheduling:\u003c\/strong\u003e monthly schedule trigger\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eNodes\/logic:\u003c\/strong\u003e if, set, code, gmail, merge, aggregate\u003c\/li\u003e\n  \u003cli\u003e\n\u003cstrong\u003eSetup:\u003c\/strong\u003e add a Gmail OAuth2 credential, set the recipient address in the email step, update \u003cem\u003etestUrl\u003c\/em\u003e, tune the request delay, and edit the bot User-Agent list as needed.\u003c\/li\u003e\n\u003c\/ul\u003e","brand":"N8N Commerce","offers":[{"title":"Default Title","offer_id":46164213694643,"sku":"N8N-20481","price":61.99,"currency_code":"GBP","in_stock":true}],"thumbnail_url":"\/\/cdn.shopify.com\/s\/files\/1\/0749\/6279\/6723\/files\/Lsph_TxYW1k7DcFNVM3Go_BFQru3LQ.png?v=1791278300","url":"https:\/\/buyflowscripts.com\/products\/audit-ai-search-bot-access-with-robots-txt-gmail-reports-n8n","provider":"N8N Commerce","version":"1.0","type":"link"}