Recursive Web Scraper n8n Workflow with Airtop & Google Sheets

Recursively scrape websites from a URL list—store page content, discover new links, and iterate automatically with n8n

This Recursive Web Scraper n8n workflow with Airtop & Google Sheets reads a starting URL from Google Sheets, scrapes each page, saves the results to a document, then extracts matching links from the scraped pages and appends them back to the sheet—repeating for your configured Depth.

What this workflow does

Built for recursive depth collection (content across multiple linked pages), the automation follows this flow:

• Seed URL: Starts by reading the Seed URL from the Google Sheet.
• Scrape and store: Scrapes each page and saves its content into the specified document (via Google Docs).
• Filter discovered links: Extracts newly discovered links from each page that match the Links must contain string.
• Update Google Sheet: Appends those newly found links back into the Google Sheet.
• Iterate by depth: Repeats scraping and link discovery for Depth - 1 additional iterations (layers of links).

Use cases

• Content aggregation: Collect structured page content across a site section by limiting link discovery with a keyword or URL fragment.
• Lead generation & research: Build a growing list of relevant pages by recursively following internal links that match your criteria.
• Competitive research: Gather documentation or topic pages by specifying a “must contain” link pattern and scraping multiple layers deep.

Technical details

Designed for n8n users and automation engineers, this workflow uses:

• Google Sheets (reads Seed URL and appends newly discovered links)
• Google Docs (stores scraped page content; requires Google credentials)
• Airtop (requires an Airtop API Key)
• n8n nodes: if, set, code, airtop, google docs, plus sticky note for workflow guidance

Setup requirements: Generate an Airtop API Key, configure Google Docs credentials in Google Console, and set your Seed URL, Links must contain, and `Depth` in the Google Sheet.