Effortless Data Scraping & Storage Automation with n8n

Transform massive multi-page websites into organized data with this comprehensive n8n workflow that automatically scrapes, caches, and stores information from TheSwiftCodes.com across all countries and pages. Perfect for data professionals who need reliable, scalable web scraping automation without manual intervention.

What This Workflow Does

This intelligent data scraping workflow systematically extracts SWIFT code information from TheSwiftCodes.com by:

• Starting with the complete country list from the browse-by-country page
• Loading each individual country page (like Albania, etc.)
• Automatically paginating through every page within each country section
• Extracting structured data from all country pages
• Storing extracted data directly into MongoDB with duplicate prevention
• Using intelligent caching system to avoid redundant requests

The workflow leverages n8n's getWorkflowStaticData('global') method to maintain pagination state across executions, ensuring seamless continuation from the previous page. Built-in cache functionality saves visited pages to local disk, dramatically reducing execution time on subsequent runs.

Key Features & Benefits

• Smart Caching: Local disk cache system prevents unnecessary web requests
• Duplicate Prevention: Validates swift_code uniqueness before MongoDB insertion
• Automated Scheduling: Compatible with Cron nodes for weekly data updates
• Proxy-Ready: Designed for proxy integration (Scrapoxy.io recommended) to avoid IP blocks
• Pagination Mastery: Handles complex multi-level pagination automatically

Ideal Use Cases

• Financial data aggregation and SWIFT code database building
• Banking industry research and compliance data collection
• Small to medium-scale data scraping projects
• Automated competitor intelligence gathering

Technical Stack

Built with essential n8n nodes: IF conditions, Set operations, HTTP requests, MongoDB integration, Function nodes for custom logic, and HTML Extract for data parsing. Optimized for small data requirements with scalable architecture.

Note: For dynamic content scraping, consider headless browser solutions as complementary tools.