AI-Powered Web Crawler for Social Media Link Extraction

Transform any website into a goldmine of social media contact data with this intelligent AI-powered web crawler that autonomously navigates websites to extract social media profile links with precision and speed.

What this workflow does

This advanced n8n automation workflow deploys an AI agent equipped with two specialized tools to systematically crawl websites and extract social media profile links. The text tool retrieves all textual content from web pages, while the URLs tool extracts every discoverable link. The AI agent intelligently navigates through website structures, identifying relevant subpages (like contact or about pages) that typically contain social media links, then extracts the specific information using natural language processing.

The workflow uses Supabase as the default storage solution for both input URLs and extracted data, though you can easily configure any database of your choice. The agent's behavior is fully customizable through prompt engineering and JSON schema modifications, allowing you to extract different types of contact information beyond social media links.

Use cases

• Lead generation: Build comprehensive social media contact databases for outreach campaigns
• Competitor analysis: Map competitor social media presence across platforms
• Partnership prospecting: Identify potential business partners through their social channels
• Content creator outreach: Compile influencer contact information for marketing campaigns
• Market research: Analyze social media adoption patterns across industry websites

Technical details

Built using core n8n nodes including set, html, merge, filter, markdown, and split out nodes. The workflow integrates seamlessly with OpenAI for AI-powered decision making and Supabase for data storage. The modular architecture allows you to split agent tools into separate workflows for enhanced performance and customization.

Setup requires connecting your database with website URLs, configuring the crawling agent with custom prompts and parsing schemas, and setting OpenAI credentials. The workflow is demonstrated in an accompanying YouTube tutorial for easy implementation.