AI Chatbot Builder: Auto-Index Websites with Firecrawl & Qdrant

Transform any website into an intelligent AI chatbot that can answer customer questions using your existing content. This n8n workflow automatically crawls and indexes your entire website using Firecrawl and Qdrant vector database, then powers a conversational RAG (Retrieval-Augmented Generation) chatbot that searches your content in real-time and escalates unresolved issues to your support team via Gmail.

What this workflow does

The indexing pipeline starts by mapping every link across your defined root domains using Firecrawl, giving you full visibility of what will be indexed before scraping begins. It removes duplicate URLs across all domains, then scrapes each unique page individually, converting content to clean markdown. Content gets chunked into overlapping segments using Recursive Character Text Splitter (1000 characters, 200 overlap) to preserve context. Mistral's codestral-embed-2505 model converts each chunk into vector embeddings, which are stored in Qdrant Cloud in batches of 100. A Wait node paces the process to avoid API rate limits on large sites.

The AI chatbot uses a public Chat Trigger that generates an embeddable URL for your website. GPT-4o-mini processes messages with a 10-message memory window for natural conversations. The AI Agent intelligently searches the Qdrant vector store only when questions require it, retrieving the top 3 most relevant content chunks per query.

Use cases

• Customer support automation for SaaS platforms and e-commerce sites
• Knowledge base chatbots for documentation and help centers
• Lead qualification bots that answer product questions instantly
• Internal company wikis with conversational search capabilities

Technical details

• Integrations: Firecrawl for web scraping, Qdrant Cloud for vector storage, Mistral AI for embeddings, OpenAI GPT-4o-mini for conversations, Gmail for support escalation
• Key nodes: Code, Set, Wait, Split Out, Gmail Tool, Sticky Note
• Features: RAG architecture, automatic content chunking, rate limiting, conversation memory, intelligent query routing