Scrape, search and browse the web with a Firecrawl AI agent webhook
Turn any prompt into structured web data. Send a POST request with a natural language prompt and an optional JSON schema, and get back clean, structured results scraped from the web by an AI agent powered by Firecrawl. Use Cases Data Enrichment: Feed company names or URLs from your CRM and get back structured firmographic data (industry, funding, team size, tech stack). Lead Generation: Ask the agent to find pricing, contact pages, or product details for a list of competitors. Market Research: Extract structured pricing plans, feature comparisons, or product catalogs from any website. Content Aggregation: Pull structured news, events, or job postings from across the web on a schedule. Sales Intelligence: Enrich prospect lists with company info, recent news, or tech stack details before outreach. How It Works Receive Scrape Request receives a POST request with and an optional . Validate Output Schema checks the schema. If none is provided, it falls back to a permissive default. If the schema is malformed, it returns a clear error via Return Schema Error. Research & Extract Web Data takes the prompt and uses the full Firecrawl toolkit to research the web: - Search (): Finds relevant pages and sources across the web. - Scrape (): Extracts clean, structured content from any URL. - Interact (interactContext, interact, interactStop): Lets the agent interact with scraped pages in a live session. After scraping a page, the agent can click buttons, fill forms, navigate dynamic content, and extract data that static scraping cannot reach, all without managing sessions manually. This combination gives the AI agent complete web navigation capabilities. It can discover sources, read pages, and interact with dynamic content autonomously. Format Response to Schema (Structured Output Parser) formats the agent's response to match the provided (or default) schema. Return Structured Results sends the structured JSON back to the caller. Setup Requirements Firecrawl API Key: Sign up at firecrawl.dev and grab your API key. Connect it in the Firecrawl credential nodes. LLM Provider: Configure your Primary Chat Model and Fallback Chat Model nodes (e.g., OpenRouter, OpenAI, Anthropic). The template uses two model nodes for reliability, plus a separate Parser Chat Model for the output parser. n8n Instance: Self-hosted or cloud. Make sure the webhook node is set to accept POST requests. API Reference Endpoint Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | | string | Yes | Natural language instruction for the agent | | | object | No | JSON Schema defining the desired output structure | Response Returns a JSON object matching the provided schema, or a flexible object if no schema was given. Testing Examples Basic Request (No Schema) The agent decides the output structure on its own. Expected output: A JSON object with whatever structure the agent finds most appropriate for the data. Since no schema was provided, the internal default () is used. Request With a Custom Schema You define exactly the shape of data you want back. Expected output: Invalid Schema (String Instead of Object) Expected output: Invalid Schema (Array Instead of Object) Expected output: Same error response as above. Invalid Schema (Missing Property) Expected output: Same error response as above. Invalid Schema (Invalid Value) Expected output: Same error response as above. Workflow Architecture Schema Validation Logic The Validate Output Schema node runs this validation before passing data to the agent: If is missing or null, the default permissive schema is used: . If is present, it must be a JSON object (not a string, array, or primitive). It must have a property with a valid value: , , , , or . If validation fails, the workflow returns an error response with a helpful message and example schema. Notes The Format Response to Schema node (Structured Output Parser) requires the schema to be passed as a JSON string. The expression handles this conversion. The agent has access to Firecrawl's full toolkit: search, scrape, and interact. With all three connected, the agent has complete web navigation powers. It can discover sources via search, extract content via scrape, and interact with dynamic JavaScript-heavy pages via interact. The interact tools let the agent scrape a page first and then continue working with it in a live session, clicking buttons, filling forms, and navigating deeper, all without manual session management. The agent autonomously decides which tools to use based on the prompt. Response times vary depending on the complexity of the prompt and how many pages the agent needs to visit. Simple lookups take a few seconds; deep research can take longer.
Workflow JSON
{"id":"NGnt5dWPVpVGTwOF","meta":{"instanceId":"ba63db282a94e75c9ed970aa4cb5b2f9d6030188b3dfa060dd78ea9e1993f61a","templateCredsSetupCompleted":true},"name":"Scrape, search and browse the web with a Firecrawl AI agent webhook","tags":[],"nodes":[{"id":"b4b5e2a2-3f60-406f-81b4-f7fcad4f9144","name":"Receive Scrape Request","type":"n8n-nodes-base.webhook","position":[224,672],"webhookId":"48e70613-6302-4edc-9364-921e1e95929f","parameters":{"path":"scrape-agent","options":{},"httpMethod":"POST","responseMode":"responseNode"},"typeVersion":2.1},{"id":"82a9cc43-5e23-4fbb-8d35-aff63475013a","name":"Validate Output Schema","type":"n8n-nodes-base.code","onError":"continueErrorOutput","position":[448,672],"parameters":{"jsCode":"const body = $input.first().json.body;\n\n// No schema provided, use permissive default\nif (!body.output_schema) {\n return [{\n json: {\n prompt: body.prompt,\n output_schema: {\n type: \"object\",\n additionalProperties: true\n }\n }\n }];\n}\n\n// Validate the provided schema\nconst schema = body.output_schema;\n\nif (typeof schema !== \"object\" || Array.isArray(schema)) {\n throw new Error(\"output_schema must be a JSON object\");\n}\n\nif (!schema.type) {\n throw new Error(\"output_schema must have a 'type' property\");\n}\n\nconst validTypes = [\"object\", \"array\", \"string\", \"number\", \"boolean\"];\nif (!validTypes.includes(schema.type)) {\n throw new Error(`output_schema.type must be one of: ${validTypes.join(\", \")}`);\n}\n\n// Schema is valid, pass it through\nreturn [{\n json: {\n prompt: body.prompt,\n output_schema: schema\n }\n}];"},"typeVersion":2},{"id":"132edac3-6602-4963-bf8a-d9a3c3d28733","name":"Return Schema Error","type":"n8n-nodes-base.respondToWebhook","position":[640,800],"parameters":{"options":{},"respondWith":"json","responseBody":"{\n \"error\": true,\n \"message\": \"Invalid output_schema: must be a JSON object with a valid 'type' property (object, array, string, number, boolean)\",\n \"example_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"name\": { \"type\": \"string\" },\n \"price\": { \"type\": \"number\" }\n }\n }\n}"},"typeVersion":1.5},{"id":"5785b8ae-1c34-4587-83d4-a072c485ef46","name":"Research & Extract Web Data","type":"@n8n/n8n-nodes-langchain.agent","position":[992,656],"parameters":{"text":"={{ $('Receive Scrape Request').item.json.body.prompt }}","options":{"maxIterations":40,"systemMessage":"Use Firecrawl's tools to search, then scrape whatever the user's prompt requires. Make sure to answer the user's query in the best way possible. Consider using search to find the URL before trying to scrape if the user does not specify one. \n \nWhen using the search tool, at least one source must be selected (web, images, or news). \n\n Do not use the search tool if you already have the URL.\n\nIMPORTANT:\nIf a URL is provided, try using the scrape tool before using the search tool.\n\nCRITICAL: In Node/JS mode use process.stdout.write(data + \"\\n\") instead of console.log() to return data (console.log output is NOT captured), use unique variable names per call to avoid re-declaration errors in the persistent REPL, and NEVER put agent-browser commands or plain English in the Code field when Language is Node — agent-browser is Bash only.\n\nIf the website has heavy JavaScript or problems while fetching data (such as requiring interaction for pagination, clicking through elements, filling forms, or navigating deeper), only then use the Interact feature. First scrape the page to get a scrapeId, then use the Interact tool with a prompt or code to perform the needed interactions. \n\nALWAYS use the \"Interact Context\" tool to get full context on how to use the interact tools before using them."},"promptType":"define","needsFallback":true,"hasOutputParser":true},"typeVersion":3.1},{"id":"0cfe18d0-f86d-4eae-80e5-d2d3672fcda9","name":"Return Structured Results","type":"n8n-nodes-base.respondToWebhook","position":[1424,656],"parameters":{"options":{}},"typeVersion":1.5},{"id":"10f8ba2e-c1f3-4d9c-ad74-48fc9c0d422a","name":"Primary Chat Model","type":"@n8n/n8n-nodes-langchain.lmChatOpenRouter","position":[864,800],"parameters":{"model":"anthropic/claude-sonnet-4.6","options":{}},"credentials":{"openRouterApi":{"id":"sUeQtILma9E44HnE","name":"OpenRouter account"}},"typeVersion":1},{"id":"4b8f8364-1692-42fa-a005-3642a60390b5","name":"Fallback Chat Model","type":"@n8n/n8n-nodes-langchain.lmChatOpenRouter","position":[1024,800],"parameters":{"model":"anthropic/claude-haiku-4.5","options":{}},"credentials":{"openRouterApi":{"id":"sUeQtILma9E44HnE","name":"OpenRouter account"}},"typeVersion":1},{"id":"e7192283-1b70-4d50-abb8-e2fae7b24ae6","name":"Parser Chat Model","type":"@n8n/n8n-nodes-langchain.lmChatOpenRouter","position":[1376,912],"parameters":{"model":"anthropic/claude-haiku-4.5","options":{}},"credentials":{"openRouterApi":{"id":"sUeQtILma9E44HnE","name":"OpenRouter account"}},"typeVersion":1},{"id":"e023509d-a032-469f-aa06-ee10d10daebc","name":"Format Response to Schema","type":"@n8n/n8n-nodes-langchain.outputParserStructured","position":[1392,816],"parameters":{"autoFix":true,"schemaType":"manual","inputSchema":"={{ JSON.stringify($('Validate Output Schema').item.json.output_schema) }}"},"typeVersion":1.3},{"id":"d1b90a28-a9b1-4552-8ee2-f81b708932ac","name":"Sticky Note2","type":"n8n-nodes-base.stickyNote","position":[160,-32],"parameters":{"width":624,"height":624,"content":"## Firecrawl Interact Agent\nSend a POST request with a URL, a natural language prompt, and an optional JSON schema. An AI agent uses Firecrawl to scrape the page, then interact with it, clicking buttons, filling forms, navigating dynamic content, and extracting structured data that matches your schema.\n**Use cases:** form-driven data extraction, authenticated page scraping, paginated content collection, dynamic SPA navigation, multi-step checkout or search flows.\n## How it works\n1. **Receive Interact Request** accepts a POST with a `url`, `prompt`, and optional `output_schema`.\n2. **Validate Output Schema** checks if the schema is valid JSON. If missing, a permissive default is used. If malformed, a clear error is returned immediately.\n3. **Scrape & Interact** first scrapes the target URL to get initial content and a `scrapeId`, then uses Firecrawl's interact feature to click, fill forms, and navigate the live page based on the prompt. No manual session management is required.\n4. **Format Response to Schema** parses the agent's output into structured JSON matching the provided or default schema.\n5. **Return Structured Results** sends the final JSON back to the caller.\n## Setup steps\n1. **Firecrawl API key:** Sign up at [firecrawl.dev](https://www.firecrawl.dev) and connect your key to all Firecrawl credential nodes.\n2. **LLM providers:** Configure the Primary Chat Model and Fallback Chat Model nodes with your preferred provider (OpenRouter, OpenAI, Anthropic, etc.). Also set up the Parser Chat Model used by the output formatter.\n3. **Activate the workflow** and copy your webhook URL.\n4. **Test it:** Send a POST with a `url` and `prompt` to confirm everything works."},"typeVersion":1},{"id":"6f01b690-3570-447f-adbf-69fb5fb65bc3","name":"Sticky Note3","type":"n8n-nodes-base.stickyNote","position":[160,608],"parameters":{"color":7,"width":624,"height":304,"content":"## Input & Validation"},"typeVersion":1},{"id":"1854ff48-aa38-493b-b42d-6919c3e5884e","name":"Sticky Note4","type":"n8n-nodes-base.stickyNote","position":[800,608],"parameters":{"color":7,"width":512,"height":304,"content":"## AI Agent & LLM Models"},"typeVersion":1},{"id":"89dedb88-3334-472f-9e81-684dd069743c","name":"Sticky Note5","type":"n8n-nodes-base.stickyNote","position":[1328,608],"parameters":{"color":7,"width":336,"height":416,"content":"## Output & Response"},"typeVersion":1},{"id":"01de6598-5edf-4bb0-a91c-032027a1d1f0","name":"Sticky Note6","type":"n8n-nodes-base.stickyNote","position":[816,1040],"parameters":{"color":7,"width":848,"height":640,"content":"## Firecrawl Tools"},"typeVersion":1},{"id":"2be3ff3c-deb6-4057-84f6-b60ad61e7143","name":"Sticky Note","type":"n8n-nodes-base.stickyNote","position":[1056,1360],"parameters":{"color":5,"width":592,"height":304,"content":"## Browser Tool"},"typeVersion":1},{"id":"626fbcc2-952a-44d9-9278-4d62182e14af","name":"Sticky Note1","type":"n8n-nodes-base.stickyNote","position":[960,1088],"parameters":{"color":4,"width":352,"height":240,"content":"## Search & Scrape"},"typeVersion":1},{"id":"db7882fd-6b95-4571-86c0-d12c37db0c0e","name":"/scrape with Firecrawl","type":"@mendable/n8n-nodes-firecrawl.firecrawlTool","position":[1008,1168],"parameters":{"url":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('URL', ``, 'string') }}","parsers":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Parsers', ``, 'string') }}","operation":"scrape","requestOptions":{}},"credentials":{"firecrawlApi":{"id":"2SnLAelRpQ8H5tju","name":"Firecrawl account"}},"typeVersion":1},{"id":"466ac39c-e56f-4c59-b6f7-236e68131f5d","name":"/search with Firecrawl","type":"@mendable/n8n-nodes-firecrawl.firecrawlTool","position":[1184,1168],"parameters":{"query":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Query', ``, 'string') }}","sources":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Sources', ``, 'string') }}","resource":"MapSearch","operation":"search","requestOptions":{}},"credentials":{"firecrawlApi":{"id":"2SnLAelRpQ8H5tju","name":"Firecrawl account"}},"typeVersion":1},{"id":"24377630-a048-4a29-8374-7e62d7a83008","name":"Interact context with Firecrawl","type":"@mendable/n8n-nodes-firecrawl.firecrawlTool","position":[1136,1408],"parameters":{"resource":"Interact","requestOptions":{}},"credentials":{"firecrawlApi":{"id":"2SnLAelRpQ8H5tju","name":"Firecrawl account"}},"typeVersion":1},{"id":"26069d9a-ae64-423e-9172-9cc2a920d918","name":"Execute interaction with Firecrawl","type":"@mendable/n8n-nodes-firecrawl.firecrawlTool","position":[1296,1488],"parameters":{"code":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Code', ``, 'string') }}","resource":"Interact","scrapeId":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Scrape_ID', ``, 'string') }}","operation":"interact","interactMode":"code","requestOptions":{}},"credentials":{"firecrawlApi":{"id":"2SnLAelRpQ8H5tju","name":"Firecrawl account"}},"typeVersion":1},{"id":"9b777540-a930-40c0-8e18-c6692b6ac8bb","name":"Stop interaction with Firecrawl","type":"@mendable/n8n-nodes-firecrawl.firecrawlTool","position":[1472,1488],"parameters":{"resource":"Interact","scrapeId":"={{ /*n8n-auto-generated-fromAI-override*/ $fromAI('Scrape_ID', ``, 'string') }}","operation":"interactStop","requestOptions":{}},"credentials":{"firecrawlApi":{"id":"2SnLAelRpQ8H5tju","name":"Firecrawl account"}},"typeVersion":1}],"active":true,"pinData":{"Receive Scrape Request":[{"json":{"body":{"prompt":"Go to airank.dev and get the top 20 first LLMs from the main table in the home page"},"query":{},"params":{},"headers":{"host":"quickconnect1.stage-app.n8n.cloud","accept":"*/*","cf-ray":"9e1e6002da0b0d0b-GRU","cdn-loop":"cloudflare; loops=1; subreqs=1","cf-ew-via":"15","cf-worker":"n8n.cloud","x-real-ip":"177.200.39.12","user-agent":"curl/8.7.1","cf-ipcountry":"BR","content-type":"application/json","content-length":"59","x-warp-trusted":"yes","accept-encoding":"gzip, br","x-forwarded-for":"177.200.39.12, 172.69.39.2","x-warp-provider":"cloudflare","cf-connecting-ip":"177.200.39.12","x-forwarded-host":"quickconnect1.stage-app.n8n.cloud","x-forwarded-port":"443","x-forwarded-proto":"https","x-forwarded-server":"traefik-stage-users-gwc-1-7cb4dbf9bf-6dqgk"},"webhookUrl":"https://quickconnect1.stage-app.n8n.cloud/webhook-test/scrape-agent","executionMode":"test"}}]},"settings":{"binaryMode":"separate","availableInMCP":false,"executionOrder":"v1"},"versionId":"823b4c27-e6a2-41a3-a7e1-ac3bec06d801","connections":{"Parser Chat Model":{"ai_languageModel":[[{"node":"Format Response to Schema","type":"ai_languageModel","index":0}]]},"Primary Chat Model":{"ai_languageModel":[[{"node":"Research & Extract Web Data","type":"ai_languageModel","index":0}]]},"Fallback Chat Model":{"ai_languageModel":[[{"node":"Research & Extract Web Data","type":"ai_languageModel","index":1}]]},"/scrape with Firecrawl":{"ai_tool":[[{"node":"Research & Extract Web Data","type":"ai_tool","index":0}]]},"/search with Firecrawl":{"ai_tool":[[{"node":"Research & Extract Web Data","type":"ai_tool","index":0}]]},"Receive Scrape Request":{"main":[[{"node":"Validate Output Schema","type":"main","index":0}]]},"Validate Output Schema":{"main":[[{"node":"Research & Extract Web Data","type":"main","index":0}],[{"node":"Return Schema Error","type":"main","index":0}]]},"Format Response to Schema":{"ai_outputParser":[[{"node":"Research & Extract Web Data","type":"ai_outputParser","index":0}]]},"Research & Extract Web Data":{"main":[[{"node":"Return Structured Results","type":"main","index":0}]]},"Interact context with Firecrawl":{"ai_tool":[[{"node":"Research & Extract Web Data","type":"ai_tool","index":0}]]},"Stop interaction with Firecrawl":{"ai_tool":[[{"node":"Research & Extract Web Data","type":"ai_tool","index":0}]]},"Execute interaction with Firecrawl":{"ai_tool":[[{"node":"Research & Extract Web Data","type":"ai_tool","index":0}]]}}}How to Import This Workflow
- 1Copy the workflow JSON above using the Copy Workflow JSON button.
- 2Open your n8n instance and go to Workflows.
- 3Click Import from JSON and paste the copied workflow.
Don't have an n8n instance? Start your free trial at n8nautomation.cloud
Related Templates
Auto-create TikTok videos with VEED.io AI avatars, ElevenLabs & GPT-4
Automate the creation and distribution of trending TikTok videos using AI avatars. This workflow connects Telegram, Perplexity, OpenAI, ElevenLabs, VEED.io, and BLOTATO to generate scripts, synthesize voice, create video, and publish across multiple social platforms. Content creators and marketers can rapidly produce engaging short-form video content without manual editing.
IT Ops AI SlackBot Workflow - Chat with your knowledge base
Empower your IT operations team to instantly access information by building an AI-powered Slackbot that chats with your Confluence knowledge base. This productivity workflow integrates OpenAI and Slack, using a webhook trigger to initiate conversations. When a direct message is received in Slack, the workflow verifies the webhook, sends an initial message, and then an AI Agent processes the user's query. The AI Agent leverages a Window Buffer Memory to maintain context and can call a Confluence Workflow Tool to retrieve relevant information from your knowledge base. After processing, the initial message is deleted, and the AI Agent sends a comprehensive response back to the user in Slack. This automation streamlines IT support, reduces response times, and frees up valuable IT staff by allowing them to quickly find answers to common questions without manual searching, ultimately improving efficiency and user satisfaction within your organization.
Manage Google Calendar and Gmail from Telegram with a Claude AI assistant
Manage your Google Calendar and Gmail directly from Telegram using a Claude AI assistant. This workflow connects Telegram for input, OpenAI and LmChatOpenRouter for AI processing, and Google Calendar and Gmail for managing events and emails. Users can schedule meetings, check their calendar, or reply to emails on the go, making personal and professional organization more efficient.