Vision-Based AI Agent Web Scraper with Google Sheets & Gemini

Overview

This workflow automates the extraction of structured data from webpages using a vision-based AI agent. It combines Google Sheets for managing input URLs and storing results, ScrapingBee for capturing full-page screenshots (and optional HTML fallback), and Google Gemini 1.5 Pro for intelligent data parsing. The AI agent primarily works by analyzing webpage screenshots—making it ideal for dynamic or JavaScript-heavy sites—and falls back to HTML scraping only when the screenshot is insufficient.

This approach is particularly useful for e-commerce scraping, product monitoring, price tracking, or any scenario where you need to extract specific fields (e.g., titles, prices, brands, promotional info) from multiple pages without writing custom selectors.

Workflow Steps

1. When clicking ‘Test workflow’ — Manual Trigger (No auth)

   • Starts the workflow on demand. Replace this with a scheduled trigger for automation.

2. Google Sheets – Get list of URLs — Google Sheets (OAuth2 / Service Account)

   • Reads a list of target URLs from a Google Sheet (List of URLs sheet). Configured with a service account credential.

3. Set fields — Set (No auth)

   • Maps the url field from the sheet row for downstream nodes.

4. ScrapingBee – Get page screenshot — HTTP Request (API Key auth)

   • Calls ScrapingBee API with api_key, url, and screenshot_full_page=true to capture a full-page screenshot. Also sends a custom User-Agent header.

5. Vision-based Scraping Agent — AI Agent (No auth)

   • The core node. It receives the screenshot URL (from the screenshot node) and uses the attached LLM and tools to extract data. The system prompt instructs the agent to extract product details from the screenshot and fall back to HTML if needed.

6. Google Gemini Chat Model — LLM (API Key auth)

   • Powers the vision-based extraction using gemini-1.5-pro-latest. Credentials require a Google AI API key.

7. HTML-based Scraping Tool — Workflow Tool (No auth)

   • A sub-workflow tool that the agent calls when screenshot extraction is insufficient. It returns HTML content (converted to Markdown) as a fallback.

8. Structured Output Parser — Output Parser (No auth)

   • Defines a JSON schema for the extracted data (e.g., product_title, product_price, promo, promo_percent). Ensures the agent returns data in a consistent format.

9. Split Out — Split Out (No auth)

   • Splits the array of extracted products (from the parser) into individual items for the next step.

10. Google Sheets – Create Rows — Google Sheets (OAuth2 / Service Account)

    • Appends each extracted product as a new row in the Results sheet of the same Google Sheet. Maps fields to columns (product_title, product_price, etc.).

11. HTML-Scraping Tool (trigger) — Execute Workflow Trigger (No auth)

    • Entry point for the sub-workflow when the agent calls the HTML tool.

12. Set fields – from AI agent query — Set (No auth)

    • Extracts the url from the agent’s query (the $json.query value).

13. ScrapingBee- Get page HTML — HTTP Request (API Key auth)

    • Retrieves the raw HTML of the target URL using the same ScrapingBee API key.

14. HTML to Markdown — Markdown (No auth)

    • Converts the returned HTML to Markdown to reduce token usage when sent back to the agent.

Setup Instructions

• ScrapingBee: Sign up at scrapingbee.com to get an API key. Insert it in the two HTTP Request nodes (<your_scrapingbee_apikey>).
• Google Sheets: Use a service account or OAuth2 credential. Create a spreadsheet with two sheets: List of URLs (column A with URLs) and Results (columns matching the output schema). An example sheet is provided.
• Google Gemini: Obtain an API key from Google AI Studio. Create an API key credential in n8n and connect it to the Gemini node.
• N8n: Import the workflow, configure credentials for Sheets and Gemini, and replace the placeholder API key in ScrapingBee nodes. Run the workflow to test.

Use Cases & Variations

• E-commerce scraping: Extract product listings, prices, and promotions from category pages.
• Competitor monitoring: Regularly scrape competitor websites for pricing changes.
• Lead generation: Scrape business directories or review sites for contact info.
• Custom data fields: Modify the Structured Output Parser JSON schema and the Google Sheets columns to extract any visible information (e.g., ratings, descriptions, availability).
• Scheduled execution: Replace the manual trigger with a Schedule Trigger to run daily or hourly.
• Multi-page scraping: Adjust the Google Sheets input to include multiple URLs and process them sequentially.

{
  "id": "PpFVCrTiYoa35q1m",
  "meta": {
    "instanceId": "b9faf72fe0d7c3be94b3ebff0778790b50b135c336412d28fd4fca2cbbf8d1f5",
    "templateCredsSetupCompleted": true
  },
  "name": "Vision-Based AI Agent Scraper - with Google Sheets, ScrapingBee, and Gemini",
  "tags": [],
  "nodes": [
    {
      "id": "90ac8845-342e-4fdb-ae09-cb9d169b4119",
      "name": "When clicking ‘Test workflow’",
      "type": "n8n-nodes-base.manualTrigger",
      "position": [
        160,
        460
      ],
      "parameters": {},
      "typeVersion": 1
    },
    {
      "id": "7a2bfc41-1527-448d-a52c-794ca4c9e7ee",
      "name": "ScrapingBee- Get page HTML",
      "type": "n8n-nodes-base.httpRequest",
      "position": [
        2280,
        1360
      ],
      "parameters": {
        "url": "https://app.scrapingbee.com/api/v1",
        "options": {},
        "sendQuery": true,
        "queryParameters": {
          "parameters": [
            {
              "name": "api_key",
              "value": "<your_scrapingbee_apikey>"
            },
            {
              "name": "url",
              "value": "={{$json.url}}"
            }
          ]
        }
      },
      "typeVersion": 4.2
    },
    {
      "id": "a0ab6dcb-ffad-40bf-8a22-f2e152e69b00",
      "name": "Structured Output Parser",
// ... truncated (copy to see full JSON)