ETL pipeline

Automate your data extraction, transformation, and loading with this robust ETL pipeline, designed to efficiently process and analyze information from various sources. This workflow begins on a schedule, fetching tweets from Twitter/X, then storing them in MongoDB for initial processing. The MongoDB data is then sent to Google Cloud Natural Language for sentiment analysis or entity extraction, with the results subsequently prepared and stored in PostgreSQL. A conditional check on the PostgreSQL data determines whether to send an alert to Slack, ensuring timely notifications for critical insights or anomalies. This powerful automation is ideal for marketing teams monitoring brand sentiment, researchers analyzing public opinion, or businesses tracking competitor activity, providing actionable intelligence without manual data handling. By automating data ingestion, enrichment, and storage, this workflow significantly reduces the time and effort spent on data preparation, allowing teams to focus on analysis and strategic decision-making while ensuring data consistency and accessibility.

9 nodesschedule trigger137 views4 copiesData

PostgreSQLMongoDBTwitter/XSlack

Workflow JSON

{"id": "6", "name": "ETL pipeline", "nodes": [{"name": "Twitter", "type": "n8n-nodes-base.twitter", "position": [300, 300], "parameters": {"limit": 3, "operation": "search", "searchText": "=#OnThisDay", "additionalFields": {}}, "credentials": {"twitterOAuth1Api": "twitter_api"}, "typeVersion": 1}, {"name": "Postgres", "type": "n8n-nodes-base.postgres", "position": [1100, 300], "parameters": {"table": "tweets", "columns": "text, score, magnitude", "returnFields": "=*"}, "credentials": {"postgres": "postgres"}, "typeVersion": 1}, {"name": "MongoDB", "type": "n8n-nodes-base.mongoDb", "position": [500, 300], "parameters": {"fields": "text", "options": {}, "operation": "insert", "collection": "tweets"}, "credentials": {"mongoDb": "mongodb"}, "typeVersion": 1}, {"name": "Slack", "type": "n8n-nodes-base.slack", "position": [1500, 200], "parameters": {"text": "=\ud83d\udc26 NEW TWEET with sentiment score {{$json[\"score\"]}} and magnitude {{$json[\"magnitude\"]}} \u2b07\ufe0f\n{{$json[\"text\"]}}", "channel": "tweets", "attachments": [], "otherOptions": {}}, "credentials": {"slackApi": "slack"}, "typeVersion": 1}, {"name": "IF", "type": "n8n-nodes-base.if", "position": [1300, 300], "parameters": {"conditions": {"number": [{"value1": "={{$json[\"score\"]}}", "operation": "larger"}]}}, "typeVersion": 1}, {"name": "NoOp", "type": "n8n-nodes-base.noOp", "position": [1500, 400], "parameters": {}, "typeVersion": 1}, {"name": "Google Cloud Natural Language", "type": "n8n-nodes-base.googleCloudNaturalLanguage", "position": [700, 300], "parameters": {"content": "={{$node[\"MongoDB\"].json[\"text\"]}}", "options": {}}, "credentials": {"googleCloudNaturalLanguageOAuth2Api": "google_nlp"}, "typeVersion": 1}, {"name": "Set", "type": "n8n-nodes-base.set", "position": [900, 300], "parameters": {"values": {"number": [{"name": "score", "value": "={{$json[\"documentSentiment\"][\"score\"]}}"}, {"name": "magnitude", "value": "={{$json[\"documentSentiment\"][\"magnitude\"]}}"}], "string": [{"name": "text", "value": "={{$node[\"Twitter\"].json[\"text\"]}}"}]}, "options": {}}, "typeVersion": 1}, {"name": "Cron", "type": "n8n-nodes-base.cron", "position": [100, 300], "parameters": {"triggerTimes": {"item": [{"hour": 6}]}}, "typeVersion": 1}], "active": false, "settings": {}, "connections": {"IF": {"main": [[{"node": "Slack", "type": "main", "index": 0}], [{"node": "NoOp", "type": "main", "index": 0}]]}, "Set": {"main": [[{"node": "Postgres", "type": "main", "index": 0}]]}, "Cron": {"main": [[{"node": "Twitter", "type": "main", "index": 0}]]}, "MongoDB": {"main": [[{"node": "Google Cloud Natural Language", "type": "main", "index": 0}]]}, "Twitter": {"main": [[{"node": "MongoDB", "type": "main", "index": 0}]]}, "Postgres": {"main": [[{"node": "IF", "type": "main", "index": 0}]]}, "Google Cloud Natural Language": {"main": [[{"node": "Set", "type": "main", "index": 0}]]}}}

How to Import This Workflow

1Copy the workflow JSON above using the Copy Workflow JSON button.
2Open your n8n instance and go to Workflows.
3Click Import from JSON and paste the copied workflow.

Don't have an n8n instance? Start your free trial at n8nautomation.cloud

Related Templates

SQL agent with memory

Empower your data analysis with the SQL agent with memory workflow, automating the process of querying databases using natural language. This powerful workflow connects OpenAI's advanced language models with your local SQL databases, allowing you to interact with your data through a conversational interface. Initially, the workflow downloads a chinook.zip example database, extracts it, and saves the chinook.db file locally, making it immediately available for querying. The AI Agent, powered by OpenAI Chat Model and supported by a Window Buffer Memory, interprets your natural language questions, translates them into SQL queries, executes them against your local chinook.db, and provides the results back to you. This is incredibly useful for data analysts, business intelligence professionals, or anyone needing quick insights from their databases without writing complex SQL queries, significantly reducing the time and specialized knowledge required for data exploration. By leveraging the Chat Trigger, users can easily initiate conversations and receive immediate, intelligent responses, streamlining data access and accelerating decision-making.

13 nodes

Transcribing Bank Statements To Markdown Using Gemini Vision AI

Automate the tedious process of transcribing bank statements into structured Markdown with this powerful n8n workflow. This solution leverages Google Gemini Vision AI to intelligently extract financial data from PDF bank statements stored in Google Drive, transforming scanned documents into easily parsable text. It begins by fetching a specified bank statement PDF from Google Drive upon manual trigger, then splits the PDF into individual image pages. These images are then resized for optimal AI processing before being fed to Google Gemini Vision AI for transcription. The AI identifies and extracts deposit table rows, and a final AI chain node converts this raw data into a clean, organized Markdown format. This workflow is ideal for financial analysts, small business owners, or anyone needing to quickly digitize and analyze physical bank records, saving significant time and reducing manual data entry errors.

20 nodes

Prepare CSV files with GPT-4

Transform raw, unstructured text into perfectly formatted CSV files using the power of GPT-4 with this n8n workflow. This automation connects OpenAI's advanced language model to process your input, then meticulously structures the output into a usable CSV format. Ideal for data analysts, marketers, or researchers, this workflow helps you extract specific information from large text datasets, such as customer reviews, survey responses, or article summaries, and prepare it for analysis in spreadsheets or databases. By automating the extraction and formatting of data, you significantly reduce manual data entry errors and save countless hours of tedious work, allowing you to focus on insights rather than data preparation. The workflow manually triggers, sending your text to OpenAI, then splits the responses into manageable batches, parses the JSON output, converts it into a structured table, and finally saves a clean, UTF-8 encoded CSV file to disk, ensuring compatibility across various systems.

11 nodes

Ready to automate with n8n?

Get affordable managed n8n hosting with 24/7 support.