Convert tour PDFs to vector database using Google Drive, LangChain & OpenAI

Workflow: Process Tour PDF from Google Drive to Pinecone Vector DB with OpenAI Embeddings Overview This workflow automates the process of extracting tour information from PDF files stored in a Google Drive folder, processes and vectorizes the extracted data, and stores it in a Pinecone vector database for efficient querying. This is especially useful for building AI-powered search or recommendation systems for travel packages. Setup: Prerequisites A folder in Google Drive with PDF tour package brochures. Pinecone account + API key OpenAI API key n8n cloud or self-hosted instance Workflow Setup Steps Trigger Manual Trigger (When clicking 'Test workflow'): Used for manual testing and execution of the workflow. Google Drive Integration Step 1: Store Tour Packages in PDF Format Upload your curated tour packages containing the tours, activities and sight-seeings in PDF format into a designated Google Drive folder. Step 2: Search Folder Node: PDF Tour Package Folder (Google Drive) This node searches the designated folder for files (filter by MIME type = application/pdf if needed). Step 3: Download PDFs Node: Download Package Files (Google Drive) Downloads each matching PDF file found in the previous step. Process Each PDF File Step 4: Loop Through Files Node: Loop Over each PDF file Iterates through each downloaded PDF file to extract, clean, split, and embed. Data Preparation & Embedding Step 5: Data Loader Node: Data Loader Reads each PDF’s content using a compatible loader. It passes clean raw text to the next node. Often integrated with document loaders like pdf-loader, Unstructured, or pdfplumber. Step 6: Recursive Text Splitter Node: Recursive Character Text Splitter Splits large chunks of text into manageable segments using overlapping window logic (e.g., 500 tokens with 50 token overlap). This ensures contextual preservation for long documents during embedding. Step 7: Generate Embeddings Node: Embeddings OpenAI Uses text-embedding-3-small model to vectorize the split chunks. Outputs vector representations for each content chunk. Store in Pinecone Step 8: Pinecone Vector Store Node: Pinecone Vector Store - Store... Stores each embedding along with its metadata (source PDF name, chunk ID, etc.). This becomes the basis for fast, semantic search via RAG workflows or agents. 🛠️ Tools & Nodes Used Google Drive (Search & Download) Searches for all PDF files in a specified Google Drive folder. Downloads each file for processing. SplitInBatches (Loop Over Items) Loops through each file found in the folder, ensuring each is processed individually. Default Data Loader (LangChain) Reads and extracts text from the PDF files. Recursive Character Text Splitter (LangChain) Splits the extracted text into manageable chunks for embedding. OpenAI Embeddings (LangChain) Converts each text chunk into a vector using OpenAI’s embedding model. Pinecone Vector Store (LangChain) Stores the resulting vectors in a Pinecone index for fast similarity search and querying. 🔗 Workflow Steps Explained Trigger: The workflow starts manually for testing or can be scheduled. Google Drive Search: Finds all PDF files in the specified folder. Loop Over Files: Each file is processed one at a time using the SplitInBatches node. Download File: Downloads the current PDF file from Google Drive. Extract Text: The Default Data Loader node reads the PDF and extracts its text content. Text Splitting: The Recursive Character Text Splitter breaks the text into chunks (e.g., 1000 characters with 50 overlap) to optimize embedding quality. Vectorization: Each chunk is sent to the OpenAI Embeddings node to generate vector representations. Store in Pinecone: The vectors are inserted into a Pinecone index, making them available for semantic search and recommendations. 🚀 What Can Be Improved in the Next Version? Error Handling: Add error handling nodes to manage failed downloads or extraction issues gracefully. File Type Filtering: Ensure only PDF files are processed by adding a filter node. Metadata Storage: Store additional metadata (e.g., file name, tour ID) alongside vectors in Pinecone for richer search results. Parallel Processing: Optimize for large folders by processing multiple files in parallel (with care for API rate limits). Automated Triggers: Replace manual trigger with a time-based or webhook trigger for full automation. Data Validation: Add checks to ensure extracted text contains valid tour data before vectorization. User Feedback: Integrate notifications (e.g., email or Slack) to inform when processing is complete or if issues arise. 💡 Summary This workflow demonstrates how n8n can orchestrate a powerful AI data pipeline using Google Drive, LangChain, OpenAI, and Pinecone. It’s a great foundation for building intelligent search or recommendation features for travel and tour data. Feel free to ask for more details or share your improvements! Let me know if you want to see a specific part of the workflow or need help with a particular node!