Index legal documents for hybrid search with Qdrant, OpenAI & BM25

Index Legal Dataset to Qdrant for Hybrid Retrieval This pipeline is the first part of "Hybrid Search with Qdrant & n8n, Legal AI". The second part, "Hybrid Search with Qdrant & n8n, Legal AI: Retrieval", covers retrieval and simple evaluation. Overview This pipeline transforms a Q&A legal corpus from Hugging Face (isaacus) into vector representations and indexes them to Qdrant, providing the foundation for running Hybrid Search, combining: - Dense vectors (embeddings) for semantic similarity search; - Sparse vectors for keyword-based exact search. After running this pipeline, you will have a Qdrant collection with your legal dataset ready for hybrid retrieval on BM25 and dense embeddings: either mxbai-embed-large-v1 or text-embedding-3-small. Options for Embedding Inference This pipeline equips you with two approaches for generating dense vectors: 1. Using Qdrant Cloud Inference, conversion to vectors handled directly in Qdrant; 2. Using external provider, e.g. OpenAI for generating embeddings. Prerequisites - A cluster on Qdrant Cloud - Paid cluster in the US region if you want to use Qdrant Cloud Inference - Free Tier Cluster if using an external provider (here OpenAI) - Qdrant Cluster credentials: - You'll be guided on how to obtain both the URL and APIKEY from the Qdrant Cloud UI when setting up your cluster; - An OpenAI API key (if you’re not using Qdrant’s Cloud Inference); P.S. - To ask retrieval in Qdrant-related questions, join the Qdrant Discord. - Star Qdrant n8n community node repo <3