Api schema extractor

This workflow automates the process of discovering and extracting APIs from various services, followed by generating custom schemas. It works in three distinct stages: research, extraction, and schema generation, with each stage tracking progress in a Google Sheet. 🙏 Jim Le deserves major kudos for helping to build this sophisticated three-stage workflow that cleverly automates API documentation processing using a smart combination of web scraping, vector search, and LLM technologies. How it works Stage 1 - Research: - Fetches pending services from a Google Sheet - Uses Google search to find API documentation - Employs Apify for web scraping to filter relevant pages - Stores webpage contents and metadata in Qdrant (vector database) - Updates progress status in Google Sheet (pending, ok, or error) Stage 2 - Extraction: - Processes services that completed research successfully - Queries vector store to identify products and offerings - Further queries for relevant API documentation - Uses Gemini (LLM) to extract API operations - Records extracted operations in Google Sheet - Updates progress status (pending, ok, or error) Stage 3 - Generation: - Takes services with successful extraction - Retrieves all API operations from the database - Combines and groups operations into a custom schema - Uploads final schema to Google Drive - Updates final status in sheet with file location Ideal for: - Development teams needing to catalog multiple APIs - API documentation initiatives - Creating standardized API schema collections - Automating API discovery and documentation Accounts required: - Google account (for Sheets and Drive access) - Apify account (for web scraping) - Qdrant database - Gemini API access Set up instructions: - Prepare your Google Sheets document with the services information. Here's an example of a Google Sheet – you can copy it and change or remove the values under the columns. Also, make sure to update Google Sheets nodes with the correct Google Sheet ID. - Configure Google Sheets OAuth2 credentials, required third-party services (Apify, Qdrant) and Gemini. - Ensure proper permissions for Google Drive access.