Evaluate AI agent response relevance using OpenAI and cosine similarity
This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the origi
At a glance
Evaluate AI agent response relevance using OpenAI and cosine similarity is a ready-made n8n workflow you import as a workflow JSON file — no build required. It connects OpenAI. It's free to download. Follow the 5-step import below to go live in minutes.
- Platform
- n8n
- Connects
- OpenAI
- Modules
- 17
- Price
- Free
- Version
- v1.0

About this workflow
This n8n template demonstrates how to calculate the evaluation metric "Relevance" which in this scenario, measures the relevance of the agent's response to the user's question. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here How it works This evaluation works best for Q&A agents. For our scoring, we analyse the agent's response and ask another AI to generate a question from it. This generated question is then compared to the original question using cosine similarity. A high score indicates relevance and the agent's successful ability to answer the question whereas a low score means agent may have added too much irrelevant info, went off script or hallucinated. Requirements n8n version 1.94+ Check out this Google Sheet for a sample data
How to import this n8n workflow
- 1
Download the workflow JSON file after purchase.
- 2
Open n8n → click the menu → Import from File.
- 3
Select the downloaded JSON and import.
- 4
Set up credentials for each node that requires them.
- 5
Click Execute Workflow to test, then activate.
Setup guide
Setup guide included
Purchase to unlock the full step-by-step guide
Related N8n workflows
Generate AI document titles for Paperless-ngx with GPT-4o Mini
Automated resume job matching engine with Bright Data MCP & OpenAI 4o mini
Trigger new articles and add rows in Google Sheets with a ChatGPT Assistant
Technical stock analysis with Telegram, Airtable and a GPT-powered AI agent
Automate assignment grading with GPT-4-Turbo and multi-format reports
Send data to ActionSend and convert HTML to text with OpenAI GPT-3
Process & summarize PDFs from email & messaging apps with OpenAI GPT
Create custom PDF documents from templates with Gemini & Google Drive
Reviews
No reviews yet
Be the first to buy and share your experience.
Leave a review
Sign in to share your experience with this workflow.
Create a free account to purchase workflows.
- JSON blueprint — instant download
- Setup guide PDF included
- 5 downloads · valid 30 days
- Works with n8n
Need help setting this up?
Book a 3-hour live setup session with an Agility consultant.
- Configure live on Google Meet / Zoom
- Free follow-up if workflow has defects
- Platform expert assigned to you