# Engine for Ranking and Comparing LLM Agents. AI Agent Connect > The Agent Benchmark Comparison Engine provides a mathematical framework to evaluate LLM agents. It normalizes core metrics like accuracy, latency, cost, and hallucination rates to calculate a precise composite score for every agent you test. You can use the MCP to generate ranked lists based on custom weights, identify category leaders, or validate your evaluation parameters before running heavy calculations. This MCP lets you move beyond simple qualitative comparisons and get hard, weighted data on agent performance. ## Overview - **Category:** analytics - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_aD2zjuWHaIJRj1pXV5vT436Hv4TzDHO7e7iXDcpX/ai-agent-connect - **Tags:** llm, ranking, metrics, performance, benchmarking ## Description Comparing LLM agents is tough. You can't just look at accuracy; you need to factor in cost, speed, and reliability. This MCP solves that. It gives you a mathematical framework to evaluate multiple agents simultaneously. Instead of guessing which model is best, you feed the MCP the metrics—accuracy, latency, cost, and hallucination rates—and it calculates a single, weighted score for each one. You can use the `calculate_agent_rankings` tool to generate a full list ranked by your custom priorities. Need to know which agent excels at speed? Use `get_agent_performance_summary` for a quick overview of category leaders. Before you run any big benchmark, you can run `validate_benchmark_config` to make sure your weights and metrics are mathematically sound. This MCP turns raw performance data into actionable, comparable scores. ## Tools ### calculate_agent_rankings Performs the complete mathematical comparison and ranking of a set of agents based on provided weights ### get_agent_performance_summary Retrieves a high-level overview of the best-performing agents for specific use cases ### validate_benchmark_config 0 and metrics are within logical bounds. Ensures that a proposed set of weights and agent metrics are mathematically valid before running heavy calculations ## Prompt Examples **Prompt:** ``` Rank these agents: AgentA (accuracy: 90, latency: 200, cost: 0.5, hallucination: 0.02), AgentB (accuracy: 85, latency: 150, cost: 0.3, hallucination: 0.05) with weights accuracy: 0.4, latency: 0.3, cost: 0.2, hallucination: 0.1. ``` **Response:** ``` AgentA: 0.82, AgentB: 0.78. Rank 1: AgentA, Rank 2: AgentB. ``` **Prompt:** ``` Who is the fastest agent among AgentX (latency: 500) and AgentY (latency: 200)? ``` **Response:** ``` AgentY is the fastest agent with a latency of 200ms. ``` **Prompt:** ``` Check if these weights are valid: accuracy: 0.5, latency: 0.5. ``` **Response:** ``` The configuration is valid as the weights sum to 1.0. ``` ## Capabilities ### Weighted Ranking The AI uses this MCP when you need to rank multiple agents based on a custom combination of metrics. ### Performance Overview The AI calls this MCP to quickly identify the top-performing agents in specific categories. ### Configuration Validation The AI uses this MCP to confirm that your input weights and metrics are mathematically sound before running a benchmark. ### Metric Normalization The MCP handles the normalization of diverse metrics, like cost and accuracy, into comparable scores. ### Composite Scoring The MCP calculates a single, precise score that combines multiple performance factors into one number. ## Use Cases ### Selecting a Production Model You need to choose between three different LLMs for a customer-facing chat bot. You use the MCP to weigh latency and hallucination rate heavily to pick the most reliable option. ### Optimizing API Costs Your team uses multiple agents for different tasks. You run the MCP to compare their cost-per-query metrics, ensuring you pick the cheapest option without sacrificing quality. ### Internal Agent Comparison You have three internal agents built by different teams. You use the MCP to generate a single, objective ranking based on a mix of accuracy and speed. ### Pre-launch Benchmark Check Before launching a new feature, you use the MCP to validate your benchmark weights, ensuring the entire testing framework is mathematically sound. ## Benefits - You move beyond subjective comparisons by relying on a deterministic, mathematical scoring system. - The MCP handles the complexity of normalizing diverse metrics like cost, speed, and accuracy. - You can adjust the weights to prioritize what matters most, whether it's low cost or high accuracy. - It prevents wasted computation time by validating your benchmark inputs before running full calculations. ## How It Works Connecting to this MCP is straightforward. You connect your AI client to the Vinkius Catalog, and the MCP becomes available for immediate use. 1. Connect your AI client to the Vinkius Catalog and select this MCP. 2. Provide the MCP with the performance metrics (accuracy, latency, cost, etc.) for the agents you want to test. 3. Specify your custom weights, telling the MCP which metrics are most important to your business. 4. Invoke the desired tool, like `calculate_agent_rankings`, and receive a final, weighted score and rank list. ## Frequently Asked Questions **Does this MCP compare models based on real-time usage?** No. This MCP uses a deterministic mathematical framework. You must provide the performance metrics—like latency and cost—as inputs; it does not run live tests against external APIs. **What metrics can I use for comparison?** The MCP is designed to handle common metrics including accuracy, latency, cost, and hallucination rates. You can assign weights to each of these factors. **Is the ranking customizable?** Yes. You control the ranking by providing custom weights. You decide if latency is twice as important as cost, for example, and the MCP calculates the score accordingly. **What if my weights don't add up to 1.0?** You should run the `validate_benchmark_config` tool first. This ensures that your proposed set of weights and metrics are mathematically sound before you run the main calculations. **Can I use this MCP with my existing data?** Yes. You feed the MCP the data you've already collected. The MCP's job is to take that raw data and apply the weighted scoring formula.