# RAG Chunk Selection Optimizer AI Agent Connect

> A deterministic engine to select the most effective RAG information chunks within token budgets.

## Overview
- **Category:** optimization
- **Price:** Free
- **Endpoint:** https://edge.vinkius.com/vk_preview_MUkr0meo1CTgmH0DxXIUM43TITGllkXwBnnhs3Kv/ai-agent-connect
- **Tags:** rag, retrieval, optimization, token-management, data-cleaning

## Description

This MCP server provides a deterministic optimization engine for RAG (Retrieval-Augmented Generation) pipelines. It allows AI agents to select the most effective subsets of retrieved information chunks by maximizing relevance and coverage while strictly adhering to context token limits. 

Key capabilities include:
- `select_chunks`: Pick optimal chunks using strategies like Top-K, Relevance Threshold, or Diversity-Based selection.
- `deduplicate_chunks`: Clean the retrieved pool by merging or removing redundant information to prevent repetition.
- `calculate_metrics`: Evaluate the quality of a selection using metrics like information density, coverage score, and marginal value.

By using this tool, agents can ensure they stay within the LLM's context window while providing the highest quality information possible.

## Tools

### calculate_metrics
Calculate efficiency and quality metrics for selected chunks

### deduplicate_chunks
Remove or merge redundant chunks

### select_chunks
Select optimal chunks from a retrieved set based on a strategy

## Prompt Examples

**Prompt:** 
```
Select the top 3 most relevant chunks from these retrieved items with a budget of 1000 tokens.
```

**Response:** 
```
The selected chunks are: chunk_001 (relevance: 0.95), chunk_042 (relevance: 0.88), and chunk_015 (relevance: 0.82). Total tokens used: 850.
```

**Prompt:** 
```
Clean up these retrieved chunks to remove any duplicates.
```

**Response:** 
```
The redundant chunks have been merged. You now have 5 unique chunks instead of 7.
```

**Prompt:** 
```
Calculate the information density for the following selected chunks.
```

**Response:** 
```
The information density for the selected set is 0.012 relevance units per token.
```

## Frequently Asked Questions

**What are the different selection strategies?**
The engine supports Top-K (picking the N most relevant), Relevance Threshold (picking chunks above a specific score), and Diversity-Based (maximizing unique sources within the budget).

**How does the optimizer handle token limits?**
The `select_chunks` tool accepts a `contextBudgetTokens` parameter. The engine ensures the sum of `tokenCount` for all selected chunks does not exceed this limit.

**Can I remove redundant information?**
Yes, you can use the `deduplicate_chunks` tool to identify and merge overlapping chunks, which helps in reducing noise and saving tokens.