• NEW DATA
  • INVENTORY
  • AUDIT
  • HUMAN

How to manage model evaluation datasets with your AI

Connect Arize AI to your agent to automate how you build and organize validation data. Once connected, your AI handles the heavy lifting of dataset management.

Ask AI about this page

Short answer

How can I use my AI to manage evaluation datasets?

Your AI uses Arize AI to create new datasets and list existing ones for model validation. This lets you move from raw data to organized evaluation sets through simple chat commands. You'll get a structured way to track how new model versions perform against specific benchmarks.

The outcomes of your request.

Where your data lands.

Depending on what you ask, your AI will produce one of these results.

NEW DATA

Ready-to-use datasets

The AI uses the creation tool to build a specific set of data for your current validation task. You get a clean dataset ready for testing.

INVENTORY

Dataset oversight

Your AI pulls a list of all your current datasets from Arize AI. You'll see exactly what's available for your evaluation runs.

AUDIT

Validation readiness

The AI organizes data so you can compare model versions. This ensures your testing environment stays consistent and reliable.

HUMAN

Manual review needed

The AI handles the data movement, but you decide which datasets are actually high-quality enough for production. You retain full control over the final selection.

The workflow

What your AI does when you ask for dataset management.

The AI acts as the bridge between your chat interface and your Arize AI environment.

  1. Scan existing data

    Your AI checks your Arize AI account to see what datasets you already have stored.

    list_datasets

  2. Build new sets

    When you describe the data you need, the AI generates a new dataset for your specific model version.

    create_dataset

  3. Organize for testing

    The AI uses the creation tool to structure data specifically for validation workflows.

    create_dataset

  4. Verify availability

    The AI confirms the new or existing datasets are correctly placed in your account.

    list_datasets

Try it

Copy these to start.

Use these prompts to trigger actions in Arize AI.

Starting points

Swap out the bracketed text for your actual dataset names or specific model version requirements.

Arize AI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_wKJSgGsb2EgJmks5PzM3V0jknb7nFZgbwB0CaRFP/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Arize AI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "arize-ai-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_wKJSgGsb2EgJmks5PzM3V0jknb7nFZgbwB0CaRFP/mcp"
    }
  }
}
Copy into chat04

  • List all my current datasets in Arize AI.

  • Create a new dataset named [Dataset Name] for my latest model version.

  • Show me the datasets I have available for the [Model Name] evaluation.

  • Create a validation dataset using the parameters from [Specific Data Source].

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Arize AI once, then ask.

Just link your account once. Your credentials stay encrypted and secure, and you can immediately start managing data through your preferred AI client.

Connect Arize AI to your AI

FAQ

How this task behaves.

  • 01

    Can the AI delete my existing datasets?

    No. The AI only has tools to list and create datasets. It cannot delete or overwrite your data.

  • 02

    Does the AI decide which model version is better?

    No. The AI manages the datasets used for the test, but you must perform the actual evaluation and decision-making.

  • 03

    Can I see the contents of a dataset through the AI?

    No. The AI can list the datasets and create new ones, but it does not have a tool to read the individual rows or entries within a dataset.

  • 04

    How do I know if a dataset was created successfully?

    The AI will confirm the action once the creation tool finishes. You can also ask it to list your datasets to verify it's there.

  • 05

    Can the AI connect to my local files to make datasets?

    No. The AI can only interact with Arize AI using the provided tools. It cannot pull data from your local hard drive or other external apps.

  • More questions about Arize AI? The Connector page answers them. See everything the Arize AI Connector can do

Connect Arize AI to Claude, Cursor, ChatGPT & more