• AUTOMATED
  • ORGANIZED
  • HUMAN
  • READY

How to create model validation datasets with your AI

Connect Arize AI to your agent to turn chat commands into structured evaluation sets. Once connected, you just describe the data you need.

Ask AI about this page

Short answer

How can I automate dataset creation for model evaluation?

You describe the dataset you want in plain English, and your agent uses Arize AI to build it. This removes the need for manual data gathering and uploads. You get ready-to-use validation sets through simple chat.

The results

Where your data lands

Your agent handles the technical setup so you can focus on testing.

AUTOMATED

New datasets ready

The AI builds your validation sets directly in Arize AI. You get a fresh dataset without touching a dashboard.

ORGANIZED

Inventory check

Your agent checks what you already have. This prevents you from creating duplicate sets by mistake.

HUMAN

Review and refine

You check the dataset contents to ensure quality. The AI handles the structure, but you confirm the data is right.

READY

Evaluation starts

Once the AI confirms creation, the dataset is live. You can immediately start running model evaluations.

The workflow

What your AI does when you ask for a dataset.

The agent translates your intent into specific API calls to Arize AI.

  1. Check existing data

    The agent looks at your current Arize AI environment to see what datasets already exist. This prevents you from accidentally creating the same set twice.

    list_datasets

  2. Build the new set

    The agent takes your description and executes the command to generate the new validation dataset. It handles the technical setup of the new entry.

    create_dataset

  3. Verify the creation

    The agent checks the list again to confirm the new dataset is visible and correctly named. It ensures the task actually finished.

    list_datasets

  4. Report back

    The agent tells you the name of the new dataset and confirms it is ready for your model evaluation. You can then proceed with your testing.

    list_datasets

Try it

Copy these to start.

Use these prompts to trigger the Arize AI connector.

Starting points

Swap the bracketed text with your specific dataset names or criteria.

Arize AI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_wCwq9adpwddtNPc4Bp17r5LJCv19cGsWZBiDUv4U/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Arize AI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "arize-ai-alternative-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_wCwq9adpwddtNPc4Bp17r5LJCv19cGsWZBiDUv4U/mcp"
    }
  }
}
Copy into chat04

  • Create a new dataset in Arize AI named 'customer_support_v1'.

  • List all my current datasets so I can see what's available.

  • Make a validation dataset called 'llm_safety_test' in Arize AI.

  • Show me the datasets I have in Arize AI to check for duplicates.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Arize AI once, then ask.

Follow the link to authorize the connection. Your credentials stay encrypted and secure, and you can start chatting with your agent immediately.

Connect Arize AI to your AI

FAQ

How the task behaves

  • 01

    Can the AI delete my existing datasets?

    No. The connector only has tools to list and create datasets. It cannot delete or modify existing data.

  • 02

    Does the AI decide which data goes into the dataset?

    The AI follows your specific instructions. You must tell it what the dataset should be named and what its purpose is.

  • 03

    Can I use this to edit the contents of a dataset?

    No. The current tools only allow for creating new datasets or listing what is already there.

  • 04

    Do I need to use the Arize AI web dashboard?

    You don't need it for creating or listing datasets, as the AI handles those tasks through the connector.

  • 05

    Can the AI see my model performance metrics?

    No. This connector is limited to dataset management tasks like creating and listing.

  • More questions about Arize AI? See everything the Arize AI Connector can do

Connect Arize AI to Claude, Cursor, ChatGPT & more