• PROJECTS
  • REFINEMENT
  • ITERATION
  • HUMAN

How to manage RLHF pipelines with your AI

Connect Scale AI to your agent to handle the heavy lifting of data labeling. You can spin up new projects or tweak existing instructions without leaving your chat window.

Ask AI about this page

Short answer

How can I manage RLHF pipelines through my AI?

You can create new labeling projects and modify project parameters to refine your model's fine-tuning. This allows you to iterate on labeling instructions and data collection tasks using natural language. You'll get direct control over your training data pipeline.

Pipeline outcomes

What you get from your agent

The AI handles the configuration so you can focus on the model.

PROJECTS

New data tasks

Your AI sets up fresh labeling projects. You get a structured environment ready for human or model labeling immediately.

REFINEMENT

Updated instructions

The AI modifies existing project parameters. This lets you fix edge cases in your labeling logic on the fly.

ITERATION

Faster fine-tuning

You bridge the gap between model errors and better data. The AI handles the setup so you can focus on performance.

HUMAN

Manual oversight

The AI manages the setup, but you decide when instructions are clear enough. You remain the final judge of labeling quality.

The workflow

What your AI does when <the thing arrives>.

The AI acts as your interface to the Scale AI platform.

  1. Initialize projects

    When you need new data, your AI builds the project structure in Scale AI. It sets the foundation for your next fine-tuning run.

    create_project

  2. Adjust parameters

    If your labeling results aren't hitting the mark, your AI updates the project settings. It changes how tasks are presented to labelers.

    update_project_params

  3. Refine instructions

    You can tell your agent to change specific labeling rules. It pushes those updates to the existing project immediately.

    update_project_params

  4. Scale data collection

    The AI manages the lifecycle of these tasks. It ensures your project settings stay aligned with your fine-tuning goals.

    create_projectupdate_project_params

Try it

Copy these to start.

Use these prompts to trigger actions in Scale AI.

Starting points

These are templates. Swap out the project names and specific instruction details to match your actual work.

Scale AI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_lsC40U9ook5OZ8hrmwvYLqntMTpGHdSfQWvhNSRl/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Scale AI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "scale-ai-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_lsC40U9ook5OZ8hrmwvYLqntMTpGHdSfQWvhNSRl/mcp"
    }
  }
}
Copy into chat04

  • Create a new labeling project in Scale AI called 'Customer Support Intent' with standard classification rules.

  • Update the parameters for my 'Safety Alignment' project to include more strict edge case instructions.

  • Start a new project for 'Code Generation' evaluation and set the initial task parameters.

  • Modify the existing 'Image Captioning' project instructions to focus more on spatial relationships.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Scale AI once, then ask.

Just click the link to connect. Your credentials stay encrypted and secure, and you can start managing your RLHF pipelines immediately.

Connect Scale AI to your AI

FAQ

How this task behaves

  • 01

    Can the AI delete my existing projects?

    No. The available tools only allow for creating new projects or updating existing ones.

  • 02

    Can my AI perform the actual human labeling?

    No. The AI manages the project configuration and instructions, but the actual labeling is done through the Scale AI platform.

  • 03

    How do I change the labeling rules?

    You tell your AI to update the project parameters, and it will apply those changes to your Scale AI project.

  • 04

    Can I see the results of the labeling in my chat?

    No. This MCP is designed for managing project setup and parameters, not for retrieving or viewing the labeled datasets themselves.

  • 05

    Does the AI handle the billing for Scale AI?

    No. All financial and billing matters are handled directly through your Scale AI account.

  • More questions about Scale AI? See everything the Scale AI Connector can do

Connect Scale AI to Claude, Cursor, ChatGPT & more