• COLLECTIONS
  • INDEXED
  • IDENTIFIED
  • HUMAN

How to track entities across video with your AI

Connect Twelve Labs to your AI client to turn raw video footage into searchable, organized data. Once connected, your agent can find and group specific subjects across your entire library.

Ask AI about this page

Short answer

How can I track people or objects in my videos using AI?

Your AI uses Twelve Labs to index your video files and group specific subjects into organized collections. You'll be able to find every instance of a person or object without watching every frame manually. Here is how the process works.

Where every finding lands.

The outcomes of entity tracking

The AI organizes your video data into these specific formats.

COLLECTIONS

Grouped entity sets

The AI builds sets of specific subjects. You get a structured way to manage all clips featuring a single person or object.

INDEXED

Searchable video assets

Your video files become searchable data. The AI processes the footage so you can query it later.

IDENTIFIED

Specific entity profiles

Individual subjects are defined as unique entities. This allows the AI to distinguish between different people or items.

HUMAN

Manual review needed

You decide which entities are important enough to save. The AI flags the subjects, but you control the final organization.

The workflow

What your AI does when <the thing arrives>.

The AI handles the heavy lifting of visual recognition and data organization.

  1. Index the footage

    The AI scans your video files to understand the visual content and prepare it for searching.

    get_indexed_asset

  2. Define a subject

    Once a subject is found, the AI creates a unique identity for that person or object.

    create_entity

  3. Build a group

    The AI gathers all relevant instances of a subject into a single, manageable set.

    create_entity_collection

  4. Verify the data

    The AI presents the identified entities and collections for you to check and refine.

    get_indexed_asset

Try it

Copy these to start.

Use these prompts to trigger the tracking process.

Starting points

These are basic starting lines. Swap out the bracketed text with your actual video folder names or subject descriptions.

Twelve Labs Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_Vgl5VVTXIogBXlRhR1Xxpl2YsHWFIR0xQ7beKVtS/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Twelve Labs capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "twelve-labs-video-understanding-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_Vgl5VVTXIogBXlRhR1Xxpl2YsHWFIR0xQ7beKVtS/mcp"
    }
  }
}
Copy into chat04

  • Find all clips in the 'Security_Footage' folder that contain a person wearing a red hat.

  • Create a new entity collection for all instances of the 'Delivery Truck' in my recent uploads.

  • Identify the specific person appearing in the 'Interview_Series' assets.

  • Look through the 'Event_Video' files and create an entity for the keynote speaker.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Twelve Labs once, then ask.

Just link your account once. Your credentials stay encrypted, and you can immediately start asking your AI to find things in your videos.

Connect Twelve Labs to your AI

FAQ

How this task behaves

  • 01

    Can the AI delete my original video files?

    No. The AI only reads and indexes the files to create metadata; it cannot modify or delete your source footage.

  • 02

    Does the AI decide which entities are important?

    No. The AI identifies potential subjects, but you must tell it which ones to turn into entities or collections.

  • 03

    Can I track objects that aren't in the video?

    No. The AI can only track and index what is physically present in the video files you provide.

  • 04

    Will the AI automatically organize my entire hard drive?

    No. You must point the AI toward specific assets or folders to begin the indexing process.

  • 05

    Can the AI rename my video files?

    No. The tools are designed for indexing and entity management, not for file system manipulation.

  • More questions about Twelve Labs? The Connector page answers them. See everything the Twelve Labs Connector can do

Connect Twelve Labs to Claude, Cursor, ChatGPT & more