How to Use the NVIDIA Vision MCP in OpenAI Agents SDK

Connect the NVIDIA Vision MCP Server to your OpenAI Agents SDK for secure, guarded image analysis and generation.

See Vinkius in Action

Works with every AI agent you already use

…and any MCP-compatible client

MCP Servers - Free for Subscribers

Connect NVIDIA Vision MCP to OpenAI Agents SDK

Create your Vinkius account to connect NVIDIA Vision to OpenAI Agents SDK and route execution through our secure gateway. The platform manages server hosting, runtime updates, and security layers. Configuration requires no manual server provisioning.

GDPR Free for Subscribers

Setup NVIDIA Vision with OpenAI Agents SDK

Ask AI about this MCP

ChatGPT

Claude

Perplexity

Run `detect_objects` inside your OpenAI Agents SDK

The `detect_objects` tool lets your agent analyze images and identify objects directly within your safety-constrained pipelines. By exposing this capability to your agent, you can extract spatial coordinates and labels without writing custom parsing code. It fits right into your agent's decision loops. When you pass this tool through the MCP Server interface, the SDK handles argument verification before the model executes. This prevents the model from passing invalid image payloads to the NVIDIA API.

Parse receipts and forms with `document_qa`

The `document_qa` tool processes scanned documents, receipts, and forms to answer complex textual questions about their layout and content. Your agent can call this tool to extract raw text and structural data in a single step. Because OpenAI Agents SDK supports direct handoffs, you can have a specialized document agent run this tool and then pass the structured text back to a billing agent. It keeps your main agent focused on logic rather than parsing messy visual data.

Generate images using `generate_image` on demand

The `generate_image` tool outputs high-resolution images using Stable Diffusion models based on text prompts from your agent. It supports standard 1024x1024 formats, letting your agent create visual mockups during conversations. You can monitor these generation steps in your OpenAI dashboard since every MCP tool call is traced automatically. This gives you complete visibility into model selection and prompt tokens.

Setup guide

Set up NVIDIA Vision MCP in OpenAI Agents SDK

Prerequisites

Python 3.10+ installed
openai-agents package (pip install openai-agents)
Active Vinkius subscription with a valid endpoint token

1

Install the SDK

Run pip install openai-agents to install the OpenAI Agents SDK. The MCP integration is built-in — no extra dependencies needed.
2

Connect via SSE transport

Use MCPServerSse with your Vinkius endpoint URL. Replace [YOUR_TOKEN_HERE] with your token from cloud.vinkius.com. The SDK auto-discovers all NVIDIA Vision tools at runtime.
3

Create your Agent

Pass the MCP to Agent(mcp_servers=[server]). The agent receives NVIDIA Vision tools as native definitions — JSON schemas resolve automatically.
4

Run the agent

Call Runner.run(agent, prompt) to execute. The agent invokes the appropriate NVIDIA Vision tools and returns structured results. Copy the full example on the right to get started.

agent.py

import asyncio
from agents import Agent, Runner
from agents.mcp import MCPServerSse

async def main():
    async with MCPServerSse(
        url="https://edge.vinkius.com/[YOUR_TOKEN_HERE]/mcp"
    ) as server:
        agent = Agent(
            name="NVIDIA Vision Agent",
            instructions="You have access to NVIDIA Vision tools.",
            mcp_servers=[server],
        )
        result = await Runner.run(agent, "List recent transactions")
        print(result.final_output)

asyncio.run(main())

Get your connection token →

Independent Platform Disclaimer: Vinkius is an independent platform and is not affiliated with, endorsed by, sponsored by, verified by, or otherwise authorized by NVIDIA. All third-party trademarks, logos, and brand names are the property of their respective owners. Their use on this website is strictly for informational purposes to identify service compatibility and interoperability.

Why Choose Vinkius

Vinkius connects your tools to AI with real-time monitoring and automatic cost savings — all from one dashboard.

Connect NVIDIA Vision now

Real-time monitoring

Live

visibility into every interaction

Connect your favorite tools to your AI and see exactly what's happening — every request, every response, in real time.

Built-in savings

60%

lower AI costs

Vinkius compresses data between your apps and your AI automatically. Lower bills every month — no configuration required.

Single dashboard

One

place for every integration

Every tool your AI connects to, managed from a single screen. One account, complete control.

Common questions about NVIDIA Vision MCP in OpenAI Agents SDK

Use the MCPServerStreamableHttp class to point to your hosted endpoint. Pass this instance in the mcp_servers list when initializing your agent.

Yes, every tool invocation is fully visible. The OpenAI Agents SDK automatically logs the inputs and outputs of tools like image_segmentation directly to your tracing history.

Setting cacheToolsList=True in your configuration caches the MCP tool definitions. This reduces latency on agent startup by preventing redundant schema lookups.

You can choose between stabilityai/stable-diffusion-3-medium and stabilityai/stable-diffusion-xl-base-1.0. Pass your choice directly to the generate_image tool.

Your images and scanned documents are processed in an ephemeral V8 sandbox. Vinkius handles the underlying authentication tokens securely, ensuring your raw visual payloads are never stored on disk.

Use it with your favorite AI tools

Connect this server to Cursor, Claude, VS Code, and more.

OpenAI Agents SDK sdk-python

Google ADK sdk-python

Pydantic AI sdk-python

Vercel AI SDK sdk-typescript