# Anyscale MCP for AI Agents AI Agent Connect > Anyscale MCP lets you manage your entire inference and backend infrastructure from your AI agent. Instead of jumping between the cloud console and your terminal, you can query models, monitor Ray services, check batch job statuses, and generate embeddings through a simple conversation. It connects your Anyscale environment directly to your workflow. ## Overview - **Category:** ai-frontier - **Price:** Free - **Endpoint:** https://edge.vinkius.com/vk_preview_TYJFZRqD4R2R9l7bu7Vxd494Pf6ounExJTsky8LF/ai-agent-connect - **Tags:** distributed-computing, llm-inference, vector-embeddings, cluster-management, scalable-ai ## Description Managing a distributed AI infrastructure usually means having dozens of browser tabs open just to see if a training job failed or to check the health of a deployed service. This Connector changes that by bringing your Anyscale environment into your AI client. You can ask your agent to pull the current status of your Ray cluster or find out which foundational models are currently active in your environment. It handles the heavy lifting of navigating the cloud dashboard for you. If you're running production inference, you can check service endpoints and monitor live configurations without leaving your workspace. It also handles the data side, letting you generate vector embeddings for text arrays on the fly. By using this Connector through the Vinkius catalog, you get a direct line to your scalable infrastructure. It's about moving from manual monitoring to conversational management, where you can debug service health metrics or inspect recent execution statuses as easily as asking a teammate for an update. ## Tools ### list_models See all foundational models currently active in your Anyscale environment. This helps you identify which models are ready for inference. ### chat_completion Send conversational messages to your hosted LLMs and get responses back. It makes it easy to interact with your private models through your agent. ### text_completion Generate text completions using your generic Anyscale completion API. This is useful for foundational instruct generation tasks. ### generate_embeddings Create semantic vector embeddings for text inputs directly through your agent. You can use this to build out your data pipeline in real time. ### list_services Get a full list of all services currently deployed on your Anyscale platform. It gives you a high level view of your deployed fleet. ### get_service Pull specific details and configuration data for a single Anyscale service. Use this to debug specific endpoint statuses or health metrics. ### list_jobs View all batch and training jobs to check their recent execution status. This allows you to monitor training progress without leaving your workspace. ## Prompt Examples **Prompt:** ``` List all active models from my Anyscale cluster. ``` **Response:** ``` I've fetched your models. Currently, you have `meta-llama/Llama-2-70b-chat-hf` and `mistralai/Mistral-7B-Instruct-v0.1` ready to receive inference traffic. ``` **Prompt:** ``` Check the status of our latest training jobs. ``` **Response:** ``` Looking up recent jobs. The 'daily_retrain_v3' job succeeded 5 hours ago. However, 'experimental_tuning_7b' was interrupted due to out-of-memory errors on node 12. Would you like me to fetch more service metadata on that specific node? ``` **Prompt:** ``` Generate vector embeddings for the text: 'The sun rises in the east and sets in the west' ``` **Response:** ``` I've passed the text to your configured Anyscale embedding model. The embedding array generated successfully with 768 dimensions. Since the array is large, I've mapped the output directly to the data pipeline. ``` ## Capabilities ### Query active foundational models See which models are currently live in your environment. ### Generate semantic vector embeddings Create embeddings for text arrays directly during your data processing. ### Monitor Ray service health Check the status and endpoint configurations of your deployed services. ### Inspect cluster batch jobs View execution statuses and training metrics for your batch jobs. ### Run conversational LLM completions Send prompts to your hosted models and get responses back through your agent. ## Use Cases ### Checking if a training job failed A data scientist needs to see if their latest run crashed. They ask the agent to check the status of a specific job using `list_jobs`. ### Verifying available models An engineer needs to know which LLMs are ready for production. They use `list_models` to get a live count of active models. ### Debugging a failing endpoint A developer notices a service is acting up. They use `get_service` to pull the configuration and health metrics for that specific endpoint. ### Batch processing text embeddings A user wants to turn a CSV of text into vectors. They ask the agent to run `generate_embeddings` on the data pipeline. ## Benefits - Stop switching tabs by checking Ray cluster status and service health metrics directly in your AI client using `list_services`. - Speed up your data pipeline by using `generate_embeddings` to create vectors for text arrays on the fly. - Reduce manual overhead for MLOps by using `list_jobs` to monitor training progress and execution errors. - Simplify model discovery with `list_models` to see what's ready for inference in your private environment. - Debug production issues faster by using `get_service` to pull live endpoint configurations without the cloud dashboard. - Run inference tasks more naturally by using `chat_completion` to talk to your hosted models via your agent. ## How It Works The bottom line is you get a conversational interface for your entire Anyscale and Ray infrastructure. 1. Subscribe to the Anyscale MCP on Vinkius. 2. Provide your Anyscale API Key and Base URL in your AI client. 3. Ask your agent to list models, check job status, or generate embeddings. ## Frequently Asked Questions **Can I use Anyscale MCP to manage my Ray clusters?** Yes, you can monitor your Ray services and check cluster job statuses directly through your AI agent. It gives you a real-time look at your distributed computing environment without needing to log into a separate dashboard. **Does Anyscale MCP support generating embeddings?** Yes, you can generate semantic vector embeddings for text inputs on the fly. This makes it easy to integrate embedding generation into your data pipelines using just natural language. **How do I see my active models with Anyscale MCP?** You can simply ask your agent to list the models currently active in your environment. It will pull the list of foundational models that are ready for inference for you. **Can I debug my services with Anyscale MCP?** Yes, you can pull specific details and configuration data for any service in your fleet. This helps you quickly check endpoint statuses and health metrics during production. **Is Anyscale MCP good for MLOps engineers?** It's built for MLOps workflows. It automates the inspection of deployed models and batch jobs, making it much easier to monitor your infrastructure during CI/CD cycles. **Can I run inference tasks using Anyscale MCP?** Yes, you can send prompts to your hosted LLMs to generate conversational or text completions. It connects your agent to your private models for secure inference. **Does Anyscale MCP work with Cursor or Claude?** Yes, it works with any MCP-compatible client. Once you connect it, you can manage your Anyscale and Ray infrastructure from your favorite AI workspace. **Can I query a Llama 3 model that is locally deployed in Anyscale?** Yes. First ask the agent to list the available model APIs using `list_models` so it can grab the precise namespace (e.g. meta-llama/Llama-3-70b-instruct). Then, ask it to run `chat_completion` pointing at that specific ID. You are now effectively chaining your local agent with an enterprise-sized foundational model in your own VPC. **Is it possible to check whether my training job timed out without opening the Anyscale Dashboard?** Absolutely. Use the `list_jobs` tool directly from your chat workflow. It will pull down the state of recent tasks (running, failed, succeeded) alongside metrics. The agent can immediately summarize issues if it sees any errors, saving you a context switch. **Can I use Anyscale to process my text chunks into Vectors inside a project pipeline?** Yes. This Connector comes with an explicit `generate_embeddings` tool mapped to your Anyscale endpoints. By providing arrays of chunks, the Anyscale fast backbone will return your high-dimensional vectors. Your custom Agent can wrap this into scripts to hydrate vector databases faster.