• LOCAL
  • VECTOR
  • MODELS
  • HUMAN

How to run local model inference with your AI

Connect your agent to Open WebUI to run prompts and generate embeddings on your own machine. Once linked, your AI handles the heavy lifting locally.

Ask AI about this page

Short answer

How can I use local Ollama models with my AI?

You can trigger local inference by giving your AI agent access to Open WebUI. It uses your local Ollama setup to generate text and create embeddings. You'll get direct control over which models run on your machine.

Where your requests land.

Local execution outcomes.

Your AI directs tasks to your local hardware based on your prompts.

LOCAL

Inference on your hardware

The AI uses your local models to answer questions or write code. This keeps your data and processing on your own machine.

VECTOR

Data embeddings

Your agent creates mathematical representations of your text. This allows for local semantic search and retrieval.

MODELS

Model management

The AI checks which models you have installed. You can see exactly what's available to run locally.

HUMAN

Manual model selection

The AI cannot install new models for you. You must download the specific models you want to use via Ollama first.

The workflow

What your AI does when a prompt arrives.

The agent translates your natural language into specific local commands.

  1. Check availability

    The agent looks at your local library to see which models are ready to work. This ensures it doesn't try to call a model you haven't downloaded.

    list_models

  2. Generate text

    Your agent sends your prompt to the selected local model. The model processes the request and returns the text response.

    ollama_generate

  3. Create embeddings

    The agent converts text into vectors. This is useful if you want to perform local RAG or search through your documents.

    ollama_embed

  4. Verify tags

    The agent checks the specific tags of your models to ensure compatibility with the requested task.

    ollama_tags

Try it

Copy these to start.

Use these prompts to kick off local tasks.

Starting points

These are opening lines. Swap out the model names or file references to match what you have on your machine.

Open WebUI Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_YhwRL8adKsCUXpVOwybG0l7vG8Iky4FGyj9DhmOn/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Open WebUI capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "open-webui-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_YhwRL8adKsCUXpVOwybG0l7vG8Iky4FGyj9DhmOn/mcp"
    }
  }
}
Copy into chat04

  • List my available Ollama models so I know what I can use.

  • Use the llama3 model to summarize this text from my local file.

  • Generate embeddings for the contents of my project folder.

  • Check which tags are available for the mistral model.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Open WebUI once, then ask.

You only need to link your Open WebUI instance once. Your credentials stay encrypted, and you can immediately start talking to your local models.

Connect Open WebUI to your AI

FAQ

How this task behaves.

  • 01

    Can the AI download new models for me?

    No. The AI can only see and use models that you have already downloaded through Ollama.

  • 02

    Does my data leave my computer?

    No. Because the inference happens through your local Ollama instance, the actual processing stays on your hardware.

  • 03

    Can the AI delete my local models?

    No. The tools only allow the AI to list, generate, and embed. It has no capability to delete or modify your local files.

  • 04

    How does the AI know which model to use?

    It uses the list of available models to identify which one matches your request or which one is best suited for the task.

  • 05

    Can I use this with any local LLM?

    It works with any model that is compatible with Ollama and visible through your Open WebUI setup.

  • More questions about Open WebUI? See everything the Open WebUI Connector can do

Connect Open WebUI to Claude, Cursor, ChatGPT & more