• QUEUE
  • STATUS
  • DATA
  • HUMAN

How to expand web crawls dynamically with your AI

Connect Apify to your agent to inject new targets into active scrapers. You can grow your datasets without restarting jobs or manual configuration.

Ask AI about this page

Short answer

Can I add new URLs to a scrape while it's running?

Yes. You can push new URLs into an active Apify queue to increase the crawl depth immediately. This allows your agent to find new links and feed them back into the scraper. You'll see how this works in the breakdown below.

Where every URL lands.

The outcomes of dynamic crawling

Your agent manages the flow of data between discovery and collection.

QUEUE

Expanded crawl depth

The AI adds new links to your active Apify run. This keeps the scraper moving through new territory.

STATUS

Real-time progress tracking

Your agent checks how many pages are left. You'll know exactly how much data is still being collected.

DATA

Fresh datasets

As the scraper hits new URLs, the AI can process the results. You get a growing pool of information to analyze.

HUMAN

Manual intervention

If a crawl hits a dead end, the AI flags it for you. You decide if the scraper needs new logic or a different target list.

The workflow

What your AI does when a new link is found.

The AI acts as the bridge between finding a URL and adding it to the workload.

  1. Identify new targets

    Your agent scans the current page content or a provided list to find relevant new URLs.

    push_to_queue

  2. Inject into the scraper

    The agent sends these new addresses directly into the active Apify job queue.

    push_to_queue

  3. Monitor the job

    The agent checks the status of the current run to ensure the new URLs are being processed.

    get_run

  4. Verify completion

    The agent confirms when the expanded crawl has finished so you can start your analysis.

    get_run

Try it

Copy these to start.

Use these prompts to trigger the Apify Connector.

Starting points

Swap out the bracketed info for your specific Apify run IDs or target domains.

Apify Connector

You're all set. Choose your MCP client and follow the setup instructions.

Connector linkhttps://edge.vinkius.com/vk_preview_kyOCHu4Rvz1CsEZ7ViQMTOM5deCOQwTatWoMs4PP/mcp

Claude Desktop

Follow the steps below to connect in seconds.

  1. 1In Claude Desktop, open Settings → Connectors.
  2. 2Click “Add custom connector” and paste the connector link above as the remote MCP server URL.
  3. 3Click Add and start a new chat — Apify capabilities are ready to use.
Configuration · claude_desktop_config.jsonCopy
{
  "mcpServers": {
    "apify-mcp": {
      "url": "https://edge.vinkius.com/vk_preview_kyOCHu4Rvz1CsEZ7ViQMTOM5deCOQwTatWoMs4PP/mcp"
    }
  }
}
Copy into chat04

  • Push these three URLs to my active Apify run: [URL 1, URL 2, URL 3]

  • Check the status of my Apify run [Run ID] and tell me how many items are left.

  • Find all product links on this page and push them to my current Apify queue.

  • Monitor my Apify run [Run ID] and let me know when the status changes to finished.

  • Claude
  • ChatGPT
  • Cursor
  • VS Code
  • Windsurf
  • Claude Code
  • JetBrains
  • Cline

Start here

Connect Apify once, then ask.

Just link your Apify account through Vinkius. Your credentials stay encrypted, and you can start managing crawls through your agent immediately.

Connect Apify to your AI

FAQ

How this task behaves

  • 01

    Can the AI start a brand new scrape from scratch?

    No. This connector is designed to interact with existing runs by pushing to a queue or checking status.

  • 02

    Will the AI delete my existing Apify datasets?

    No. The AI only has permission to push new items to a queue or read the status of a run.

  • 03

    Does the AI handle the actual web scraping logic?

    No. Apify performs the scraping. The AI simply manages the list of URLs being fed into that process.

  • 04

    Can I stop a run using the AI?

    No. The current tools only allow for pushing to a queue and getting run status.

  • 05

    How does the AI know which run to add URLs to?

    You must provide the specific Run ID in your prompt so the agent knows which queue to target.

  • More questions about Apify? See everything the Apify Connector can do

Connect Apify to Claude, Cursor, ChatGPT & more