Using AI Agents for Web Scraping: How to Set Up a Proxy

Sites decide how agents get in

Websites set their own rules for automated visitors, and those rules increasingly mention AI agents by name.

The direction is clear: each site decides which automated visitors it lets in. A good agent reads those rules and follows them.

When a proxy helps, and when it doesn't

A proxy helps with tasks you're allowed to do:

• Checking your own listings and prices.
• Monitoring public pages within a site's terms and robots.txt.
• Logged-in work on accounts you own or manage.
• Seeing a site the way a visitor in another country does.

It doesn't help when the rules say no:

• A proxy doesn't change what a site allows.
• If a site's terms or robots.txt rule out automated access, respect that and look for an official API.

Short version: A proxy changes the IP your agent uses, not the rules it has to follow.

Static ISP or rotating residential for agents?

Ours in the US are static US ISP proxies in Ashburn, Virginia, and New York. For the rotating side, see residential proxies.

Set up a proxy for your agent

Web scraping using AI agents comes down to one thing on the network side: every request your agent makes should leave through the proxy you picked. Our proxies support HTTP and HTTPS only for now, and most agent frameworks and headless browsers accept them.

The proxy URL. Every proxy is a host, a port, a username and a password, written as one URL:

http://user:pass@host:port

Environment variables. Many tools read the standard proxy variables, so setting them once covers the whole process:

export HTTP_PROXY="http://user:pass@host:port"
export HTTPS_PROXY="http://user:pass@host:port"

In code. Credentials come from environment variables, never from the code. A short Python sample with the requests library:

import os
import requests
proxy = os.environ["QP_PROXY_URL"]  # http://user:pass@host:port
proxies = {"http": proxy, "https": proxy}
r = requests.get("https://example.com/", proxies=proxies, timeout=30)
print(r.status_code)

In a headless browser. Most headless browsers take a proxy setting when they launch: the server as http://host:port, plus the username and password. In outline:

launch options:
proxy server:   http://host:port
proxy username: read from QP_PROXY_USER
proxy password: read from QP_PROXY_PASS

Map each agent to its own proxy. Give every agent or task one proxy URL, so each session keeps its IP from start to finish:

import os
AGENT_PROXIES = {
"price-checker": os.environ["QP_PROXY_A"],
"account-agent": os.environ["QP_PROXY_B"],
"listing-monitor": os.environ["QP_PROXY_C"],
}
def proxies_for(agent):
url = AGENT_PROXIES[agent]
return {"http": url, "https": url}

Test it first. Before the agent starts work, call an IP-echo endpoint through its proxy and check that the IP and country are the ones you expect:

IP_ECHO_URL = os.environ["IP_ECHO_URL"]  # any IP-echo service you trust
r = requests.get(IP_ECHO_URL, proxies=proxies_for("price-checker"), timeout=30)
print(r.text)  # check the IP and country before the agent starts

Manage proxies from your AI assistant (MCP)

Quantum runs a hosted MCP server, so you can manage your proxies from your AI assistant in plain English. Per our MCP docs, your assistant can:

• List your residential plans and generate residential proxies.
• Read residential usage and 30-day stats.
• List your static ISP plans with their proxy lists.
• Check subscription status.

What it doesn't do: it doesn't fetch or parse pages. The workflow has two steps: your assistant pulls the proxy list over MCP, then your agent uses those proxies over HTTP/HTTPS.

API keys are issued by our team, not self-serve. Ask on Discord or at [email protected]. Connection steps for each client are in the MCP docs.

Where to run your agent

Your own computer is fine for light use. A VPS keeps the agent on all day.

Our VPS:

• Our VPS run in Frankfurt, Germany.
• Ubuntu, Debian or Windows Server 2022.
• Unlimited fair traffic.
• Built for uptime.

Plans and sizes are on VPS. They're CPU-based plans, sized for agents that call hosted AI models.

Be honest about location: our VPS is in Frankfurt. If your agent needs a US IP, pair the VPS with static US ISP proxies in Ashburn, Virginia, and New York.

A good-citizen checklist for agent builders

• Respect each site's terms and robots.txt.
• Keep request rates modest.
• Identify your agent the way the site asks, for example in its user-agent string or sign-in flow. A proxy doesn't replace that.
• Stop when a site denies access.
• Keep API keys and proxy credentials in environment variables.
• Log what your agent does.

You're responsible for following the laws that apply to you and each website's terms. See our Terms of Service.