David Fox

Your agent is reading a different web than you are

· docker, gluetun, vpn, llm, mcp, networking

An agent I was working on fetched a documentation page, summarized it, and told me something confidently wrong. The page it summarized was a 403.

That's the whole reason this post exists. The tool did its job, the fetch returned 200-ish looking bytes, the model read what it was handed, and nothing anywhere in the chain said "by the way, the thing you just read was a block page." I'd spent an entire post on the tastytrade MCP server arguing that you should never let a model launder missing data, and then got caught by the same failure one layer down, in the network.

So I wanted to actually measure it: does the web serve me something different depending on where I come from? Here's the setup I use, and what it turned up.

What I'm actually measuring

Two things need saying up front, because a lot of writing on this topic is sloppy about both.

This does not get you residential IPs. AirVPN exit nodes live in datacenter ranges that every serious anti-bot vendor has had catalogued for years. Routing through them makes you look more like a bot, not less. That's the point. I want to measure the penalty that datacenter and VPN egress carries, because that's the penalty my server-side tools are already paying every day. Residential proxy networks are a different product with a different ethical profile, and they're not what this is.

Keep the client constant. If you compare your browser against curl-through-a-VPN, you've changed the ASN, the TLS fingerprint, the HTTP/2 settings, the user agent, and the behavioral profile all at once, and you've learned nothing about which one mattered. Every request below uses the same curl, the same flags, and the same UA string. The only variable is where the packets come out.

The compose file

Three exit nodes, each running an HTTP proxy on its own host port. No VPN on the host, nothing touching your default route.

# compose.yaml
x-exit: &exit
  image: qmcgaw/gluetun:v3
  cap_add: [NET_ADMIN]
  devices: ["/dev/net/tun:/dev/net/tun"]
  restart: unless-stopped

x-exit-env: &exit-env
  VPN_SERVICE_PROVIDER: airvpn
  VPN_TYPE: wireguard
  WIREGUARD_PRIVATE_KEY: ${AIRVPN_PRIVATE_KEY}
  WIREGUARD_PRESHARED_KEY: ${AIRVPN_PRESHARED_KEY}
  WIREGUARD_ADDRESSES: ${AIRVPN_ADDRESSES}
  SERVER_COUNTRIES: United States
  HTTPPROXY: "on"

services:
  exit-chi:
    <<: *exit
    container_name: exit-chi
    environment:
      <<: *exit-env
      SERVER_CITIES: Chicago
    ports: ["8888:8888"]

  exit-lax:
    <<: *exit
    container_name: exit-lax
    environment:
      <<: *exit-env
      SERVER_CITIES: Los Angeles
    ports: ["8889:8888"]

  exit-mia:
    <<: *exit
    container_name: exit-mia
    environment:
      <<: *exit-env
      SERVER_CITIES: Miami
    ports: ["8890:8888"]

Every container runs the same image with the same credentials and differs only in which city it lands in and which host port its proxy answers on. The container-side proxy port stays 8888 in all three; the host mapping is what varies.

Your .env comes straight out of AirVPN's config generator. Pick WireGuard, generate a config, and lift three values:

AIRVPN_PRIVATE_KEY=...
AIRVPN_PRESHARED_KEY=...
AIRVPN_ADDRESSES=10.x.x.x/32

One AirVPN key works across all three containers.

Bring it up:

docker compose up -d
for p in 8888 8889 8890; do
  echo -n "$p: "
  curl -s -x "http://localhost:$p" https://ipinfo.io/json | jq -c '{ip, city, org}'
done

Do not skip this step. SERVER_CITIES values have to match the names in gluetun's embedded server list, and a name it doesn't recognize gets you a connection to somewhere else entirely rather than an error. Gluetun also exposes a control server on port 8000 with a /v1/publicip/ip endpoint that reports the same thing, though recent versions gate those routes behind an auth config file, so the Using ipinfo for the round-trip is a quicker way to check.

The probe

#!/usr/bin/env bash
# probe.sh <url>
set -euo pipefail
URL="$1"
UA="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36"

printf "%-10s %6s %9s %8s  %s\n" EXIT CODE BYTES SECS NOTES

probe() {
  local name="$1" proxy="$2"
  local out hdrs
  hdrs="$(mktemp)"
  out="$(curl -sS -A "$UA" -o /dev/null -D "$hdrs" \
        -w '%{http_code} %{size_download} %{time_total}' \
        ${proxy:+-x "$proxy"} "$URL" 2>/dev/null || echo "000 0 0")"
  local notes
  notes="$(grep -iE '^(cf-mitigated|cf-ray|server|x-served-by|retry-after):' "$hdrs" \
           | cut -d: -f1 | paste -sd, -)"
  printf "%-10s %6s %9s %8s  %s\n" "$name" $out "${notes:-none}"
  rm -f "$hdrs"
}

probe direct ""
probe chicago "http://localhost:8888"
probe la      "http://localhost:8889"
probe miami   "http://localhost:8890"

Four requests, same client, four egress points. Status code, response size, wall time, and which interesting headers came back.

Output looks something like this:

EXIT         CODE     BYTES     SECS  NOTES
direct        200     48219    0.412  server
chicago       403      1102    0.388  cf-mitigated,cf-ray,server
la            403      1102    0.401  cf-mitigated,cf-ray,server
miami         200     31447    0.455  cf-ray,server

Pay attention to the BYTES column. If you get a 200 status but the response is much smaller than what you see on your home connection, that's a red flag--your code won't catch it.

What actually turns up

Four categories, roughly in order of how easy they are to notice.

Hard blocks. A 403, or a 402 if the site has opted into Cloudflare's pay-per-crawl. The 402 is worth internalizing: Payment Required has gone from a reserved curiosity to a status code you will actually see in production, and it means the site has priced access for automated clients and you haven't paid. These are the loud failures.

Challenges. A 200 carrying an interstitial instead of content. cf-mitigated: challenge is the tell. Your HTTP client sees success, your parser sees markup, and your model sees a page about enabling JavaScript.

Soft degradation. A real 200 with real content, minus things. Missing JSON-LD, truncated listings, no prices, a stub instead of the article body. This is the category that ruins agents, because there's no signal at all--the response is well-formed, plausible, and incomplete.

Rate limit asymmetry. The same request budget that lasts all day from home gets you a If you connect from a VPN or datacenter ASN, you'll start seeing retry-after headers much sooner—sometimes after just a few requests. On your normal connection, good reputation means you get more leeway. But you don't from these ranges.

Why the ASN is doing so much work

Detection in 2026 is layered and scored, not just one check. IP and ASN reputation, TLS fingerprint, HTTP/2 frame ordering, user agent, and behavior over time all feed a number, and you have to clear the bar on the total.

ASN reputation is the cheapest layer to run and the hardest for you to change. Cloudflare, DataDome, and Akamai all flag datacenter ranges by default, and the pressure behind that has gotten substantially worse: AI-driven traffic is up sharply year over year, agentic-browser traffic especially. Since mid-2025, Cloudflare has started blocking AI crawlers by default on new domains. For a site owner, blocking datacenter traffic is an easy way to cut out most of the bots and noise. But for the rest of us, it means your code is stuck with an IP reputation you didn't make and can't do anything about.

Notice also that my Miami exit behaved differently from Chicago and LA in the sample above. Exit nodes on the same provider don't share a reputation. Individual ranges get burned by whatever else was running through them last week, which is a good argument for probing several rather than assuming "VPN" is one condition.

The part that matters if you build agents

Every server-side tool that reads the web has this problem, and almost none of them report it.

A WebFetch-style tool, an MCP server that pulls a page, a RAG pipeline refreshing its corpus, a scheduled job enriching records--they all run from a datacenter, which means they all get the datacenter tier of the web. And the standard implementation checks response.ok, gets true, hands the body to a model, and the model does exactly what models do with a challenge page, which is describe it fluently.

The fix is the same one I keep arriving at: make the tool's uncertainty structural instead of implied.

  • Check size against a baseline, not just status. A response 80% smaller than the last successful fetch of that host is a finding, whatever the code says.
  • Surface the block signals. cf-mitigated, retry-after, and a 402 belong in the tool's return value where the model can see them, not swallowed in a client library.
  • Give "I was blocked" a name. My stock screener returns INSUFFICIENT_DATA as a first-class outcome rather than an empty list. A fetch tool needs the same thing. BLOCKED_AT_EDGE beats an empty string every single time, because an empty string looks like a successful answer.
  • Fail loudly on challenge markup. Detecting an interstitial is a substring match. Doing it costs nothing and turns a silent wrong answer into an error you can act on.

None of that gets you around a block, but at least you know when it happens.

Pointing it at your own site

The same rig works in reverse, and this is the use I'd defend to anyone.

If you run bot rules, a WAF, or a CDN with a managed ruleset, you have no idea what they do to real clients until you look from outside your own network. Point probe.sh at your own domain and check that a plain curl from three ASNs gets the page. I ran it against this site and against the self-hosted GoatCounter endpoint behind it, mostly to confirm that my analytics beacon isn't quietly failing for anyone browsing through a VPN. On a static site with no WAF it's a boring result, which is the correct result.

It's a better check than it sounds, because the failure mode it catches is invisible from your desk: you browse from a residential IP with warm reputation, everything looks fine, and a slice of your actual audience is getting a challenge page.

Housekeeping

Kill the stack when you're done rather than leaving three tunnels up:

docker compose down

Two limits worth stating plainly. Latency through a VPN exit tells you very little about real user latency from that city, so this is a correctness harness rather than a performance one. And curl's TLS fingerprint is nothing like a browser's, so what you're measuring throughout is how the web treats an obvious automated client from various ASNs. That happens to be exactly the question I care about, because it's what my tools are.

Obvious but worth writing down: this measures how sites respond, and that's all I use it for. Read robots.txt, keep your request volume somewhere near sane, and if a site has priced automated access and you want that access, pay for it. I didn't come out of this wanting a way around the wall. I came out of it knowing my tooling couldn't tell when it hit one.