What an AI agent looks like in your logs
· observability, loki, logql, security, llm, bots
I wrote two posts recently that turn out to be halves of the same thing.
One was about what the web serves an agent: route the same request through a few datacenter exit nodes and watch the responses change. That was the outbound direction, me looking at how the internet treats my code.
The other was about running Loki and Alloy to get logs off a handful of machines and into one place I can query.
Point the second at the first and you get the inbound question. Other people's agents are hitting your services right now. Can you see them?
Most people can't, and the reason is boring. Default access logs throw away the fields that would tell you.
Why this is worth an afternoon
Nobody sets out to be blind to this. It happens because access logs were built to answer "did the request succeed" and nothing else, and because "was that a person" was not an interesting question until recently.
It is now. Well-behaved crawlers announce themselves with something like a dozen distinct user agents, most of which did not exist two years ago. The badly behaved ones announce nothing. And some share of what used to be human traffic is a model fetching your page for somebody, which quietly changes what your analytics mean without changing what they say.
You don't have to do anything about any of it. You should be able to answer the question.
Log the fields you'll need
The nginx combined format was designed in the nineties and it shows. You get an IP, a timestamp, a request line, a status, a byte count, a referer, and a user agent. That's most of what you need and none of what makes it queryable.
Switch to structured output. In nginx:
log_format json_access escape=json
'{'
'"time":"$time_iso8601",'
'"ip":"$remote_addr",'
'"method":"$request_method",'
'"path":"$uri",'
'"status":$status,'
'"bytes":$body_bytes_sent,'
'"ua":"$http_user_agent",'
'"referer":"$http_referer",'
'"accept":"$http_accept",'
'"proto":"$server_protocol",'
'"ims":"$http_if_modified_since",'
'"inm":"$http_if_none_match",'
'"ms":$request_time'
'}';
access_log /dev/stdout json_access;Four of those fields are unusual, and they're the ones that carry the signal. The accept header,
the protocol version, and the two conditional-request headers separate real browsers from HTTP
clients pretending to be browsers, which I'll come back to.
Logging to stdout means Alloy picks it up through the same Docker discovery as everything else, so
there's nothing new to configure on the collector beyond adding the logging label to the proxy
container.
One thing to get right, and it's the same trap as before. Keep ip and ua inside the log line
where they belong. Promoting either to a Loki label creates a stream per client and your instance
falls over inside a week. Query them with | json and leave the label set small.
The ones that tell you
Start with the easy population. A lot of crawlers identify themselves honestly, and you can count them with a single query:
sum by (bot) (
count_over_time(
{app="proxy"} | json
| ua =~ "(?i).*(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider|Meta-ExternalAgent|Google-Extended|Amazonbot|Applebot-Extended).*"
| label_format bot=`{{ regexReplaceAll ".*(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider|Meta-ExternalAgent|Google-Extended|Amazonbot|Applebot-Extended).*" .ua "${1}" }}`
[1h]
)
)Ugly, and it works. You get a stacked graph of who is crawling you and how hard.
These split into two behaviors that people conflate. Training crawlers like GPTBot and CCBot harvest broadly on their own schedule. User-triggered fetchers like ChatGPT-User and Claude-User fire when somebody asks a question your page answers, so they arrive one request at a time and track real human interest. A lot of the second kind is a signal about what to write next.
The ones that don't
Declared bots are the polite minority. The rest show up as python-requests/2.32, curl/8.5,
Go-http-client/2.0, node-fetch, axios, or a copy-pasted Chrome string from 2023.
Generic clients are trivial to catch:
sum by (ua) (
count_over_time(
{app="proxy"} | json
| ua =~ "(?i).*(python-requests|httpx|aiohttp|curl|wget|go-http-client|node-fetch|axios|okhttp|scrapy).*"
[24h]
)
)The interesting population is what's left over. Clients claiming a browser user agent that aren't browsers.
Three tells that don't depend on the user agent
A user agent string is a claim, and anyone can type anything into it. These are observations about behavior, which is harder to fake and mostly nobody bothers.
Nobody loads your CSS
This is the strongest single signal and it costs nothing to compute. Load a page in a browser and it immediately fetches the stylesheet, the fonts, the favicon, a script or two, and the images. An HTTP client fetches the HTML and stops.
So compare, per client, how many document requests it made against how many asset requests:
sum by (ip) (count_over_time({app="proxy"} | json | path !~ ".*\\.(css|js|png|jpg|svg|woff2?|ico)$" [1h]))
/
sum by (ip) (count_over_time({app="proxy"} | json | path =~ ".*\\.(css|js|png|jpg|svg|woff2?|ico)$" [1h]) > 0)A browsing human sits somewhere below one. An agent runs high, and a client that fetched forty pages and zero stylesheets is not a person no matter what its user agent says.
The same idea shows up in the accept header. Browsers send a long negotiated string starting with
text/html,application/xhtml+xml. Libraries send */* or nothing.
{app="proxy"} | json | accept = "*/*" | path !~ ".*\\.(css|js|png|jpg|svg|woff2?|ico)$"Nothing ever gets cached
Browsers revalidate. Once they've seen a page they send If-None-Match or If-Modified-Since and
take a 304 when nothing changed. Almost no automated client does this, because caching is work
nobody implements until they have to.
sum(count_over_time({app="proxy"} | json | inm = "" and ims = "" [1h]))A client that has requested the same URL five times and never once sent a conditional header is either automated or having a very strange day.
The timing is inhuman
People read. They land on a page, spend a while, click something. Agents fetch in bursts with sub-second gaps, or on suspiciously regular intervals.
sum by (ip) (count_over_time({app="proxy"} | json | path !~ ".*\\.(css|js|png|jpg|svg|woff2?|ico)$" [1m]))Twenty documents in sixty seconds from one address is not reading. Regularity is the other half of this. A request every three hundred seconds, forever, is a cron job wearing a browser costume.
Bringing the ASN back
The outbound post was largely about ASN reputation, and it applies just as well pointing the other way. Traffic from AWS, GCP, Azure, Hetzner, or OVH is running on a server. That doesn't make it hostile, and combined with the behavioral signals above it's close to conclusive.
Alloy can annotate this during ingestion with a MaxMind GeoLite2 ASN database:
loki.process "enrich" {
stage.json {
expressions = { ip = "ip" }
}
stage.geoip {
source = "ip"
db = "/etc/alloy/GeoLite2-ASN.mmdb"
db_type = "asn"
}
forward_to = [loki.write.default.receiver]
}Now every log line carries the network it came from, and separating datacenter traffic from residential traffic becomes a filter rather than a research project.
Resist the urge to make the ASN a label. There are tens of thousands of them.
The dashboard
Four panels have earned their place on mine.
Declared crawler volume over time, split by bot, which tells you who has discovered you. Unknown automated clients over time, using the asset-ratio query, which is the population worth watching. Requests grouped by ASN organization, which surfaces anything running at scale from one provider. And a table of the top clients by document count with their asset ratio next to it, which is where you actually look when something seems off.
Alerting on any of this is usually premature. The volume is small, the consequences are minor, and you'd be paging yourself because a search engine indexed your blog. Build the panels, look at them occasionally, and add an alert only once you know what normal is.
What I don't do with it
Blocking, mostly. This blog is a static site on GitHub Pages and the crawlers are welcome to it.
Most of what shows up is search infrastructure doing its job, and the AI crawlers that respect
robots.txt will respect it if I ever decide I care.
The value is in knowing. Being able to answer "is this real traffic or a scraper" without guessing changes how you read an analytics spike. Recognizing that a burst of ChatGPT-User requests means people are asking questions your content answers is genuinely useful information about what to write next.
Blocking is available if you need it, and it works better against declared crawlers than anything else, because a bot that identifies itself is a bot that can be told no.
Where this falls down
Determined automation defeats every signal above. A headless browser loads your CSS, sends real Accept headers, revalidates its cache, waits a plausible interval, and runs through a residential proxy. What you're catching here is the lazy majority, and the lazy majority is most of the traffic.
Static hosting gives you nothing. GitHub Pages and most CDN tiers don't hand you access logs at all, which is why my own blog contributes nothing to this and everything above runs against services I host myself. If your site is on a platform without log access, the technique is unavailable and no amount of Loki config changes that.
IP addresses are a weak identifier. Mobile carriers NAT thousands of users behind one address and cloud clients rotate constantly, so per-IP aggregation is a heuristic rather than an identity.
Do it anyway. Going from no idea to a rough idea is a bigger jump than any refinement you make afterward.
Both halves come down to the same thing. Your code is an HTTP client that other people's servers are quietly scoring, and your server is being read by other people's code. Neither shows up unless you go looking, and going looking costs one nginx config change and about six queries.