AI crawler visibility

GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot fetch your pages to build the indexes and training sets that AI assistants answer from. None of them execute JavaScript, so no pixel can see them — ours, Shopify’s, or anyone else’s. Your server is the only place those fetches exist. Reporting them tells you which of your pages AI systems actually read.

Next.js

proxy.ts

import { withAgentShopCrawlerCapture } from "@agentshop/seo/next/proxy";

export const proxy = withAgentShopCrawlerCapture();

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Next 16 renamed middleware.ts to proxy.ts and the exported function from middleware to proxy. On Next 15 and below, name the file middleware.ts and export it as middleware — the wrapper itself is unchanged. Next ships a codemod: npx @next/codemod@canary middleware-to-proxy .

Already have one? Pass it in. Yours runs first and its response is returned untouched:

proxy.ts

import { withAgentShopCrawlerCapture } from "@agentshop/seo/next/proxy";
import { NextResponse } from "next/server";

export const proxy = withAgentShopCrawlerCapture(async (request) => {
  // your existing logic
  return NextResponse.next();
});

Reporting is deferred with event.waitUntil, so it never delays a response — and, unlike a bare unawaited fetch, it isn’t dropped when a serverless invocation freezes. Use the matcher to skip static paths rather than expecting the wrapper to do it. Routing is the framework’s job and a matcher costs nothing at runtime.

Cloudflare

If your storefront sits behind Cloudflare, the Next proxy only sees requests your app actually renders — responses served from the edge cache never reach it. A Worker sees all of them:

src/index.js

import { reportIfCrawler } from "@agentshop/seo";

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    // The whole capture recipe — isbot gate, origin+pathname stripping (no
    // query strings, no fragments), referrer handling — in one call. The
    // Worker performed the fetch itself, so it has a real status to report.
    ctx.waitUntil(
      reportIfCrawler(request, {
        apiKey: env.AGENTSHOP_API_KEY,
        baseUrl: env.AGENTSHOP_SEO_URL,
        statusCode: response.status,
      }),
    );
    return response;
  },
};

Set AGENTSHOP_API_KEY as a Worker secret (wrangler secret put AGENTSHOP_API_KEY), route it at *yourdomain.com/*, and bind it to the zone serving your storefront.

Any other host

Both recipes above are the same HTTPS call, so any server or edge runtime works — Netlify, Fastly, CloudFront, a plain Node server:

curl -X POST https://api.useagentshop.com/api/v1/crawler-hits \
  -H "Authorization: Bearer ask_xxxxxxxx…" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://yourstore.com/products/tee",
    "method": "GET",
    "statusCode": 200,
    "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"
  }'

See the endpoint reference for the full field list.

What gets sent, and what doesn’t

You report what you saw — url, method, status, user agent, referrer. Identifying which crawler that was happens server-side, so the list stays current as AI platforms add and rename their agents, with no redeploy on your side and nothing to keep in sync at your edge. On both recipes, url and referrer are stripped to origin + pathname before they ever leave your server — no query string, no hash. A magic-link or password-reset URL carries its token in the query string, and a link-preview bot fetches exactly that token-bearing URL the moment a teammate pastes it in chat; this guarantees that token never reaches analytics. A referrer on a non-http(s) scheme (an app deep link, for example) is dropped from the report entirely rather than sent malformed. The API re-strips server-side and only accepts http/https URLs — see the endpoint reference. If any of your routes carry secrets in the path itself (/reset/<token>, signed downloads, invite links), exclude them at the capture layer — the Next matcher supports negative matching for exactly this, and a Worker route pattern can do the same. reportIfCrawler is the same helper the Next proxy wrapper uses — any runtime with a standard Request (Remix loader, Express with a Request shim, Fastly, Netlify) can call it directly.

No IP address is collected, sent, or accepted. A crawler fetch needs no visitor identity, and not collecting one removes the question of how to anonymize it.

The edge check uses isbot — a maintained, public bot list, the same one Shopify’s Hydrogen depends on. It is deliberately broader than the AI crawlers we report on: it filters out ordinary human traffic before anything leaves your server, and anything that isn’t an AI crawler is acknowledged and dropped on our side.

Status codes

The Cloudflare Worker reports a real status because it performs the fetch itself. The Next.js proxy omits statusCode: NextResponse.next() carries a sentinel 200 meaning “continue routing”, not the page’s final status, which is decided after the proxy has already returned. Recording it would stamp 200 on every crawler hit including 404s — precisely the number you’d want when asking whether AI crawlers are hitting dead URLs.

Headless SEO SDK

Framework guides

Analytics

Artifacts

Reference

Security & troubleshooting

AI crawler visibility

Next.js

Cloudflare

Any other host

What gets sent, and what doesn’t

Status codes

Next steps

Storefront analytics

Endpoint reference

​Next.js

​Cloudflare

​Any other host

​What gets sent, and what doesn’t

​Status codes

​Next steps

Storefront analytics

Endpoint reference

Next.js

Cloudflare

Any other host

What gets sent, and what doesn’t

Status codes

Next steps