> ## Documentation Index
> Fetch the complete documentation index at: https://docs.useagentshop.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawler visibility

> Report GPTBot, ClaudeBot, and PerplexityBot hits to AgentShop from a Next.js proxy or a Cloudflare Worker, since AI crawlers never run JavaScript.

GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot fetch your pages to build the indexes and training sets that AI assistants answer from. None of them execute JavaScript, so **no pixel can see them** — ours, Shopify's, or anyone else's.

Your server is the only place those fetches exist. Reporting them tells you which of your pages AI systems actually read.

## Next.js

```ts proxy.ts theme={null}
import { withAgentShopCrawlerCapture } from "@agentshop/seo/next/proxy";

export const proxy = withAgentShopCrawlerCapture();

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};
```

<Note>
  Next 16 renamed `middleware.ts` to `proxy.ts` and the exported function from `middleware` to `proxy`. On Next 15 and below, name the file `middleware.ts` and export it as `middleware` — the wrapper itself is unchanged. Next ships a codemod: `npx @next/codemod@canary middleware-to-proxy .`
</Note>

Already have one? Pass it in. Yours runs first and its response is returned untouched:

```ts proxy.ts theme={null}
import { withAgentShopCrawlerCapture } from "@agentshop/seo/next/proxy";
import { NextResponse } from "next/server";

export const proxy = withAgentShopCrawlerCapture(async (request) => {
  // your existing logic
  return NextResponse.next();
});
```

Reporting is deferred with [`event.waitUntil`](https://nextjs.org/docs/app/api-reference/file-conventions/proxy#waituntil-and-nextfetchevent), so it never delays a response — and, unlike a bare unawaited `fetch`, it isn't dropped when a serverless invocation freezes.

Use the `matcher` to skip static paths rather than expecting the wrapper to do it. Routing is the framework's job and a matcher costs nothing at runtime.

## Cloudflare

If your storefront sits behind Cloudflare, the Next proxy only sees requests your app actually renders — responses served from the edge cache never reach it. A Worker sees all of them:

```js src/index.js theme={null}
import { reportIfCrawler } from "@agentshop/seo";

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    // The whole capture recipe — isbot gate, origin+pathname stripping (no
    // query strings, no fragments), referrer handling — in one call. The
    // Worker performed the fetch itself, so it has a real status to report.
    ctx.waitUntil(
      reportIfCrawler(request, {
        apiKey: env.AGENTSHOP_API_KEY,
        baseUrl: env.AGENTSHOP_SEO_URL,
        statusCode: response.status,
      }),
    );
    return response;
  },
};
```

Set `AGENTSHOP_API_KEY` as a Worker **secret** (`wrangler secret put AGENTSHOP_API_KEY`), route it at `*yourdomain.com/*`, and bind it to the zone serving your storefront.

## Any other host

Both recipes above are the same HTTPS call, so any server or edge runtime works — Netlify, Fastly, CloudFront, a plain Node server:

```bash theme={null}
curl -X POST https://api.useagentshop.com/api/v1/crawler-hits \
  -H "Authorization: Bearer ask_xxxxxxxx…" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://yourstore.com/products/tee",
    "method": "GET",
    "statusCode": 200,
    "userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"
  }'
```

See the [endpoint reference](/sdk/reference/endpoints#post-crawler-hits) for the full field list.

## What gets sent, and what doesn't

You report **what you saw** — url, method, status, user agent, referrer. Identifying which crawler that was happens server-side, so the list stays current as AI platforms add and rename their agents, with no redeploy on your side and nothing to keep in sync at your edge.

On **both recipes**, `url` and `referrer` are stripped to origin + pathname before they ever leave your server — no query string, no hash. A magic-link or password-reset URL carries its token in the query string, and a link-preview bot fetches exactly that token-bearing URL the moment a teammate pastes it in chat; this guarantees that token never reaches analytics. A referrer on a non-http(s) scheme (an app deep link, for example) is dropped from the report entirely rather than sent malformed. The API re-strips server-side and only accepts `http`/`https` URLs — see the [endpoint reference](/sdk/reference/endpoints#post-crawler-hits).

If any of your routes carry secrets in the **path itself** (`/reset/<token>`, signed downloads, invite links), exclude them at the capture layer — the Next `matcher` supports [negative matching](https://nextjs.org/docs/app/api-reference/file-conventions/proxy#negative-matching) for exactly this, and a Worker route pattern can do the same. `reportIfCrawler` is the same helper the Next proxy wrapper uses — any runtime with a standard `Request` (Remix loader, Express with a Request shim, Fastly, Netlify) can call it directly.

<Warning>
  No IP address is collected, sent, or accepted. A crawler fetch needs no visitor identity, and not collecting one removes the question of how to anonymize it.
</Warning>

The edge check uses [`isbot`](https://www.npmjs.com/package/isbot) — a maintained, public bot list, the same one Shopify's Hydrogen depends on. It is deliberately broader than the AI crawlers we report on: it filters out ordinary human traffic before anything leaves your server, and anything that isn't an AI crawler is acknowledged and dropped on our side.

## Status codes

The Cloudflare Worker reports a real status because it performs the fetch itself. The Next.js proxy **omits** `statusCode`: `NextResponse.next()` carries a sentinel `200` meaning "continue routing", not the page's final status, which is decided after the proxy has already returned. Recording it would stamp `200` on every crawler hit including 404s — precisely the number you'd want when asking whether AI crawlers are hitting dead URLs.

## Next steps

<Columns cols={2}>
  <Card title="Storefront analytics" icon="chart-line" href="/sdk/analytics">
    Report AI-referred human visits alongside crawler traffic.
  </Card>

  <Card title="Endpoint reference" icon="globe" href="/sdk/reference/endpoints">
    Auth, fields, and status codes for every endpoint.
  </Card>
</Columns>


## Related topics

- [Next.js API](/sdk/reference/next.md)
- [Endpoint reference](/sdk/reference/endpoints.md)
- [Core API](/sdk/reference/core.md)
- [Security](/sdk/security.md)
- [Products: AI Visibility Scoring Across Your Shopify Catalog](/user-guide/screens/products.md)