# Sitedex — Full Reference

> Sitedex is the live web index for AI agents. Every public website is crawled, scored, and served as raw markdown over MCP, a REST API, a CLI, and WebMCP. Reads are free. Audits cost 1 credit. Anyone can index a new site for free.

This is the long-form companion to [sitedex.dev/llms.txt](https://sitedex.dev/llms.txt). It inlines the docs, the API reference, and the pricing so an agent can answer any question about Sitedex without another fetch.

## What Sitedex is

Sitedex indexes public websites and serves them to AI agents.

Search the index. Read pages as markdown. Score a site. Run a fresh audit. Submit a new domain. Everything is one tool call away — over MCP, over REST, or from the command line.

Two audiences, one product:

- **Agent builders** get a live, citation-rich web index. Free for every read tool. No setup beyond connecting an MCP client.
- **Site owners** get a public scorecard for how well agents can read their site, with a ranked fix list — and an MCP endpoint of their own.

## Surfaces

- **MCP (hosted)** — `https://mcp.sitedex.dev/mcp`. Streamable HTTP. Public read tools. Server name: `sitedex`. Namespace: `dev.sitedex/mcp`. Listed at [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io).
- **MCP (per-site)** — `https://mcp.sitedex.dev/s/{slug}/mcp`. One endpoint per indexed site. The slug is the URL form (`fireflies-ai`), matching `^[a-z0-9][a-z0-9-]{0,127}$`.
- **MCP (local)** — `npx -y @sitedex/cli mcp`. Same registry as hosted, plus audit and account tools once signed in.
- **REST API** — `https://api.sitedex.dev/v1/*`. Bearer-key auth. Manage keys at [app.sitedex.dev/settings/api-keys](https://app.sitedex.dev/settings/api-keys).
- **CLI** — `npm install -g @sitedex/cli`. Or `npx -y @sitedex/cli <command>`. Ships `sitedex` and `sitedex-mcp` binaries.
- **WebMCP** — every `sitedex.dev` page registers WebMCP tools through `navigator.modelContext`, so any browser implementing the W3C draft (Chrome 146+ Origin Trial) sees them automatically. Manifest: [sitedex.dev/.well-known/webmcp.json](https://sitedex.dev/.well-known/webmcp.json).

## Tools — hosted MCP

Seven tools, no sign-in required.

- **`sitedex_search(query, site?, top_k?)`** — search every indexed site. Returns ranked markdown passages with the source URL, heading path, and score. Pass `site` to scope to one domain.
- **`sitedex_list_sites(limit?)`** — every indexed domain with name, category, page count, and score.
- **`sitedex_get_site(site)`** — profile and headline score for one domain.
- **`sitedex_list_pages(site, limit?, offset?)`** — paths Sitedex has indexed on a domain.
- **`sitedex_get_page(site, path)`** — full markdown of one page.
- **`sitedex_get_visibility_score(site)`** — the full audit: per-question diagnosis, capabilities, protocol checklist. Use before suggesting fixes.
- **`sitedex_index_url(url)`** — submit a domain Sitedex doesn't have yet. First-time indexing is free. If the domain is already indexed, returns a pointer to its existing report.

## Tools — local MCP adds

Run `npx -y @sitedex/cli mcp` and sign in to also get:

**Audit and account**

- **`sitedex_audits_create(url, force?)`** — run a fresh audit. 1 credit. Anyone can index a new site free, no account needed; signed-in callers spend a credit. Pass `force` to run a fresh audit on a site that's already been indexed (charges another credit).
- **`sitedex_audits_get(id)`** — fetch one audit by id.
- **`sitedex_audits_list(site, limit?)`** — audit history for a site.
- **`sitedex_credits_balance()`** — your wallet.
- **`sitedex_whoami()`** — your account.

**Sign in from inside the agent**

- **`sitedex_login_start()`** — returns a verification URL and code. Open the URL in your browser to authorize.
- **`sitedex_login_finish()`** — completes the flow and saves your credentials. Subsequent tool calls pick up the new sign-in.
- **`sitedex_logout()`** — clears local credentials.

## Tools — per-site MCP

Same surface, smaller scope. Each tool acts on the bound site only — no `site` argument and no `sitedex_` prefix:

- **`search(query, top_k?)`**
- **`get_info()`**
- **`list_pages(limit?, offset?)`**
- **`get_page(path)`**

## REST API

Base URL: `https://api.sitedex.dev`. Auth: `Authorization: Bearer sdx_…` or `x-api-key: sdx_…`.

Errors:

```json
{
  "error": {
    "type": "invalid_request_error",
    "code": "domain_required",
    "message": "Domain is required",
    "param": "domain"
  }
}
```

`type` is one of `invalid_request_error | authentication_error | rate_limited | api_error`. Branch on `code`. `message` is human-readable.

### Public

| Method | Path | Description |
|---|---|---|
| `POST` | `/v1/audits` | Run an audit. Anonymous calls require a Turnstile token. Signed-in calls cost 1 credit. Submitting a site that's already indexed returns the existing report (`reused: true`, no charge); pass `force: true` to crawl again. |
| `GET` | `/v1/sites/:domain` | Site profile and headline score. |
| `GET` | `/v1/sites/:domain/report` | Full audit: per-question results, capabilities, protocol checklist, discovery files. |
| `POST` | `/search` | Search the index. Body: `{ query, site?, top_k? }`. |
| `GET` | `/sites/:domain/pages/:path` | One page as markdown. |

### Signed in

| Method | Path | Description |
|---|---|---|
| `GET` | `/v1/auth/whoami` | Account info. |
| `GET` | `/v1/audits` | Your audit history. |
| `GET` | `/v1/audits/:id` | One audit by id. |
| `GET` | `/v1/sites` | Sites you've claimed. |
| `GET` | `/v1/sites/:domain/audits` | All audits for a site. Verified owners see every audit; otherwise scoped to your account. |
| `GET` | `/v1/credits/balance` | Wallet snapshot. |
| `GET` | `/v1/credits/ledger` | Transaction history. |
| `POST` | `/v1/credits/purchase` | Returns a Stripe Checkout URL for one credit. |
| `*` | `/v1/auth/tokens` | Manage API keys. |

### Idempotency

Mutating routes accept an `Idempotency-Key` header. Same key + same payload returns the cached response. Same key + different payload returns `409`.

### Sunset

Pre-`/v1` paths (`/analyze`, `/sites/*`, `/billing/payg/*`) still respond and include a `Sunset` header pointing at the new route. They stop responding on 2026-08-04.

## CLI

```bash
npm install -g @sitedex/cli
# or, one-off:
npx -y @sitedex/cli <command>
```

**Auth**

```bash
sitedex login                  # default: opens your browser
sitedex login --token=sdx_…    # paste an API key (CI / headless)
sitedex logout
```

**Commands**

- `sitedex audits create <url> [--force]`
- `sitedex audits list <domain> [--limit=N]`
- `sitedex audits get <id>`
- `sitedex sites list`
- `sitedex sites get <domain>`
- `sitedex report <domain>` — full audit report (per-question, capabilities, protocol)
- `sitedex credits balance`
- `sitedex credits ledger`
- `sitedex whoami`
- `sitedex search <query> [--site=<domain>] [--top-k=N]`
- `sitedex pages get <domain> <path>`
- `sitedex mcp` — start the local MCP server

The CLI, the REST API, and the MCP server share the same client. Behaviour is identical across surfaces.

## WebMCP

Two pieces — the runtime and the discovery file. Use both.

### Runtime — drop-in widget

```html
<script async src="https://sitedex.dev/widget.js"></script>
```

Any browser implementing `navigator.modelContext` registers Sitedex's per-site tools (`search`, `get_info`, `list_pages`, `get_page`) on your domain. No backend code, no auth, no agent setup.

The widget renders an "AI-indexed by Sitedex" pill at the bottom of the page. Hide it with `[data-sitedex] { display: none !important; }`.

### Discovery — static manifest

WebMCP doesn't define a discovery format. The community has converged on `/.well-known/webmcp.json`. Sitedex generates one for you:

```bash
curl https://api.sitedex.dev/sites/your-domain.com/webmcp.json > webmcp.json
# host at https://your-domain.com/.well-known/webmcp.json
```

## The audit

Every audit produces:

- **A score, 0–100.** Sitedex asks the site the buyer questions a real customer would and grades each answer:
  - **Answered** — a clear, specific answer is on the site.
  - **Partial** — the topic is covered but the detail a buyer needs is thin.
  - **Vague** — the topic is mentioned but the specifics are missing.
  - **Collapsed** — the right page exists but the body is empty (a JS accordion that didn't render, an SSR gap).
  - **Missing** — no page covers it.
  - **Contradictory** — pages disagree.
  - **Silent** — nothing on the site touches it.

  Every answered question cites the page and heading.

- **A protocol check.** Whether AI crawlers can actually reach the site, in plain terms:

  | Category | Check |
  |---|---|
  | Discoverability | `robots.txt`, `sitemap.xml`, `llms.txt` |
  | Access | AI-crawler allow rules, content-signal directives, crawl success rate |
  | Rendering | Markdown content negotiation, server-side rendering rate |
  | Interaction | MCP card, OpenAPI spec, WebMCP signal |
  | Commerce (if applicable) | Payment, webhook, and auth docs |

- **A fix list.** Every gap with the predicted score uplift if you ship the fix and a copy-pasteable file or snippet.

Composite: `0.7 × content + 0.3 × protocol`, with a guardrail that protocol can lift a weak content score but not more than 2×.

## How indexing works

Crawl, extract markdown, chunk, embed, serve.

- The crawler identifies as Sitedex. It honors `robots.txt`. It renders pages in a real browser, so SPAs work.
- Free-tier crawls have a per-domain page cap. Verified owners get a deeper crawl.
- Every result carries a `last_crawled` timestamp.

## Pricing

- **Reads** — free. Search, list, profile, page, audit history, score.
- **Index a new site** — free, no account needed. Anyone can submit a URL Sitedex doesn't have yet.
- **Audits** — 1 credit each. 1 credit = $5 today (we may offer bulk packs later).
- **New accounts** — 1 free credit on signup.
- **No subscription, no seats.**

## Domain claims

Anyone can index any public site. Reports are public.

Claim your domain ([app.sitedex.dev/team/domains](https://app.sitedex.dev/team/domains)) with a DNS TXT record and you get:

- **Audit history.** Every audit anyone has run on your domain, not just yours.
- **Deeper crawl.** More pages indexed per audit.
- **Workspace controls.** Trigger re-crawls, edit your description, manage API keys.

The site report URL is the same whether you're a buyer or the owner — `app.sitedex.dev/sites/{domain}`. No sign-in to view.

## Discovery records

- [`sitedex.dev/openapi.json`](https://sitedex.dev/openapi.json) — OpenAPI 3.1 spec
- [`sitedex.dev/.well-known/mcp.json`](https://sitedex.dev/.well-known/mcp.json) — MCP server card
- [`sitedex.dev/.well-known/webmcp.json`](https://sitedex.dev/.well-known/webmcp.json) — WebMCP manifest
- [`sitedex.dev/.well-known/agentroot.json`](https://sitedex.dev/.well-known/agentroot.json) — agent discovery record
- [`sitedex.dev/.agents/skills/sitedex/SKILL.md`](https://sitedex.dev/.agents/skills/sitedex/SKILL.md) — Agent Skill
- [`sitedex.dev/llms.txt`](https://sitedex.dev/llms.txt) — short summary
- [`sitedex.dev/llms-full.txt`](https://sitedex.dev/llms-full.txt) — this file

## Contact

- CLI on npm: [`@sitedex/cli`](https://www.npmjs.com/package/@sitedex/cli)
- Email: `hello@sitedex.dev`