# Sitedex

> Sitedex is two products on one engine. For AI agents: a live web index — every public website crawled, scored, and served as raw markdown over MCP, a REST API, and a CLI. For site owners: an audit that knows what AI agents need to read a site clearly and finds exactly what's missing — vague claims, contradictions, content gaps, missing technical signals — then ships the fix list and the files AI agents look for. Reads are free. Audits cost 1 credit. Anyone can index a new site for free.

## Surfaces

- **MCP (hosted)** — `https://mcp.sitedex.dev/mcp`. Connect from any MCP client over Streamable HTTP. No setup.
- **MCP (per-site)** — `https://mcp.sitedex.dev/s/{slug}/mcp`. One endpoint per site, with the site already bound. Slug is the URL form (`fireflies-ai`, not `fireflies.ai`).
- **MCP (local)** — `npx -y @sitedex/cli mcp`. Adds audit, credit, and account tools once you sign in.
- **REST API** — `https://api.sitedex.dev/v1/*`. Authed with a Bearer key from [app.sitedex.dev/settings/api-keys](https://app.sitedex.dev/settings/api-keys).
- **CLI** — `npm install -g @sitedex/cli`. Sign in once, then run audits, search the index, and pull reports from the terminal.

## Tools (hosted MCP — no sign-in)

- **`sitedex_search`** — search every indexed site, scope to one domain with `site`.
- **`sitedex_list_sites`** — every indexed domain with category, page count, score.
- **`sitedex_get_site`** — site profile and headline score.
- **`sitedex_list_pages`** — paths Sitedex has on a domain.
- **`sitedex_get_page`** — full markdown of one page.
- **`sitedex_get_visibility_score`** — full audit: which buyer questions the site answers, which it misses, and what to fix.
- **`sitedex_index_url`** — submit a new domain. First-time indexing is free.

## Tools (local MCP — adds audit + account)

Run `npx -y @sitedex/cli mcp` and sign in to also get:

- **`sitedex_audits_create`** — run a fresh audit. 1 credit. Anyone can index a new site free, no account needed.
- **`sitedex_audits_get` / `sitedex_audits_list`** — your audit history.
- **`sitedex_credits_balance` / `sitedex_whoami`** — your wallet and account.
- **`sitedex_login_start` / `sitedex_login_finish` / `sitedex_logout`** — sign in from inside the agent.

## Tools (per-site MCP)

Same surface, smaller scope. Each tool acts on the bound site only — no `site` argument:

- **`search`**, **`get_info`**, **`list_pages`**, **`get_page`**

## REST API

Base URL: `https://api.sitedex.dev`. Auth: `Authorization: Bearer sdx_…`.

Errors:

```json
{ "error": { "type": "...", "code": "...", "message": "..." } }
```

### Public

| Method | Path | Description |
|---|---|---|
| `POST` | `/v1/audits` | Run an audit. |
| `GET` | `/v1/sites/:domain` | Site profile and headline score. |
| `GET` | `/v1/sites/:domain/report` | Full audit report. |
| `POST` | `/search` | Search the index. |
| `GET` | `/sites/:domain/pages/:path` | One page as markdown. |

### Signed in

| Method | Path | Description |
|---|---|---|
| `GET` | `/v1/auth/whoami` | Account info. |
| `GET` | `/v1/audits` / `/v1/audits/:id` | Audit history and a single audit. |
| `GET` | `/v1/sites` / `/v1/sites/:domain/audits` | Sites you've claimed and their audit history. |
| `GET` | `/v1/credits/balance` / `/v1/credits/ledger` | Wallet and transactions. |
| `POST` | `/v1/credits/purchase` | Buy a credit. |
| `*` | `/v1/auth/tokens` | Manage API keys. |

Pre-`/v1` paths return a `Sunset` header pointing at the new route. They stop responding on 2026-08-04.

## Pricing

- **Reads** — free. Search, browse, fetch markdown, view the score.
- **Index a new site** — free, no account needed. Anyone can submit a URL Sitedex doesn't have yet.
- **Audits** — 1 credit each. Every new account gets 1 free credit. Credits never expire.
- **Credit packs** — Single $5 (1 credit · $5.00/credit), Starter $50 (15 credits · $3.33/credit), Pro $100 (40 credits · $2.50/credit). Bigger packs cost less per credit; same product everyone gets.
- **No subscription, no seats.**

## Site owners

[Index your site](https://app.sitedex.dev/audits/new). The first audit is on us.

[Claim your domain](https://app.sitedex.dev/team/domains) with a DNS TXT record to manage it, see all audit history, and unlock deeper crawls.

The site page at `https://app.sitedex.dev/sites/{domain}` is the same page buyers see. Score, gaps, citations, and a fix list. No sign-in required to view.

### Files Sitedex builds for you

Generated from your indexed pages, downloadable from your site's fix list. Same data the audit produces, packaged for install.

- **`llms.txt`** — the file ChatGPT, Claude, and Perplexity read first. Free starter for any URL at [sitedex.dev/llms-txt](https://sitedex.dev/llms-txt); enhanced per-domain version once indexed.
- **AI-crawler `robots.txt` block** — per-bot rules for GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, OAI-SearchBot, and Googlebot-Extended. Append to your existing `robots.txt`.
- **MCP server-card** — `/.well-known/mcp/server-card.json`. Points agents that probe your domain at Sitedex's hosted endpoint with your site bound. No server to run.
- **WebMCP manifest** — `/.well-known/webmcp.json` + a one-line script tag. Discovery for the W3C draft `navigator.modelContext` runtime registrations.
- **Organization JSON-LD** — `<script type="application/ld+json">` block drop-in for `<head>`. Drives AI Knowledge Graph and how agents introduce your company.
- **Canonical / meta description / `<html lang>` snippets** — three one-line fixes most sites miss. Copy-paste ready.

## Free tools (browser, no sign-in)

Two standalone diagnostic pages. Paste a URL, get an answer, nothing stored. Both run on SitedexBot, our signed crawler.

- **fetchcheck** — [sitedex.dev/tools/fetchcheck](https://sitedex.dev/tools/fetchcheck). Sends one signed SitedexBot request to any site and shows exactly what came back: allowed, blocked, challenged, redirected, or a page with no readable content. Answers "can a trusted crawler get in, and is your content actually there?" before you run a full audit.
- **keycheck** — [sitedex.dev/tools/keycheck](https://sitedex.dev/tools/keycheck). Lints your own Web Bot Auth signing-key directory, key by key, for the mistakes that stop verification — wrong key type or curve, an expiring key, a leaked private key.

SitedexBot's published identity and public signing keys: [sitedex.dev/bot](https://sitedex.dev/bot).

## How it works

We crawl. We extract markdown. We embed it. We serve it.

The audit is where Sitedex's intelligence shows up. For each site we generate a bank of buyer questions specific to its category — the actual questions a real customer asks during evaluation, not generic "what does X do" prompts. We then ask a real LLM to answer each one from the site's content, and grade the answer on a sharp diagnosis taxonomy:

- **answered** — content addresses the question specifically. Cite-worthy.
- **vague claim** — topic mentioned, specific detail (number, threshold, policy edge case) missing.
- **collapsed content** — page exists, body hidden behind JavaScript that didn't render.
- **missing page** — no dedicated page covers the topic.
- **contradictory pages** — two or more pages disagree; agents pick one at random.
- **genuinely absent** — topic isn't anywhere on the site.

Every miss gets a fix hint a writer or engineer can ship. The protocol layer separately checks the technical signals that decide whether AI agents can read the site at all — `llms.txt`, sitemap, robots-for-AI-crawlers, MCP card, WebMCP, JSON-LD.

Score: `0.7 × content + 0.3 × protocol`, capped so protocol can't more than double a weak content score.

Levels: 80+ Agent-Native, 60+ Agent-Ready, 40+ Agent-Accessible, 20+ Basic Presence, below 20 Invisible.

## Discovery

- [Install guide](https://sitedex.dev/docs)
- [OpenAPI 3.1 spec](https://sitedex.dev/openapi.json)
- [MCP server card](https://sitedex.dev/.well-known/mcp.json)
- [Agent discovery](https://sitedex.dev/.well-known/agentroot.json)
- [CLI on npm](https://www.npmjs.com/package/@sitedex/cli)
- [Long-form docs for AI agents](https://sitedex.dev/llms-full.txt)

## Contact

`hello@sitedex.dev`