---
name: find-competitor-keyword-gaps
description: Compare supplied domains across two observed Google result pages for ten buying keywords and produce a page-level refresh or gap-investigation queue.
---

# Find keywords competitors win

Give an SEO lead ten buying keywords, an own domain and competitor domains. The agent returns evidence for the first two Google result pages per keyword, then chooses **refresh existing URL** or **investigate gap**; it does not claim search volume, rank beyond the window, or that a page is absent from the web.

The default cap is 20 attempts, at most **$0.00300** at $0.15 per 1,000 successful calls, not a measured run. A retry uses an attempt and may leave a planned page unchecked; do not promise 20 distinct observations when retries consume the cap.

## 1. Set the fixed comparison

Require exactly ten buyer-intent queries, `OWN_DOMAIN`, `COMPETITOR_DOMAINS`, `gl`, `hl`, and `device`. Check the key and cap before any work:

```bash
curl -s https://api.litescrape.com/api/keys/status -H "Authorization: Bearer $LITESCRAPE_API_KEY"
```

Use one locale and device for all 20 requests. Keep raw responses and the exact keyword list; results are a time-stamped observation, not a universal rank. Ask which buying intents matter most commercially; if no priority is supplied, use first-page competitor evidence before second-page evidence, never an invented search-volume or revenue estimate.

## 2. Collect the first page

For each keyword, request organic results only. `fast_mode=true` deliberately excludes AI and other result modules.

```bash
curl -sG https://api.litescrape.com/api/google/search \
  -H "Authorization: Bearer $LITESCRAPE_API_KEY" \
  --data-urlencode 'q=best payroll software for small business' \
  --data-urlencode 'gl=us' --data-urlencode 'hl=en' \
  --data-urlencode 'device=desktop' --data-urlencode 'fast_mode=true' \
  --data-urlencode 'start=0' > keyword-page-1.json
```

Read `organic_results[].position`, `link`, `title` and `snippet`. Normalize a host only to match it against supplied domains; retain the original link. Positions can restart on each page, pages can contain fewer than ten results, and URLs can repeat. Preserve both `page_offset` and the raw `position`; do not label offset plus position as a measured global rank. Order appearances by offset then position and retain the earliest appearance of each URL for comparisons.

## 3. Collect the second page

Repeat every keyword at `start=10` (another 10 calls). Google can return fewer than ten organic rows. If a page adds no new organic rows, mark that keyword’s second page exhausted and do not fabricate a continuation.

```bash
curl -sG https://api.litescrape.com/api/google/search \
  -H "Authorization: Bearer $LITESCRAPE_API_KEY" \
  --data-urlencode 'q=best payroll software for small business' \
  --data-urlencode 'gl=us' --data-urlencode 'hl=en' \
  --data-urlencode 'device=desktop' --data-urlencode 'fast_mode=true' \
  --data-urlencode 'start=10' > keyword-page-2.json
```

Stop at 20 attempts, a 402 balance response, or the agreed cap. Retry a transient 429/5xx once only within that cap, honoring `Retry-After`; never treat an error as an absent ranking. Save exact pending query/offset pairs so a later approved run can resume without repurchasing successful observations. Record the request timeout and distinguish transport failures from API errors.

## 4. Make the action queue

For each keyword, record all own and competitor appearances in the observed window. First assign `query_state=complete`, `exhausted`, or `incomplete` from successful page coverage. An error, timeout or capped page is incomplete, never exhausted. For incomplete queries use **insufficient coverage**, retaining observed rivals as provisional evidence without an absence-based recommendation. For complete/exhausted queries recommend **refresh existing URL** only when a competitor precedes an observed own URL. Where own precedes competitors, recommend **retain / monitor**. Where a competitor appears and own does not, recommend **investigate gap**. Where no competitor appears, mark **no comparison signal**. Investigate the existing site's assets before deciding whether any new content is needed.

- `keyword-gap.csv` / keyword, page offset, observed positions, host, URL, evidence title and action / prioritize refreshes or investigations.
- `evidence.md` / result links, locale, device, retrieval time and exhausted pages / review the claim before editing content.

Never turn “own domain not observed” into “no page exists.” Return actual calls, estimated cost and any incomplete keywords.

## Run record

Write `manifest.json` with the ten queries, domains, locale/device, page offsets, attempt cap, attempted/successful/failed calls, estimated cost, pending query/offset pairs, source run/request IDs for reused evidence, and a `stop_reason` for every keyword (`second_page_complete`, `no_new_rows`, `cap`, `balance`, or `error`). `keyword-gap.csv` columns are `keyword,query_state,available_offsets,provisional,page_offset,domain,url,position,title,action,action_reason,retrieved_at`; escape spreadsheet cells. Send `LITESCRAPE_API_KEY` only as the bearer header to `https://api.litescrape.com`; never save it. Stop on 401/403/402, retry a 429 or transient 5xx once within cap and honoring `Retry-After`, and record failures rather than inferring a rank.