// field guide 001
Build a Google Maps scraper with Python and Playwright
Run a standalone Python and Playwright scraper that scrolls Google Maps, extracts place details, resumes from JSONL checkpoints, and exports CSV without an API key.
This standalone scraper drives a local Chromium browser with Playwright. It scrolls the same Google Maps results panel a person sees, visits each place, and writes the data to disk. It does not call an API or private endpoint and requires no API key. The full Python file appears in section seven.
01 / outcome
What you will build
The example searches for coffee shops near central Austin and collects up to 50 results. Change the query, center, zoom, and limit from the command line. Each row contains the search query, business name, category, address, phone number, website, rating, review count when exposed, hours label, coordinates, source URL, and scrape time.
The script writes two files. The CSV works well for spreadsheets and imports. The JSONL file is the resumable checkpoint and preserves numbers and null values that CSV flattens into text or empty cells. The script appends one JSON object after each place. If the browser stops on place 38, the first 37 records remain on disk. Run the same command again and the script skips their URLs.
Google Maps removes off-screen result cards from the DOM as you scroll. Scraping each card in place can make the feed jump or lose its position. The script first collects stable destination URLs, then opens them in a second tab. The search feed retains its position while the detail pages load.
02 / page model
How Google Maps renders search results
Maps renders search results in a JavaScript application. As the map moves, it fetches data and places result cards inside a scrollable element with role="feed". The browser keeps only part of that list mounted; scrolling the feed causes more cards to appear.
A request library such as requests can download the initial document, but it cannot execute the JavaScript that builds and updates the feed. Playwright controls a browser engine and queries the live DOM after Maps updates it.
Your locator choices determine how often this scraper breaks. Generated classes such as .Nv2PK or .DUwDvf may work today, but they say nothing about the element and can change during a frontend deploy. This script starts with accessibility roles and semantic attributes. Playwright's locator guide gives the same advice: prefer roles and user-facing names, then use CSS when no better contract exists.
| Page element | Locator | Reason |
|---|---|---|
| Search results | [role="feed"] | The scrollable results region exposed to assistive technology. |
| Place destination | a[href*="/maps/place/"] | The script can retain a real link after Maps removes its card from the DOM. |
| Business name | h1 | The detail panel has one primary place heading. |
| Address | [data-item-id="address"] | The attribute names the contact field instead of its presentation. |
| Phone | [data-item-id^="phone:"] | The fixed prefix comes before the phone number in the value. |
| Website | [data-item-id="authority"] | The link target gives the actual website URL. |
These locators are Google Maps implementation details, not an official data contract. Google can change them, although accessible roles tend to outlive styling classes. Before a large job, run a small fixture and check field completeness. A production job should stop when completeness drops; otherwise a selector failure can produce hundreds of blank values without raising an exception.
03 / setup
Install Python, Playwright, and Chromium
Use Python 3.10 or newer. A virtual environment keeps Playwright and its browser driver out of the rest of your Python installation. The final command downloads the Chromium build matched to the installed Playwright version.
python3 -m venv .venvsource .venv/bin/activatepython -m pip install --upgrade pip playwrightplaywright install chromium On Linux CI hosts, playwright install --with-deps chromium also installs the operating-system libraries Chromium needs. It may require elevated package permissions. On a laptop, start with the four commands above.
04 / discovery
First pass: collect result links
The command accepts both a natural-language query and an optional map center. A query such as "coffee shops in Austin, TX" gives Google geographic context. --center 30.2672,-97.7431 --zoom 14 makes the initial camera position explicit. Maps still ranks the results, may include places outside the visible bounds, and does not promise exhaustive coverage within a radius.
make_search_url() URL-encodes the query and produces a normal Maps URL. The script opens it with wait_until="domcontentloaded", handles the consent screen that appears in some regions, and waits up to 20 seconds for the feed. Fixed delays are unreliable because load times vary.
Once the feed is visible, collect_place_links() repeats the same loop:
- Read every place link currently mounted inside the feed.
- Put each URL in a dictionary keyed by the URL before its query string.
- Scroll the feed element to its current
scrollHeight. - Wait 1.2 seconds for Maps to render another batch.
- Stop at the requested limit, the end-of-list marker, or four no-growth rounds.
The script deduplicates links as it collects them. Virtualized lists reuse space and can expose the same card several times while you scroll, so counting DOM nodes overstates progress. Normalized destination URLs give the loop a stable measure instead.
05 / extraction
Second pass: open each place and read its details
Result cards often omit fields. Phone, website, hours, and the full address tend to live in the selected place panel. The script keeps the search tab open and creates a second page in the same browser context. That detail page visits each saved URL in sequence.
scrape_place() waits for the place's h1, then reads contact values from semantic attributes. The address and phone selectors return accessible labels such as "Address: 401 Congress Ave." The helper clean_label() removes only the known prefix. Website comes from the anchor's href, not its shortened display text.
Rating and review count need a different parser. Maps exposes them through accessible labels, but the exact wrapping element varies. rating_and_reviews() collects the page's aria-label values, looks for phrases such as "4.7 stars" and "1.2K reviews," and converts compact counts to integers. Some signed-out or limited views omit the review count, in which case the field stays null. The script sets the browser context to en-US because those regular expressions assume English labels and punctuation. Supporting another locale requires corresponding parser changes; changing the URL alone is insufficient.
coordinates_from_url() prefers the place coordinates embedded as !3d...!4d... and falls back to the visible @latitude,longitude segment. The distinction matters because @ can describe the camera center. If a navigation redirect removes both coordinate patterns, the script uses the URL collected from the feed. Missing fields remain empty or null. An empty phone number may reflect a business that did not publish one rather than a scrape error.
| Field | Source | Stored form |
|---|---|---|
| Name | Place heading | Text |
| Category | Category button | Text |
| Address | Address button label | Text without "Address:" |
| Phone | Phone button label | Display-formatted text |
| Website | Authority link | Absolute URL |
| Rating | Accessible rating label | Float |
| Reviews | Accessible reviews label | Integer when parseable |
| Coordinates | Place URL | Latitude and longitude floats |
06 / durability
Durable checkpoints and CSV export
A navigation can time out, the page can change, a laptop can sleep, or a consent page can appear halfway through a run. Writing records only at the end would lose the entire run when one of those failures occurs.
After each successful detail page, append_jsonl() opens the checkpoint in append mode and writes one complete JSON object. On startup, load_jsonl() reads those objects and rebuilds the set of completed URLs. It skips malformed lines with a warning in case the last write was interrupted.
After the browser closes, write_csv() regenerates the CSV from every valid checkpoint record. This avoids partial CSV quoting problems and makes JSONL the resume record. Pass --fresh to delete the matching JSONL and CSV files and start over. Otherwise, rerunning the same output path resumes the job.
The retry wrapper catches only Playwright timeouts and operating-system I/O errors. It waits two seconds after the first failure and four after the second; a third failure skips that place. A robot-check page raises RuntimeError and ends the run. Retrying a challenge would send more traffic after the site has asked the client to stop.
07 / complete file
The complete browser-automation scraper
Copy this into scrape_google_maps.py. The file includes a runnable CLI and the browser, extraction, retry, checkpoint, and export code described above.
from __future__ import annotations import argparseimport asyncioimport csvimport jsonimport reimport sysfrom datetime import datetime, timezonefrom pathlib import Pathfrom typing import Any, Awaitable, Callable, TypeVarfrom urllib.parse import quote from playwright.async_api import ( Locator, Page, TimeoutError as PlaywrightTimeoutError, async_playwright,) FEED_SELECTOR = '[role="feed"]'PLACE_LINK_SELECTOR = 'a[href*="/maps/place/"]'CSV_FIELDS = [ "query", "name", "category", "address", "phone", "website", "rating", "reviews", "hours", "latitude", "longitude", "source_url", "scraped_at",]BLOCK_TEXT = ( "our systems have detected unusual traffic", "unusual traffic from your computer network", "verify you're not a robot",) T = TypeVar("T") def clean_label(value: str | None, *prefixes: str) -> str: """Remove an accessible-label prefix such as ``Address:``.""" text = (value or "").strip() lowered = text.casefold() for prefix in prefixes: if lowered.startswith(prefix.casefold()): return text[len(prefix) :].lstrip(" :·") return text def compact_number(value: str) -> int | None: """Turn ``1,234`` or ``1.2K`` into an integer.""" match = re.search(r"([\d.,]+)\s*([KMB]?)", value, re.IGNORECASE) if not match: return None number = match.group(1) suffix = match.group(2).upper() if suffix: base = float(number.replace(",", "")) multiplier = {"K": 1_000, "M": 1_000_000, "B": 1_000_000_000}[suffix] return int(base * multiplier) return int(number.replace(",", "").replace(".", "")) def coordinates_from_url(url: str) -> tuple[float | None, float | None]: # The !3d/!4d pair identifies the place. An @ pair can be only the camera center. match = re.search(r"!3d(-?\d+(?:\.\d+)?).*?!4d(-?\d+(?:\.\d+)?)", url) if not match: match = re.search(r"@(-?\d+(?:\.\d+)?),(-?\d+(?:\.\d+)?)", url) if not match: return None, None return float(match.group(1)), float(match.group(2)) def place_key(url: str) -> str: """Drop tracking parameters while keeping the place path and data blob.""" return url.split("?", 1)[0].rstrip("/") def make_search_url(query: str, center: str | None, zoom: int) -> str: path = f"https://www.google.com/maps/search/{quote(query, safe='')}" if center: latitude, longitude = center.split(",", 1) path += f"/@{float(latitude)},{float(longitude)},{zoom}z" return f"{path}?hl=en" async def first_text(page: Page, selectors: list[str]) -> str: for selector in selectors: locator = page.locator(selector).first try: if await locator.count(): text = (await locator.inner_text(timeout=2_000)).strip() if text: return text except PlaywrightTimeoutError: continue return "" async def first_attribute( page: Page, selectors: list[str], attribute: str,) -> str: for selector in selectors: locator = page.locator(selector).first try: if await locator.count(): value = await locator.get_attribute(attribute, timeout=2_000) if value: return value.strip() except PlaywrightTimeoutError: continue return "" async def with_backoff_async( func: Callable[..., Awaitable[T]], *args: Any, max_try: int = 3, wait: float = 2, max_wait: float = 8,) -> T: """Retry transient browser failures with capped exponential backoff.""" for attempt in range(1, max_try + 1): try: return await func(*args) except (PlaywrightTimeoutError, OSError) as error: if attempt == max_try: raise print( f"{type(error).__name__} on attempt {attempt}/{max_try}; retrying in {wait:g}s", file=sys.stderr, ) await asyncio.sleep(wait) wait = min(wait * 2, max_wait) raise RuntimeError("unreachable") async def dismiss_consent(page: Page) -> None: """Handle the consent page shown in some regions.""" for name in ("Reject all", "Accept all"): button = page.get_by_role("button", name=name) try: if await button.count() and await button.first.is_visible(): await button.first.click() await page.wait_for_timeout(750) return except PlaywrightTimeoutError: pass async def raise_if_blocked(page: Page) -> None: try: body = (await page.locator("body").inner_text(timeout=5_000)).casefold() except PlaywrightTimeoutError: return if any(marker in body for marker in BLOCK_TEXT): raise RuntimeError( "Google returned a traffic or robot-check page. Stop the run, wait, " "and retry with --headed. This script does not bypass challenges." ) async def links_in_feed(feed: Locator) -> list[dict[str, str]]: links = feed.locator(PLACE_LINK_SELECTOR) return await links.evaluate_all( """ elements => elements.map(element => ({ url: element.href, name: element.getAttribute('aria-label') || element.textContent || '' })) """ ) async def collect_place_links( page: Page, search_url: str, limit: int,) -> list[dict[str, str]]: """Scroll Google's virtualized result feed and retain unique place URLs.""" await page.goto(search_url, wait_until="domcontentloaded", timeout=60_000) await dismiss_consent(page) await raise_if_blocked(page) feed = page.locator(FEED_SELECTOR).first await feed.wait_for(state="visible", timeout=20_000) found: dict[str, dict[str, str]] = {} stagnant_rounds = 0 while len(found) < limit and stagnant_rounds < 4: before = len(found) for item in await links_in_feed(feed): url = item.get("url", "") if not url: continue found.setdefault( place_key(url), {"url": url, "name": item.get("name", "").strip()}, ) if len(found) >= limit: break if len(found) == before: stagnant_rounds += 1 else: stagnant_rounds = 0 print(f"Found {len(found)} place links", file=sys.stderr) end_marker = page.get_by_text(re.compile(r"reached the end of the list", re.IGNORECASE)) if await end_marker.count(): break await feed.evaluate("element => element.scrollTo(0, element.scrollHeight)") await page.wait_for_timeout(1_200) await raise_if_blocked(page) return list(found.values())[:limit] async def rating_and_reviews(page: Page) -> tuple[float | None, int | None]: labels: list[str] = await page.locator("[aria-label]").evaluate_all( "elements => elements.map(element => element.getAttribute('aria-label') || '')" ) rating: float | None = None reviews: int | None = None for label in labels: if rating is None: rating_match = re.search(r"([\d.]+)\s+stars?", label, re.IGNORECASE) if rating_match: rating = float(rating_match.group(1)) if reviews is None: review_match = re.search(r"([\d.,]+\s*[KMB]?)\s+reviews?", label, re.IGNORECASE) if review_match: reviews = compact_number(review_match.group(1)) if rating is not None and reviews is not None: break return rating, reviews async def scrape_place( page: Page, target: dict[str, str], query: str,) -> dict[str, Any]: """Open one place panel and extract its visible business fields.""" await page.goto(target["url"], wait_until="domcontentloaded", timeout=60_000) await dismiss_consent(page) await raise_if_blocked(page) await page.locator("h1").first.wait_for(state="visible", timeout=20_000) name = await first_text(page, ["h1", '[role="main"] h1']) category = await first_text( page, [ 'button[jsaction*="category"]', 'button[aria-label*="Category"]', "button.DkEaL", ], ) address_label = await first_attribute(page, ['button[data-item-id="address"]'], "aria-label") phone_label = await first_attribute(page, ['button[data-item-id^="phone:"]'], "aria-label") hours_label = await first_attribute( page, ['button[data-item-id^="oh"]', '[aria-label^="Hours"]'], "aria-label", ) website = await first_attribute(page, ['a[data-item-id="authority"]'], "href") rating, reviews = await rating_and_reviews(page) latitude, longitude = coordinates_from_url(page.url) if latitude is None: latitude, longitude = coordinates_from_url(target["url"]) return { "query": query, "name": name or target.get("name", ""), "category": clean_label(category, "Category"), "address": clean_label(address_label, "Address"), "phone": clean_label(phone_label, "Phone"), "website": website, "rating": rating, "reviews": reviews, "hours": clean_label(hours_label, "Hours"), "latitude": latitude, "longitude": longitude, "source_url": target["url"], "scraped_at": datetime.now(timezone.utc).isoformat(), } def load_jsonl(path: Path) -> list[dict[str, Any]]: if not path.exists(): return [] records: list[dict[str, Any]] = [] for line_number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): if not line.strip(): continue try: records.append(json.loads(line)) except json.JSONDecodeError: print( f"Skipping malformed JSONL line {line_number} in {path}", file=sys.stderr, ) return records def append_jsonl(path: Path, record: dict[str, Any]) -> None: path.parent.mkdir(parents=True, exist_ok=True) with path.open("a", encoding="utf-8") as file: file.write(json.dumps(record, ensure_ascii=False) + "\n") def write_csv(path: Path, records: list[dict[str, Any]]) -> None: path.parent.mkdir(parents=True, exist_ok=True) with path.open("w", newline="", encoding="utf-8") as file: writer = csv.DictWriter(file, fieldnames=CSV_FIELDS, extrasaction="ignore") writer.writeheader() writer.writerows(records) async def run(args: argparse.Namespace) -> int: output: Path = args.output checkpoint = output.with_suffix(".jsonl") if args.fresh: checkpoint.unlink(missing_ok=True) output.unlink(missing_ok=True) records = load_jsonl(checkpoint) seen = {place_key(record["source_url"]) for record in records} search_url = make_search_url(args.query, args.center, args.zoom) async with async_playwright() as playwright: browser = await playwright.chromium.launch( headless=not args.headed, slow_mo=args.slow_mo, ) context = await browser.new_context( locale="en-US", viewport={"width": 1440, "height": 1000}, ) search_page = await context.new_page() detail_page = await context.new_page() try: targets = await collect_place_links(search_page, search_url, args.max_results) pending = [target for target in targets if place_key(target["url"]) not in seen] print( f"Collected {len(targets)} links; {len(pending)} still need details", file=sys.stderr, ) for index, target in enumerate(pending, 1): try: record = await with_backoff_async( scrape_place, detail_page, target, args.query, ) except RuntimeError: raise except (PlaywrightTimeoutError, OSError) as error: print( f"Skipping {target['url']} after retries: {error}", file=sys.stderr, ) continue append_jsonl(checkpoint, record) records.append(record) seen.add(place_key(target["url"])) print( f"[{index}/{len(pending)}] {record['name'] or target['url']}", file=sys.stderr, ) await detail_page.wait_for_timeout(args.delay_ms) finally: await context.close() await browser.close() write_csv(output, records) print(f"Saved {len(records)} places to {output}") print(f"Raw checkpoints: {checkpoint}") return 0 def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description="Scrape visible Google Maps place results with Playwright.") parser.add_argument("query", help='For example: "coffee shops in Austin, TX"') parser.add_argument( "--center", help="Optional map center as latitude,longitude (for example 30.2672,-97.7431)", ) parser.add_argument("--zoom", type=int, default=14) parser.add_argument("--max-results", type=int, default=50) parser.add_argument("--output", type=Path, default=Path("google_maps_results.csv")) parser.add_argument("--delay-ms", type=int, default=900) parser.add_argument("--slow-mo", type=int, default=0) parser.add_argument("--headed", action="store_true") parser.add_argument( "--fresh", action="store_true", help="Delete the matching CSV and JSONL checkpoint before starting.", ) args = parser.parse_args() if args.max_results < 1: parser.error("--max-results must be at least 1") if args.zoom < 1 or args.zoom > 22: parser.error("--zoom must be between 1 and 22") if args.delay_ms < 0: parser.error("--delay-ms cannot be negative") if args.center: try: latitude, longitude = (float(part) for part in args.center.split(",", 1)) except ValueError: parser.error("--center must be latitude,longitude") if not (-90 <= latitude <= 90 and -180 <= longitude <= 180): parser.error("--center is outside valid latitude/longitude bounds") return args if __name__ == "__main__": raise SystemExit(asyncio.run(run(parse_args())))08 / first run
Run it headed, then move to headless
The command below opens a visible Chromium window, pins the map camera to central Austin, and requests up to 50 results. A narrow query or viewport may return fewer places.
python scrape_google_maps.py \ "coffee shops in Austin, TX" \ --center 30.2672,-97.7431 \ --zoom 14 \ --max-results 50 \ --output austin_coffee.csv \ --headed The output above is from that run, abridged only between places 2 and 49 with the file paths shortened to match the command. Watch your first headed run: you should see the results panel scroll, followed by a sequence of place pages in the second tab. Inspect austin_coffee.jsonl while it runs, then check ten CSV rows against their place pages.
After verifying the output, remove --headed. Keep the same query, center, zoom, locale, and browser viewport for comparable runs. Headless execution or a different host can change the consent page, load timing, and DOM that the scraper receives.
To resume, run the same command with the same --output. To delete both output files and rerun every detail page, add --fresh. The JSONL checkpoint has no locking, so only one process should write to a given output path.
09 / debugging
Diagnose common failures
Google Maps may present a different page because of regional consent, experiments, localization, slow networks, bot defenses, or DOM changes.
| Symptom | Likely cause | First check |
|---|---|---|
Timeout waiting for [role="feed"] | Consent screen, single-place result, slow load, or changed role | Rerun with --headed and inspect the visible page. |
| Only a handful of links | Narrow query, end marker, or feed not loading another batch | Scroll manually and confirm that more results exist. |
| Names exist but contact fields are empty | Detail selectors changed or businesses did not publish the fields | Compare several records; inspect data-item-id and labels. |
Rating parser returns null | Locale differs or accessible wording changed | Print candidate aria-label values from one place page. |
Robot-check RuntimeError | Google challenged the browser's traffic | Stop the run, reduce volume and frequency, and do not add a challenge bypass. |
| Duplicate businesses | Distinct URLs, branch listings, or an unstable long-term key | Review source URLs and add the stable identifier your dataset needs. |
When a selector breaks, replace it with the narrowest stable locator you can find: a role, accessible name, data-item-id, link destination, or repeated text pattern that describes the field. Avoid 12-level CSS paths copied from DevTools. Test the replacement against several place categories, including one without a website or phone number.
Save evidence for failed records. A production variant should write the current URL, the exception, a screenshot, and the page HTML to a timestamped debug directory. Keep those artifacts out of public logs if they can contain personal or session data. This example prints failures and preserves completed rows. A deployed scraper also needs an artifact-retention policy.
10 / operations
What the browser costs when this becomes a job
The code covers a scraper on one machine. Multiple workers add browser binaries, memory, CPU, temporary storage, monitoring, and shared checkpoint storage. Detail pages dominate wall time because the script performs one real navigation per place.
Measure before adding concurrency. Record link-discovery time, median detail-page time, rows per successful run, missing-field rates, bytes transferred, retry count, and browser memory. Although two detail tabs may improve throughput, twenty can raise failure rates enough to reduce it. A semaphore caps simultaneous pages. Upstream capacity and usage permissions stay the same as concurrency rises.
Keep discovery and extraction separate
Queue collected place URLs so a failed detail worker does not lose the search.
Version selectors and schemas
Store which extractor produced each row so a DOM change is traceable.
Set quality gates
Stop a batch when name or address completion drops below its expected range.
Budget for browser minutes
Include retries, cold starts, binary downloads, and failed runs in cost per row.
A few authorized searches may require only the local script. Before scheduling a data pipeline, compare the full cost of owning browsers with an official or managed data source: engineering time, compute, proxying if appropriate and permitted, monitoring, breakage, and reruns. A managed source can cost less overall even when its unit price is higher.
This browser implementation provides a field-level baseline for evaluating official or managed replacements.