scrapercity-api

ScraperCity REST API

Overview

This skill turns "call ScraperCity from my code" into a client that starts each paid run once, waits the right way for it, downloads the CSV inside the 48-hour window, pages the databases without skipping or repeating rows, and stays under 100 requests a minute. The deliverable is an API integration spec plus working code from references/code-samples.md.

Everything here follows the published scrapercity npm package (lib/client.mjs is the reference for HTTP behavior) and the prices in references/scrapercity-catalog.md.

The mistakes it prevents:

• A retry loop around POST /api/v1/scrape/{slug} that buys the same run twice after a timeout.
• Polling an Apollo run every 15 seconds instead of registering a webhook.
• GET /api/v1/downloads/{runId} (404) instead of /api/downloads/{runId}.
• Paging the Lead Database with page=2, 3, 4 and excludeDelivered=true, which skips records.
• A 20KB query string from a long saved-search URL that the server refuses.
• Treating a webhook POST as proof a run succeeded without calling the status endpoint.

When to Use

• "What's the endpoint and body to start a Google Maps scrape?"
• "Write a Python script that pulls new leads from the Lead Database every morning."
• "Build a webhook receiver for Apollo runs."
• "I got a 409 / 429 / 503. What do I do?"
• "How do I page through 30,000 local businesses?"
• "My Apollo URL is too long for a GET."
• "Give me curl for wallet, status and download."
• "Enrich this 5,000-row CRM export: validate what has emails, find the missing ones, write back to HubSpot."
• "How big should each batch be, and how do I rerun a failed one without paying twice?"

Use a different skill for nearby jobs:

Prerequisites

• An API key from app.scrapercity.com/dashboard/api-docs, one per automation.
• The plan: Starter ($49/mo) covers scrapers and enrichment; the database endpoints need Growth ($149/mo) or Scale ($649/mo). Confirm with GET /api/v1/wallet.
• Runtime: curl, Python 3.8+ (standard library only) or Node 18+ (global fetch).
• For webhooks: a public HTTPS URL that can answer within a few seconds.
• Volume and schedule, so the rate budget and cost table can be filled in.

Authoritative Foundations

Routing and detail: references/framework-notes.md.

Workflow

1. Confirm key and plan

GET /api/v1/wallet. Read plan.name and wallet.total_balance_dollars. A 401 means a bad key (check for quotes, spaces, a line break). If the job needs a database and the plan is Starter, switch to the scraper path or stop.

2. Pick the endpoint for each step

Full slug list, bodies and response fields: references/endpoint-reference.md.

3. Count and estimate before any paid start

• Databases: same filters with limit=1, read pagination.total. No wallet charge.
• Scrapers: compute the ceiling from the body (for example searchStringsArray length x maxCrawledPlacesPerSearch x $0.0099 for scrape_maps). The formula table is in references/scrapercity-agent-patterns.md.
• scrape_maps and scrape_apollo start responses can carry local_business_database or lead_database with matches already in the databases. On Growth or Scale, pull those from the database first.
• Before any paid run: show the estimate, say what will be left in the wallet, and get a yes (or stay under the per-run amount the user set).

4. Start each paid run exactly once

• POST the body. Save runId before doing anything else.
• Pass fileName where the tool accepts it (scrape_apollo, scrape_builtwith, scrape_store_leads, scrape_property_lookup) so /api/v1/runs can find the run by file_name.
• 409: the same search was accepted in the last couple of minutes. Find it in GET /api/v1/runs?hours=1 and reuse it.
• 429: look in /runs first. If nothing matches, wait 30 s and retry once.
• Timeout or reset: look in /runs (the samples wait 10 s first), resubmit only if nothing matches.
• Keep at most 10 runs in flight.

5. Wait the right way

The npm client polls every 15 s with a 1-hour timeout; AGENT_DOCS gives each tool's delivery time.

Status is RUNNING, then SUCCEEDED, FAILED or CANCELLED. Show handled / requested as progress. On FAILED, read statusMessage and GET /api/v1/scrape/logs/{runId} before changing inputs.

6. Webhooks

Register the receiver URL at app.scrapercity.com/dashboard/webhooks (dashboard only). ScraperCity POSTs when a run reaches SUCCEEDED or FAILED, with runId, fileName and leadsCount in the JSON body. No signature or retry schedule is documented, so:

1. Answer 200 immediately.
2. Call GET /api/v1/scrape/status/{runId}. Download only on SUCCEEDED.
3. Record the runId as handled before importing, so a second delivery is ignored.
4. Run an hourly sweep of GET /api/v1/runs?hours=48 for SUCCEEDED runs not yet handled.
5. Use HTTPS and a long random path.

7. Download within 48 hours

GET /api/downloads/{runId} with the same Bearer header. No /v1. Name the file {tool}-{segment}-{YYYYMMDD}-{runId first 8}.csv. Dedupe locally before loading anywhere: lowercase email for people, normalized domain for companies, phone plus postal code for local businesses.

8. Page the databases

GET arrays use repeated keys; POST arrays are JSON arrays; booleans go as "true". Identical requests within 30 s return 429, so a loop must change after each call or wait. A 503 means the search ran too long: narrow it, do not hammer it. Each database allows 100,000 new records a day, reset at midnight UTC.

9. Bulk files: run the staged pipeline

When the input is a file the user already has, follow references/bulk-enrichment-pipeline.md: intake and clean, suppress and filter before paying, route each row by its gap (database first, then pay-on-find tools), one run per tool per batch of about 1,000, a batch ledger row per run, routing for every result status, a 20-row QA sample, and one-way CRM write-back that never overwrites owner, stage or deal fields.

10. Write the spec and test it

Fill templates/output-template.md, run the test plan (wallet, count, a 3-email validator run for $0.01, a fake webhook post sent twice), then python3 scripts/check-output.py spec.md.

ScraperCity Implementation

curl

# Start, then poll, then download
curl -sS -X POST https://app.scrapercity.com/api/v1/scrape/email-validator \
  -H "Authorization: Bearer $SCRAPERCITY_API_KEY" -H "Content-Type: application/json" \
  -d '{"emails":["[email protected]","[email protected]","[email protected]"]}'
curl -sS https://app.scrapercity.com/api/v1/scrape/status/RUN_ID -H "Authorization: Bearer $SCRAPERCITY_API_KEY"
curl -sS --fail -o validated.csv https://app.scrapercity.com/api/downloads/RUN_ID \
  -H "Authorization: Bearer $SCRAPERCITY_API_KEY"

# Local Business Database count, GET with repeated keys
curl -sS -G https://app.scrapercity.com/api/v1/database/local-businesses \
  -H "Authorization: Bearer $SCRAPERCITY_API_KEY" \
  --data-urlencode "keyword=dentist" --data-urlencode "location=Austin, TX" \
  --data-urlencode "location=Round Rock, TX" --data-urlencode "hasEmail=true" --data-urlencode "limit=1"

Python

import scrapercity_client as sc          # references/code-samples.md

APOLLO_URL = "https://app.apollo.io/#/people?personTitles[]=cto&personLocations[]=United%20States"

w = sc.wallet()
total = sc.db_page(sc.LEADS, {"title": "CTO", "country": ["United States"], "hasEmail": True, "limit": 1})
rows = list(sc.db_all(sc.LEADS, {"title": "CTO", "country": ["United States"],
                                 "hasEmail": True, "excludeDelivered": True}, max_rows=500))
run_id = sc.start("apollo", {"url": APOLLO_URL, "count": 2000, "fileName": "apollo-cto-us-202611"},
                  match=lambda r: r.get("file_name") == "apollo-cto-us-202611")

Node

import * as sc from './scrapercity-client.mjs' // references/code-samples.md

const runId = await sc.start('maps', { searchStringsArray: ['roofing contractor'], locationQuery: 'Denver, CO', maxCrawledPlacesPerSearch: 200 })
await sc.waitFor(runId, { everyMs: 15000, giveUpMs: 1800000 })
await sc.download(runId, `maps-roofing-denver-co-${runId.slice(0, 8)}.csv`)
for await (const biz of sc.dbAll(sc.LOCAL, { keyword: ['dentist'], location: ['Austin TX'], hasEmail: true })) { /* load */ }

All three clients in references/code-samples.md were syntax-checked and run against a local mock of every endpoint they call, including the 409 reuse path, the 429 retry, the POST switch for a 7,000-character URL, and a webhook delivered three times that downloads once.

Worked cost estimate

A monthly job for a SaaS outbound agency on Growth: 500 new VP Sales leads a weekday from the Lead Database, validated, plus one Apollo run of 2,000 leads for a technology segment the database cannot filter.

That fits inside the $159 of monthly Growth credits. The Apollo run plus validation is $11.18, shown and approved before the first start. Requests: 5 database calls a day at 1 a second, up to 40 status checks per validator run at 15 s over its 1 to 10 minute delivery time, one webhook-driven download for Apollo. Peak 10 a minute against a limit of 100.

Output Format

An API integration spec in Markdown from templates/output-template.md:

1. Request Summary: goal, runtime, plan, balance, volume, destination.
2. Endpoints: step, method, path, body or params, response fields used.
3. Auth and Keys: header, storage, one key per automation, rotation.
4. Run Handling: completion signal, poll interval and timeout per tool.
5. Database Paging: cursor, excludeDelivered, count, daily volume.
6. Error Handling: action and retry rule per status code.
7. Rate Budget: peak requests a minute, concurrent scrapes, pacing.
8. Cost Estimate: catalog prices, total.
9. Test Plan and Next Steps.

Plus the client files from references/code-samples.md adapted to the job. Check the spec with python3 scripts/check-output.py spec.md.

A single-endpoint question (a path, a body, what a 409 means) gets the call and its handling in a few lines, not a spec. The spec is for integrations that run unattended.

• Each fact once; every call and script written out in full (no "..." values, no loop described in a comment); no counts or results marked assumed. If nothing has run, say so once at the top.
• What the reader receives is the deliverable only: no internal file paths, reference section numbers or runbook IDs; another skill is named only in a handoff line that tells the user what to run next; no tables of checks that were not run (one "Not checked:" line instead); and no repeated identical rows where one template plus a list will do.

Quality Check

• Every path matches references/endpoint-reference.md; downloads use /api/downloads/{runId}.
• Scraper starts are POST; wallet, runs, status, logs and downloads are GET; cancel is POST.
• No API key appears in code, config committed to a repo, logs or the spec.
• Any code path that resubmits a POST start checks /api/v1/runs first.
• 409 is handled by reuse; 401, 402 and 403 stop the job.
• Apollo uses a webhook plus an hourly sweep, or polls every few minutes after the first 15.
• Database reads use after for multi-page pulls and never pair excludeDelivered=true with page above 1.
• Requests over 6,000 characters of query string go as POST bodies.
• Peak load stays under 100 requests a minute and 10 concurrent scrapes.
• The cost table uses catalog prices, and every paid step was approved before it ran.
• python3 -m py_compile / node --check pass on the delivered files.

Common Pitfalls

Execution Artifacts

Related Skills

• scrapercity-integrations: the same API inside Clay, n8n, Zapier, Make, Pipedream, Sheets, Airtable, CRMs and senders.
• scrapercity-mcp and scrapercity-cli: the other two access paths.
• scrapercity-lead-database, scrapercity-local-business-database, scrapercity-real-estate-data: what to send in the request.
• email-list-cleaner: what to do with validator output.
• crm-enricher: enriching records already in a CRM.
• using-scrapercity-skills: router.