scrapercity-cli

ScraperCity CLI

Overview

This skill turns a ScraperCity job into commands and scripts that run unattended: a count check, a cost estimate, a start command that captures the runId, a poll, a download to a named file, and a schedule. It covers every command and flag in scrapercity 1.0.17, read from bin/cli.mjs and lib/client.mjs and exercised against a stubbed network on 2026-10-13.

The mistakes it prevents:

• Scripting around a flag that does not exist, or a command the CLI does not have (there is no property-lookup command in 1.0.17).
• Treating db-leads output as an export. It prints the total, the next cursor and the first 3 rows only.
• Combining flags that the 1.0.17 parser mixes into other values, such as scrapercity builtwith Shopify --name x, which searches for the technology "Shopify x".
• A cron job that dies silently because cron has neither a PATH to node nor SCRAPERCITY_API_KEY.
• A daily pull that re-buys or re-downloads yesterday's records because it skipped --exclude-delivered or paged with --page.

When to Use

Use it when the user says:

• "Give me the command to scrape plumbers in Denver" or "what does --after do?"
• "Write a script that pulls new leads every morning."
• "Run ScraperCity in GitHub Actions" or "in a Makefile."
• "Validate this CSV's emails from the terminal, then push the good ones into Instantly."
• "scrapercity says Unknown command" or "No API key."
• "Is there a CLI command for property lookup / the ecommerce database / webhooks?"

Use a different skill for nearby jobs:

Prerequisites

• Node.js 18 or newer. Run with npx -y [email protected] <command> (pinned, no install prompt) or install once with npm i -g [email protected] and call scrapercity <command>.
• An API key from app.scrapercity.com/dashboard/api-docs in SCRAPERCITY_API_KEY, or saved by scrapercity login.
• jq for scripts that write database rows to files. curl for the REST calls the CLI does not cover.
• For db-leads and db-local: Growth ($149/mo) or Scale.

Authoritative Foundations

Details and routing: references/framework-notes.md.

Workflow

1. Set up the key once

npx -y [email protected] login          # prompts for the key, writes ~/.scrapercityrc (owner-only)
export SCRAPERCITY_API_KEY=...           # or this; the variable wins over the file
npx -y [email protected] wallet         # proves the key; prints plan, balance, used, renewal

On a shared machine or CI, skip login and use the environment variable from a secret store.

2. Map the job to commands

Use references/command-reference.md. If a tool has no command or a needed flag is missing (see references/gaps-and-quirks.md), use the REST call shown there, or the MCP tool.

3. Count before you pull

A total of 0 prints as ? total leads. Treat ? as zero.

4. Estimate and confirm

Multiply billable units by catalog price for every step, including the CLI defaults you did not override (maps 500 places, apollo 1,000 leads, store-leads 1,000 leads). Show the table and get a go-ahead before any paid run. For a recurring job, show the monthly total too.

5. Start, capture the run id, poll, download

RUN=$(npx -y [email protected] maps -q "roofing contractor" -l "Dallas, TX" --limit 300 | awk '/Run ID:/ {print $3}')
npx -y [email protected] poll "$RUN" --interval 30
npx -y [email protected] download "$RUN" -o "maps-roofing-dallas-tx-$(date +%Y%m%d)-${RUN:0:8}.csv"

poll checks every --interval seconds (default 15), stops after one hour, exits 1 on FAILED, CANCELLED or timeout, and does not download. Use --interval 30 or more. Apollo never goes through poll: register a webhook at app.scrapercity.com/dashboard/webhooks and check status every few minutes.

6. Schedule the recurring part

Daily new-record pulls use --exclude-delivered with an --after cursor that starts at 0 each day. The CLI prints the next cursor as Next page: --after N; loop until it stops printing one or the daily target is reached. To write the rows to a file, use the REST loop in references/scripting-patterns.md, since the CLI shows only 3 rows per call. Run cron after 00:00 UTC, when the allowance resets.

7. Handle failure in the script

Every failure exits 1 and prints ✗ plus the server's message to stderr. The HTTP code is not printed (except HTTP N when the server sent no message), so branch on the message text. Use the error table in references/scrapercity-agent-patterns.md.

8. Deliver the spec

Fill templates/output-template.md and run python3 scripts/check-output.py spec.md. The checker rejects commands and flags that 1.0.17 does not have.

ScraperCity Implementation

Commands in 1.0.17 (27 plus help)

Not in 1.0.17: a property lookup command, an ecommerce database command, webhook management, an Apollo status command, JSON output mode. Workarounds in references/gaps-and-quirks.md.

Database count and cursor loop (CLI only)

npx -y [email protected] db-leads --title "VP Sales, Head of Sales" --not-title "Assistant, Intern" \
  --company-country "United States" --min-employees 50 --max-employees 200 \
  --keywords saas --has-email --exclude-delivered --after 0 --limit 100
# 4812 total leads, page 1 of 49
# Next page: --after 98765

List flags take commas: --industry, --country, --state, --seniority, --department, --company-country, --company-state, --not-industry (db-leads) and --category, --country, --not-category (db-local). db-local --location takes semicolons because places contain commas: --location "Houston, TX; Austin, TX; 78701".

Worked cost estimate

Request: a morning job on Growth that adds 400 new VP Sales leads and 100 new Texas dentists a day, validated before they reach the sequencer, for 30 days.

Per day that is $0.85 and the month is $25.35, which is 16% of Growth's $159 monthly credit. Show the monthly figure once and get an explicit approval before turning on cron. Database pulls use 400 of the Lead Database's 100,000 a day allowance and 100 of the Local Business Database's.

Output Format

A CLI run spec in Markdown, built from templates/output-template.md:

1. Environment: package version, Node version, key source, working folder.
2. Commands: one row per command with what it maps to and its billable units.
3. Cost Estimate: line items, total, approval line.
4. Script: the full bash script, ready to save.
5. Schedule: cron line or CI schedule, time zone, allowance reset.
6. Failure Handling: each failure, how the script detects it, what it does.
7. Verification: dry-run results.

A one-command question ("what does --after do?", "the command to validate this file") gets the command and two or three lines on what it bills and prints, not a spec. The spec is for scripts, schedules and multi-step jobs.

• Each fact once; every call and script written out in full (no "..." values, no loop described in a comment); no counts or results marked assumed. If nothing has run, say so once at the top.
• What the reader receives is the deliverable only: no internal file paths, reference section numbers or runbook IDs; another skill is named only in a handoff line that tells the user what to run next; no tables of checks that were not run (one "Not checked:" line instead); and no repeated identical rows where one template plus a list will do.

Quality Check

• Every command exists in 1.0.17 and every flag is one that command reads.
• No API key appears in the script; it comes from SCRAPERCITY_API_KEY or ~/.scrapercityrc.
• The script starts with set -euo pipefail and checks wallet before paid steps.
• --exclude-delivered is paired with --after, never with --page above 1.
• Every start command captures the runId; every download names its file with -o.
• No apollo run is polled with poll; a webhook is named.
• Cost lines use catalog prices, CLI defaults are counted, and any paid total (per run or per month) has an approval line.
• Version is pinned in scheduled jobs.

Common Pitfalls

Execution Artifacts

Related Skills

• using-scrapercity-skills: router for every ScraperCity job.
• scrapercity-mcp: the same package as an MCP server inside AI clients.
• scrapercity-api and scrapercity-integrations: REST and no-code paths, including endpoints the CLI lacks.
• scrapercity-lead-database and scrapercity-local-business-database: filter design for db-leads and db-local.
• email-list-cleaner and cold-email-list: what happens to the files next.