linkedin-post-research

LinkedIn Post Research

Search LinkedIn for posts matching keywords via the Crustdata API, deduplicate results, and output sorted by engagement.

Quick Start

Requires requests and CRUSTDATA_API_TOKEN environment variable.

# Single keyword search
python3 skills/linkedin-post-research/scripts/search_posts.py \
  --keyword "AI sourcing" \
  --time-frame past-week

# Multiple keywords, output CSV
python3 skills/linkedin-post-research/scripts/search_posts.py \
  --keyword "talent sourcing tools" \
  --keyword "recruiting automation" \
  --keyword "AI sourcing" \
  --time-frame past-week \
  --output csv \
  --output-file results.csv

# Keywords from file, multiple pages
python3 skills/linkedin-post-research/scripts/search_posts.py \
  --keywords-file keywords.txt \
  --time-frame past-week \
  --pages 3 \
  --output json \
  --output-file results.json

# Summary only (prints to stderr)
python3 skills/linkedin-post-research/scripts/search_posts.py \
  --keyword "recruiting stack" \
  --output summary

Inputs

• Keywords: Search terms — pass via --keyword flags or --keywords-file (one per line)
• Time frame: past-day, past-week, past-month, past-quarter, past-year, all-time (default: past-month)
• Pages: Number of pages per keyword (default: 1, ~5 posts/page). Max 20.
• Sort by: relevance or date (default: relevance)

CLI Reference

Flag	Default	Description
--keyword, -k	required	Keyword to search (repeatable)
--keywords-file, -f	—	File with one keyword per line (lines starting with # are ignored)
--time-frame, -t	past-month	Time filter
--sort-by, -s	relevance	Sort order
--pages, -p	1	Pages per keyword (~5 posts per page)
--limit, -l	—	Exact number of posts per API call (1-100)
--output, -o	json	Output format: json, csv, summary
--output-file	stdout	Write output to file
--max-workers	6	Max parallel API calls

How It Works

1. Takes keywords via CLI args or file
2. Calls Crustdata's /screener/linkedin_posts/keyword_search API in parallel for all (keyword × page) combinations
3. Deduplicates posts across keywords by backend_urn
4. Sorts by total_reactions descending
5. Outputs JSON, CSV, or summary

Output Schema (JSON)

{
  "author": "Jane Smith",
  "keyword": "AI sourcing",
  "reactions": 142,
  "comments": 28,
  "date": "2026-02-20",
  "post_preview": "First 200 chars of the post text...",
  "url": "https://www.linkedin.com/posts/...",
  "backend_urn": "urn:li:activity:123456789",
  "num_shares": 12,
  "reactions_by_type": "{\"LIKE\": 100, \"EMPATHY\": 30, \"PRAISE\": 12}",
  "is_repost": false
}

Output Columns (CSV)

Column	Description
author	LinkedIn post author name
keyword	Which search keyword matched this post
reactions	Total reaction count
comments	Total comment count
date	Post date (YYYY-MM-DD)
post_preview	First ~200 characters of the post
url	Direct link to the LinkedIn post
backend_urn	Unique post identifier
num_shares	Number of shares

Crustdata API Details

Endpoint: GET https://api.crustdata.com/screener/linkedin_posts/keyword_search

Parameters:

• keyword — Search term
• page — Page number (1-based, ~5 posts per page)
• sort_by — relevance or date
• date_posted — Time filter (past-day, past-week, etc.)
• limit — Exact number of posts to return (1-100)

Auth: Authorization: Token <CRUSTDATA_API_TOKEN>

Credit usage: 1 credit per post returned.

Rate limits: Searches run in parallel (default 6 workers). The script handles 429 responses with automatic retry.

Environment Variables

Variable	Required	Description
CRUSTDATA_API_TOKEN	Yes	Crustdata API token

Cost

~1 credit per post returned. Searching 10 keywords × 1 page = ~50 posts = ~50 credits.

Chaining with Other Skills

After getting the post list, common next steps:

1. Extract commenters → linkedin-commenter-extractor — pass post URLs to find warm leads
2. Research authors → web search on high-engagement post authors
3. Qualify leads → lead-qualification — score extracted people against ICP

Example pipeline:

# Step 1: Search posts
python3 skills/linkedin-post-research/scripts/search_posts.py \
  --keyword "AI sourcing" --keyword "recruiting automation" \
  --time-frame past-week --output json --output-file posts.json

# Step 2: Extract post URLs for commenter extraction
cat posts.json | python3 -c "
import json, sys
posts = json.load(sys.stdin)
# Filter: keep posts with 5+ comments
for p in posts:
    if p['comments'] >= 5:
        print(p['url'])
" > post_urls.txt

# Step 3: Extract commenters from those posts
while read url; do
  python3 skills/linkedin-commenter-extractor/scripts/extract_commenters.py --post-url "\$url" --output json
done < post_urls.txt