campaign-analyzer

Campaign Analyzer

Analyze Instantly email campaign performance across personas, signal types, and copy patterns. Produces a comprehensive Word document with statistical rigor and actionable recommendations.

Overview

This skill connects to the Instantly MCP API, pulls all campaign and analytics data, and generates a detailed performance analysis. It covers:

• Performance rankings by persona and signal type
• Statistical significance testing (two-proportion z-tests)
• Copy pattern analysis (what works vs what doesn't)
• Cross-tab breakdown (persona x signal matrix)
• Confidence intervals for reply rates
• Specific, actionable improvement recommendations

Prerequisites

• An Instantly MCP connector must be connected. The skill will auto-detect which MCP prefix to use by listing campaigns and finding the workspace with signal-type campaigns (names containing [COLD], [ANNIV], [UPCMG], [LITIG], [M&A], [SHUTDOWN]).
• The docx skill should also be available for Word document generation, though this skill includes its own document creation script as a fallback.

How It Works

Step 1: Identify the correct Instantly MCP

The user may have multiple Instantly workspaces connected. To find the right one:

1. Search deferred tools for Instantly-related MCPs (look for list_campaigns tools)
2. Call list_campaigns on each MCP and check the campaign names
3. The correct workspace has campaigns named like [COLD] Junior Eng, [ANNIV] Senior AE, etc. - with signal prefixes in brackets
4. If only one Instantly MCP exists, use it directly

Store the MCP prefix (e.g., mcp__<server-id>) for all subsequent calls.

Step 2: Pull campaign data

Fetch all campaigns with pagination:

list_campaigns(limit=100, skip=0)   # page 1
list_campaigns(limit=100, skip=100) # page 2 if needed

The response structure is {result: JSON_STRING} where the JSON string parses to {items: [...]}. Each campaign item has id, name, status, and timestamp_created.

Step 3: Parse campaign metadata from names

Campaign names follow the pattern: [SIGNAL] Persona Name

Signal types: COLD, ANNIV, UPCMG, LITIG, M&A, SHUTDOWN

Personas (14 total): Junior Eng, Senior Eng, Staff Eng, Eng Leadership, Junior AE, Senior AE, GTM Leadership, Junior CS, Senior CS, CS Leadership, BDR, FDE, Growth, Solution Architect

Extract signal and persona from each campaign name using regex: \[([A-Z&]+)\]\s*(.+)

Skip campaigns that don't match this pattern (they're not signal-based outbound campaigns).

Step 4: Fetch analytics

Use get_campaign_analytics with a list of all non-archived campaign IDs. The API accepts up to ~100 IDs at a time.

Response structure: {result: JSON_STRING} where the JSON string parses to an array of analytics objects with fields including campaign_id, sent, contacted, new_leads_contacted, replies, unique_replies, bounced, opportunities, opportunity_value, open, unique_opened.

Step 5: Fetch campaign sequences (for copy analysis)

For the top 10 and bottom 10 campaigns by reply rate, call get_campaign with each campaign ID to retrieve the full sequence (email steps with subject lines and body content).

Sequence steps contain variants - most campaigns use A/B testing with 2 variants on steps 1 and 3, and single variants on steps 2 and 4. Variants include subject and body fields. Body content often uses spintax format: {{RANDOM | option1 | option2 | option3}}.

Step 6: Run analysis

Use the bundled Python script to process all data:

python3 <skill-path>/scripts/analyze_campaigns.py \
  --data <path-to-merged-json> \
  --output <output-directory>

The script produces:

• analysis_results.json - Full analysis data
• statistical_tests.json - All z-test results with p-values and confidence intervals

If the script is not available, perform the analysis inline using Python with scipy.stats for statistical testing.

Analysis Components

Per-Persona Rollup: Aggregate sent, replies, reply rate, opens, bounces, opportunities, and pipeline value for each persona.

Per-Signal Rollup: Same aggregation by signal type.

Cross-Tab (Persona x Signal): Matrix showing reply rate for each combination, highlighting cells with sufficient volume for statistical inference.

Statistical Significance Testing:

• Two-proportion z-test comparing each persona's reply rate against the overall average
• Pairwise z-tests between signal types
• 95% confidence intervals for each persona's true reply rate
• Flag results as significant at p < 0.05, p < 0.01, or p < 0.001
• Note where sample sizes are too small for reliable inference (fewer than ~200 sends)
• Mention Bonferroni correction awareness for multiple comparisons

Copy Pattern Analysis:

• Compare subject lines and body content between top and bottom performers
• Identify patterns in: subject line length and style, greeting tone, specificity of value proposition, email length, follow-up progression, use of personalization tokens, spintax variety
• Categorize findings into "winning patterns" and "losing patterns"

Subject Line Performance Analysis:

• Extract all step 1 subject lines from campaign sequences (both A/B variants)
• Deduplicate subject lines and associate each with the campaign's reply rate
• Rank by reply rate and output top 10 and bottom 10 subject lines
• Analyze patterns across all subject lines:
  • firstName personalization usage rate
  • Lowercase vs mixed case frequency
  • Short (under 5 words) vs long subject lines
  • Role title mentions
  • Spintax usage
  • Question format usage
• Output pattern summary with counts and percentages
• Note: Reply rates are at campaign level (Instantly API does not expose per-variant analytics)

Opening Sentence Performance Analysis:

• Extract the first 2 lines of body content (after the greeting) from step 1 sequences
• Strip HTML tags and skip greeting lines ("hey {{firstName}}" etc.)
• Deduplicate and associate each opening with the campaign's reply rate
• Rank by reply rate and output top 10 and bottom 10 openings
• Analyze patterns across all openings:
  • companyName personalization token usage
  • "work with" or "partner with" phrasing
  • Founder/founding team mentions
  • Compensation or equity mentions
  • Short (under 80 chars) vs long openings
  • Spintax usage
• Output pattern summary with counts and percentages

Step 7: Generate the Word document

Read the docx skill (at /sessions/gifted-upbeat-shannon/mnt/.claude/skills/docx/SKILL.md or wherever it's installed) and follow its instructions for creating the output document. The document should include:

1. Executive Summary - Key metrics, top/bottom performers, headline findings
2. Performance by Signal Type - Table with metrics per signal, ranked by reply rate
3. Performance by Persona - Table with metrics per persona, ranked by reply rate
4. Statistical Significance - Which differences are real vs noise, confidence intervals
5. Reply Rate Cross-Tab - Persona x Signal matrix with color coding
6. Top/Bottom Campaigns - Lists with campaign names, metrics, and copy excerpts
7. Copy Pattern Analysis - What winning copy does differently from losing copy
8. Subject Line Performance - Top 10 and bottom 10 subject lines ranked by reply rate, plus pattern analysis table showing which subject line traits correlate with performance
9. Opening Sentence Performance - Top 10 and bottom 10 opening sentences ranked by reply rate, plus pattern analysis table showing which opening traits correlate with performance
10. Recommendations - Specific, actionable items based on the data

Use color coding in tables:

• Green (#2E7D32) for high performers (top quartile)
• Red (#C62828) for low performers (bottom quartile)
• Navy (#1F4E79) header backgrounds with white text

Save the document to the user's workspace folder.

Step 8: Present findings

After generating the document, provide a brief summary in chat covering:

• The single most important finding
• Any statistically significant surprises
• The top 2-3 recommendations
• A link to the full document

Handling Edge Cases

• No Instantly MCP connected: Tell the user they need to connect their Instantly workspace first.
• Multiple Instantly workspaces: Auto-detect by campaign naming pattern, or ask the user which workspace to analyze.
• Campaigns with zero sends: Exclude from percentage calculations to avoid division by zero.
• Low-volume campaigns: Flag personas or signals with fewer than 200 total sends as having insufficient data for statistical conclusions.
• Missing analytics: Some campaigns may not return analytics data - note these in the report and exclude from calculations.
• API pagination: Always check if more pages exist by comparing returned count to the limit.

Tips for Better Analysis

• Reply rate is the primary metric, but also look at opportunity conversion (replies that become opportunities) as a quality signal
• Pipeline value per send is a useful compound metric for comparing economic efficiency
• When comparing copy, focus on step 1 (initial email) since that's what most recipients see
• Spintax options within a single variant don't have separate tracking, so treat the whole variant as one unit
• A/B variant-level analytics aren't available through the Instantly API, so compare at the campaign level