messaging-ab-tester

Messaging A/B Tester

Stop debating which message is better — test it. Generate messaging variants, deploy them through real channels, and measure which framing actually resonates with your ICP.

Core principle: At seed/Series A, you don't have enough traffic for website A/B tests. But you do have enough LinkedIn impressions and cold email sends to test messaging angles fast.

When to Use

• "Which of these value props should we lead with?"
• "Test our messaging angles and tell me which works"
• "I can't decide between [message A] and [message B]"
• "What messaging resonates most with [ICP]?"
• "Run a messaging test for [product/feature]"

Phase 0: Intake

What to Test

1. Core value prop — The claim or positioning you want to test (e.g., "We help growth teams run outbound 10x faster")
2. Test goal — What are you deciding? (Headline for website, cold email angle, LinkedIn content strategy, ad copy direction)
3. ICP — Who should this resonate with? (Title, company type, stage)
4. Current messaging — What are you using today? (Baseline to beat)

Test Channel

5. Where to test:
   • LinkedIn organic — Post variants across consecutive days, compare engagement
   • Cold email — A/B test subject lines or opening hooks via Smartlead
   • Both — Run in parallel for fastest signal
6. Sample size available:
   • LinkedIn: followers/typical impressions per post
   • Email: list size available for testing

Constraints

7. Number of variants — 3-5 recommended (more = slower signal)
8. Test duration — How long to run? (Default: 1 week for LinkedIn, 3-5 days for email)

Phase 1: Generate Messaging Variants

Create 3-5 variants that test different angles, not just different words. Each variant should represent a distinct strategic bet:

Variant Types

Type	What It Tests	Example
Outcome-driven	Leading with the result	"3x your pipeline in 30 days"
Pain-driven	Leading with the problem	"Tired of spending 4 hours a day on manual prospecting?"
Identity-driven	Leading with who they are	"Built for growth teams who move fast"
Proof-driven	Leading with evidence	"How [Customer] went from 10 to 50 demos/month"
Contrast-driven	Leading with what you're not	"Not another CRM. An outbound engine."

Variant Template

For each variant:

VARIANT [N]: [Type — e.g., "Outcome-driven"]

Hypothesis: This framing will resonate because [reasoning tied to ICP psychology]

LinkedIn post version:
---
[Full post copy — 100-200 words, native LinkedIn format]
---

Email subject line version:
[Subject line — max 50 chars]

Email opening hook version:
[First 2 sentences of an email]

Headline version:
[Website headline — max 10 words]

Phase 2: Deploy Tests

Option A: LinkedIn Organic Test

Setup:

1. Schedule variants as consecutive posts (1 per day, same time of day)
2. Each post should be similar length and format (control for post structure)
3. Don't boost any posts — organic only for clean comparison

Measurement (after 48 hours per post):

• Impressions
• Reactions (likes, celebrates, etc.)
• Comments
• Comment sentiment (positive/negative/neutral)
• Profile visits (if trackable)
• DMs received mentioning the post

Option B: Cold Email A/B Test

Setup via Smartlead:

1. Create campaign with all variants as A/B test sequences
2. Split list evenly across variants (minimum 50 per variant for signal)
3. Same send time, same sender, same CTA — only the messaging changes

Measurement (after 5 days):

• Open rate (tests subject line)
• Reply rate (tests full message resonance)
• Positive reply rate (tests conversion quality)
• Click rate (if link included)

Option C: Both (Recommended)

Run LinkedIn and email in parallel. Different channels may show different winners — that's valuable signal about where each message works best.

Phase 3: Analyze Results

Scoring Framework

Metric	Weight (LinkedIn)	Weight (Email)
Engagement rate	30%	—
Comment quality	30%	—
Open rate	—	30%
Reply rate	—	40%
Positive reply rate	—	30%
Impressions	20%	—
Profile visits / clicks	20%	—

Statistical Significance Check

For email tests:

• Minimum sends per variant: 50 (for directional signal), 200+ (for confident decisions)
• Minimum difference to call a winner: >20% relative difference in primary metric

For LinkedIn tests:

• Minimum posts per variant: 1 (you're testing with limited data — treat as directional)
• Minimum impressions: 500 per post to be comparable

Winner Selection

WINNER: Variant [N] — [Type]

Primary metric: [X] (vs average of [Y] across other variants)
Relative improvement: [Z%] over baseline

Why it won:
[1-2 sentences on what this tells us about ICP messaging preferences]

Runner-up: Variant [N]
[1 sentence on when this might work better — different channel, different segment]

Phase 4: Output Format

# Messaging A/B Test Results — [DATE]
Value prop tested: [description]
ICP: [target audience]
Test duration: [dates]

---

## Test Design

| Variant | Type | Hypothesis |
|---------|------|-----------|
| A | [Type] | [Hypothesis] |
| B | [Type] | [Hypothesis] |
| C | [Type] | [Hypothesis] |

---

## Results

### LinkedIn Test

| Variant | Impressions | Reactions | Comments | Engagement Rate | Score |
|---------|------------|-----------|----------|----------------|-------|
| A | [N] | [N] | [N] | [X%] | [weighted] |
| B | [N] | [N] | [N] | [X%] | [weighted] |
| C | [N] | [N] | [N] | [X%] | [weighted] |

### Email Test

| Variant | Sends | Opens | Open Rate | Replies | Reply Rate | Positive | Score |
|---------|-------|-------|-----------|---------|------------|----------|-------|
| A | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |
| B | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |
| C | [N] | [N] | [X%] | [N] | [X%] | [N] | [weighted] |

---

## Winner: Variant [N] — "[Headline]"

**Why it won:** [Analysis — what does this tell us about how our ICP thinks?]

**Recommended deployment:**
- Website headline: "[adapted version]"
- Sales deck opening: "[adapted version]"
- LinkedIn bio: "[adapted version]"
- Cold email default: "[adapted version]"

---

## Variant Details & Copy

### Variant A: [Full copy used in test]
### Variant B: [Full copy used in test]
### Variant C: [Full copy used in test]

---

## What to Test Next

Based on these results, the next messaging test should explore:
1. [Angle suggested by results — e.g., "test more specific proof points since proof-driven won"]
2. [Segment test — e.g., "test winning message against different ICP segment"]

Save to clients/<client-name>/product-marketing/messaging-tests/[test-slug]-[YYYY-MM-DD].md.

Cost

Component	Cost
Variant generation	Free (LLM reasoning)
LinkedIn posting	Free (organic)
Email testing (via Smartlead)	Included in Smartlead plan
Results analysis	Free (LLM reasoning)
Total	Free (assuming existing Smartlead subscription)

Tools Required

• Smartlead — for email A/B testing (optional — only if testing via email)
• No other tools required for LinkedIn organic testing
• Pure reasoning for variant generation and analysis

Trigger Phrases

• "Test which messaging angle works best for [ICP]"
• "Run a messaging A/B test for [value prop]"
• "Which of these messages should we lead with?"
• "Help me decide between these positioning options"