Systems Lab

Agent skill

ab-testing-messaging

Design and run A/B tests on messaging variables with proper statistical discipline — Bayesian or frequentist depending on volume, minimum sample sizes, and one-variable-at-a-time design.

slowingNeeds a keyActs undeclared458 words

Filed under Analytics and reporting and Positioning and messaging.

From devangk003/gtm-agent-skills · 32 skills · 0 · pushed 2026-06-18

What it does when it runs

Design and run A/B tests on messaging variables with proper statistical discipline — Bayesian or frequentist depending on volume, minimum sample sizes, and one-variable-at-a-time design. Use when the user says "A/B test subject lines", "test opener variants", "which variant wins", or "sample size for A/B test.

Read from the skill and the 5 files bundled beside it. A skill’s own description is written to be selected by an agent, so it describes the job and not the dependencies.

Keys and connectors you must supply
  • AGENTIC_APP_TOKEN
Hosts it reaches
No third-party host appears in the skill or its bundled files.
Tool permissions it declares
No allowed-tools in the frontmatter. It does act, so it runs under whatever permissions your session already grants.
Actions present in the files
shell

Ask about ab-testing-messaging

Opens your assistant with this page's verified links already in the prompt.

Is this safe to install?ClaudeChatGPT
Adapt it to my stackClaudeChatGPT
What else do I need for it to workClaudeChatGPT
Rather ask a human? Talk to Cheetah

Install it

View source on GitHub ↗
git clone --depth 1 --filter=blob:none --sparse https://github.com/devangk003/gtm-agent-skills.git /tmp/gtm-agent-skills
git -C /tmp/gtm-agent-skills sparse-checkout set "ab-testing-messaging"
mkdir -p ~/.claude/skills/ab-testing-messaging
cp -R "/tmp/gtm-agent-skills/ab-testing-messaging/." ~/.claude/skills/ab-testing-messaging/

Picked up without a restart. A project skill of the same name is shadowed by your personal one. For one repository only, swap ~/.claude/skills for .claude/skills. Claude Code docs ↗

The folder is the same in every client that implements the format — 46 of them — so if yours is not above, only the destination changes.

Before you install: this skill will not complete its job on a bare agent. It needs AGENTIC_APP_TOKEN, which you have to obtain separately.

The skill

Source on GitHub ↗

Reproduced in full from devangk003/gtm-agent-skills/blob/6b9a1b0094ffd83d6c02bc03b7ce1610661a1644/ab-testing-messaging/SKILL.md, which is licensed MIT (skill frontmatter). 458 words, 6 headings.

A/B Testing — Messaging

Design and run A/B tests on outreach messaging variables (subject lines, openers, hooks, CTAs, sequence length, send-time) with proper statistical discipline. Hard rule: never ship a "winner" on n<50/arm.

When to Use

  • Active campaign needs messaging optimization
  • New copy hypothesis needs validation before scaling
  • Reply rate plateaued — copy refresh in flight
  • Sample-size estimate needed before launching a test
  • User says "A/B test subject lines" or "which variant wins"

Quick Reference

ConceptValue
One-variable ruleOnly ONE thing differs between A and B
Primary metricReply rate (open rate is noise — Apple MPP)
Low-volume regime (<1k/arm/14d)Bayesian + MIN_EFFECT_SIZE 3.0pp
High-volume regime (≥5k/arm/14d)Frequentist z-test + MIN_EFFECT_SIZE 1.0pp
Bayesian thresholdP(B>A) >0.85 (house); 0.95 industry-canonical
Frequentist thresholdTwo-proportion z-test p<0.05 AND effect ≥1pp
Min n per arm50; below = call inconclusive
Max test duration14d; longer = drift confounds

Procedure

  1. Validate hypothesis + variable. ONE variable differs. Multi-variable → reject. Confirm testable (not "tone"). See ${HERMES_SKILL_DIR}/references/regime-selection.md.
  2. Compute sample size + pick regime. Given baseline rate + expected volume, compute MDE and n required. Pick low-volume or high-volume regime explicitly.
  3. Pick significance method. Apply Quick Reference regimes against expected 14-day volume per arm: <1k → Bayesian (P(B>A)>0.85, MIN_EFFECT 3.0pp); ≥5k → frequentist z-test (p<0.05, MIN_EFFECT 1.0pp). 1k–5k is a judgment zone — default Bayesian unless the user prefers frequentist. Hard floor n≥50/arm applies in either regime.
  4. Design test document. Hypothesis + variants + split + metrics + thresholds + max duration + min n. This is the test contract. See ${HERMES_SKILL_DIR}/references/test-design.md.
  5. Launch via channel skill. Hand off to cold-email-sequence/linkedin-outreach with split + allocation rule.
  6. Monitor + check stop conditions. Daily/weekly: pull metrics per arm. Check min-n, significance, max duration.
  7. Significance call. Significance + effect ≥ threshold + min-n → winner. Ship 100% remaining to winner. Max duration without → inconclusive (ship control). See ${HERMES_SKILL_DIR}/references/significance-calls.md.
  8. Push to CRM. Test design at launch; result at call. See ${HERMES_SKILL_DIR}/scripts/push_to_crm.py.

Pitfalls

  • Multi-variable changes labeled "A/B test" — can't attribute the lift
  • Optimizing open rate — Apple MPP noise; reply rate only
  • Calling winners early on small n — n<50 = noise
  • Effect-size-blind significance — p<0.05 with 0.3pp lift = ship-worthless
  • Allocation drift mid-test — hash-based; same recipient always same variant

Verification

  1. Pre-launch sample-size estimate documented
  2. One-variable rule enforced
  3. Significance call references method + raw numbers + effect size
  4. Min-n + max-duration stop conditions honored
  5. Winner handoff to channel skill executed

Files bundled with it

These load only when the skill asks for them, so they cost nothing until it runs.

Other skills for the same job

Different authors, same problem. Matched on the words in the skill name, across every library in the catalogue except this one.

Need help setting it up?

This page tells you what ab-testing-messaging does and what it needs. Cheetah builds the agent setup it runs inside: data, CRM, sequencing and the guardrails.

Book a call →

The directory stays free. There is nothing gated behind this.