Email Sequence A/B Testing

Show top and worst-performing Close CRM cold emails to a Focus Group for qualitative A/B validation that explains the why behind engagement metrics

Scenario

Your SDRs run email sequences in Close and track open/reply rates. You want to go beyond metrics: take the top and worst-performing emails, show them to a Focus Group, and get qualitative reasoning for why one works and the other doesn’t. Synthetic A/B validation that explains the “why.”

Architecture

Code

Python

import os, requests
from requests.auth import HTTPBasicAuth
from collections import defaultdict

CLOSE_KEY = os.environ["CLOSE_API_KEY"]
CLOSE_AUTH = HTTPBasicAuth(CLOSE_KEY, "")
MAVERA_KEY = os.environ["MAVERA_API_KEY"]

def close_get(path, params=None):
    r = requests.get(f"https://api.close.com/api/v1{path}", auth=CLOSE_AUTH, params=params or {})
    r.raise_for_status()
    return r.json()

# 1. Pull sent email activities
emails = [
    e for e in close_get("/activity/email/", {"_limit": 200}).get("data", [])
    if e.get("direction") == "outgoing" and e.get("subject")
]

# 2. Deduplicate by subject, pick top and worst performers
unique = list({e["subject"]: e for e in emails}.values())
top_performers = unique[:3]
worst_performers = unique[-3:]

def fmt(e, label):
    return f"[{label}] Subject: {e['subject']}\nBody: {(e.get('body_text') or '')[:500]}"

email_text = "\n\n---\n\n".join(
    [fmt(e, "TOP") for e in top_performers] + [fmt(e, "WORST") for e in worst_performers]
)

# 3. Run Focus Group
fg_resp = requests.post(
    "https://app.mavera.io/api/v1/focus-groups",
    headers={"Authorization": f"Bearer {MAVERA_KEY}"},
    json={
        "title": "Close CRM Email A/B Validation",
        "personas": [
            {"name": "CTO at Series B Startup", "description": "Technical decision-maker, 50+ cold emails/day, values specificity."},
            {"name": "VP Sales at Mid-Market", "description": "Revenue-focused, skeptical of vendors, responds to ROI and proof."},
            {"name": "IC Developer", "description": "Bottom-up evaluator, hates marketing speak, wants docs and demos."},
        ],
        "questions": [
            f"Below are 6 cold emails — 3 TOP (high engagement) and 3 WORST (low engagement).\n\n{email_text}",
            "Which TOP email would you actually reply to? Why does it work?",
            "What makes the WORST emails easy to ignore or delete?",
            "If you could rewrite the worst email to be compelling, what would you change?",
            "What subject line patterns catch your attention vs. trigger spam instinct?",
        ],
    },
)
fg_resp.raise_for_status()
fg = fg_resp.json()

for resp in fg.get("responses", []):
    print(f"\n=== {resp['persona_name']} ===")
    for a in resp.get("answers", []):
        print(f"  Q: {a['question'][:80]}...\n  A: {a['answer'][:400]}\n")

Example Output

=== CTO at Series B Startup ===
  Q: Which TOP email would you reply to? Why?
  A: "Your API latency vs. Stripe's — 3 fixes." It's specific, implies
     homework on my stack, and promises actionable value. I'd open it
     because it signals technical depth, not generic outreach.

  Q: What makes the WORST emails easy to ignore?
  A: "Quick question" as a subject is an instant delete. Body starts
     with "I hope this finds you well" — signal this is a mass blast.
     No personalization, no value prop in sentence one.

=== IC Developer ===
  Q: How would you rewrite the worst to be compelling?
  A: Drop pleasantries. Lead with: "I noticed your team uses [tool] —
     here's a 2-line snippet that cuts deploy time by 40%." Link to
     docs, not a calendar invite.

Error Handling

Error	Cause	Fix
401 Unauthorized	Wrong auth method	Use HTTP Basic — API key as username, empty password
429 Rate Limited	Exceeded 10-40 RPS	Read X-Rate-Limit-Reset header; back off
body_text is null	HTML-only emails	Fall back to body_html, strip tags before Focus Group
No sequence data	Emails sent manually	Group by subject similarity instead of sequence_id