Email A/B Winner → Focus Group Why

Pull Mailchimp A/B test results and run a Mavera Focus Group to understand the psychological drivers behind why one subject line outperformed another

Scenario

Mailchimp ran an A/B test on subject lines (or content) and one variant won. You know which subject line had a higher open rate, but not why. You pull both variants and the performance results, then run a Mavera Focus Group asking personas which subject line they’d open and why — the psychological drivers behind the click.

Architecture

Code

Python

import os, requests, time

MC_KEY = os.environ["MAILCHIMP_API_KEY"]
MC_DC = os.environ["MAILCHIMP_DC"]
MV = os.environ["MAVERA_API_KEY"]
MB = "https://app.mavera.io/api/v1"
MH = {"Authorization": f"Bearer {MV}", "Content-Type": "application/json"}
MC_BASE = f"https://{MC_DC}.api.mailchimp.com/3.0"
mc_auth = ("anystring", MC_KEY)

AB_CAMPAIGN_ID = os.environ.get("MAILCHIMP_AB_CAMPAIGN_ID", "")

if AB_CAMPAIGN_ID:
    r = requests.get(f"{MC_BASE}/campaigns/{AB_CAMPAIGN_ID}", auth=mc_auth)
    r.raise_for_status()
    campaign = r.json()
    ab_settings = campaign.get("variate_settings", {})
else:
    ab_settings = {
        "subject_lines": [
            "Your Q1 numbers are in — here's what they mean",
            "We analyzed your Q1 data (3 surprises inside)"
        ],
        "winner_criteria": "open_rate",
    }
    campaign = {
        "report_summary": {"open_rate": 0.32, "click_rate": 0.048},
    }

subject_a = ab_settings.get("subject_lines", ["Subject A", "Subject B"])[0]
subject_b = ab_settings.get("subject_lines", ["Subject A", "Subject B"])[1] if len(ab_settings.get("subject_lines", [])) > 1 else "Subject B"

report_r = requests.get(f"{MC_BASE}/reports/{AB_CAMPAIGN_ID}/ab", auth=mc_auth) if AB_CAMPAIGN_ID else None
if report_r and report_r.status_code == 200:
    ab_report = report_r.json()
    a_stats = ab_report.get("a", {})
    b_stats = ab_report.get("b", {})
else:
    a_stats = {"open_rate": 0.28, "click_rate": 0.038, "unique_opens": 1400}
    b_stats = {"open_rate": 0.36, "click_rate": 0.052, "unique_opens": 1800}

winner = "B" if b_stats.get("open_rate", 0) > a_stats.get("open_rate", 0) else "A"

PERSONA_IDS = os.environ.get("AB_PERSONA_IDS", "").split(",")
if not PERSONA_IDS[0]:
    for name, desc in [
        ("Newsletter Regular", "Opens emails 2-3x per week. Skims subject lines quickly. Values clarity over cleverness."),
        ("Busy Executive", "Gets 200+ emails/day. Opens only what seems urgent or directly relevant. Reads on mobile."),
        ("Curious Explorer", "Opens most marketing emails. Enjoys discovering new ideas. Clicks through to articles."),
    ]:
        p = requests.post(f"{MB}/personas", headers=MH, json={"name": name, "description": desc}).json()
        PERSONA_IDS.append(p["id"])
        time.sleep(0.3)

fg = requests.post(f"{MB}/focus-groups", headers=MH, json={
    "name": "Mailchimp A/B Subject Line Investigation",
    "persona_ids": [pid for pid in PERSONA_IDS if pid],
    "questions": [
        f'You see two emails in your inbox:\n  A: "{subject_a}"\n  B: "{subject_b}"\nWhich would you open first? Why? What about the winning subject line caught your attention?',
        f'Subject A got {a_stats.get("open_rate", 0):.0%} opens and Subject B got {b_stats.get("open_rate", 0):.0%}. Does the winner ({winner}) surprise you? Why do you think it performed better?',
        "What elements in a subject line make you most likely to open: (1) Personalization, (2) Curiosity gap, (3) Urgency, (4) Social proof, (5) Specificity (numbers/data)? Rank and explain your #1.",
        "Write a subject line that would outperform BOTH of these. Explain your strategy.",
    ],
    "context": f"""Mailchimp A/B test results:

VARIANT A: "{subject_a}"
  Open rate: {a_stats.get('open_rate', 0):.0%} | Clicks: {a_stats.get('click_rate', 0):.0%} | Unique opens: {a_stats.get('unique_opens', 0):,}

VARIANT B: "{subject_b}"
  Open rate: {b_stats.get('open_rate', 0):.0%} | Clicks: {b_stats.get('click_rate', 0):.0%} | Unique opens: {b_stats.get('unique_opens', 0):,}

Winner: Variant {winner} (by {ab_settings.get('winner_criteria', 'open_rate')})
Margin: {abs(a_stats.get('open_rate', 0) - b_stats.get('open_rate', 0)):.0%} difference""",
    "responses_per_persona": 2,
}).json()

for _ in range(24):
    time.sleep(5)
    data = requests.get(f"{MB}/focus-groups/{fg['id']}", headers=MH).json()
    if data.get("status") == "completed":
        break

print(f"Focus Group: {data.get('id')} — {data.get('status')}\n")
for resp in data.get("responses", []):
    print(f"[{resp.get('persona_id','?')}] {resp.get('question','')[:70]}")
    print(f"  → {resp.get('answer','')[:300]}\n")

Example Output

Focus Group: fg_mc_ab_5k8n — completed

[Newsletter Regular] Which would you open first?
  → B: "We analyzed your Q1 data (3 surprises inside)". The number 3
    tells me this is a quick read, and "surprises" creates a curiosity
    gap I can't ignore. Subject A feels like a report I should read —
    Subject B feels like a secret I want to know.

[Busy Executive] Does the winner surprise you?
  → No. B has specificity (3 surprises) and implied exclusivity ("your
    data"). A sounds like a generic newsletter. When I'm scanning 200
    subjects, the one that says "we did work FOR YOU" wins over "here's
    information."

[Curious Explorer] Write a subject line that outperforms both
  → "Your Q1 had 3 anomalies — #2 changes your strategy." It combines
    personalization ("your"), specificity ("3"), curiosity ("#2 changes"),
    and stakes ("your strategy"). The incomplete list (naming #2 but
    not #1 or #3) creates irresistible curiosity.

Error Handling

A/B test campaign structure

Mailchimp stores A/B test variants in variate_settings.subject_lines (for subject tests) or variate_settings.contents (for content tests). The winning criteria is in variate_settings.winner_criteria.

A/B report endpoint

The /reports/{id}/ab endpoint returns variant-level stats. Not all campaigns have this — only those created as “A/B Test” type. Regular campaigns return 404 on this endpoint.

Focus Group timing

3 personas × 4 questions × 2 responses = 24 total. Allow 60–120s. The loop provides ~120s.

---

What’s Next

Mailchimp Integration

Back to Mailchimp integration overview

Campaign Content Optimization

Train brand voice from winning campaigns

Interest Content Generation

Generate content by audience interest

Focus Groups API

Full reference for POST /api/v1/focus-groups