The Experiment Decision Journey

The Loop, Annotated

    ┌──────────────────────────────────────────────────────┐
    │                                                      │
    ▼                                                      │
 SEGMENT ─────► EXPERIMENT ─────► ANALYZE ─────► DISCOVER  │
 "Who are       "What should      "Did it        "Where    │
  our            we test,          work?"         does it   │
  customers?"    for whom?"                       differ?"  │
    ▲                                               │      │
    │                          ┌────────────────────┤      │
    │                          ▼                    ▼      │
    │                    REFINE SEGMENTS       ASK WHY     │
    │                    "What new             "Why did     │
    │                     boundary             that group   │
    │                     did we find?"         respond     │
    │                          │                differently?"│
    │                          └───────┬────────┘          │
    │                                  ▼                   │
    └──────────────── RICHER MODEL ────────────────────────┘
                      "What do we understand now
                       that we didn't before?"

Phase 1: SEGMENT — "Who are our customers?"

The operator needs to understand the current customer landscape before designing experiments.

Key insight: Segments are a practice, not a model. The quality of segments improves through loop iterations. First iteration uses rough cuts (new vs. returning). Later iterations incorporate boundaries discovered by Tyche.

Phase 2: EXPERIMENT — "What should we test, for whom?"

The operator designs an experiment: choosing a surface, forming a hypothesis, selecting metrics, and configuring targeting.

Key insight: The experiment config is a contract between the operator and the system. It declares: "for visitors matching these targeting rules, deliver these variants, and measure these metrics." The system enforces the contract.

Phase 3: ANALYZE — "Did it work?"

The experiment has run. The operator needs to understand the overall result.

Key insight: The answer to "did it work?" is always a distribution, not a yes/no. The expected loss distribution makes the decision explicit: "shipping variant B when it's actually worse would cost us $X/visitor, and there's a Y% chance of that."

Phase 4: DISCOVER — "Where does it differ?"

Beyond the overall result, the operator needs to understand treatment effect heterogeneity.

Key insight: The n-problem is the central tension. Small segments have wide posteriors — that's uncertainty made visible, not a failure. "We found a difference for returning customers, but we only have 47 visitors in that segment, so the posterior is wide." This drives the next iteration: target that segment specifically to narrow the posterior.

Phase 5: REFINE — "What boundary did we find?"

Discovery produced segment-level results. The operator decides which discoveries to incorporate.

Key insight: This is where the loop actually closes — or doesn't. The segment registry connects Tyche's discovery output to dbt's segment rules and the experiment engine's targeting. Without it, the operator must manually translate between three representations of the same concept.

Phase 6: ASK WHY — "Why did that group respond differently?"

Not every discovery has an obvious explanation. The operator needs qualitative signal.

Key insight: This phase is the human part of the loop. Quantitative analysis tells you what happened; qualitative feedback tells you why. The VoC system enriches the loop but doesn't block it — the loop can turn without it, but turns better with it.

Phase 7: RICHER MODEL — "What do we understand now?"

After one or more loop iterations, the operator reflects on what's changed.

Key insight: The richer model is not a deliverable — it's an emergent property of the loop turning. Each iteration deposits understanding. The value compounds. Iteration 5's experiment is qualitatively different from iteration 1's because the segmentation model, the hypotheses, and the operator's intuition have all improved.

← Back to Stoa Stack