Track real experiment exposure in PostHog
Experiment assignment and actual exposure are different events. Choose the randomization unit, define a variant-independent exposure boundary, record the assigned variant with that exposure, and count outcomes at the same unit so repeated member activity cannot distort an account-level experiment.
Inside this article 5 sections +
A feature flag can be evaluated while the application loads even when the user never reaches the feature being tested. If that evaluation is treated as exposure, the experiment includes people who had no opportunity to experience the change. In a B2B product, repeated activity by several account members adds another source of confusion.
Separate assignment, exposure, and outcome. Assignment determines which variant a unit receives. Exposure records a defined opportunity to encounter the changed experience. Outcome records the behavior the experiment is intended to influence. The identifiers and timing must connect all three.
Choose the unit before instrumenting events
For a shared reporting feature, account-level randomization may be appropriate because members collaborate and can influence one another. All members should then receive a consistent account variant. Randomizing each person independently could expose one account to both versions and make the treatment difficult to interpret.
PostHog supports group-based analytics and targeting, but configuration alone does not establish the right unit for your question. Define whether the intended outcome is per person, per account, or per workspace. Keep the denominator at that grain.
Resolve the account identity before evaluating the flag when the experiment depends on it. A flag evaluated anonymously and then reevaluated after identification can create conflicting exposures. Test account switching, multiple browser sessions, and people who belong to several workspaces.
Define exposure at a shared product boundary
For a report-layout experiment, exposure might occur when an eligible account member opens the report editor and the assigned layout is rendered. That is more specific than application startup. The boundary should be available to both variants and should not itself depend on an outcome caused by the treatment.
PostHog uses feature-flag evaluation events for default exposure and supports custom exposure events when a more precise boundary is needed. For a custom event, include the correct feature-variant property and ensure the experiment configuration selects that event.
{
"event": "report_editor_exposed",
"distinct_id": "user-demo-12",
"properties": {
"account_id": "account-demo-A",
"experiment_id": "report-layout-v1",
"$feature/report-layout": "test",
"surface_version": "editor-v6",
"exposure_reason": "eligible_editor_rendered"
}
}
This synthetic capture envelope illustrates the event contract. Configure the SDK's group association and experiment aggregation separately; an arbitrary account_id property alone does not automatically establish account-level randomization.
Be careful when choosing an exposure filter after seeing results. Restricting analysis to a post-treatment behavior can select different kinds of users in each variant. Define the boundary before launch and retain assignment data for a broader intention-to-treat view when appropriate.
Inspect a small ledger before trusting the dashboard
| Account | Assigned variant | Observed behavior | Exposure interpretation |
|---|---|---|---|
| A | Control | Editor opened; one report completed | One exposed account, one outcome |
| B | Test | Two members opened editor; one report completed | One exposed account, not two |
| C | Test | Flag evaluated at login; editor never opened | Assigned but not exposed under this definition |
| D | Control | Editor opened; no report completed | One exposed account, no outcome |
Check that repeated exposures do not multiply the chosen unit. Preserve raw events for diagnostics, but aggregate according to the experiment definition. If the outcome is account activation, several member actions should not create several activated accounts.
Record conflicting variants and investigate them. They may result from identity changes, flag configuration changes, or incorrect group assignment. Use an explicit analysis policy for those cases rather than silently choosing whichever variant makes the result cleaner.
Connect outcomes to time and opportunity
Define the outcome event, qualifying conditions, and observation window before launch. An account that completed a report before exposure should not count as a post-exposure conversion for this feature. A delayed event should retain its occurrence time and be handled under a documented late-arrival policy.
Distinguish an attempted action from a completed outcome. A client click on Publish can fail validation; a backend report-published event provides a stronger completion signal. Keep both if they answer different questions, but do not swap them midway through the experiment.
Use guardrail measures such as error rate, latency, and support burden alongside the primary outcome. A new layout might increase completion while causing a serious regression for a smaller group. The event design should make those effects observable without inventing a composite score that hides them.
Test the measurement before the treatment decision
Run synthetic accounts through both variants, including login without editor entry, multiple members, account switching, duplicate captures, and delayed outcomes. Compare the raw ledger with the configured PostHog experiment. Resolve differences in identity and exposure before interpreting statistical results.
Watch assignment balance, exposure balance, missing variant properties, and unexpected multiple-variant units. A sample-ratio mismatch can indicate instrumentation or rollout problems. It should trigger investigation rather than a quick adjustment to the report filter.
Keep the experiment definition, flag configuration, exposure contract, and metric version together. A later reviewer should be able to explain exactly who could enter the analysis and which behavior counted as success.
Read account identity modeling and activation funnel definitions. Our software implementation services connect those foundations to trustworthy experiment instrumentation.