Using control groups in one-shot campaigns
What is a "Control Group"?
A control group is a subset of contacts randomly selected from the audience of a campaign, who are excluded from sending the message for that specific campaign .
The goal is to compare the behavior of contacts exposed to the campaign with that of similar contacts who did not receive it. This comparison allows us to measure the incremental impact of the campaign.
In other words, the Control Group helps to answer the question:
"What happened as a result of this campaign?"
rather than:
"What revenue is associated with this campaign?"
1. How does a control group work?
A control group campaign consists of two populations:
Exposed group
Targeted contacts by the campaign after exclusion from the Control Group.
This includes the entire audience targeted by the campaign, including blocklisted, unreachable or undeliverable contacts.
Control group
Contacts intentionally excluded from the mailing.
They are selected completely randomly from the campaign audience to ensure a reliable and unbiased comparison.
2. When to use a control group?
Recommended use cases
Measuring the incremental lift of a promotional campaign.
Evaluate the real impact of a CRM action.
Addressing allocation challenges with Finance or Management teams.
Validate a strategy using a "test & learn" approach before large-scale deployment.
Less suitable use cases
Small audiences, for which the results may lack statistical significance.
Contexts where marketing pressure does not allow for the exclusion of a portion of the audience.
3. Define the audience
The audience estimate displayed in the builder and the control group are not correlated.
The audience estimation removes blocklisted contacts as well as unreachable contacts.
The control group, on the other hand, is calculated based on the total audience, which includes:
the contact details that can be reached;
unreachable contacts;
contacts in blocklist.
4. Understanding statistics
The campaign statistics only reflect the performance of the exposed Group .
Contacts belonging to the Control Group are excluded from standard indicators such as:
open rate;
click-through rate;
conversion rate;
overall campaign performance.
This data is visible in the Campaign Performance Dashboard (CPD) .
Control Group Analysis
The analysis of the Control Group is available in a specific tab.
It compares the results obtained by the exposed group and the control group in order to estimate the incremental impact of the campaign.
5. Understanding metrics
Metric | Definition | Formula |
|---|---|---|
Total audience | All the people who could have received your campaign. | This is the same value as the total audience in the Statistics tab (which includes unreachable contacts, blocklisted contacts and the Control Group). |
Exposed group | All targeted individuals (total audience less control group). | Exposed group = total audience - Control group |
Control group | A random sample excluded from sending the campaign (defined as a % or a specific number). | Control group = % or volume of the total audience |
Buyers | Number of people who actually made a purchase in each (unique) group. | Total of all unique buyers |
Conversion rate | Number of unique buyers compared to the total number of contacts. | Buyers ÷ Contacts × 100 |
Confidence interval | Indicates the range in which the real effect of your campaign is likely to fall, as the test is based solely on a sample of users. | It takes into account conversions and contacts. |
Average basket | Average amount spent per order. | Total revenue ÷ Total number of orders |
Revenue | Total amount spent by each group. | Sum of all total_prices of orders |
RPM | Revenue generated per 1,000 messages sent. For the control group, this is the revenue that would have occurred anyway. | Revenue ÷ Contacts × 1,000 |
Incremental conversions | Additional conversions generated by the Exposed Group compared to the Control Group. | (Conversion of the Exposed Group - Conversion of the Control Group) ÷ Conversion of the Control Group |
Incremental revenue | Incremental revenue estimates the campaign's impact on revenue by comparing the exposed group to the control group. It takes into account the impact on conversion rates plus the impact on average order value. | Exposed Group × (RPM of Exposed Group - RPM of Control Group) ÷ 1000 |
6. Interpret the results
Significant results
If the confidence intervals of the Exposed Group and the Control Group do not overlap, the campaign had a statistically significant impact.
Results not significant
If the confidence intervals overlap, the results are considered inconclusive.
When the control group performs better
If the control group performs better than the exposed group, this may indicate:
that the campaign did not have the expected effect;
that other external factors influenced the results.
7. FAQ
Q: Does the control group stop receiving all marketing communications?
A: No. The contacts in the Control Group are only excluded from the campaign in question. They may continue to receive other marketing campaigns.
Q: Can I manually select the contacts included in the Witness Group?
A: No. To ensure reliable and unbiased results, contacts are randomly selected by the system. You can only define the size of the control group as a percentage or as the number of contacts.
Q: Can I adjust the observation period to measure the results?
A: Yes. The observation period is configurable. By default, it is set to 7 days, but it can be extended up to 30 days, even after the campaign has been sent.
Q: Can I use control groups in automated campaigns?
A: No. Control groups are currently only available for one-off campaigns. Automated campaigns are not supported due to constraints related to statistical analysis.