Applied Science

A binomial A/B experiment is simulated via Monte Carlo on n observations per group.

Data generation. For each unit an independent Bernoulli draw is created:
Control ∼ Bernoulli(\(p_C + \text{bias}\))
Treatment ∼ Bernoulli(\(p_C + \Delta\))

Instant estimators. After each batch the rates \(\hat{p}_C, \hat{p}_T\), the difference \(\widehat{\Delta} = \hat{p}_T - \hat{p}_C\), and its standard error \(\text{SE}= \sqrt{\tfrac{\hat{p}_C(1-\hat{p}_C)}{n} + \tfrac{\hat{p}_T(1-\hat{p}_T)}{n}}\) are updated.

Confidence intervals. The trajectory of \(\widehat{\Delta}\) is plotted together with its CI at level \(1-\alpha\):
\(\widehat{\Delta} \pm z_{1-\alpha/2}\,\text{SE}\), with \(z\) corresponding to 90 %, 95 % or 99 % coverage.

Final test. A two-sided z-test for the difference of proportions with pooled variance is applied:
\(z=\dfrac{\hat{p}_T-\hat{p}_C}{\sqrt{\hat{p}(1-\hat{p})\,2/n}}\), where \(\hat{p}=(\hat{p}_C+\hat{p}_T)/2\). The p-value, significance (α = 0.05), final rates and successes are reported.

Visualisations

Bar chart: final rates for Control vs Treatment.

Time chart: evolution of \(\widehat{\Delta}\) and its CI.

These views allow you to explore power, type I error, biases, and the speed at which evidence accumulates in a classic A/B test.

Control

↔