Statistical Test Cheat Sheet
This article serves as a guide and reference for those already familiar with the fundamentals of statistical testing. It is designed to function as a cheat sheet, offering a quick refresher on key concepts.
Identifying Terminologies
Hypothesis testing is testing if our theory or new idea is true or not.
If we do X it will result in Y.
If we increase the discount rate the conversion rate will increase.
The Null Hypothesis is the truth or the base that is assumed to be true before testing.
If we increase the discount rate the conversion rate won’t change.
The Alternative Hypothesis is the new idea we test against the old one (null hypothesis). If it is tested to be true we replace it with the new one, or if it is tested to be false we don’t replace it and we keep the old one.
If we increase the discount rate the conversion rate will change.
Which test to use?
First, we need to know the distribution shape of the dataset to decide if it’s parametric (coming from a normally distributed dataset) or non-parametric (coming from a non-normally distributed dataset).
Second, we need to decide if the datasets are paired (when comparing measurements or observations taken on the same subjects or items under different conditions or at different time points, for example, the same group before and after) or unpaired ( when comparing two distinct groups with no relationship between them)
Let’s say we’re measuring the mean between two data samples.
Null hypothesis H0
when the population mean is equal to the hypothesis mean.
H0:μ0=μ ➡️μ0-μ = 0
Alternative hypothesis Ha
is when there is a difference between the population mean and the hypothesis mean.
1. Two-tailed alternative hypothesis when the direction of the effect/difference is unknown.
Ha:μ0#μ ➡️ μ0>μ or μ0<μ
For example, applying a 50% discount will have an effect (could be higher or lower) on the conversion rate.
2. Upper tail alternative hypothesis when the direction of the effect/difference is to the right side of the distribution curve.
Ha:μ>μ0
For example, applying a 50% discount will result in a higher conversion rate.
3. Lower tail alternative hypothesis when the direction of the effect/difference is to the left side of the distribution curve.
Ha:μ<μ0
For example, applying a 50% discount will result in a lower conversion rate.
Decision Making
Now, to make a decision on whether to reject or accept the Null hypothesis, we compare the p-value obtained from the test:
Assuming alpha α = 0.05 (alpha is the probability that we will make the mistake of rejecting the null hypothesis when in fact it is true, think of it as the error allowed in our test usually between 0.01–0.05, in case of a two-tailed alternative hypothesis splits alpha into two 0.025, allocating half of α to the lower tail and half of α to the upper tail of the distribution).
p-value > α ➡️ Accept the Null hypothesis, we don’t have enough evidence to reject the null hypothesis.
p-value < α ➡️ Reject the Null hypothesis, results are statistically significant
