Power indicates the probability with which a statistical test detects an actually existing effect.
It is used to assess whether the existing data base is sufficient to reliably detect relevant differences, variances, or changes in rates.
Unlike the classical hypothesis test, power does not focus on the decision between the alternative hypothesis and the null hypothesis, but rather on how powerful the test is under the given assumptions:
- high power → a relevant effect is detected with high probability
- low power → an existing effect may remain undetected despite investigation
Important: When calculating power, data summarized in AlphadiTab is entered. Therefore, raw data is not required, but metrics such as sample size, mean, standard deviation, or number of events.
In product development, a new tomato sauce formulation is being tested. The goal is to determine whether the viscosity of the new formulation differs on average from the previous formulation.
For this purpose, viscosity measurements are conducted on samples of the old formulation and the new formulation.
Using the t-test (2 samples), it was checked whether the observed difference in means is statistically significant or can be explained by random variation.
The calculated p-value is significantly above the significance level of 0.05, so the null hypothesis is retained. Based on this test, no significant difference between the average viscosity of the old and new formulation is found. With the calculation of the power, we now want to assess how reliably an actual difference can be detected.
For this, we need to determine which difference in mean viscosity we want to detect. The key is which difference is practically relevant. In this example, a difference in means of 0.5 MPa is to be detected. This difference corresponds to the effect. We can take the sample size and standard deviation from the conducted test. The standard deviation was rounded to 5.5; the less favorable of the two standard deviations is used here.
Interpretation of the results:
The power is 0.059. This corresponds to a β-error of 94.1%. Under these conditions, the test cannot reliably detect an actual difference of 0.5 MPa. From the power curve, it is evident that an effect of about 5 MPa can be detected with this test with approximately 80% power. The β-error would then be about 20%.
Explanations of the graphic:
- The curve shows the power for different effects (differences)
- The point represents the current input.
The following are the considerations and steps necessary to calculate the power for various statistical tests.
Preparation
- Select the appropriate test (e.g., t-test, F-test, test of proportions, or ANOVA).
- Determine which difference, variance ratio, or change in proportions is practically relevant.
- Set the significance level (usually α = 0.05).
- Determine the desired power or beta error.
- Determine the direction of the hypothesis (two-sided, greater, or less), if the method supports this choice.
Usage in AlphadiTab
- In the Analyze Phase, select the Power tool.
- Select the appropriate type of test, e.g., t-test, F-test, test of proportions, or ANOVA.
- Enter the summarized metrics into the input fields.
- Conduct the analysis by “Create New”.
Interpretation
- Check if the calculated power is sufficiently high.
- In practice, 80% or 90% power is often used as a guideline.
- If the power is low, check whether a larger sample size or a more focused study design is necessary.
Important: Power is not a retrospective confirmation of a p-value. It describes how likely a test is to detect an assumed effect under the chosen inputs.
Test types
- t-Test: Sensitivity for comparing means.
- F-Test: Sensitivity for comparing variances or dispersions.
- Test of proportions: Sensitivity for comparing ratios or binary events.
- ANOVA: Sensitivity for comparing means from two or more groups.
- Test on variances: Sensitivity for comparing variances or dispersions for two or more samples, optionally non-parametric.
In hypothesis tests with 2 samples, the direction for the alternative hypothesis can be selected.
Choose two-sided if a difference is to be detected without specifying a particular direction.
- This setting is useful when both higher and lower values would be relevant.
Example:
Is the average delivery time different between two processes?
Choose greater if an effect is only relevant in the direction of higher values.
- This setting is useful, for example, if it is to be tested whether a success rate increases.
Example:
Is the sales rate higher with a new sales approach than before?
Choose smaller if an effect is only relevant in the direction of lower values.
- This setting is useful, for example, if it is to be tested whether a processing time or defect rate decreases.
Example:
Is the average processing time after an optimization lower than before?
Filling quantity tomato sauce – Machine A vs. Machine B
In production, several filling machines are used. It should be assessed whether the available data is sufficient to reliably detect a difference between Machine A and Machine B.
t-Test:
Differences in means: Is the power sufficient to detect a difference of 5 ml in the average filling quantity?
Interpretation
For the t-test, the power shows the probability with which a difference in the average filling quantity is detected. The power is 1.000. Thus, the test can detect the assumed difference with very high probability under the chosen inputs. The β-error is correspondingly close to 0%.
F-Test:
Differences in variances: Is the power sufficient to detect a doubling of the variance of the filling quantity?
Interpretation
For the F-test, the power shows the probability with which a difference in variance is detected. The power is 0.91389. A doubling of the standard deviation can thus be detected with approximately 91.4% power. The β-error is approximately 8.6%.
Response time of the IT helpdesk vs. location ANOVA
In the IT service desk, tickets are processed at multiple locations. Response times are regularly evaluated to identify differences in service quality.
In the example of IT tickets, data from three locations is available. The sensitivity is used to assess whether a difference of 5 minutes in response time would be detected with the ANOVA.
Interpretation of the results
The test setup is suitable for detecting a difference of 5 minutes with three levels and a standard deviation of 4. The sensitivity is sufficiently high; the β-error is correspondingly low.
Test of Proportions - Loss Rate by Sales Approach
In sales, two sales approaches are compared. It should be assessed whether the existing offer numbers are sufficient to reliably detect a relevant change in the loss rate. The test of proportions showed that the sales methods do not differ significantly.
Now, using the power calculation, it should be determined whether a change from 35% to 30% would be detected.
Interpretation
The power indicates the probability with which the test detects a relevant change in the loss rate. The power is low at 0.11703. This corresponds to a β-error of approximately 88.3%. The fact that no significant difference could be detected with the test of proportions may be due to the fact that too few offers were evaluated.
Delivery time after logistics center
In the logistics department, customer orders are picked and shipped. To increase efficiency, new forklifts have been introduced.
It should be investigated whether an improvement in delivery time by 1 day can be detected with a one-tailed t-test. The variation in delivery time is estimated at 2.5 days. 30 measurements are to be taken.
Interpretation
The power is 81.775%. Thus, a difference of 1 day can be detected with approximately 81.8% power. The β error is approximately 18.2%. A power of at least 80% is often used as a guideline. Nevertheless, it should be determined in advance what minimum power should be achieved.
Supplier Comparison
In purchasing, components are sourced from multiple suppliers. It should be assessed whether the existing tests are sufficient to detect a 1.5-fold difference in the standard deviation between suppliers. There are 50 data sets per supplier.
Interpretation
The sensitivity helps to assess whether the existing number of deliveries is sufficient to reliably detect relevant supplier differences. With low sensitivity, the β-error would be high; an actually existing difference could then remain undetected.
Forecast deviation
In production planning, demand forecasts are created for different planning periods. To assess the quality of the forecast, the forecast deviation is calculated.
It should be assessed whether the existing sample is sufficient to detect differences between short-term and long-term planning horizons with a t-test. A difference of 2% should be detected. The standard deviation is known from previous analyses and is 0.9%. For each horizon, 10 forecasts should be created and compared.
Interpretation
With the t-test, a difference of 2% can be reliably detected if the standard deviation is 0.9%. The power is 99.68%. The β error is therefore approximately 0.32%.
Discriminatory Power / Power: Probability with which a test detects an actually existing effect.
Effect: Practically relevant difference, e.g., mean difference, variance ratio, or change in odds.
n = Sample size: Number of observations, parts, offers, or trials within a group.
α = Significance level: Preset probability of error with which the null hypothesis is incorrectly rejected.
β = Type II error: Probability of not detecting an actually existing effect.
Power = 1 - β: Counter-probability to Type II error.
x̄ = Sample mean: Average value of the collected measurement data.
s = Sample standard deviation: Measure of the dispersion of the data around the mean.
p̂ = Sample proportion: Proportion of events in a sample; calculated as events / number of trials.
Event: The case of interest in a proportion analysis, e.g., leaking, lost, on time, or passed.
Factor: Categorical influencing variable in ANOVA, e.g., logistics center, supplier, or formulation.
Groups / Levels: Manifestations of the factor, e.g., Center A, Center B, and Center C.
Two-sided: It is checked whether a difference can be detected in both directions.
Greater: It is checked whether an effect can be detected in the direction of greater values.
Smaller: It is checked whether an effect can be detected in the direction of smaller values.
The following formulas describe the calculation logic of the discrimination power used in the tool. The required distribution and quantile functions are not implemented manually but are evaluated using the statistical libraries CenterSpace NMath and – as specified – Numerics.NET.
Basic Principle
The discrimination power, also called power, describes the probability that a statistical test will detect an actually existing effect.
A high discrimination power means that an existing effect is detected with high probability. A low discrimination power means that the β error is correspondingly larger.
t-Test for Two Samples
In the t-test for two samples, the probability of detecting an assumed difference between two means is calculated. The tool assumes equally sized samples.
The non-centrality parameter is:
The degrees of freedom are:
For the calculation, quantiles of the central t-distribution and the distribution function of the non-central t-distribution are used. These functions are evaluated using CenterSpace NMath. Ft,v,λ(x) denotes the distribution function of the non-central t-distribution.
Power = 1 − Ft,v,λ(t1−α,v)
Power = Ft,v,λ(tα,v)
Power = Ft,v,λ(tα/2,v) + 1 − Ft,v,λ(t1−α/2,v)
F-Test for Two Variances
In the F-test, the discrimination power for an assumed difference between the variances of two groups is determined. The tool uses the ratio of standard deviations as input:
For the calculation, the ratio of variances is formed:
For equally sized samples, the following degrees of freedom apply:
The required quantiles and distribution functions of the F-distribution are evaluated using CenterSpace NMath. Fk1,k2(x) denotes the distribution function of the F-distribution.
Power = 1 − Fk1,k2 (F1−α,k1,k2 / ρ²)
Power = Fk1,k2 (Fα,k1,k2 / ρ²)
Power = Fk1,k2 (Fα/2,k1,k2 / ρ²) + 1 − Fk1,k2 (F1−α/2,k1,k2 / ρ²)
Levene's Test for Two Variances
For the power calculation of Levene's test, the tool uses an approximation based on the ratio of standard deviations.
The degrees of freedom of the used t-distribution are:
The critical t-value is determined using the quantile function of the t-distribution:
Additionally, the calculation uses the following auxiliary quantities:
c = √(π/2 − 1)
SE = c·√(ρ²/n₁ + 1/n₂)
From this, the two z-values are determined:
z₂ = −tkrit/θ − (ρ−1)/SE
The power is then determined using the standard normal distribution. The t- and normal distribution functions are evaluated using CenterSpace NMath.
Test of Proportions for Two Samples
In the test of proportions, the probability of detecting an actual difference between two proportions is calculated.
The difference in proportions is:
The common proportion for the calculation under the null hypothesis is:
The standard error under the null hypothesis is:
The standard error under the alternative hypothesis is:
The quantiles and the distribution function of the standard normal distribution are evaluated using CenterSpace NMath.
Power = Φ((d − z1−α·SE₀) / SE₁)
Power = 1 − Φ((d + z1−α·SE₀) / SE₁)
Power = 1 − Φ((d + z1−α/2·SE₀) / SE₁) + Φ((d − z1−α/2·SE₀) / SE₁)
One-Way ANOVA
The one-way ANOVA tests whether the means of several groups differ.
H₁: At least one group mean differs.
In the tool, the maximum difference between group means is used as the effect:
For the calculation of the non-centrality parameter, it is assumed that two groups form the maximum distance Δ. The non-centrality parameter used in the tool is:
The degrees of freedom are:
The critical F-quantile is determined using CenterSpace NMath:
For the evaluation of the non-central F-distribution, the tool Numerics.NET is used.
One-sample t-test
In the one-sample t-test, the power is calculated for a given effect, a sample size, and an assumed standard deviation.
The non-centrality parameter is:
The degrees of freedom are:
The required quantiles of the t-distribution and the distribution function of the non-central t-distribution are evaluated using CenterSpace NMath.
Power = 1 − Ft,v,λ (t1−α,v)
Power = Ft,v,λ (tα,v)
Power = Ft,v,λ (tα/2,v) + 1 − Ft,v,λ (t1−α/2,v)
One-sample variance test
In the one-sample variance test, the power is determined for an assumed ratio between the actual standard deviation and the hypothetical standard deviation.
The squared ratio is used for the calculation:
The degrees of freedom of the chi-square distribution are:
The required quantiles and distribution functions of the chi-square distribution are evaluated using CenterSpace NMath.
Power = 1 − Fχ²,v (χ²1−α,v / r²)
Power = Fχ²,v (χ²α,v / r²)
Power = Fχ²,v (χ²α/2,v / r²) + 1 − Fχ²,v (χ²1−α/2,v / r²)
One-sample proportion test
In the proportion test, the power is calculated for an actual comparison proportion p against a hypothetical proportion p₀.
The standard error under the null hypothesis is:
The standard error under the alternative hypothesis is:
The required quantiles and distribution functions of the standard normal distribution are evaluated using CenterSpace NMath.
Power = Φ((d − z1−α·SE₀) / SE₁)
Power = Φ((−d − z1−α·SE₀) / SE₁)
Power = Φ((d − z1−α/2·SE₀) / SE₁) + Φ((−d − z1−α/2·SE₀) / SE₁)