A test asks how the data compare with a specified null model. It does not directly assign a probability to that model.
By the end: Compute a discrete tail p-value, distinguish false positives from missed effects, and interpret power conditional on an alternative.
Specify the direction before observing the result
Consider twenty independent trials. The null model has p = 0.5. To look for a success probability above 0.5, use the success count as a statistic and treat larger counts as more extreme. If fifteen successes occur, the one-sided p-value is the null probability of fifteen or more, about 0.0207.
A p-value is the probability, under the specified null model and analysis plan, of a statistic at least as extreme as observed. It is not the probability the null is true, not the probability the result was caused by chance, and not the size of the effect. A two-sided question needs an appropriate two-sided rule; it cannot be silently substituted after seeing a convenient direction.
Build the tail from possible datasets
Suppose a packaging machine should place a marked label on half its packets. An audit samples twenty packets independently under stable conditions. The null model says each packet has marking probability 0.5. The proposed alternative says that probability is higher. The statistic is the number marked, so a higher count is stronger evidence in the planned direction. This setup specifies both a data-generating model and an ordering of results before any count is observed.
Under the null, every particular twenty-packet sequence has probability one divided by 1,048,576. That denominator comes from multiplying two choices twenty times. However, a count such as fifteen represents many sequences. The five unmarked positions can occupy 15,504 different sets of positions. Thus exactly fifteen marked packets has probability 15,504 divided by 1,048,576, about 1.479%. Counts are groups of equally weighted sequences, not twenty-one equally weighted outcomes. The combinatorial multiplicity belongs in the calculation.
The planned tail includes fifteen and every higher count. The sequence counts for fifteen through twenty are 15,504, 4,845, 1,140, 190, 20, and 1. Their sum is 21,700. Dividing that sum by 1,048,576 gives about 2.0695%. This is the one-sided p-value for an observed count of fifteen. Counting only the observed count would omit more extreme results that the same rule would treat as at least as strong evidence against the null.
Add sequences for the entire upper tail
Marked count
Sequences
Null probability
15
15,504
1.47858%
16
4,845
0.46206%
17
1,140
0.10872%
18
190
0.01812%
19
20
0.00191%
20
1
0.000095%
Tail total
21,700
2.06947%
Twenty independent fair trials give 1,048,576 equally likely sequences. Each row groups sequences by count. Rounded percentages need not sum exactly to the printed total.
A particular sequence can be unlikely without being evidence of an unusual marking rate. For example, any specified alternating sequence has the same null probability as any specified sequence containing fifteen marks. If the question concerns the marking rate, we group sequences by their counts. If instead the question concerns serial patterns, a different statistic might be appropriate. Its null distribution would need a fresh calculation. Small probability for a fully specified dataset is not by itself a useful discrepancy measure.
Direction changes the event being measured. A prespecified two-sided rule for this symmetric fair model can treat counts at least five away from ten as extreme. After fifteen marks, that rule includes both zero through five and fifteen through twenty. Symmetry makes the two tail weights equal, giving about 4.1389%. Doubling works here because the chosen statistic and null are symmetric. It is not a general definition for every discrete test or every two-sided scientific question.
Compare the two questions before interpreting their answers. The one-sided audit asks whether marking is excessive; the two-sided audit asks whether marking departs from half in either direction. A shortage matters to the second question but not to the first rejection rule. Choosing whichever direction looks favorable after inspecting the count changes the procedure. The arithmetic can still be correct for each displayed tail while the claimed planned error rate no longer describes the actual selection process.
The direction determines which counts enter the tail
Circles mark integer count locations, not equal probability masses. For the symmetric null p = 0.5, the upper selected tail has mass 2.06947%; both selected tails together have mass 4.13895%.
Compare two kinds of error
A rule that rejects the null at fifteen or more successes has a false-rejection probability of about 2.07% at p = 0.5. This is its actual size in the illustrated point-null model. Lowering the cutoff increases false rejections but can detect more real increases.
Power is the rejection probability under a specified alternative. If p = 0.75, the same rule has power about 61.72%. Thus it misses the increase roughly 38.28% of the time. Power depends on the effect, sample size, model, and rejection rule; a test does not have one universal power for all alternatives.
Apply one rejection region to different generating rates
Fix the rule to reject when at least fifteen of twenty packets are marked. Under a generating probability of 0.5, its rejection chance is 2.0695%. This chance is the test's actual size at the stated point null. A nominal 5% test need not reject exactly 5% of the time. Counts are discrete, so changing an integer cutoff can jump over the desired error budget. Randomizing at a boundary could fill that gap, but this lesson's rule uses a fixed integer cutoff.
To see the jump, lower the cutoff to fourteen. Exactly fourteen has null probability 38,760 divided by 1,048,576, about 3.6964%. Adding it to the tail from fifteen gives 5.7659%. Thus the nonrandomized cutoff fifteen stays below 5%, whereas fourteen exceeds it. Calling either rule a 5% procedure without checking its actual rejection event would hide the difference. The cutoff is part of the test, not a harmless display choice made after the data arrive.
Now keep the cutoff at fifteen but change the generating probability to 0.75. A particular sequence with fifteen marks has probability 0.75 multiplied fifteen times, then 0.25 multiplied five times. Multiply by 15,504 to include all such sequences, and add the probabilities of counts sixteen through twenty. The result is about 61.7173%. That is power against this particular alternative. The complementary 38.2827% is the chance of failing to reject despite a true marking rate of 0.75.
Keep the selected counts and change their weights
Both charts use the same count and probability scales. Height is the mass of one count, so selected heights sum to size or power. The dashed boundary separates counts fourteen and fifteen. All trials are independent within each generating model.
Size and power therefore share a rejection event but use different probability weights. They do not divide reported findings into true and false claims. Imagine 1,000 hypothetical audits all generated at 0.5. About 20.7 rejections are expected. Among another 1,000 generated at 0.75, about 617.2 are expected. Actual repetitions fluctuate. Neither number states how many of a mixed collection of rejected audits came from the null, because the mixture proportions have not yet been specified.
For that reverse question, stipulate that 90% of audits come from rate 0.5 and 10% from rate 0.75, with no other possibilities. Among 10,000 hypothetical audits, the expected null rejections are 9,000 times 0.020695, about 186.3. Alternative rejections are 1,000 times 0.617173, about 617.2. The null fraction among rejections is then about 186.3 divided by 803.4, or 23.2%. This conditional answer requires the extra mixture assumption; it does not equal the test's size.
The rejection event has likelihoods 0.020695 under the null and 0.617173 under the alternative. Their ratio favors the alternative by about 29.8 to one. A posterior also needs prior odds, such as the stipulated one to nine here. Moreover, this event discards the exact observed count. Updating after exactly fifteen marks would use the likelihoods of exactly fifteen, not the probabilities of the whole rejection region. Evidence compression and reverse conditioning are separate operations.
A reverse conditional needs the mixture of audits
Generating rate
Expected audits
Rejection chance
Expected rejections
0.5, null
9,000
2.06947%
186.2526
0.75, alternative
1,000
61.71727%
617.1727
All
10,000
Mixture-dependent
803.4252
Null among rejections
Prior null weight 90%
186.2526 / 803.4252
23.1823%
Expected frequencies are not guaranteed counts. These two generating rates are stipulated to exhaust the possibilities. The rejection event has different likelihoods under the two states; posterior conditioning also uses their prior weights.
Work through the mechanism
Use one rejection rule on two distributions
False rejection at p = 0.5: 2.07%. Power at p = 0.75: 61.72%.
Gold bars meet the count ≥ 15 rejection rule; blue bars do not. Dashed lines mark the start of the rejection region. Both axes and sample sizes match. Independence and fixed p hold within each model. Moving the cutoff replaces the planned rule for comparison; selecting it after seeing real data would need its own error analysis. Power is specific to the displayed alternative.
Planned rule: count ≥ 15 of 20
Rejection chance
Null p = 0.5
≈ 2.07%
Alternative p = 0.75
≈ 61.72%
The shaded tails use the same cutoff but different generating probabilities.
Compare planned cutoffs without selecting one from the result
The experiment displays two complete count distributions simultaneously. The left assumes marking probability 0.5, and the right assumes 0.75. Both use twenty independent trials and the same horizontal count scale. Each rectangle's height is the probability mass of one integer count. These are not densities over intervals of marking probability. Adding the heights for selected counts gives the rejection chance. The gold region is the decision rule applied to the statistic, not a region of posterior probability over hypotheses.
Start at the baseline cutoff fifteen and identify the same selected counts in both panels. Predict the effect of choosing the strict cutoff eighteen before using that preset. Removing counts fifteen, sixteen, and seventeen must decrease both displayed rejection chances, because every removed count has positive probability under either model. This creates fewer false rejections at 0.5 but also fewer detections at 0.75. No change to measurement quality has occurred. Only the definition of a rejecting result has changed.
The strict cutoff has null rejection probability 211 divided by 1,048,576, about 0.0201%. The numerator is 190 plus 20 plus 1, the sequence counts for eighteen through twenty. Its power at 0.75 is about 9.1260%. Thus it misses about 90.8740% of these alternatives. A rule can achieve an impressively small false-rejection probability simply by rarely rejecting anything. Assessing whether that rule is useful requires looking at the alternative panel as well as the null panel.
Next predict what happens with the liberal cutoff ten. Under the fair model, symmetry gives equal probability above and below ten, while the central count has probability 184,756 divided by 1,048,576. Half the remaining probability plus that central mass gives a rejection chance of about 58.8099%. Under 0.75, power is about 99.6058%. The high detection chance comes with frequent rejection when the null is true. It does not mean that the method has become nearly infallible.
Use the slider to locate the smallest displayed cutoff with null rejection below 5%, then compare its power with the neighboring cutoff. This exploration reconstructs the discrete boundary between fourteen and fifteen. The lesson lets you compare hypothetical plans; it does not observe packets and optimize a cutoff against their realized count. In a real audit, choosing a threshold after seeing whether the observed count crosses it would make the count influence both the evidence and the rule used to judge that evidence.
Only the cutoff changes in this experiment. The number of trials and the two generating probabilities remain fixed. You therefore cannot infer how power changes with sample size by treating a slider movement as extra observations. Nor does the display include a distribution over unknown generating probabilities. It shows repeated-dataset probabilities under two specified settings. To answer a new question about another effect size, another sampling design, or posterior uncertainty, construct the corresponding model rather than relabeling these two panels.
Moving a planned boundary changes both errors
Reject at count
Size at p = 0.5
Power at p = 0.75
Miss chance at p = 0.75
At least 10
58.8099%
99.6058%
0.3942%
At least 14
5.7659%
78.5782%
21.4218%
At least 15
2.0695%
61.7173%
38.2827%
At least 18
0.0201%
9.1260%
90.8740%
All rows use twenty independent trials. These are simultaneous comparisons of possible plans, not a rule for selecting a cutoff after observing a real count. A miss is failure to reject at the displayed alternative, not a posterior error probability.
A threshold does not replace an estimate
Failing to reject does not establish equality or prove that any effect is negligible. A small dataset may be compatible with both meaningful effects and no effect. Conversely, a large dataset can make a very small effect statistically detectable. Report estimated magnitude, uncertainty, and practical consequences.
A test of association also does not establish a causal mechanism. Randomization, measurement, selection, and model assumptions remain relevant. The ASA statement warns against reducing scientific or policy conclusions to whether a p-value crosses a fixed threshold.
Separate model discrepancy from effects and actions
An observed marking fraction of fifteen out of twenty is 75%, an estimated increase of 25 percentage points above half. The p-value answers a different question about how unusual the count is under the fair model. A larger sample can make a much smaller departure detectable because its sampling distribution is narrower. A smaller p-value therefore does not order effects by practical importance. The estimated rate and its uncertainty describe the possible magnitude; the test describes compatibility with the specified null procedure.
Interval reasoning can make the distinction concrete without inventing a posterior. For a planned one-sided test, consider each candidate rate and ask whether its upper-tail probability for the observed count is below the chosen level. Rates rejected in that direction fall below a corresponding confidence bound. This inversion connects tests and confidence procedures under matching conventions. It still describes repeated-sampling behavior, not a probability that the fixed unknown rate lies above the bound. A credible bound requires a prior and a likelihood instead.
For an operational decision, define a tolerable departure separately. Suppose the factory regards rates between 45% and 55% as practically acceptable. A result compatible with both 50% and 65% does not show acceptability, even if a test of exactly 50% fails to reject. It leaves an unacceptable rate possible under the analysis. Conversely, a sufficiently precise estimate near 52% could distinguish it from exactly 50% while remaining within the acceptable band. Equality and practical adequacy are different claims.
An exact null and a practical acceptance band differ
Horizontal position is marking rate on a linear 0–100% scale. Shading denotes the stipulated acceptable band, not probability density or confidence. The dots are example true rates, not measured estimates or intervals.
The null also includes collection assumptions. Twenty measurements on packets from one machine jam may share a common cause. A binomial calculation that treats them as independent can then understate how often extreme counts occur. Repeating a reading on the same stored image does not create a new independent packet outcome. Conditional independence would require the relevant shared conditions to remove the remaining dependence, not merely a second collection time. Diagnose the physical and recording process before treating the nominal sample size as twenty independent trials.
Sampling and assignment address different gaps in the evidence. Randomly sampling packets helps estimate the marking rate in the population from which they were drawn. Randomly assigning comparable production runs to two machine settings can help attribute a difference to the setting. A low p-value in a before-and-after comparison could instead reflect changed materials, operator practices, or temperature. The probability calculation does not choose among those causal explanations. Its null describes observed variation under assumptions, not a complete intervention mechanism.
Finally, rejecting the null does not specify whether to halt production. That action needs consequences such as missed defects, shutdown costs, and the effectiveness of repair. Those belong in a loss model, not inside the definition of a p-value. A well-calibrated forecast of future faults would also need checks across comparable predictions, while separating risky from safe runs is discrimination. Neither property follows from one significant test. Keep the observed discrepancy, estimated effect, causal explanation, and action comparison as distinct conclusions supported by distinct evidence.
Different claims require different evidence
Claim
Relevant evidence
What a small p-value does not supply
Rate differs from 0.5
Planned statistic and valid null distribution
Effect magnitude
Departure is operationally important
Estimated rate and uncertainty against practical band
A practical tolerance
Changing a setting changes marking
Causal design, such as randomized assignment
Removal of confounding
Halting production is worthwhile
Consequences, repair effectiveness, and losses
An action threshold
Forecasts identify risky runs
Discrimination plus calibration evaluation
Individual prediction quality
Sampling addresses which population is represented; assignment addresses causal comparisons. These are methodological comparisons, not numerical performance claims about an actual factory.
Exercises
A tail is not a single outcome
For four independent fair trials, you predefine unusually high counts as evidence against fairness. If all four succeed, what is the one-sided p-value? What if three succeed?
Hint
Add all outcomes at least as high as the observation.
Worked answer
Observed count
Null tail
p-value
4
1/16
0.0625
3
4/16 + 1/16
0.3125
The extremeness rule was specified before the data.
Four successes gives 1/16 = 0.0625. Three successes gives P(3) + P(4) = 4/16 + 1/16 = 0.3125. The latter is not just the probability of exactly three, and neither number is the posterior probability of fairness.
A nonsignificant result
A study’s interval includes no effect and a large useful benefit. Is “the treatment does not work” established?
Hint
Ask whether important effects were excluded.
Worked answer
Compatible effect
Included?
No effect
Yes
Useful benefit
Yes
Conclusion
Useful benefit has not been excluded
Statistical detectability and practical importance are different questions.
No. The result is inconclusive about a useful benefit under the model. To argue that effects are too small to matter, define a practical equivalence range and use a design and analysis that can exclude effects outside it. A lack of rejection alone does not do this.
Sources and further reading
The explanations and exercises are complete without these optional links. The references distinguish probability identities from inference procedures and reporting guidance. The examples are constructed teaching models, not empirical estimates or professional decision advice.
The ASA’s six principles on interpretation, effect size, transparency, and avoiding threshold-only conclusions. This linked PDF is the announcement and summary, not the full technical article.