Compare probability forecasts with outcomes and a reference.
By the end: Separate calibration, sharpness, skill, and decision value.
A 25% rain forecast can be followed by rain without being logically wrong. Calibration asks whether events occur about one quarter of the time among many comparable cases forecast at 25%. A single event cannot answer that question.
Sharpness concerns how concentrated or decisive forecasts are before observing outcomes. It is useful when justified, but confidently predicting the wrong outcome is not better. Resolution concerns distinguishing groups with different observed event frequencies.
A verification case pairs an issued forecast with the outcome for exactly the target it predicted. For site rainfall, that means the same location, accumulation interval, threshold, and comparison convention. A forecast issued before the interval begins is different from a forecast updated halfway through it, even if both refer to tomorrow's total. Lead time is how far ahead the forecast was issued. Longer lead times generally involve a different prediction problem. Mixing them without labels can make a system appear to improve merely because the archive contains more short-lead forecasts than before.
An observation also needs an operational definition. A gauge reading represents rainfall at its collecting location, while a model may represent an average across a grid cell. A localized shower can create disagreement without either number being a careless measurement. Before converting totals into zero or one, decide which quantity defines the event. A missing observation is not evidence of a dry outcome. Coding it as zero would create false successes for dry forecasts and false failures for wet ones. The lab avoids these complications by stipulating eight complete binary outcomes rather than processing real instrument records.
| Forecast definition | Candidate observation | Comparable? |
|---|---|---|
| Site, 24-hour total | Same site and window | Yes, subject to measurement error |
| Site, 24-hour total | Same site, 1-hour peak | No, different quantity |
| Cell-average total | One point gauge | Needs representativeness treatment |
Reliability groups cases by what was forecast, rather than by what happened afterward. In a separate constructed archive, suppose a 0.3 probability is issued for ten comparable cases and the event occurs in three. The observed frequency is three divided by ten, or 0.3. That group agrees with its issued probability in this archive. It does not mean each forecast was individually correct in a yes or no sense. Seven cases did not have the event, as the group forecast allows. The comparison concerns frequencies across repeated predictions of a specified kind.
Now suppose ten cases forecast at 0.7 contain only five events. Their observed frequency is 0.5, below the issued 0.7. That group overforecasts event occurrence in this constructed sample. The first group can agree while the second disagrees, so an overall event count alone can conceal conditional errors. Reliability tables keep these groups separate and show their denominators. They should not silently treat three out of ten and three hundred out of a thousand as equally precise evidence, even though both frequencies are 0.3. Sample size affects what can reasonably be inferred beyond the archive.
Sharpness requires no outcomes to calculate. A system issuing only probabilities near one half is less decisive than one frequently issuing values near zero or one. Yet decisiveness cannot establish reliability. A forecaster who always assigns probability one before a dry outcome is maximally decisive and repeatedly wrong. Resolution asks a different question, whether forecast groups separate cases with different observed frequencies. The 0.3 and 0.7 groups just described have frequencies 0.3 and 0.5, so they separate outcomes somewhat despite the second group's probability error. Reliability and resolution can therefore differ within the same archive.
| Issued probability | Events / cases | Observed frequency |
|---|---|---|
| 0.3 | 3/10 | 0.3 |
| 0.7 | 5/10 | 0.5 |
| 0.3 | 300/1000 | 0.3 |
The eight-case lab makes these distinctions inspectable without claiming statistical evidence about a weather service. Its choices replace probabilities while holding the outcomes fixed. This is the comparison required to attribute score differences to the forecast probabilities rather than to an easier set of cases. The groupwise frequency table then conditions on each selected probability. Selecting a different system can change group membership even when the verifying events remain identical. There is no random resampling or arrival of new observations when a button changes the table, and no elapsed animation time represents an additional forecast lead time.
The Brier score averages squared differences between forecast probability p and the binary event indicator, one for occurrence and zero otherwise. Smaller is better. A perfect deterministic forecast scores zero; a certain wrong forecast contributes one.
The lab contains eight invented cases. The informative forecast uses 0.25 for four cases with one event, and 0.75 for four cases with three events. Its Brier score is 0.1875. Always forecasting 0.5 gives 0.25. Reversing the informative probabilities gives 0.4375, despite equal sharpness.
The binary indicator represents occurrence by one and nonoccurrence by zero. A probability can lie anywhere between them. Subtracting the indicator from the probability gives a signed error, but averaging signed errors can let wet and dry mistakes cancel. Squaring makes both directions contribute nonnegative amounts and penalizes larger distances more strongly. The Brier score averages those contributions across all included cases. It is dimensionless because both probability and indicator are dimensionless. Millimeters define the event before scoring; they do not become units of the probability error after the event has been coded.
Take a novel four-case example with probabilities 0.1, 0.4, 0.6, and 0.9, and outcomes zero, one, zero, and one. The signed differences are 0.1, minus 0.6, 0.6, and minus 0.1. Their sum is zero, despite two large realized errors. Squaring gives 0.01, 0.36, 0.36, and 0.01. The sum is 0.74 and the mean is 0.185. This example explains why an apparently unbiased average signed error does not imply accurate probability forecasts. The opposing errors remain visible in the squared score instead of canceling each other out.
Brier contribution for one case
Issued event probability, dimensionless
For one unknown binary outcome, a forecast probability p has two possible score contributions. If the event occurs, the contribution is the square of one minus p. Otherwise it is the square of p. Plotting both against p shows the tradeoff. Moving toward one lowers the event contribution while raising the non-event contribution. These are score curves, not weather trajectories and not empirical estimates of event frequency. Their shapes follow directly from squaring. No probability can minimize both possible contributions simultaneously, because the outcome cannot be known at the time of a genuinely prospective forecast.
To see which probability is justified before an outcome, suppose the event has a stipulated true chance q. Weight the event contribution by q and the non-event contribution by one minus q. Algebra gives expected score equal to q times one minus q, plus the square of p minus q. The second term cannot be negative and becomes zero when p equals q. Thus honest reporting of the stipulated chance minimizes expected Brier score. This is an expectation over repeated outcomes under an assumed chance, not a promise that the honest probability wins on every single observed case.
p is reported probability; q is stipulated event chance; s is one squared-error contribution. E[s] means mean score across the assumed outcome probabilities, dimensionless.
Expanding either side gives q − 2qp + p²; the final square is minimized at p = q.
For a worked value, set q to 0.4. The unavoidable expected contribution is 0.4 times 0.6, or 0.24. Reporting p equal to 0.4 adds zero. Reporting 0.1 adds the square of minus 0.3, or 0.09, giving expected score 0.33. Reporting 0.7 adds the same amount because its error has the same magnitude in the opposite direction. These calculations use a stipulated chance to explain the scoring rule. A real verification archive does not reveal the true chance for each individual case, so it estimates performance through observed outcomes rather than inserting known q values.
Reporting p = 0.4 gives 0.24. These are expected scores under an assumed chance, not observed archive scores.
Expected Brier contribution per case
Reported probability, dimensionless
This property does not make the Brier score a universal measure of forecast usefulness. It evaluates probabilities for one binary definition with quadratic penalties. A user concerned about severe damage may care differently about missed events and unnecessary precautions. The score also discards how far rainfall lies above the chosen threshold once occurrence is coded as one. A 13 mm total and a 70 mm total can receive identical binary labels for the same event. Their practical consequences can differ. Such differences require another target or decision analysis rather than an unsupported interpretation of the Brier score as total value.
p is a probability from zero to one; o is zero or one. Average these contributions over all cases for the Brier score.
Changing the system replaces probabilities for the same eight events. It does not select a more favorable set of outcomes. The reference score remains 0.25.
| Case | Forecast p | Event, 0 or 1 | Squared error |
|---|---|---|---|
| 1 | 0.25 | 0 | 0.0625 |
| 2 | 0.25 | 0 | 0.0625 |
| 3 | 0.25 | 0 | 0.0625 |
| 4 | 0.25 | 1 | 0.5625 |
| 5 | 0.75 | 0 | 0.5625 |
| 6 | 0.75 | 1 | 0.0625 |
| 7 | 0.75 | 1 | 0.0625 |
| 8 | 0.75 | 1 | 0.0625 |
Brier score: 0.1875. Skill against constant 50%: 0.25.
| Forecast group | Cases | Observed events | Observed frequency |
|---|---|---|---|
| 0.25 | 4 | 1 | 0.25 |
| 0.75 | 4 | 3 | 0.75 |
| Forecast | Brier score | Skill against 0.25 |
|---|---|---|
| Informative | 0.1875 | 0.25 |
| Constant 0.5 | 0.25 | 0 |
| Reversed | 0.4375 | −0.75 |
The lab's reference is a constant probability of one half on the same eight outcomes as every selected forecast. Because either binary outcome lies one half away, each reference contribution is 0.25. That makes its mean independent of outcome order in this particular example. Other constant references do not have that symmetry. Their score depends on the event frequency in the verification sample. The informative and reversed options use the same probability values with different assignment to cases. They are equally sharp, but their different relationships with outcomes produce different scores and reliability tables.
A separate twenty-case construction makes the score components explicit. Ten cases receive probability 0.3 and contain three events. Ten receive 0.7 and contain seven events. The overall frequency is ten divided by twenty, or one half. Both groups match their assigned probabilities in this sample. Within the low group, seven non-events contribute 0.09 each and three events contribute 0.49 each. The total is 2.1, giving mean 0.21. Symmetry gives the same mean in the high group, so the overall Brier score is also 0.21 for these deliberately arranged cases.
Ten cases forecast at 0.3 contain three events. S is a sum of dimensionless squared errors; the high group forecast at 0.7 has the same sum.
| Forecast group | Events / cases | Sum of squared errors |
|---|---|---|
| Probability 0.3 | 3/10 | 2.1 |
| Probability 0.7 | 7/10 | 2.1 |
| Combined | 10/20 | 4.2 |
The overall binary frequency supplies an uncertainty term. Multiply the sample frequency by one minus that frequency. Here one half times one half gives 0.25. For each group, compare its observed frequency with the overall frequency, square the difference, and weight by its share of cases. The two differences are minus 0.2 and plus 0.2; both squares are 0.04. Their weighted average is 0.04, the resolution term. Reliability error is zero because each group's forecast matches its frequency. For these exact probability groups, score equals uncertainty minus resolution plus reliability, or 0.25 minus 0.04.
U is uncertainty; D is resolution; L is reliability error. All score terms are dimensionless. Overall event frequency is 10/20 = 0.5.
L = 0 because each exact probability matches its group's observed frequency. The identity uses exact repeated probabilities, not broad bins.
Keep the same groups and outcomes but report 0.2 and 0.8 instead. Each assigned probability is now 0.1 away from its group's observed frequency. The reliability term becomes 0.01, while resolution stays 0.04 because the groups and their outcomes have not changed. The resulting score is 0.22. More extreme forecasts increased sharpness but worsened probability accuracy in this constructed archive. This calculation uses exact repeated probability values. With broad bins containing varied probabilities, a simple grouped decomposition can lose information and should not be presented as an exact reconstruction of the original unbinned score.
Only the reports change to 0.2 and 0.8; the same groups still have observed frequencies 0.3 and 0.7.
Skill measures relative improvement over a stated reference score. In the twenty-case example, a constant 0.5 reference scores 0.25. A score of 0.21 reduces that by 0.04, and dividing by 0.25 gives skill 0.16. It means a 16% reduction in mean squared probability error on these cases. It is not an event probability, a percentage of forecasts that were correct, or a guarantee for future samples. A reference score of zero would make this ratio undefined. The benchmark must therefore be named and checked rather than treated as an invisible normalization constant.
Skill is relative reduction in mean squared probability error, dimensionless; it is not an event probability.
| Term | Report 0.3 and 0.7 | Report 0.2 and 0.8 |
|---|---|---|
| Uncertainty | 0.25 | 0.25 |
| Resolution | 0.04 | 0.04 |
| Reliability error | 0 | 0.01 |
| Brier score | 0.21 | 0.22 |
Use the lab options to compare the same two aspects separately. The contribution table shows which cases create score differences. The grouped table shows whether different issued probabilities correspond to different observed frequencies. A reversed assignment can preserve separation between groups while giving the probabilities the wrong meaning. A single scalar score summarizes their combined effect but does not replace the conditional comparison. The new twenty-case table is arithmetic outside the controls, not an additional selectable dataset. The controls retain the original eight outcomes and three authored forecast systems throughout every comparison on this page.
Skill is improvement relative to a reference such as climatology or persistence. Here Brier skill relative to 0.25 is one minus the score divided by 0.25. The informative set has skill 0.25. That is a 25% reduction in this sample score, not a 25% probability of being correct.
Evaluate forecasts on cases not used to tune them. Match location, season, lead time, event threshold, and observation definition. Report sample uncertainty and rare-event limitations. Eight designed cases illustrate arithmetic; they cannot certify a real forecasting system.
A forecast system can earn a small raw Brier score without providing much information about rare events. Suppose a constructed archive has two events in one hundred cases. Always forecasting zero scores two contributions of one and ninety-eight contributions of zero, for mean 0.02. Always reporting the archive frequency of 0.02 gives mean 0.0196. The second score is lower even though both numbers look small. Most cases are easy for a dry forecast. The event frequency and benchmark reveal why a low number alone does not establish useful ability to distinguish risky cases from ordinary ones.
Using the observed sample frequency as a reference is useful for explaining arithmetic, but prospective comparisons need a reference available without seeing future outcomes. A seasonal climatology estimated from an earlier training period is one possibility. Persistence, which uses a recent state, addresses a different benchmark question. Each forecast and its reference should be evaluated on the same held-out cases. If one is scored only during dry months and the other during stormy months, a difference can reflect the sample rather than forecast quality. Fair verification holds the target and case selection fixed before judging competing probabilities.
| Constant probability | Cases with event | Mean Brier score |
|---|---|---|
| 0 | 2/100 | 0.02 |
| 0.02 | 2/100 | 0.0196 |
| 0.5 | 2/100 | 0.25 |
Postprocessing and model selection can accidentally use verification outcomes as training information. Suppose several probability adjustments are tried and only the lowest scoring adjustment on one archive is reported. That archive has helped select the adjustment, so its score no longer provides a clean test of performance on unseen cases. Reserve a later period or another appropriate untouched set for evaluation. The split must respect the prediction task. Randomly mixing nearly identical forecasts from the same storm between training and evaluation can leave substantial dependence across the split and make apparent replication less informative than the case count suggests.
Sampling uncertainty is especially visible in reliability groups with few cases. One event among four gives frequency 0.25. One additional event would raise it to 0.5, while one fewer gives zero. One hundred events among four hundred also gives 0.25, but changing one outcome alters the frequency by only 0.0025. This comparison does not calculate confidence intervals or assume independent weather cases. It shows why denominators belong beside frequencies. Repeated forecasts from a single persistent storm can be dependent, so even four hundred recorded cases need not carry the information of four hundred independent events.
Pooling can hide a systematic error that changes with context. Imagine two equal groups of ten hypothetical cases, all forecast at 0.5. One group has two events and the other has eight. Pooling gives ten events in twenty, apparently matching 0.5. Within groups the observed frequencies are 0.2 and 0.8, so the system misses a potentially useful distinction if group identity was known at issue time. These groups could represent different stipulated conditions, not measured weather categories in this lesson. Stratifying by relevant known conditions can expose weaknesses, but excessive subdivision leaves too few cases for stable conclusions.
| Known group | Events / cases | Issued probability |
|---|---|---|
| Group A | 2/10 | 0.5 |
| Group B | 8/10 | 0.5 |
| Pooled | 10/20 | 0.5 |
A defensible report therefore includes the event definition, issue lead time, verification cases, observation treatment, score, and reference, with limits on what the sample supports. Reliability and resolution explain different reasons for a score, while separate decisions determine practical value. No single diagnostic certifies every aspect of forecasting. The lab supplies complete invented outcomes and fixed probabilities so that arithmetic can be checked without those empirical ambiguities. Applying its reasoning to a real service would require an appropriate forecast archive and observation record. Neither the displayed score nor the added examples provide such a validation or current hazard guidance.
A forecast gives p = 0.2 and rain occurs. What is its Brier contribution? What if rain does not occur?
Square the difference from one or zero.
| Outcome | Difference | Squared difference |
|---|---|---|
| Rain, 1 | −0.8 | 0.64 |
| No rain, 0 | 0.2 | 0.04 |
With rain, the contribution is (0.2 − 1)² = 0.64. Without rain it is (0.2 − 0)² = 0.04. Both cases must be included in a representative evaluation; selecting only successes distorts the score.
A system scores 0.08 while a suitable reference scores 0.05. Is its skill positive?
Compute one minus their ratio.
| Step | Value |
|---|---|
| Score ratio | 0.08/0.05 = 1.6 |
| Skill | 1 − 1.6 = −0.6 |
| Interpretation | 60% higher mean squared probability error |
No. The skill score is 1 − 0.08/0.05 = −0.6. A small raw score can still be worse than a strong reference, especially when the event is rare. The benchmark and event base rate are essential.
Selected reference passages support the definitions and modeling distinctions below. The examples, explanations, and implementations are original. Terminology and assumptions are stated within each lesson; source review is not an independent audit of this course. Optional links are not runtime dependencies.
Selected sections on reliability, resolution, Brier scores, skill, and forecast value. The eight-case sample is deliberately too small to establish real calibration.