Risk literacy boosts

Being well informed about risks is instrumental to making good decisions. Yet sometimes risks are communicated in ways that can be highly misleading.

In 1995, the UK Committee on Safety of Medicines issued a warning that the third-generation oral contraceptive increased the risk of blood clots by 100%. Many women panicked and stopped taking it, leading to a marked increase of unwanted pregnancies and abortions (an estimated 12,400 additional births and 13,600 additional abortions in England and Wales in 1996 compared to 1995; Furedi, 1999).

The study the warning was based on showed that out of every 7,000 women who took the second-generation pill, 1 had a thrombosis, and that this number increased to 2 in 7,000 for women who took the third-generation pill. Had the absolute increase in risk (1 in 7,000 to 2 in 7,000) been reported, or had more women known the importance of finding out the missing base rate information (100% more than what?), it is likely that fewer women would have stopped taking the pill.

Risk literacy boosts promote informed decision making in many domains, including health, weather, and finances. They foster the competence to understand and evaluate statistical information about risks. Risk literacy can be boosted through transparent and balanced risk communication, or through brief sessions that teach people how to bring clarity to opaque statistical representations.

Fact boxes: Concisely summarizing the best available evidence

What is the boost?

A fact box is a tabular summary of the best available evidence about the benefits and harms of a medical procedure, medical treatment, or health behavior (see McDowell et al., 2016, for a guide for creating a fact box).

This is a fact box for PSA screening as a tool for early detection of prostate cancer. Information about the benefits and harms of screening are presented side by side, making it easy to compare outcomes between men who were screened and men who were not. Information about risks is summarized in absolute numbers so that people can weigh the benefits against the harms. Other features include a short summary; the reference class, age range, and time frame of the study; and any caveats that may influence how people interpret the evidence.

What challenges does the boost tackle?

Representations of statistical information that are incomplete (e.g., excluding benefits, harms, or the base rate), unbalanced (e.g., emphasizing either benefits or harms), or opaque (e.g., presenting risks in relative numbers) can lead to poor understanding of risks and thus to poor decisions.

What competences does the boost foster?

Risk literacy: the competence to understand and assess statistical information about risks.

What is the evidence behind the boost?

Fact boxes are understood better, and lead to more long-lasting knowledge retention, than the same content provided in text (Brick et al., 2020; McDowell et al., 2019).

Key reference

Brick, C., McDowell, M., & Freeman, A. L. J. (2020). Risk communication in tables versus text: A registered report randomized trial on “fact boxes.” Royal Society Open Science, 7(3), Article 190876. https://doi.org/10.1098/rsos.190876

Natural frequencies: Faciliting Bayesian inferences

What is the boost?

Unlike conditional probabilities, natural frequencies are not normalized with respect to the base rates. They therefore make it easier for people to apply Bayes’s rule to determine, for instance, how likely it is that a person who tests positive for breast cancer does, in fact, have the disease (i.e., the posterior probability of the disease given a positive test result; Gigerenzer & Hoffrage, 1995).

The two tree diagrams above summarize the use of mammography to detect breast cancer. The four probabilities on the left are conditional probabilities; the four frequencies on the right are natural frequencies.

Note that natural frequencies are not the same as the normalized frequencies displayed in fact boxes. Natural frequencies carry information about the prevalence of an event (e.g., a disease), while normalized frequencies do not.

What challenges does the boost tackle?

Conditional probabilities require people to apply a complex Bayesian algorithm to determine a test’s positive predictive value (i.e., Bayes' theorem). This is because the information about the prevalence of, for instance, a disease, has to be reintroduced by multiplying it by each conditional probability. There is abundant evidence showing that people have difficulty carrying out these computations (see literature review in McDowell & Jacobs, 2017).

What competences does the boost foster?

Natural frequencies simplify computations and specify the pieces of numerical information and how they relate to each other. They thus help people make Bayesian inferences, which are more difficult to make based on conditional probabilities. The improvement in performance is a short-term, “local fix,” that is, there is no aspiration to improve Bayesian reasoning beyond the current context.

What is the evidence behind the boost?

A meta-analysis showed that, on average, 24% of participants correctly solved Bayesian inference problems in the natural frequency format, whereas only 4% did so in the conditional probability format. These proportions imply an odds ratio of 7.1, which can be considered a strong effect (McDowell & Jacobs, 2017).

Key references

• Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. http://dx.doi.org/10.1037/0033-295X.102.4.684

• Hoffrage, U., Lindsey, S., Hertwig, R., & Gigerenzer, G. (2000). Communicating statistical information. Science, 290, 2261–2262. https://doi.org/10.1126/science.290.5500.2261

• McDowell, M., & Jacobs, P. (2017). Meta-analysis of the effect of natural frequencies on Bayesian reasoning. Psychological Bulletin, 143(12), 1273–1312. https://doi.org/10.1037/bul0000126

Teaching how to construct natural frequency trees out of probability information

What is the boost?

Knowing how to construct natural frequency trees out of probability information helps people successfully make Bayesian inferences (e.g., to understand what a positive test result really means).

The boost above involves four steps:

1. Read the problem description represented in probabilities.
2. Draw a natural frequency tree. The root node (at the top) represents the total number of cases and is then broken down into four subclasses.
3. Fill in the numbers. For convenience, set the root node to 100 patients. Insert the base-rate frequency in the “sepsis” node by calculating 10% of 100 patients (= 10). Fill in the “no sepsis” node by subtracting the patients with sepsis from the total number of patients (100 - 10 = 90). Divide the 10 patients in the sepsis node into 8 showing the symptoms (80%) and 2 not showing the symptoms (the remaining 20%). Divide the 90 patients in the “no sepsis” node into 9 showing the symptoms (90%) and 81 not showing the symptoms (the remaining 10%).
4. Calculate the posterior probability. With all numbers filled in, the probability of sepsis given the presence of symptoms can be easily calculated: $8/(8 + 9) = .47$ or 47%.

What challenges does the boost tackle?

Risks that are communicated using opaque statistical representations based on conditional probabilities can be difficult to grasp (see the natural frequencies section).

What competences does the boost foster?

The ability to translate probability information into a natural frequency tree, then to apply Bayes’s rule to calculate a posterior probability.

What is the evidence behind the boost?

Sedlmeier and Gigerenzer (2001) offered training in natural frequency trees. They compared it with training in conditional probability trees and rule-based training in Bayes’s rule. Although the immediate effect was strong for all training programs, only the training in natural frequency trees showed no decay over time: The immediate training effect of a median of 93% Bayesian solutions, compared to a median of 14% at baseline, remained stable over a period of 15 weeks.

Key reference

Sedlmeier, P., & Gigerenzer, G. (2001). Teaching Bayesian reasoning in less than two hours. Journal of Experimental Psychology: General, 130(3), 380–400. https://doi.org/10.1037/0096-3445.130.3.380

Boosting people’s ability to compare two risks using simple, step-by-step online tutorials

What is the boost?

A brief step-by-step online tutorial can teach people how to accurately calculate and compare the event rates of two risks (e.g., case-fatality rates for the flu versus COVID-19) by illustrating how to divide the number of key events (e.g., number of deaths among people infected with disease X) by the total number of events (e.g., total number of people infected with disease X).

What challenges does the boost tackle?

If people focus on the key event (e.g., number of deaths among people infected with disease X), then a common, but less dangerous risk (e.g., a common, but not very lethal disease) can seem more risky than a less common, but more dangereous risk (e.g., a less common, but more lethal disease). The training decreased the likelihood that people mistakenly focused just on the number of key events (e.g., number of deaths among people infected with disease X) without dividing by the total number of events (e.g., total number of people infected with disease X) and can thus improve people’s ability to correctly identify the more dangereous risk (e.g. which of two diseases has a higher case-fatality risk).

What competences does the boost foster?

Risk literacy: the competence to understand statistical information about risks. Ameliorating whole number bias errors can not only help people think more accurately about, say, COVID-19 statistics expressed as rational numbers, but also about novel future health crises, or any other context that involves information expressed as rational numbers.

What is the evidence behind the boost?

1297 participants were randomly assigned to an intervention or control condition. Thompson et al. (2021) showed that participants in the intervention condition, relative to those in the control condition, were more accurate and less likely to explicitly mention whole-number-bias errors in their strategy reports as they solved COVID-19-related math problems. At the time of study (mid-March 2020), the statistics for the US were 22,000 flu deaths (relative to 36,000,000 infections = a case-fatality ratio of 22,000/36,000,000 = 0.00061 = .06%) and, at that point, only 9,318 COVID-19 deaths (relative to 227,743 infections = a case-fatality ratio of 9,318/227,743 = 0.041 = 4.1%).

Key reference

Thompson, C. A., Taber, J. M., Sidney, P. G., Fitzsimmons, C. J., Mielicki, M. K., Matthews, P. G., Schemmel, E. A., Simonovic, N., Foust, J. L., Aurora, P., Disabato, D. J., Seah, T. H. S., Schiller, L. K., & Coifman, K. G. (2021). Math matters: A novel, brief educational intervention decreases whole number bias when reasoning about COVID-19. Journal of Experimental Psychology: Applied, 27(4), 632–656. https://doi.org/10.1037/xap0000403