Simpsons Paradox and the Ecological Fallacy [Data Science]

Simpsons Paradox

I’m currently studying for a Master’s Degree in Global Health at The University of Manchester, and I’m absolutely loving it. Right now, we’re studying epidemiology and study design, which also involves a great deal of statistical analysis.

Some data was presented to us from an ecological study (a type of scientific study, that looks at large-scale, population level data) called The WHO MONICA Project that showed mean cholesterol vs mean height, grouped by population centre (E.g. China-Beijing or UK-Glasgow).

In this chart, you can see a positive correlation between height and cholesterol, with a coefficient of 0.36, suggesting that height may be a potential risk factor for higher cholesterol.

However, when the analysis was re-run using raw data (not averaged for each of the population centres), the correlation coefficient was -0.11.

So, when using mean measures of each population centre, it appears that height could be a risk factor for higher cholesterol, whilst the raw data actually shows the opposite is slightly more likely to be true!

This is known as an “ecological fallacy” – because it takes population level data and makes erroneous assumptions about individual effects.

This is a great example of Simpsons Paradox.

Simpsons paradox is when a trend appears in several different groups of data but disappears or reverses when the groups are combined.

Table 1 in Wang (2018) is a relatively easy example. (This is fictional test score data for two schools.)

(Also, please ignore for a moment the author’s possible bias in scoring male students higher – maybe this is a test about ability to grow facial hair.)

male

male

female

female

School

n

average

n

Average

Alpha (1)

80

84

20

80

Beta (2)

20

85

80

81

It’s clear if you look at the numbers that the Beta school have higher average scores (85 and 81 for male students and female students respectively).

However, if you calculate the averaged scores for individuals in the schools, Alpha school has an average score of 83.8 and Beta has just 81.8.

So whilst Beta school *looks* like the highest performing school when broken down by gender, it is actually Alpha school that has the highest average scores.

In this case, it’s quite clear why: if you only look at the average scores by gender, it’s easy to assume that the proportion of male and female pupils for each school is roughly the same, when in fact 80 pupils at Alpha school are male (and 20 female), but only 20 are male at the Beta school, with 80 female.

Using gender to segment the data hides this disproportion of gender between the schools. This may be appropriate to show in some cases, but can lead to false assumptions being made.

The same issue can be seen in Covid-19 Case Fatality Rate (CFR) data when comparing Italy and China. Kegelgen et al (2020) found that CFRs were lower in Italy for every age group, but higher overall (see table (a)) in the paper.

The reason, when you see table (b), is clear. The CFR for the 70-79 and 80+ groups are far higher than for all other age groups, and these age groups are significantly over-represented in Italy’s confirmed cases of Covid-19. This means that Italy’s overall CFR is higher than China’s only by dint of recording a “much higher proportion of confirmed cases in older patients compared to China.” China simply didn’t report as many Covid-19 cases in older individuals, and the fatality rate is far higher in older individuals. Italy has a more elderly population (median age of 45.4 opposed to China’s 38.4), but other factors such as testing strategies and social dynamics may also be playing a part.

Another example of Simpsons Paradox is in gender bias among graduate admissions to University of California, Berkeley, where it was used in reverse. In 1973, the admission figures appeared to show that men were more likely to be admitted than women, and the difference was significant enough that it was unlikely to be due to chance alone. However, the data for the individual departments showed a “small but statistically significant bias in favour of women”. (Bickel et al, 1975). Bickel et al’s conclusions were that women were applying to more competitive departments such as English, whilst men were applying to departments such as engineering and chemistry, that typically had higher rates of admission.

(Whether this still constitutes bias is the subject of a different debate.)

The crux of Simpsons Paradox is: If you pool data without regard to the underlying causality, you could get the wrong results.

References:

Bokai WANG, C. (2018) “Simpson’s Paradox: Examples”, Shanghai Archives of Psychiatry, 30(2), p. 139. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5936043/ (Accessed: 21 October 2020).

Julius von Kugelgen, Luigi Gresele, Bernhard Scholkopl, (2020) “Simpson’s paradox in Covid-19 case fatality rates: a mediation analysis of age-related causal effects.” Arxiv.org. Available at: https://arxiv.org/pdf/2005.07180.pdf (Accessed: 21 October 2020).

P.J. Bickel, E.A. Hammel and J.W. O’Connell (1975). “Sex Bias in Graduate Admissions: Data From Berkeley”(PDF). Science. 187 (4175): 398–404. doi:10.1126/science.187.4175.398. PMID 17835295. https://homepage.stat.uiowa.edu/~mbognar/1030/Bickel-Berkeley.pdf

WHO MONICA Project Principal Investigators (1988) “The world health organization monica project (monitoring trends and determinants in cardiovascular disease): A major international collaboration” Journal of Clinical Epidemiology 41(2) 105-114. DOI: 10.1016/0895-4356(88)90084-4

Psychological Safety and Westrum’s Organisational Typologies

“Culture Eats Strategy for Breakfast”

A statement famously attributed to Peter Drucker, by which he means that however much you work on your strategy, you ultimately cannot ignore the “people factor”. It is people that execute your strategy and it is through people that it will succeed or fail.

People Create Culture

The most important aspect for any organisation is people. Not strategy, not processes, not operations, and not even finance. An organisation is made of people, and people create culture. If strategy consists of the rules of the game, culture will determine how the game is played.

Psychological safety is often an emergent property of great organisational culture, but that doesn’t mean you can’t explicitly and purposefully work towards it and state that one of your goals for the organisation is to possess a great degree of psychological safety. Indeed, the first step in an intelligent journey to build psychological safety is often stating your goal and asking for help in getting there.

Psychological Safety and Culture

I’ve previously written about how to measure psychological safety, but measuring culture can be more challenging. Dr. Ron Westrum wrote in 2003 about The Typologies of Organisational Cultures that reflect how information flows through an organisation. He wrote: “organisational culture bears a predictive relationship with safety and that particular kinds of organisational culture improve safety…” That is to say, because information flow is influential and indicative of other aspects of culture, it can be used to predict how organisations or parts of them will behave when problems arise.

Westrum was focussed on real-world safety measures in the realm of healthcare and aviation, but in our technology world we should strive to adopt the same diligent approach to safety for the sake not just of the products we build but for the humans on our teams as well.

Culture is the almost intangible aspect of an organisation that so often reflects the CEO’s personality or the stance of the board members. As Westrum states: “Culture is shaped by the preoccupations of management.”

For example, if management, particularly senior management, are most concerned about exposure to risk, the organisational culture will reflect that, with processes and checks in place to ensure risk is reduced wherever possible; this usually results in a decreased focus on innovation, lower speed to market, and a low appetite for change.

Westrum’s Organisational Typologies

See the table below for Westrum’s organisational typology model. Each column describes a broad cultural typology: Pathological, Bureaucratic, or Generative, and six aspects of those cultures. It is clear from the table that the Generative culture that Westrum describes is a broadly psychologically safe culture where team members cooperate, share their fears, admit failure and continually improve.

 

Pathological Bureaucratic Generative
Power oriented Rule oriented Performance oriented
Low cooperation Modest cooperation High cooperation
Messengers “shot” Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented

The Westrum organisational typology model: How organizations process information ( Ron Westrum, “A typology of organisation culture),” BMJ Quality & Safety 13, no. 2 (2004), doi:10.1136/qshc.2003.009522.)

By surveying people across the organisation, you can establish the broad typology in which your organisational culture sits, and identify measures to improve. Ask respondents to rate their agreement on a 1-5 scale (1 being not at all, 5 being complete agreement) with the below statements:

  • On my team, information is actively sought.
  • On my team, failures are learning opportunities, and messengers of them are not punished.
  • On my team, responsibilities are shared.
  • On my team, cross-functional collaboration is encouraged and rewarded.
  • On my team, failure causes enquiry.
  • On my team, new ideas are welcomed.

These 6 statements are from Dr Nicole Forsgren’s research into high performing teams at DORA.

Each of these statements align with a row in the table above, so by collecting and analysing the average scores, you can quantitatively determine where your organisation resides in Westrum’s Typologies. Analyse the standard deviation of the scores to determine both the range of scores and the degree of statistical significance of the results.

Intra-Organisational Psychological Safety

There are many ways to improve the psychological safety of your team and your organisation, but sometimes as a leader, your influence may not extend very far outside of your team, and as a result, you may decide to build a high-performing, psychologically safe team within an environment of much lower psychological safety. This is admirable, and most likely the best course of action, but it is one of the most difficult places to put yourself as a leader.

Consider the “safety gradient” between your team boundary and the wider organisation. In a pathological or bureaucratic organisation, with varying degrees of toxic culture, that safety gradient is steep, and can be very hard to maintain as the strong leader of a high performing team. You may elect as your strategy to lead by example from within the organisation, and hope that your high-performing, psychologically safe team highlights good practice, and combined with a degree of evangelicalism and support, you can change the culture from “bottom-up”, not “top-down”.

This can work, and it will certainly be rewarding if you succeed, but a more effective strategy may be to build your effective team whilst lobbying, persuading and influencing senior management with hard data and a business case for psychological safety that demonstrates the competitive advantage that it can bring.

Create Psychological Safety

Take a look at my Psychological Safety Action pack, with a ready-made business case and background information, workshops and templates, to give yourself a shortcut to influencing and build a high performance, psychologically safe, and Generative organisational culture.

For any further information on how to build high performing organisations and teams, get in touch at tom@tomgeraghty.co.uk.

 

Three Simple Psychological Safety Exercises

It’s a long and worthwhile journey to build high levels of psychological safety in your team, and much of the hard work involves excellent leadership, clarity of direction, effective support, vulnerability, curiosity and much more.

However there are some simple exercises that you can carry out with your teams that directly build psychological safety. See below for four effective exercises and practices to build psychological safety, cohesion and performance.

1 – Run a Values and Behaviours Workshop

Workshop with your team to establish and refine the main values that all members of your team endorse. From these values, extrapolate the behaviours, with your team members, that reflect these values and help the team work together to achieve their goals.

For example, “blamelessness” could be one of your team values, and a behaviour that reflects this could be “Taking collective responsibility for mistakes.”

Sharing common expectations of behaviour is fundamental for psychological safety in a team.

As a result of carrying out this Values and Behaviours workshop:

  • Team members understand what is expected of them and others.
  • Team cohesion and performance improves.
  • The team are aligned to the values of the organisation.
  • Boundaries regarding acceptable behaviours are agreed.
  • The degree of psychological safety of team members increases.

2 – Hold a “Fear Conversation”

Whilst psychological safety is not about existential or external threats, it is very much about being able to show vulnerability and emotion. This exercise encourages that behaviour and builds psychological safety by making openness a norm for the team. It also provides some actionable outcomes to deal with real-world risks and threats.

On a white board or flip chart, create three columns – one for “Fear”, one for “Mitigations” and one for “Target State”.

In the fear column, write down some of the fears that you and team members possess in the team, such as “missing deadlines” or “making mistakes”. Ask everyone to contribute, but make sure that as the team leader, you go first.

Then, as a team, come up mitigations to these fears, which consist of practical things team members can do to reduce the risk of the fears becoming real. Or, in case those fears are inevitable, instead write down ways that the impact can be reduced.

Finally, discuss and write down your “Target State” – this is your team’s utopia, where “everyone can make mistakes without fear of repercussions” or “we never miss a deadline”. This helps the team cohere around common goals and aspirations, which are essential to building psychological safety.

3 – Run Retrospectives

Carrying out regular retrospectives to find the systemic root cause of failures, problems or mistakes is one of the most valuable things you can do as a leader in your journey to building psychological safety.

Ensure that any retrospective is given enough time and is carried out in an appropriate setting. Team members need to feel able to be honest and as vulnerable as possible, so carry it out in a non-public area and certainly don’t record it if you’re carrying out over a video call.

Highlight, discuss, and deep dive into the things that went well, the things you need to change as a team, any lessons learned or anything still to be discovered.

Identifying root causes of failure without apportioning blame is crucial to psychological safety, because team members need to know that they can take intelligent risks without fear of repercussions, humiliation or punishment.

For more detailed guides in the above workshops, along with templates and examples, download the psychological safety Action Pack.