Post Hoc Tests Explained: Tukey, Bonferroni and Dunn, with Real Data

Home › Research Methodology › Post Hoc Tests Explained

Below, we explain post hoc tests step by step and run them live on real data from ScholarsColab.

Common post hoc tests at a glance

Worked example: which venue groups differ in citations?

This continues the example from our ANOVA guide, where we found that citations differ between types of venue. Now we find out which types differ. Every result is calculated live, and you can run the same code yourself.

The data: 41,792 papers, mostly published between 1998 and 2026, by 967 ScholarsColab mentors (data updated 15 October 2026). The file you will load is anonymised: no paper titles, journal names or researcher identities.

Step 1: Load the data

import numpy as np
import pandas as pd
from datetime import date

# The ScholarsColab Research Methods Index (papers level): anonymised, updated daily
DATA = "https://scholarscolab.com/wp-json/scholarscolab/v1/research-methods-data?level=papers"
papers = pd.read_csv(DATA, comment="#")   # lines starting with # hold the citation

def p_text(p):
    return "< 0.001" if p < 0.001 else f"= {p:.3f}"

print(papers.shape)
papers.head()

(41792, 15)

Step 2: Choose the papers and the groups

We use the same five venue groups and the same publication window as in the ANOVA example.

THIS_YEAR = date.today().year
# Compare papers that have had a similar time to collect citations

window = papers[papers["year"].between(THIS_YEAR - 16, THIS_YEAR - 4)].copy()

groups = ["IEEE Transactions / Journals", "Open-access mega-journals", "Other journals",
          'Generic "International Journal of"', "Conference"]
window = window[window["venue_group"].isin(groups)]

summary = window.groupby("venue_group")["citations"].agg(papers="count", median="median", mean="mean")
summary.reindex(groups).round(1)

Step 3: Compare every pair, and correct for multiple testing

With five groups there are ten possible pairs. Because citations are skewed, we compare each pair with the Mann-Whitney U test. Running ten tests raises the chance of a false alarm, so the Bonferroni correction multiplies each p-value by ten.

from itertools import combinations
from scipy import stats

pairs = list(combinations(groups, 2))
rows = []
for a, b in pairs:
    x = window.loc[window["venue_group"] == a, "citations"]
    y = window.loc[window["venue_group"] == b, "citations"]
    u, p = stats.mannwhitneyu(x, y, alternative="two-sided")
    rows.append({"pair": f"{a} vs {b}", "median A": x.median(), "median B": y.median(),
                 "p": p, "p (Bonferroni)": min(1, p * len(pairs))})

result = pd.DataFrame(rows).set_index("pair")
result["significant"] = result["p (Bonferroni)"] < 0.05
result.round(4)

How to choose and run a post hoc test

1. Run the overall test first (ANOVA or Kruskal-Wallis).
2. Choose the post hoc test that matches it and your data (see the table above).
3. Decide which comparisons you need: all pairs, or only each group against a control.
4. Report adjusted p-values and the size of each difference.
5. Present results clearly, for example with letters or a table showing which groups differ.

Common mistakes reviewers point out

• Running many unadjusted t-tests instead of a proper post hoc test.
• Using a post hoc test that does not match the overall test or the data.
• Reporting only “significant” pairs without the differences themselves.
• Running post hoc tests after a non-significant overall test and reporting chance findings.

Frequently asked questions

Use this data in your own work

More research methodology guides

• Independent vs Dependent Variables
• Control Group vs Experimental Group
• Mediator vs Moderator Variables
• Validity vs Reliability (and Generalizability) in Research, with Real Data
• Retrospective vs Prospective Studies
• Data Collection Methods in Research
• Types of Study Design
• Research Hypothesis
• Literature Review vs Systematic Review vs Meta-Analysis, with Real Data
• Sampling Methods in Research
• Experimental vs Quasi-Experimental Design
• Correlation vs Causation
• Deductive vs Inductive Reasoning in Research, with Real Examples
• Incidence vs Prevalence
• Descriptive vs Inferential Statistics, Explained with Real Data
• P-Value and Statistical Significance
• Operationalization
• Surveys and Questionnaires in Research
• Qualitative Research Methods
• Bias in Research
• Research Gap, Problem Statement and Research Question, Explained with an Example
• Pearson Correlation Coefficient (Pearson’s r)
• Chi-Square Test Explained
• ANOVA and the Kruskal-Wallis Test, Explained with Real Data
• Regression Analysis Explained
• Standard Deviation and Variance
• Confidence Intervals Explained
• t-Test vs Mann-Whitney U Test
• Effect Size Explained
• Heteroskedasticity Explained