ANOVA and the Kruskal-Wallis Test, Explained with Real Data

Home › Research Methodology › ANOVA and the Kruskal-Wallis Test, Explained with Real Data

Below, we explain ANOVA in plain language and then work through a complete analysis, notebook-style, on real research data: does the type of venue a paper is published in affect how often it is cited? Every result on this page is calculated live from the ScholarsColab dataset, and you can run the same code yourself.

ANOVA at a glance

Why not just run several t-tests? Each test carries a 5% risk of a false alarm; running many of them multiplies that risk. ANOVA tests all groups at once.

Worked example: does the venue affect citations?

The data: 41,792 papers, mostly published between 1998 and 2026, by 967 ScholarsColab mentors (data updated 15 October 2026). Each paper’s venue has been grouped from its journal or conference name. No titles, journal names or researcher identities are included in the dataset you will load.

Step 1: Load the data

We load the anonymised paper-level data straight from ScholarsColab. Each row is one published paper.

import numpy as np
import pandas as pd
from datetime import date

# The ScholarsColab Research Methods Index (papers level): anonymised, updated daily
DATA = "https://scholarscolab.com/wp-json/scholarscolab/v1/research-methods-data?level=papers"
papers = pd.read_csv(DATA, comment="#")   # lines starting with # hold the citation

def p_text(p):
    return "< 0.001" if p < 0.001 else f"= {p:.3f}"

print(papers.shape)
papers.head()

(41792, 15)

Step 2: Choose the papers and the groups

Citations build up over time, so comparing a 2024 paper with a 2012 paper would be unfair. We keep papers published in the same window of years, and compare five groups of venues.

THIS_YEAR = date.today().year
# Compare papers that have had a similar time to collect citations

window = papers[papers["year"].between(THIS_YEAR - 16, THIS_YEAR - 4)].copy()

groups = ["IEEE Transactions / Journals", "Open-access mega-journals", "Other journals",
          'Generic "International Journal of"', "Conference"]
window = window[window["venue_group"].isin(groups)]

summary = window.groupby("venue_group")["citations"].agg(papers="count", median="median", mean="mean")
summary.reindex(groups).round(1)

Step 3: Look at the data with a box plot

Before running any test, always look at the data. A box plot shows the middle half of each group (the box), the median (the thick line) and the spread (the whiskers).

import matplotlib.pyplot as plt

# Citations are very skewed, so plot them on a log scale (log10 of citations + 1)

window["log_citations"] = np.log10(window["citations"] + 1)
data = [window.loc[window["venue_group"] == g, "log_citations"] for g in groups]

fig, ax = plt.subplots(figsize=(9, 4.5))
ax.boxplot(data, vert=False, showfliers=False)
ax.set_yticks(range(1, len(groups) + 1), groups)
ticks = [0, 10, 100, 1000]
ax.set_xticks(np.log10(np.array(ticks) + 1), ticks)
ax.set_xlabel("Citations (log scale)")
ax.invert_yaxis()
plt.tight_layout()
plt.show()

Step 4: Check the shape of the data

ANOVA assumes the data in each group are roughly normal. Skewness measures how lopsided the data are: 0 is symmetric, and values above about 1 are strongly skewed.

from scipy import stats

print(f"Skewness of citations:     {stats.skew(window['citations']):.1f}")
print(f"Skewness of log citations: {stats.skew(window['log_citations']):.2f}")

Skewness of citations:     38.5
Skewness of log citations: 0.34

Step 5: Run the one-way ANOVA

samples = [window.loc[window["venue_group"] == g, "log_citations"] for g in groups]
F, p = stats.f_oneway(*samples)

# Eta squared: the share of variation explained by the groups

grand_mean = window["log_citations"].mean()
ss_between = sum(len(s) * (s.mean() - grand_mean) ** 2 for s in samples)
ss_total = ((window["log_citations"] - grand_mean) ** 2).sum()

print(f"One-way ANOVA: F = {F:.1f}, p {p_text(p)}")
print(f"Eta squared = {ss_between / ss_total:.3f}")

One-way ANOVA: F = 450.6, p < 0.001
Eta squared = 0.081

Step 6: Confirm with the Kruskal-Wallis test

Because citations are so skewed, it is good practice to confirm the result with a test that uses ranks instead of values.

# The rank-based alternative: suitable for skewed data such as citations

H, p = stats.kruskal(*[window.loc[window["venue_group"] == g, "citations"] for g in groups])
print(f"Kruskal-Wallis: H = {H:.1f}, p {p_text(p)}")

Kruskal-Wallis: H = 1610.2, p < 0.001

Both tests agree: the type of venue is clearly associated with how often papers are cited. Which venue groups differ from which? That is the job of the post hoc tests.

How to run a one-way ANOVA

1. Plot the groups with box plots before testing.
2. Check assumptions: independent observations, roughly normal data in each group, similar spreads (Levene’s test).
3. If assumptions fail, transform the data (as we did with logs), use Welch’s ANOVA, or use Kruskal-Wallis.
4. Report F, both degrees of freedom, the p-value and an effect size such as eta squared.
5. If significant, run post hoc tests to see which groups differ.

Common mistakes reviewers point out

• Running many t-tests instead of one ANOVA.
• Testing before looking at the data, and missing heavy skew or outliers.
• Claiming that all groups differ from a significant ANOVA alone.
• Reporting p without an effect size.

Frequently asked questions

Use this data in your own work

More research methodology guides

• Independent vs Dependent Variables
• Control Group vs Experimental Group
• Mediator vs Moderator Variables
• Validity vs Reliability (and Generalizability) in Research, with Real Data
• Retrospective vs Prospective Studies
• Data Collection Methods in Research
• Types of Study Design
• Research Hypothesis
• Literature Review vs Systematic Review vs Meta-Analysis, with Real Data
• Sampling Methods in Research
• Experimental vs Quasi-Experimental Design
• Correlation vs Causation
• Deductive vs Inductive Reasoning in Research, with Real Examples
• Incidence vs Prevalence
• Descriptive vs Inferential Statistics, Explained with Real Data
• P-Value and Statistical Significance
• Operationalization
• Surveys and Questionnaires in Research
• Qualitative Research Methods
• Bias in Research
• Research Gap, Problem Statement and Research Question, Explained with an Example
• Pearson Correlation Coefficient (Pearson’s r)
• Chi-Square Test Explained
• Regression Analysis Explained
• Standard Deviation and Variance
• Post Hoc Tests Explained
• Confidence Intervals Explained
• t-Test vs Mann-Whitney U Test
• Effect Size Explained
• Heteroskedasticity Explained