Did your teachers grade your work unfairly? This is a common feeling among students, and there have been many scientific studies on the topic. The classic approach is to look at students' grades when their name is known to the teacher who grade them, and compare that to the grades of the same students at anonymous tests. Any impressive number of studies found systematic favoritism towards girls compared to boys, especially coming from female teachers. A few studies also found bias against ethnic minorities, but this has been studied less often than the gender bias.

Unfortunately, it is still hard to draw any solid conclusion, because (afaik) nobody has done a systematic meta-analysis of the research. Thus, here is what I suggest: let's do our own meta-analysis!

What's a meta-analysis?

Sometimes, different studies on the same topic give different results. This is either due to random sampling error, or because the studies were done in different places, at different times, or used slightly different methodologies. The danger is cherry-picking: if you want to defend a particular viewpoint, it is easy to pick just the studies that back up your narrative, even if there are many other studies that found opposite results.

Meta-analysis is a solution to this problem: it aims to find all the studies ever done on a topic, and aggregate them into a single big-picture. To put it in a simple way, the goal is to pool all the existing samples into one big sample and look at the overall effect.

Meta-analysis can also bring some new insight: there are specific methods to find publication bias (when researchers don't publish their results because they are negative, or contradict their expectations - this happens more often than you think). We can also see how the results evolved over time. Additionally, we could also look at the media coverage of these studies, to see if studies that find bias against some population x or y receive more attention from the media.

Why is that important? Who cares about your middle-school grades?

The research above shows that school grading has important consequences on people's life. It affects what field people end up studying, which university they get into, whether they drop out, or what diploma they get in the end (there is a study finding bias on teacher's accreditation exams, for example). Conversely, unfair grading may push people to work harder, or, if they are aware of the unfair grading, cause mistrust in the system and make people work in jobs where they don't depend on someone else's evaluation. In short, school and university grading plays a systemic role in the production of gender roles and stereotypes, that's why it's important to have clear data about it.

What there is to be done

You don't need any particular skill or experience to participate in the project, as long as you are motivated to help. You don't need to be familiar with statistics or academic writing (I'm happy to help). There are guidelines called PRISMA that describe precisely how to do a good meta-analysis, so we can follow their advice. Here is a rough roadmap:

​

  • Find all the studies ever done on the topic (this means coming up with a transparent search strategy, e.g. which keywords to use in search engines)
  • Extract the results of each study, as well as all the relevant variables (year, sample size, country, level of education...) to make a clean dataset
  • Check that the studies were well-conducted: did they follow statistical "good practices", like correcting for multiple testing?
  • Combine them into a global measurement of discrimination, do some statistical analysis to quantify publication bias
  • Make plots
  • Write the text of the paper, including a short review of the literature, describing the methodologies and explain how we did the meta-analysis, a detailed result section, a conclusion...
  • Go through the publication process (peer-review etc.), if we ever get there.

Some parts of the project might be somewhat tedious, if done by a single person. By making it a collective project, I hope to make it more fun and exciting. Also, there is no big commitment here, any small contribution is already very valuable.

It's important to note that the goal is not to lobby for any particular agenda. The goal is to find out the truth. Maybe we will find that there is no discrimination at all, or that it's negligibly small, or that it was just publication bias from the beginning etc. If that's the case, we still need to report it accurately.

Will this be published?

I hope so. I we actually manage to produce something of worth, the least we can do it to put it on a pre-print server like SocArXiv. Then, we might go all the way and try to publish it in an actual academic journal, but that's a more complicated adventure (we might have to crowd-fund publication costs, for instance).

How do I participate?

Here is a subreddit that I created a long time ago, but never used for anything. Unless anybody has a better idea, I suggest we use it for organizing:

https://www.reddit.com/r/EBSJ/

If you feel like helping or just standing by to see how thing progress, don't hesitate to join!