Someone asked about the process of using a second reviewer in a meta-analysis to abstract data from studies. The rationale for a second reviewer, of course, is to establish that there is no serious subjectivity involved with the recording of information from individual studies. By showing that two independent reviewers produced roughly comparable data set, you have established objectivity in the data abstraction step. The question arises, though, do you have to use the second reviewer on all studies, or can you just do this for a certain percentage of the studies. If so, is there a certain percentage that is generally accepted?
As far as I know, there are no “official” rules on this. If you do a subset, there is a risk that the statistics that come out may make you wish that you had done everything with two reviewers. Don’t do a subset unless you are pretty sure that the agreement will be high enough to justify not doing 100% of the data abstraction with two reviewers.
If you do a certain percentage, there are some numbers (10%, 20%) that appear often enough in the literature that you can cite them as examples. For example, “Following the process in [reference], we had a second reviewer abstract the same data from 20% of the studies. I call this the lemming school of research.
If you wanted to do this a bit more rigorously, you would justify the sample size by showing that the statistics produced by your comparison of the two raters (typically a correlation of some type) have sufficiently narrow confidence intervals.
You can find an earlier version of this page on my original website.