Meta-Analysis
A traditional literature review as discussed above involves reading, summarizing, and interpreting research in a given field. You simply summarize the earlier findings and draws conclusions about the state of the literature in a given area. The conclusions thus drawn are mostly subjective, based on your critical evaluation of the literature. Hence, there is the possibility that such subjective conclusions may not accurately reflect the actual strength of the subject matter under review. The following are the major problems with the traditional literature review:
- It tends to be a “biased” sample of the full range of the literature on the subject.
- It is usually undertaken through the perspective of the reviewer who gathers interprets the literature in a given field.
- The reasons for including some studies and excluding others are often not made explicit and may reflect the biases of the researcher.
- Included references may be used to support the “expert opinion” while another reference that contradicts this opinion may be excluded from the review.
- If the search strategy and inclusion criteria have not been made explicit, it will not possible for the review to be replicated by another researcher.
- Usually, the individual studies are not quality assessed before inclusion in the review, therefore there may be no differentiation between methodologically “sound” “unsound” studies.
This problem existing in the traditional literature review can be minimized by adding a new analysis to the review.
What is Meta-analysis?
A more rigorous alternative to the traditional review is the meta-analysis. A meta-analysis differs from a traditional review in that its methods are explicit and open to scrutiny. It seeks to identify all the available evidence with respect to a given theme. Meta-analysis has the advantage of including all the studies in a field, so the readers can judge using the totality of evidence whether the evidence supports or refutes a given hypothesis.
There is not a long history of the use of meta-analysis techniques in social science research. In the 1950s and 1960s, social science researchers explored completely different statistical approaches for undertaking meta-analyses. This was significantly thus within the fields of psychology and education. Social scientists published several texts within the 1970s and 1980s on statistical approaches to meta-analysis and information synthesis. It was only in relatively recent times that the other social scientists realized the merits of undertaking meta-analysis.
In the past two decades, the popularity of this research method has increased. Today, it is used as a means for overcoming the problem of subjective interpretation of reviews and providing a more objective method of doing such a review. Additionally, meta-analysis enables you to identify potential moderating variables between an independent and a dependent variable.
A meta-analysis combines all the research on one topic into one large review. The following are some useful definitions of meta-analysis:
- Meta-analysis is the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic.
- Meta-analysis is the statistical synthesis of the data from separate but similar (comparable) studies leading to a quantitative summary of the pooled results.
- Meta-analysis is a type of data analysis in which the results of several studies are lumped together and analyzed as if they were the results of one large study.
- Meta-analysis refers to the statistical analysis of a large collection of results from individual studies for the purpose of integrating the findings.
- Meta-analysis is a systematic review that uses quantitative methods to summarize the literature.
Meta-analysis is thus a more objective method of reviewing literature in a field. It is also known as an aggregate quantitative review. This review is generally centered on the relationship between one explanatory and one response variable. It basically involves comparing or combining the results of related studies. This form of literature review enables you to look at the validity of findings from a comprehensive set of individual studies and then apply a formula to them to determine if they consistently produced similar results (Bordens & Abbott, 2006). If results prove to be consistent, it allows you to conclude more confidently that validity is generalizable.
Steps in Meta-analysis
There are three basic steps in conducting a meta-analysis: identifying relevant variables, locating relevant research to review, and conducting the meta-analysis (Bordens & Abbott, 2006).
Identifying Relevant Variables
The variables to be analyzed have to be identified first. The process of identifying the variables may not be easy especially in a research area in which there is a wide body of research. You must be very specific in identifying the variables. For example, you might choose to meta-analyze the impact of age on employees’ commitment. Here you are limiting yourself to a segment of the commitment literature.
After the scope of analysis is narrowed down, you must decide what variables to record as you review each study. This decision will be guided by the research question. In addition to recording the variables in each study, you should also record for each study the full reference (author, title, date, journal, issue, page numbers, etc.) as well as the nature of the subject sample and procedures.
Locating and Searching Relevant Research to Review
This is a vital step in a meta-analysis. The literature existing in the selected area must be thoroughly searched. You should also uncover those studies that exist but are not published. A questionnaire can be administered, asking knowledgeable persons to share information about such published and unpublished research studies. To avoid your bias, all the existing studies thus located, should be included for meta-analysis.
Doing the Meta-Analysis
After gathering relevant literature and data, you are ready to apply the meta-analysis statistical technique. There are two techniques of Meta-analysis: comparing and combining the results of studies. The first technique is for comparing studies. This comparison is done when you want to determine whether two studies produce significantly different effects. The second technique shows that you can also combine studies to determine the average effect of a variable across studies. Looking at the columns, you can evaluate studies by comparing or combining either p values or effect sizes. The heart of Meta-analysis is the statistical combination of results across studies (Bordens & Abbott, 2006).
Drawbacks to Meta-Analysis
Many researchers question the concept of meta-analysis and its usefulness. The following are some of the drawbacks of meta-analysis (Bordens & Abbott, 2006).
- Assessing the quality of the research reviewed. It is difficult to assess the quality of different research studies. No reliable indicators have developed yet to assess the quality of published research. The assessment of research in new areas of study cannot be the same as the assessment of the quality of research in a well-researched area.
- Putting everything together creates problems. Me analysis adds together apples and oranges. As a consequence, over-generalization may appear in the review.
- Combining and comparing studies using different methods. It is difficult to understand how studies with widely varying materials, measures, and methods can be compared and combined. The other issue is whether averaging should be done across heterogeneous studies.
- Practical problems. Due to the application of a wide variety of approaches to research, many studies cannot be brought under meta-analysis. Likewise, many research studies do not provide the necessary information to conduct a meta-analysis. Hence, there is no choice but to eliminate them for the purpose of meta-analysis.
When Can You Do Meta-Analysis?
In view of the above problems, it is not advisable to go for meta-analysis always. As pointed out by Wilson (1999), meta-analysis is appropriate to be used to collections of research that:
- are empirical, rather than theoretical.
- produce quantitative results, rather than qualitative findings.
- examine the same constructs and relationships.
- have findings that can be configurated in a comparable statistical form (such as effect sizes, correlation coefficients, etc.).
- are comparable” given the research questions at hand?