Teachers Matter: A Meta-Analysis

• Název práce v češtině: Teachers Matter: A Meta-Analysis
• Název v anglickém jazyce: Teachers Matter: A Meta-Analysis
• Akademický rok vypsání: 2022/2023
• Typ práce: bakalářská práce
• Jazyk práce: angličtina
• Ústav: Institut ekonomických studií (23-IES)
• Vedoucí / školitel: doc. PhDr. Zuzana Havránková, Ph.D.
• Řešitel: skrytý - zadáno vedoucím/školitelem
• Datum přihlášení: 16.03.2023
• Datum zadání: 27.06.2023

Seznam odborné literatury

Andrews, I., & Kasy, M. (2019). Identification of and Correction for Publication Bias. American Economic Review, 109(8), pp. 2766-2794.
Bom, P. R. D., & Rachinger, H. (2019). A Kinked Meta-Regression Model for Publication Bias Correction. Research Synthesis Methods, 10(4), pp. 497-514.
Darling-Hammond, L. (1999). Teacher quality and student achievement: A review of state policy evidence. Education Policy Analysis Archive, 8(1), pp. 1-44.
De Paola, M. (2009). Does Teacher Quality Affect Student Performance? Evidence from an Italian. Bulletin of Economic Research, 61(4), pp. 353-377.
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), pp. 629-634.
Elliott, G., Kudrin, N., & Wuthrich, K. (2022). Detecting p-hacking. Econometrica, 90(2), pp. 887-906.
Furukawa, C. (2019). Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method. Working paper, Massachusetts Institute of Technology.
Glass, G. V., & Smith, M. L. (1979). Meta-Analysis of Research on Class Size and Achievement. Educational Evaluation and Policy Analysis, 1(1), pp. 2-16.
Graham, L. J., White, S. L. J., Cologon, K., & Pianta, R. C. (2020). Do Teachers' Years of Experience Make a Difference in the Quality of Teaching? Teaching and Teacher Education, 96.
Hanushek, E. A., & Rivkin, S. G. (2006). Chapter 18 Teacher Quality. Handbook of the Economics of Education, pp. 1051-1078.
Hanushek, E. A. (2011). The Economic Value of Higher Teacher Quality. Economics of Education Review, 30(3), pp. 466-479.
Hassan, A., Jamaludin, N. S., Sulaiman, T., & Baki, R. (2010). Western and Eastern Educational Philosophies. In 40th Philosophy of Education Society of Australasia Conference, Murdoch University, Western Australia.
Havranek, T., Stanley, T., Doucouliagos, H., Bom, P., Geyer-Klingeberg, J., Iwasaki, I., Reed, W. R., Rost, K., & Van Aert, R. (2020). Reporting Guidelines for Meta-Analysis in Economics. Journal of Economic Surveys, 34(3), pp. 469-475.
Ioannidis, J. P., Stanley, T. D., & Doucouliagos, H. (2017). The Power of Bias in Economics Research. Economic Journal, 127(605), pp. 236-265.
John P. Papay, Matthew A. Kraft (2015). Productivity Returns to Experience in the Teacher Labor Market: Methodological Challenges and New Evidence on Long-Term Career Improvement. Journal of Public Economics, 130, pp. 105-119.
Pham, L. D., Nguyen, T. D., & Springer, M. G. (2021). Teacher Merit Pay: A Meta-Analysis. American Educational Research Journal, 58(3), pp. 527-566.
Stanley, T. D. (2005). Beyond Publication Bias. Journal of Economic Surveys, 19(3), pp. 309-345.
Steel, M. F. (2020). Model Averaging and Its Use in Economics. Journal of Economic Literature, 58(3), pp. 644-719.
Van Aert, R. C., & Van Assen, M. (2021). Correcting for Publication Bias in a Meta-Analysis with the p-uniform* Method. Working paper, Tilburg University & Utrecht University, available online at osf.io/preprints/metaarxiv/zqjr9/download (accessed on December 22, 2021).

Předběžná náplň práce v anglickém jazyce

Research question and motivation

The main research question I intend to study is how teacher quality affects student achievement via meta-analysis.

The thesis's primary objective is to provide a thorough evaluation of the returns on investing in teacher quality as it pertains to improving student achievement. By conducting a meta-analysis of existing research on the impact of teacher characteristics on student achievement, this thesis will provide a comprehensive and up-to-date understanding of the returns to teachers' quality in terms of student achievement. The findings of this thesis will have important implications for policy makers. By demonstrating the value of investing in teachers, particularly in terms of increasing their experience and qualifications, this analysis will provide evidence-based guidance for improving the quality of education in schools. As stated by De Paola (2009), an increase in teacher experience or in teacher research productivity has a positive effect on student performance. Numerous methods exist for enhancing the quality of education. Glass and Smith (1979) state, "There is little doubt that, other things equal, more is learned in smaller classes" (p. 15). However, reducing class size can be rather expensive and it is much better to have just one good teacher for larger class than having more teachers with lesser quality. Research (Darling-Hammond, 1999) indicates that the quality of teacher education and teaching practices has a more significant impact on student achievement compared to factors such as class size, overall spending levels, or teacher salaries. Consequently, this thesis aims to explore the potential cost-effectiveness of investing in teachers' quality directly.

Hanushek (2011) asserts that teachers represent the most influential factor in determining student achievement, with no other attribute of schools yielding a similar magnitude of impact. Empirical evidence indicates that the productivity of teachers typically improves beyond the initial years of their career. Despite this, policymakers have historically overlooked the significance of teacher experience. However, recent research findings suggest that the observed correlation between teacher wages and experience may genuinely reflect productivity growth (Papay & Kraft, 2015). According to Graham et al. (2020) the mere accumulation of years of experience does not inherently result in superior quality of teaching and to boost the quality of teaching among all educators, not just those at the beginning of their careers, they need better support and access to professional development opportunities. The principal-agent theory posits that incentivizing teachers based on their contribution to student achievement can potentially stimulate them to exert additional effort toward enhancing student performance outcomes. Closest to this topic is a meta-analysis that suggests that linking financial incentives to student outcomes has a positive and statistically significant effect (Pham et al., 2021).

Contribution

To the best of my knowledge, there is currently no existing meta-analysis that addresses the specific effect in question. As a result, I will undertake the task of constructing a comprehensive dataset by systematically reviewing and synthesizing relevant literature. This dataset will consist of estimates about the impact of teachers' quality (measured in terms of experience) on student achievement (measured by test results). Additionally, I will analyze the presence of publication bias in the literature using state-of-the-art techniques. To gain a deeper understanding of the variability observed in the estimated effects, I intend to utilize Bayesian and Frequentist model averaging approaches. I will also give particular emphasis to examining the causal estimates reported in the literature. Finally, I will conduct a synthetic study to estimate the best-practice effect, which will be adjusted to account for any biases identified during the analysis.

Methodology

The effect in question represents a relationship between teachers´ quality and pupils´ educational outcomes. Both components of the effect can be measured using various metrics. For instance, in defining teachers´ quality, one can consider measures such as the number of years the teacher has already spent teaching (ie, experience), the number of years of teachers' education, their achieved degrees, and the number of specialized certifications (ie., education), peer evaluations, wages, or any outcome-based measures of teacher effectiveness (see Hanushek & Rivkin, 2006). To define educational outcomes, one can consider factors such as study habits (including class attendance or time spent studying), the decision to pursue further/higher education, or more commonly, test results. These distinct measures indicate I will possibly have to refocus on a more restricted sample of data with more homogenous effects. To construct the dataset, I will create a search query for Google Scholar that will identify the relevant studies while following the PRISMA reporting standards (Havranek et al. 2020). I will track down the references of the best-published articles from the last 10 years to ensure that my search does not miss anything important. From the identified studies, I will collect the effects in question, their standard errors (or other measures that can be recalculated to standard errors such as t-statistics, p-value, or confidence interval), and code for different aspects of the study design.

Once the data is collected, I will examine the presence of publication bias in the literature. Publication selection bias (more in Stanley, 2005) usually occurs when only results that have statistical significance or that are in line with some commonly accepted theory get published. That is not to say that such bias equals cheating: often, a badly chosen model, measurement errors in data, or small samples produce estimates that are possibly erroneous, and, on an individual study level, it makes sense to discard them. The intuitive answer to the question of how teachers´ quality affects pupils´ educational outcomes is: positively. If authors treat large positive estimates indifferently but discard any negative estimates, publication bias might be present: any systematic disregard would distort the whole literature. A basic visual test used for the detection of publication bias is the so-called funnel plot (Egger et al., 1997). It is a scatter plot of point estimates on the horizontal axis against the estimates' precision (the inverse of the standard error) on the vertical axis. In the absence of publication bias (and systematic heterogeneity), the most precise estimates should concentrate around the mean underlying effect. As the precision decreases, estimates become more dispersed from the mean. Consequently, the scatter plot will have the shape of an inverted funnel. If some negative estimates are disregarded (or discarded), the funnel plot will no longer be symmetrical around the mean. The symmetry thus serves as a basic test of publication bias and is rigorously tested by the funnel asymmetry test (Stanley 2005). If publication bias is a linear function of the standard error and if there is no correlation between estimates and standard errors in the absence of publication bias, the degree of publication bias is easily identifiable.

The linearity assumption is well motivated: in the presence of large standard errors (low statistical significance), researchers strive for larger estimates (to achieve statistical significance). The uncorrelation assumption, on the other hand, assumes that the estimates and standard errors are statistically independent quantities, which is implied by most techniques used in this literature. Nevertheless, both of these assumptions can get violated, so I will use other methods that do not impose at least one or both: the weighted average of adequately powered studies (Ioannidis et al. 2017), the Andrew-Kasy selection model (Andrews & Kasy 2019), the stem-based method (Furukawa 2019), the endogenous kink model (Bom & Rachinger 2019), the p-uniform* method (van Aert & van Assen 2018), or p-hacking (Elliott et al. 2022). Additionally, I will use the robust Bayesian model averaging (Maier et al. 2022) method, which combines the aforementioned techniques for publication bias detection into one estimate of the true mean value and the extent of publication bias while accounting for between-study heterogeneity.

Finally, I plan to deal with the variation coming from the context and design of studies in the empirical literature. I will identify factors responsible for context variation which I will code into explanatory variables that will further enter my analysis. There could be dozens of such variables and I will remedy the implied model uncertainty using model averaging methods: Bayesian and Frequentist model averaging methods (more in Steel, 2020). As the bottom line of my analyses, I will compute the implied effects for different contexts. Each region has its unique educational framework, resulting in distinct approaches to teaching and learning. Consequently, the role and impact of teachers within these diverse systems differ accordingly. For example, Western education emphasizes a student-centered approach, while in Eastern education teachers are entrusted with the primary responsibility for ensuring the effectiveness of the classroom experience (Hassan et al., 2010).

Outline

1. Introduction
2. Teachers' quality effect on student achievement
a. Significance of the topic
b. Previous research (literature review)
c. My contribution
3. Collecting the dataset for meta-analysis
a. Selection criteria and the final dataset
b. Basic summary statistics, what do summary statistics suggest
4. Presence of publication and endogeneity bias in the literature
a. Why it could be present, conducting linear (for example the funnel asymmetry test by Stanley (2005)) and non-linear test that use relationship between estimate and standard error
5. Heterogeneity
a. Why estimates in the literature differ
b. The best-practice estimate
6. Conclusion