Research project assessments and supervisor marking
By Anosh Butt
Introduction to Problem
This case study investigates research conducted by academics and researchers from Newcastle University. The research investigates undergraduate report marking involving two assessors; however, marking models are not consistent as some institutions use two independent assessors and do not use the project supervisor (McQuade et al., 2020). The authors highlight that critics point out that the latter model may create a supervisor bias, while its supporters suggest that it ensures subject expertise, which is crucial in the assessment process. The undergraduate biosciences programs at Newcastle University utilise the supervisor as one of the report assessors, while the postgraduate programs do not. The research explores the impact of supervisor marking, compares undergraduate and postgraduate project marks between the period: 2011/12 – 2016/17, and analyses two different reconciliation processes quantitatively.
Solution
The authors present that the project marking data was associated with undergraduate and postgraduate biosciences programmes delivered between the academic years 2011/12 – 2016/17. During this period, the total number of undergraduate projects was 1460, with 897 relating to four academic years 2011/12 – 2014/15 (reconciliation method I), and 563 related to two years 2015/16 – 2016/17 (reconciliation method II). Postgraduate marking data related to period 2011/12 – 2015/16 and represented 894 project reports.
The undergraduate project reports were presented in scientific paper format and had a word limit of 5000 words, while the postgraduate project reports were presented in a thesis report format and had a word limit of 8000 words. The pass mark for the undergraduate and postgraduate projects were 40% and 50%, respectively. Project supervisors for undergraduate and postgraduate research projects involved up to 150 academics, spread across five different research institutes and three schools (McQuade et al., 2020).
The first reconciliation method consisted of discussions among the two markers, and they were required to submit an independent mark, supported by written comments. It was used for undergraduate report reconciliations between 2011/12 – 2014/15 and postgraduate reports during 2011/12 – 2015/16. The second reconciliation method comprised discussions between the two markers, and they were required to reconcile their marks, so they return a single agreed mark and one set of supporting comments. It was used for undergraduate report reconciliations between 2015/16 – 2016/17.
The research data were divided into three groups, which consisted of undergraduate project report marking 2011/12 – 2014/15, postgraduate project report marking 2011/12 – 2015/16, and undergraduate project report marking 2015/16 – 2016/17. The data was analysed using the difference between the grades awarded by markers one and two. For undergraduate reports, marker one was undergraduate supervisor and marker two was a faculty academic, and for postgraduate reports, marker one was an internal examiner with appropriate subject knowledge, and marker two was an external examiner with appropriate expertise.
Measurable impact
The analysis of undergraduate marking data over a four-year period evidenced a mean difference between supervisors and faculty academic of +2.3% (McQuade et al., 2020). This evidence reflects independent double marking and supports an agreement between markers, although skewed towards supervisors, and shows that supervisors mark marginally higher than second markers. There were some instances, which showed differences between markers of up to 30%, and showed less robustness in marking consistency. These discrepancies were not limited to the supervisors’ marks.
An interesting observation regarding postgraduate dissertations is that with the same internal pool of academics as first markers, there were still discrepancies of up to 30%, although the internal markers did not know the postgraduate students. Factors contributing to the marking bias include the level of subject knowledge of the markers, report marking experience, and first and second marker subconsciously favouring certain elements of marking. It is important to note that the second markers of the undergraduate report and the external assessors of the postgraduate reports each marked more dissertations than the supervisors/internal markers, and the diversity of project topics that they had to grade was significant, which may compromise their subject knowledge and marking reliability.
The authors report that there is evidence that subject knowledge may not be essential for marking student dissertations, and these scenarios relate to the assessment of more general components such as document structure, presentation, style of writing, and referencing rather than scientific content. The authors also highlight that assessors without relevant subject expertise became more cautious and are less likely to use the full marking range, but on average, they marked significantly lower than those with subject-specific knowledge.
McQuade et al. (2020) acknowledge that reconciliation comments evidence content relating to student originality, breadth, and depth of scientific knowledge, independent critical thinking, and scientific argument. Finally, the authors comment that there is a pressure in higher education to move away from the undergraduate supervisor in the dissertation marking process, and the drive for change is due to the perceived supervisor leniency associated with student contact and familiarity caused by superior subject knowledge of the supervisor.
FAQ
Q: How do the reconciliation methods affect the final grades of the projects?
A: The reconciliation methods play a crucial role in ensuring fairness and consistency in the marking process. While the blog post doesn't provide detailed outcomes of each method's impact on final grades, it's reasonable to infer that the first method, involving independent marks supported by written comments, allows for a more transparent understanding of each marker's perspective, potentially leading to a more balanced final grade. The second method, requiring markers to agree on a single mark, likely promotes consensus but may dilute individual marker's viewpoints. In both cases, the student voice is indirectly represented through the marks and comments, reflecting their understanding and engagement with the subject matter.
Q: What are the implications of these findings for future marking policies at Newcastle University and potentially other institutions?
A: The findings suggest a need for ongoing review and possibly the refinement of marking policies to ensure they are as fair and unbiased as possible. For Newcastle University and other institutions, this might mean reconsidering the role of the supervisor in the marking process, to mitigate any potential bias, while still valuing their subject expertise. Policies may also evolve to standardise reconciliation methods, ensuring they adequately reflect the student voice in terms of the quality and originality of their work. Overall, these findings could lead to a more robust and transparent assessment process, enhancing the credibility of the qualifications awarded.
Q: Were there any qualitative feedback from students or markers regarding the marking process and outcomes?
A: The blog post does not mention the collection or analysis of qualitative feedback from students or markers. However, incorporating student voice through feedback mechanisms is essential for understanding their perception of the fairness and clarity of the marking process. Likewise, markers' perspectives on the efficacy of reconciliation methods and marking criteria can provide valuable insights into improving the assessment process. Engaging with both students and markers in this way would help institutions like Newcastle University to refine their marking practices, ensuring they are aligned with the needs and expectations of all stakeholders involved in the educational process.
References
[Source Paper] McQuade, R., Kometa, S., Brown, J., Bevitt, D. and Hall, J., 2020. Research project assessments and supervisor marking: maintaining academic rigour through robust reconciliation processes. Assessment & Evaluation in Higher Education, 45(8), pp.1181-1191.
DOI: 10.1080/02602938.2020.1726284
Related Entries