Marjolijn Verspoor

Comparative judgment in peer assessment in a hybrid learning environment

Comparative judgment (CJ) is a valid and reliable method for proficiency rating (Thwaites et al., 2024). However, CJ in peer assessment — intended to engage students in internal feedback (Nicol, 2020) — remains unexplored. This study investigates how CJ can be used in a hybrid learning setting— a mix of online teaching, live teaching, peer assessment, and expert feedback — within an English for Academic Purposes context. 

Data were collected from two consecutive cohorts (N=18 and N=9, respectively) of first-year Master’s students enrolled in a genre-based Academic Language Use course. Students were trained by completing eight short writing assignments during a 14-week semester. Each assignment was uploaded to Comproved — a CJ tool — and students were asked to compare pairs of their peers' writing samples. Comproved then generated ranking reports for each assignment. The next class, the instructor showed on screen in a think-aloud session how she might edit the texts of two different students anonymously (all students had one turn). 

The effectiveness of this multimodal course was operationalized as score gains between one of the first and the last writing assignments. The first-year data showed that almost all students improved their academic writing skills. The second-year data await to be completed and analyzed. Our preliminary findings suggest that the course yields positive outcomes for developing academic writing skills.  Surveys and interviews indicated that the students found peer assessment useful, but really learned most from the online editing sessions. The instructor was pleased to be able to assign multiples writing assignments but spend relatively less time grading and still give each student detailed feedback on one text.