News Archive

Assessing student work in performance based learning areas is now easier thanks to research developed by Dr Stephen Humphry, Senior Lecturer in Educational Assessment, Measurement and Evaluation at UWA’s Graduate School of Education and Dr Sandy Heldsinger. Humphry and Heldsinger began their research on assessment methods in 2008 with grant funding from the Australian Research Council.  The impetus behind their research stemmed from a number of observations. These include:

• deficiencies identified in rubric assessment methods
• the lack of fine-grained information about student learning provided by existing method
• a complete lack of valid and reliable assessment methods for some difficult-to-assess learning areas and
• concerns from the teaching fraternity about assessment methods such as multiple choice.

“The goal of the research was to help teachers obtain valid, reliable, and detailed information in a range of difficult-to-assess areas in a way they can use it to inform their teaching,” says Dr Humphry.

“When assessing student work for tasks such as reading comprehension and mathematics, school teachers use multiple choice and short responses. In areas of learning that involve extended performances such as sport, visual arts, oral, musical and dance performances or those involving critical thinking such as information reporting, the process of assessing student work can be challenging.”

Following the completion of the research project in 2013, Stephen and Sandy established the company Pairwise in 2014 and successfully developed and commercialised the Brightpath software.  Applying what they had learned from the research, the software was a breakthrough in the assessment of students in a range of learning areas including those not amenable to large scale testing.

How does it work?

The method of pairwise comparisons dates back over a century.  In educational contexts, for each comparison teachers simply judge which of two performances is better. There is a growing body of research in which teachers use long series of pairwise comparisons to assess student work. Although reliable results are produced in a wide range of areas, the process is very time consuming.

Harnessing their reliability in marking and the advanced abilities of teachers, the researchers developed a novel two-stage approach, built on pairwise comparisons:

Stage 1: Pairwise comparison scales were developed by a small group of teachers for each area. The results were then calibrated and teachers were provided with exemplars. The WA Principals Association were instrumental to this process helping to coordinate work by teachers in many schools to collect student work samples and subsequently compare performances.

Stage 2 : Based on both statistical and qualitative analysis, sets of exemplars are then selected as anchor points on a scale, which is referred to as the “teacher’s ruler". The teacher’s job then then becomes simpler and more efficient.  When assessing work, they simply determine where the child slots in on the scale. The Brightpath system also supports teachers by using brief descriptors and annotations in conjunction with the exemplars.

By harnessing teacher judgement and involving them when setting up the scales, the outcome is an assessment method that is as straightforward as possible for teachers, accurate, reliable and applicable in a wide range of learning areas.

The impact

The School Curriculum and Standards Authority, which is responsible for assessments, standards and reporting in WA for Kindergarten to year 12 want comparable grades for student assessments in all learning areas (i.e. B means represents the same range of performance levels at each school).  The Brightpath program is working towards achieving this goal of comparability across the board, with a particular long-term focus on difficult-to-assess areas such as Oral Language and Critical Reasoning.

The greatest impact is realised by teachers who will have an effective tool that can assess a broad range of areas.  Teachers also receive data as feedback which they can use to compare students' work and track student learning over a period of time, such as a year. In conjunction with targeted teaching points, it is hoped the approach will assist teachers in improving student progress.

The assessment method is currently operating in collaboration with School Curriculum and Standards Authority (SCASA), Western Australian Primary Principals’ Association (WAPPA), Association of Independent Schools of Western Australia (AISWA) and Catholic Education Western Australia (CEOWA). The Brightpath system is currently being adopted by approximately 250 schools across Western Australia, with a significant trial recently planned for South Australia. Pairwise is also in discussion with stakeholders in other states and territories of Australia.

“There is an appetite for such a solution—one that has impact both in Australia and even internationally.  As well, the impact of this research could translate across other industries, wherever performance assessments are used such as in job selections, auditions and so on.  For now however, the focus is on the education sector,” says Dr Humphry.

Media references

Siaw Chai         Marketing and Communications Officer         +61 (08) 6488 2382