In 2023, ACER conducted a bookmark standard-setting workshop to assess whether it is possible to align PAT scale scores in Maths and Reading with the achievement standards for a subset of Victorian F–10 curriculum levels. The bookmark exercise was designed to determine where cut scores for each achievement standard level would lie. The exercise was restricted to Victorian F–10 achievement levels 1 to 6 for each of Mathematics and English.
Read the full report here: Aligning Victorian Curriculum achievement standards with PAT Maths and Reading scales
This article covers the following topics:
Background
PAT assessments report scale scores for students that reflect student achievement (ability estimate) in the learning area. Each PAT test item also has a scale score that reflects the difficulty of the item (item difficulty estimate) relative to the scale. In PAT Reading and PAT Maths assessments, students and items are mapped onto the same Reading or Maths scale, respectively. The levels of knowledge and skills required to correctly respond to an item increase as students make progress on the scale.
The Victorian Curriculum achievement standards provide descriptions of substantively different levels of progress along a continuum of learning as defined by the Victorian Curriculum F–10. The achievement standards describe what students can do and understand when they have attained that level.
Because PAT items have well-established scale score difficulties, the bookmark standard-setting procedure was used to find cut scores on PAT scales for six achievement standard levels in each of the two domains.
Method
During a standard-setting session raters use an ordered item booklet, where items from the PAT assessments are provided in their order of difficulty, with the easiest items at the beginning of the list and most difficult items at the end of the list.
Expert panellists were sought from ACER’s extensive connection with teachers and schools through the PAT assessment program, and through other ACER-run marking programs that require the input of qualified and experienced teachers. Ideally, selected participants were educators with a good
knowledge of primary level Mathematics or Reading and a sound understanding of what students can typically do at given year levels.
Data collection took place over two rounds. In the first round, raters used ACER Signum to assess each item in the ordered item booklet against the skills and knowledge detailed in achievement standards at each level they were assigned. For each bookmark level, the raters considered whether the borderline, or target, student at that level would be more likely than not to respond correctly to the item under
consideration. In this sense, raters were required to make an on-balance judgement as to where the bookmark would be set. The data from this first round were collated and the initial bookmark placements analysed for presentation prior to the second round of judgements, undertaken at an in-person workshop designed to facilitate discussion amongst each group on the placement of the bookmarks and their assessment around the skills and knowledge required to respond to each item with reference to the achievement standard level that was being considered.
As part of the process of evaluating the position of the bookmarks, the panels of raters were provided with some evidence of the impact of the location of their bookmarks when considering the typical PAT achievement of students at relevant stages of their schooling. The PAT Maths and PAT Reading Australian norms were used to show the distribution of PAT achievement relative to the bookmark standards that had been set during the workshop. The norms provide a reference for the distribution of achievement at each year level nationally.
Results
Maths
Using the mean bookmark placement and incorporating a margin of two standard errors around that
point, we can estimate a scale score range in which each achievement standard level sits. For example, we can estimate that a student who has achieved a PAT Maths scale score of at least 97.5 (± 1.0) is likely to be reaching Achievement Standard Level 1. Similarly, a student achieving a scale score of at least 134.9 (± 3.8) is likely to be reaching Achievement Standard Level 6.
For more detailed discussion of the results, including about the possible effects of the introduction of the Victorian Mathematics Curriculum v2.0 on the bookmark placements, read the article Aligning Victorian Curriculum achievement standards with PAT Maths and Reading scales.
Mathematics achievement level bookmarks and PAT Maths Australian norms
| Victorian Curriculum Mathematics Achievement Standard Level |
PAT Maths bookmark placement (scale score) |
PAT Maths Australian mean achievement (scale score) |
|---|---|---|
| 1 | 97.5 | 99.5 (year 1) |
| 2 | 107.6 | 108.3 (year 2) |
| 3 | 112.9 | 115.4 (year 3) |
| 4 | 120.4 | 121.1 (year 4) |
| 5 | 127.4 | 125.5 (year 5) |
| 6 | 134.9 | 128.9 (year 6) |
Reading
The Reading team reached a reasonable consensus, and the resultant achievement standard level cut scores are fairly well distributed along the PAT scale. But there are some issues with the alignment between the achievement standard levels, which describe expected end of year development, and the median PAT Reading scale scores for each year level, according to the Australian norms. These findings raise some interesting questions about how well curriculum standards and descriptions can be aligned with an empirical scale when it comes to Reading.
The main difficulty in this standard-setting exercise concerned the broadness of the achievement standards and curriculum descriptions and the lack of guidance they provide regarding how Reading skills progress. For example, achievement standard Level 2 for Reading and Viewing states that students “identify literal and implied meaning, main ideas and supporting detail.” (Vic. Curriculum 2022) In PAT Reading, test items provide a distinction between simple and more complex interpretations. The PAT Reading achievement band description (90–99) for the strand ‘Interpret Implied’ indicates that students at this level “use their everyday knowledge to infer simple explanations for familiar events and predict likely outcomes.” The progression of this skill is evident from the description of PAT Reading achievement band 100–109, at which students “infer main ideas when clues are scattered and there is some competing information.”
Given the issues raised in the discussion, it is not appropriate to release the alignment of the Victorian Curriculum achievement standards to the PAT Reading scale at this stage.
Questions remain about how we can align Reading assessments with the curriculum in a way that provides meaningful information to teachers about where students are in their learning and how they can be supported to progress. These questions are worthy of further scrutiny, as they concern how teachers can plan their lessons and collect evidence of student achievement in a way that best supports student development while still meeting the requirements of the curriculum.