This article provides information about a recent ACER study into the effect of different factors on students' scale score gain between tests for PAT Maths and PAT Reading.
The results of this study have been implemented as a 'Predicted gain' feature within the Progress report of the ACER Data Explorer to support you in better understanding your students' results.
The following questions are covered:
- What was the goal of the study?
- What data were used in the study?
- What does 'predicted' gain mean?
- What does predicted gain look like?
- How should predicted gain be used?
More detailed information about the study can be found in this document:
Predicting scale score gain - PAT Reading and PAT Maths (pdf)
What was the goal of the study?
The study was designed to illustrate and provide general advice on what scale score gain between assessments looks like in PAT Reading and Maths.
Specifically, the study looked at how differences in the following factors affect the scale score gain between two tests completed by the same student:
Year level: The student's year level at the time the first of the two tests was administered.
Prior scale score: The scale score achieved by the student on the first test of their two tests.
Time: The interval between the two tests administered.
Using actual PAT results data, the study modelled the effects for each year level of prior scale score and time between tests on students' scale score gain. These models can then be applied to 'predict' scale score gain for a given student.
What data were used in the study?
ACER psychometricians analysed a large set of results from PAT Maths 4th Edition and PAT Reading 5th Edition tests completed by Australian students in 2020 and 2021. In particular, the study used results from students who had completed two tests in the same learning area within a period of between 6.5 and 24 months.
| Year level | Maths tests | Reading tests |
|---|---|---|
| 1 | 4167 | 797 |
| 2 | 36 327 | 13 795 |
| 3 | 81 767 | 28 212 |
| 4 | 97 467 | 31 951 |
| 5 | 99 127 | 31 345 |
| 6 | 150 318 | 72 521 |
| 7 | 46 407 | 38 604 |
| 8 | 60 916 | 61 887 |
| 9 | 59 632 | 59 525 |
| 10 | 53 374 | 52 586 |
| Total | 689 502 | 391 223 |
What does 'predicted' gain mean?
Our conception of ‘predicted gain’ for the purpose of this study, can be understood as a response to the following question:
For students in the same year level whose initial (prior) scale scores were the same or very similar, what is the typical scale score gain achieved after a given time interval?
This is a statistical prediction based on a student's year level and current scale score only. For example, the predictions do not account for any of the following factors:
- curriculum standards or achievement levels
- your school or classroom goals
- individual student context or demographic details except year level
- prior learning progress or achievement trends.
Importantly, a student’s predicted gain does not constitute the student’s expected gain, or what they should achieve after a given time interval. It is not a benchmark or standard to be measured against.
What does predicted gain look like?
The reporting goal was to provide an indication of the range of scale scores that are typically achieved by students, given a particular scale score starting point and the time between assessments, at a particular year level. The study provided sufficient evidence that there is a statistically significant relationship between students’ prior scale scores and time between assessments, and the results may be used to predict scale score gain in Reading and Maths over time.
Within the online ACER Data Explorer, a student’s predicted PAT Maths or PAT Reading scale score gain is shown along with an inner and outer range that represent a degree of uncertainty around the prediction.
- Related article: Reporting predicted gain in the ACER Data Explorer
When a student's most recent test was completed within the past 12 months, reports will show 'future' predicted gain (in orange), indicating the gain students in the same year level and with a similar prior scale score typically achieve after 12 months. Within the chart, this projection extends from the star representing the student's most recent test. 'Future' predicted gain will not appear if the most recent test is more than 12 months old.
If the student has also completed a test 9–18 months previously, a 'past' prediction (in blue) will also be shown in the chart. This allows you to compare their actual achievement on the most recent test to their predicted achievement based on an earlier test.
The inner, darker ranges of each projection represent one standard deviation, or the 68% confidence interval, either side of the prediction. In other words, 68% of the time the observed gain for this student will fall within that interval. The broader, lighter ranges represent two standard deviations, or the 95% confidence interval, either side of the prediction. This interval is wider and so we expect that 95% of the time, the observed gain for this student will fall within the wider interval.
How should predicted gain be used?
The gain prediction between two successive assessments over a relatively short period should not form the basis of a value judgement about the quality of a student’s broader learning progress. Instead, these gain predictions can be used as an additional piece of information to assist you in understanding your assessment data and student achievement on PAT generally.
The predictions, as represented in the online reports, are useful in visualising the variability of assessment results, especially when looking at only two points of measurement. Students' long-term learning trajectory is always the most important consideration, and no single assessment should be relied upon as the sole measure of student achievement or learning growth. Information about students' learning should be gathered regularly and from multiple sources and considered together through the lens of your professional judgement.
'Negative' gain
In some cases, due to the relationship between student year level, prior scale score and the time interval between measurements, the study predicts that the typical student’s scale score will decline. Again, it is necessary to differentiate here between the statistical prediction and the actual, individual student. In cases where a student’s scale score does, in fact, decline between two assessments, we would not assume that their learning progress has reversed. Rather, we take it that no progress has been measured and our professional judgement is required to understand and evaluate this result within the student’s broader context, including through observations, classwork, and other formal and informal assessments.
Your knowledge of students and your professional judgement are crucial when analysing student achievement and setting suitable targets for learning progress.
Predicted scale score decline is not represented within the online reporting.
This video from the PAT core concepts series discusses student progress and PAT: