As part of ACER’s standard quality assurance process, we have reviewed PAT Science 2nd Edition to validate its performance in assessing students’ science understanding.
This article covers the following topics related to the validation exercise:
Background
ACER puts great effort into pre-release trialling for new assessments, but post-release reviews of live test data are essential to ensure quality and identify any improvements that will strengthen the assessments. All new or updated assessments are reviewed 12 months after release or when sufficient data are available. PAT Science 2nd Edition was released in January 2023 and has seen considerable use by schools across Australia in that time.
Review
A complete check of all available PAT Science 2nd Edition results data was undertaken by ACER’s psychometric experts to identify any irregularities, while subject matter experts reviewed the test items themselves to ensure they were sound.
Changes
The key outcomes from the review are as follows:
- Difficulty scale scores have been adjusted for 46 test items (questions) across all 8 test forms. These changes are reflected in updates to the average difficulty of the affected test forms.
- The changes in item difficulty scores also affect the calculation of students’ scale score achievement.
- Minor changes have been made to one item in Test 3 to improve clarity.
- Two items in Test 8 were found not to be performing as intended and have been removed.
As a result of the changes to these item and test difficulty scores, students’ achieved scale scores have been re-calculated in the online reports. In the vast majority of cases, students' reported scale scores have changed by no more than ±2 scale score points, if at all.
In a small number of cases, specifically where students achieve very close to 100% correct on Test 8, reported scale scores may change by more than 2 points.
Online reports generated on or after Wednesday 17 April 2024 display the updated scale scores for all students and items.
The average test difficulty scores before and after the changes were applied are shown in the table below.
|
No. items |
Average item difficulty | Standard deviation | ||||
|---|---|---|---|---|---|---|
| Test | Before | After | Before | After | Before | After |
| 1 | 25 | 25 | 110.6 | 111.2 | 7.7 | 9.1 |
| 2 | 25 | 25 | 112.8 | 112.8 | 9.1 | 8.0 |
| 3 | 30 | 30 | 116.6 | 116.6 | 9.5 | 9.6 |
| 4 | 30 | 30 | 117.8 | 117.0 | 10.1 | 10.4 |
| 5 | 35 | 35 | 124.4 | 124.1 | 7.3 | 7.5 |
| 6 | 35 | 35 | 130.2 | 129.4 | 11.6 | 10.6 |
| 7 | 40 | 40 | 131.2 | 131.2 | 8.3 | 8.9 |
| 8 | 40 | 38 | 133.8 | 133.2 | 7.2 | 7.4 |