The following topics are covered in this article:
- Assessment overview
- Development process and validity
- When to assess
- Administering the tests
- Using the results
- Supporting documents
PAT Critical Reasoning is a set of assessments that allow teachers to accurately and efficiently measure students’ abilities in critical reasoning, to diagnose gaps, strengths, and weaknesses in student learning, and monitor student progress over time. The assessments have been developed especially, but not exclusively, for use in Australian schools. The PAT Critical Reasoning construct is appropriate for broad international use.
PAT Critical Reasoning assesses a fundamental subset of the skills that constitute critical thinking, namely those that relate to the analysis and evaluation of ideas and arguments. The assessments have been developed to minimise reading load so that as far as possible students are being assessed for their thinking skills rather than their higher-level reading skills. Where more technical concepts and subject specific information are introduced, they are explicitly defined so that students are being assessed for their processing of ideas rather than any prior knowledge.
The assessments target core critical reasoning skills likely to suit the ability of students from Year 5 to Year 10. All three tests could be suitable for middle or upper primary students, with higher performing students allocated the more challenging assessments, based on teacher judgement. Likewise, all three tests could be suitable for lower or middle secondary students.
Development process and validity
The PAT Critical Reasoning tests are planned and constructed to assess critical reasoning skills through tasks that require skills in Conceptual reasoning, Basic logic, and Argument analysis. In constructing the tests, care was taken to include a range of these skill areas to ensure that the breadth of students’ critical reasoning abilities are captured. All items were subjected to intensive scrutiny, review and revision by panels of experts and psychometricians.
The items in the assessment were developed by experienced test developers who review and panel the items until they are ready for trial. A rigorous process of quality checking, proofreading and formatting then takes place. The psychometric team provide a trial design based on the items (number, distribution of strand and item type) to ensure that the most valid and reliable data is available from the trial. PAT Critical Reasoning items were trialled in standalone trial test forms, with items offered across a range of year levels to determine the appropriate targeting and difficulty for each item.
The calibration procedures identified items that appeared to be measuring skills other than those measured by the other items at trial. Items ‘misfitting’ in this way were not retained. The items retained for PAT Critical Reasoning were shown to fit the Rasch measurement model satisfactorily. All items retained could be regarded as measuring a student’s location on a single underlying continuum of critical reasoning skills.
When to assess
PAT assessments provide measures of attainment and progress that also describe the skills, knowledge and abilities required to progress over time. Most schools administer their PAT assessments towards the end of each school year to identify the skills students have attained, to identify specific areas where students need support in their learning and to measure growth from the previous year. Some schools choose to use PAT at the beginning of the year, placing a greater emphasis on teachers acting on the diagnostic information. There are also schools that use PAT both at the beginning and end of the year in order to maximise the diagnostic information available to monitor growth throughout the year.
ACER strongly recommends that schools are clear on their purpose for using PAT assessments and carefully plan their assessment approach, especially if an aim is the monitoring of student learning progress over time.
For the purpose of monitoring student progress, a gap of 9 to 12 months between testing sessions is recommended. Learning progress may not be reflected in a student’s scale scores over a shorter period of time. Longitudinal growth should be measured over a minimum of two years of schooling, or three separate testing sessions, in most contexts. This will help account for possible scale score variation, for example where external factors may affect a student’s performance on a particular testing occasion.
Administering the tests
Schools should take care in planning the administration of tests to ensure consistency. PAT Critical Reasoning is a standardised assessment that should be completed in 35 minutes. Time must be managed by the teacher invigilating the assessment. Students will not be automatically timed out by the online system.
Students’ focus and energy levels are important factors in their capacity to accurately demonstrate their ability on the assessment. For this reason, it is generally best to test students in the morning and not immediately before or after an exciting school event.
The following steps need to be completed ahead of time. For more detailed instructions on how to complete each step, please refer to the Help pages within your school’s online account, or speak to the person responsible for managing your school’s account.
- Check the technical requirements and run the browser exam from a student device to identify any potential technical issues.
- Schedule your testing date and time. ACER recommends that tests are administered in the morning and not immediately before or after an exciting school event.
- Ensure that all students are listed within your school’s online assessment account and have been assigned the necessary tests.
- Download or print a list of your students’ login details from the Students page within your school’s online assessment account.
- Make note of your school’s online assessment login page, or make sure that the URL is saved on student devices, or available to your students as a link. The address will be similar to https://oars.acer.edu.au/your-school-name.
PAT Critical Reasoning test administration instructions
- Students are permitted 35 minutes to complete the test.
- PAT Critical Reasoning should be administered under standard testing conditions with invigilation.
- Students’ screens should be monitored as part of test invigilation.
- Students are permitted to use pen/pencil and paper to make notes during the test.
|Generally suitable for
|No. of items
|Year 5, Year 6
|Year 7, Year 8
|Year 9, Year 10
All three tests could be suitable for middle or upper primary students, with higher performing students allocated the more challenging assessments, based on teacher judgement. Likewise, all three tests could be suitable for lower or middle secondary students.
To make decisions about which test is most appropriate for a particular student or group of students, it is essential that the teacher previews and becomes familiar with the content of the tests.
- Students will not be automatically locked out of the tests after the allowed time passes. You must monitor and manage the time, including accommodating toilet breaks or other interruptions that may occur.
- Student responses are automatically saved each time they navigate to another question.
- If technical problems cause the need to postpone the completion of the tests, students may close the browser without losing their progress.
iPads and tablet devices must be held in landscape orientation.
If you determine that some students require changed testing conditions due to specific learning needs, these changes should be recorded for future reference. The process for determining and implementing any changes to test conditions should be consistent between classes and across the school.
Using the results
- PAT scale score
- A PAT scale score is a numerical value given to a student whose achievement has been measured by completing a PAT assessment. A student’s scale score lies at a point somewhere on the specific PAT scale, and it indicates that student’s level of achievement in that particular learning area — the higher the scale score, the more able the student.
- Regardless of the test level or items administered to students, they will be placed on the same scale for the learning area. This makes it possible to directly compare students’ achievement and to observe students’ progress within a learning area by comparing their scale scores from multiple testing periods over time.
- A score on the PAT Critical Reasoning scale has no meaning on the PAT Reading scale or any other PAT scale. The units of the scale have different meanings for each scale.
- Achievement bands
- Students in the same achievement band are operating at approximately the same achievement level within a learning area regardless of their school year level.
- Viewing student achievement in terms of achievement bands may assist teachers to group students of similar abilities. By referencing the PAT achievement band descriptions, teachers can understand the types of skills typical of students according to their PAT band.
- A PAT Critical Reasoning scale score of 120 could be considered to be at the upper end of achievement band 110–119 or at the lower end of achievement band 120–129. In cases like these, it is important to reference the descriptions of both achievement bands to understand the student’s abilities.
- Item difficulty
- Item difficulty is a measure of the extent of skills and knowledge required to be successful on the item. This makes it possible to allocate each PAT Critical Reasoning test item a score on the same scale used to measure student achievement. An item with a high scale score is more difficult for students to answer correctly than a question with a low scale score. It could generally be expected that a student is able to successfully respond to more items located below their scale score than above.
- By referencing the difficulty of an item, or a group of items, and the proportion of correct responses by a student or within a group, it may be possible to identify particular items, or types of items, that have challenged students.
Some complex PAT Critical Reasoning items allow students to achieve 'partial credit' for their responses, so for example, different students may score 0, 1, 2, 3 or 4 points on the same item. Full credit suggests a more substantial demonstration of the skill, where partial credit suggests consolidation is still taking place.
The Group Report displays question-level information about student achievement.
To accommodate partial credit scoring, the Group Report for PAT Critical Reasoning differs slightly from that of other online PAT assessments.
A summary of the key features of the PAT Critical Reasoning Group Report is included below.
Click the images to view full-size.
For each question in PAT Critical Reasoning, the Group Report displays the following information:
- Question difficulty
- Expressed as a score on the PAT Critical Reasoning scale.
For partial-score questions, this difficulty is expressed as the average scale score difficulty of a partially or fully correct response.
- Identifies the strand being assessed by each question (Conceptual reasoning, Basic logic, or Argument analysis)
- Question number
- The chronological order of appearance for each question within the test
- Average question score
- The average score achieved for each question by the students in the report
- For single-score questions, the average score will always be between 0 and 1 and may be considered as representing the 'percentage correct within the group' for those questions.
- For partial-score questions, this average score will be between 0 and 4.
- Maximum question score
- The maximum achievable score for each question.
Each test includes several partial-score questions for which students may score a maximum of either 2 or 4.
The Group Report displays students' responses to each question in the following ways:
- Fully correct
- Green box with score achieved ('1' for single-score questions, '2' or '4' for partial-score questions)
- Partially correct
- Faded green box with '1', '2' or '3' (only applicable for partial-score questions)
- Pink square
- For simple multiple-choice questions, student's responses (A, B, C, D) will be displayed.
- For complex multiple-choice, hotspot, or calculator questions, a score of '0' will be displayed.
- Not answered
- White box with 'NA'
Test administration instructions
Instructions for teachers and staff members responsible for administering and invigilating PAT Critical Reasoning tests.
Strands and achievement band descriptions
Descriptions of the strands assessed by tests on the PAT Critical Reasoning scale and the typical skills associated with each achievement band.
Detailed overview of the PAT Critical Reasoning framework, including the assessment rationale, construct, design and reporting.
PAT Critical Reasoning is a newly developed assessment, released in Term 4, 2021.
Assessment items were written in 2017, and trialled in 2018.
Dan Vine was the lead test developer, with the other members of the test development team including Jude Alexander, Prue Anderson, Melissa Hughes and Doug McCurry.
Kathy He was the lead Research Officer. Declan Weeding and Sarah Kreibich also contributed to the early trial development, as well as the test authoring and development of scoring rules.
Eunjung Lee was the lead for the psychometrics and reporting, with Clare Ozolins responsible for psychometrics across all PAT products.
ACER Creative Services created item templates and artwork.