PAT Critical Reasoning

The following topics are covered in this article:

• Assessment overview 
• Development process and validity
• When to assess 
• Administering the tests 
• Using the results 
• Acknowledgements 
• Supporting documents

Assessment overview 

PAT Critical Reasoning is a set of assessments that allow teachers to accurately and efficiently measure students’ abilities in critical reasoning, to diagnose gaps, strengths, and weaknesses in student learning, and monitor student progress over time. The assessments have been developed especially, but not exclusively, for use in Australian schools. The PAT Critical Reasoning construct is appropriate for broad international use.

PAT Critical Reasoning assesses a fundamental subset of the skills that constitute critical thinking, namely those that relate to the analysis and evaluation of ideas and arguments.  The assessments have been developed to minimise reading load so that as far as possible students are being assessed for their thinking skills rather than their higher-level reading skills. Where more technical concepts and subject specific information are introduced, they are explicitly defined so that students are being assessed for their processing of ideas rather than any prior knowledge.

The assessments target core critical reasoning skills likely to suit the ability of students from Year 5 to Year 10. All three tests could be suitable for middle or upper primary students, with higher performing students allocated the more challenging assessments, based on teacher judgement. Likewise, all three tests could be suitable for lower or middle secondary students.

Development process and validity

The PAT Critical Reasoning tests are planned and constructed to assess critical reasoning skills through tasks that require skills in Conceptual reasoning, Basic logic, and Argument analysis. In constructing the tests, care was taken to include a range of these skill areas to ensure that the breadth of students’ critical reasoning abilities are captured. All items were subjected to intensive scrutiny, review and revision by panels of experts and psychometricians.

The items in the assessment were developed by experienced test developers who review and panel the items until they are ready for trial. A rigorous process of quality checking, proofreading and formatting then takes place. The psychometric team provide a trial design based on the items (number, distribution of strand and item type) to ensure that the most valid and reliable data is available from the trial. PAT Critical Reasoning items were trialled in standalone trial test forms, with items offered across a range of year levels to determine the appropriate targeting and difficulty for each item.

The calibration procedures identified items that appeared to be measuring skills other than those measured by the other items at trial. Items ‘misfitting’ in this way were not retained. The items retained for PAT Critical Reasoning were shown to fit the Rasch measurement model satisfactorily. All items retained could be regarded as measuring a student’s location on a single underlying continuum of critical reasoning skills.

When to assess 

PAT assessments provide measures of attainment and progress that also describe the skills, knowledge and abilities required to progress over time. Most schools administer their PAT assessments towards the end of each school year to identify the skills students have attained, to identify specific areas where students need support in their learning and to measure growth from the previous year. Some schools choose to use PAT at the beginning of the year, placing a greater emphasis on teachers acting on the diagnostic information. There are also schools that use PAT both at the beginning and end of the year in order to maximise the diagnostic information available to monitor growth throughout the year.

ACER strongly recommends that schools are clear on their purpose for using PAT assessments and carefully plan their assessment approach, especially if an aim is the monitoring of student learning progress over time.

For the purpose of monitoring student progress, a gap of 9 to 12 months between testing sessions is recommended. Learning progress may not be reflected in a student’s scale scores over a shorter period of time. Longitudinal growth should be measured over a minimum of two years of schooling, or three separate testing sessions, in most contexts. This will help account for possible scale score variation, for example where external factors may affect a student’s performance on a particular testing occasion.

Administering the tests 

Schools should take care in planning the administration of tests to ensure consistency. PAT Critical Reasoning is a standardised assessment that should be completed in 35 minutes. Time must be managed by the teacher invigilating the assessment. Students will not be automatically timed out by the online system.

Students’ focus and energy levels are important factors in their capacity to accurately demonstrate their ability on the assessment. For this reason, it is generally best to test students in the morning and not immediately before or after an exciting school event.

Preparation

The following steps need to be completed ahead of time. For more detailed instructions on how to complete each step, please refer to the Help pages within your school’s online account, or speak to the person responsible for managing your school’s account.

• Check the technical requirements and run the browser exam from a student device to identify any potential technical issues.
• Schedule your testing date and time. ACER recommends that tests are administered in the morning and not immediately before or after an exciting school event.
• Ensure that all students are listed within your school’s online assessment account and have been assigned the necessary tests.
• Download or print a list of your students’ login details from the Students page within your school’s online assessment account.
• Make note of your school’s online assessment login page, or make sure that the URL is saved on student devices, or available to your students as a link. The address will be similar to https://oars.acer.edu.au/your-school-name.

Administration instructions

 PAT Critical Reasoning test administration instructions

• Students are permitted 35 minutes to complete the test.
• PAT Critical Reasoning should be administered under standard testing conditions with invigilation.
• Students’ screens should be monitored as part of test invigilation.
• Students are permitted to use pen/pencil and paper to make notes during the test.

All three tests could be suitable for middle or upper primary students, with higher performing students allocated the more challenging assessments, based on teacher judgement. Likewise, all three tests could be suitable for lower or middle secondary students.

To make decisions about which test is most appropriate for a particular student or group of students, it is essential that the teacher previews and becomes familiar with the content of the tests.

Delivery

• Students will not be automatically locked out of the tests after the allowed time passes. You must monitor and manage the time, including accommodating toilet breaks or other interruptions that may occur.
• Student responses are automatically saved each time they navigate to another question.
• If technical problems cause the need to postpone the completion of the tests, students may close the browser without losing their progress.

iPads and tablet devices must be held in landscape orientation.

Note

If you determine that some students require changed testing conditions due to specific learning needs, these changes should be recorded for future reference. The process for determining and implementing any changes to test conditions should be consistent between classes and across the school.

Using the results 

PAT scale score

A PAT scale score is a numerical value given to a student whose achievement has been measured by completing a PAT assessment. A student’s scale score lies at a point somewhere on the specific PAT scale, and it indicates that student’s level of achievement in that particular learning area — the higher the scale score, the more able the student.

Regardless of the test level or items administered to students, they will be placed on the same scale for the learning area. This makes it possible to directly compare students’ achievement and to observe students’ progress within a learning area by comparing their scale scores from multiple testing periods over time.

A score on the PAT Critical Reasoning scale has no meaning on the PAT Reading scale or any other PAT scale. The units of the scale have different meanings for each scale.

Achievement bands 

Students in the same achievement band are operating at approximately the same achievement level within a learning area regardless of their school year level.

Viewing student achievement in terms of achievement bands may assist teachers to group students of similar abilities. By referencing the PAT achievement band descriptions, teachers can understand the types of skills typical of students according to their PAT band.

A PAT Critical Reasoning scale score of 120 could be considered to be at the upper end of achievement band 110–119 or at the lower end of achievement band 120–129. In cases like these, it is important to reference the descriptions of both achievement bands to understand the student’s abilities.