A Design for Comparing CTT and IRT in Test Assembly, Scoring, and Argumentation: Differences Among Reliability, Information, and Validation

Volume 13 Issue 2 August - October 2019

Research Paper

A Design for Comparing CTT and IRT in Test Assembly, Scoring, and Argumentation: Differences Among Reliability, Information, and Validation

Abdulelah Mohammed Alqarni*

Department of Psychology and Counselling, Faculty of Educational Graduate Studies, King Abdulaziz University, Jeddah, Saudi Arabia.

Alqarni, A. M. (2019). A Design for Comparing CTT and IRT in Test Assembly, Scoring, and Argumentation: Differences Among Reliability, Information, and Validation. i-manager’s Journal on Educational Psychology, 13(2), 1-9.

Abstract

This study compared the psychometric properties of reliability in Classical Test Theory (CTT), item information in Item Response (IRT), and validation from the perspective of modern validity theory for the purpose of bringing attention to potential issues that might exist when testing organizations use both test theories in the same testing administration. It was found that reliability, instead using corrected item-total test score correlations, and item information functions are only grossly similar and their conjoint use should be compartmentalized in the processes of test assembly, pre-testing, and scoring broadly speaking. For validity, only minor differences attributable to scoring processes are conceivable, but the main problem is that too much subjectivity by way of different arguments being constructed using the same test and test scores engenders a lack of consensus in the meaning of arguments. A checklist is presented for consideration by test validators that will produce greater consensus of arguments by improving argument comprehensiveness.