Computerized Adaptive Testing From Concept to Implementation

Used worldwide in assessment and professional certification contexts, computerized adaptive testing (CAT) offers a powerful means to measure individual differences or make classifications. This authoritative work from CAT pioneer David J. Weiss and Alper Şahin provides a complete how-to guide for planning and implementing an effective CAT to create a test unique to each person in real time. The book reviews the history of CAT and the basics of item response theory used in CAT. It walks the reader through developing an item bank, pretesting and linking items, selecting required CAT options, and using simulations to design a CAT. Available software for CAT delivery is described, including links to free and commercial options. Engaging multidisciplinary examples illustrate applications of CAT for measuring ability, achievement, proficiency, personality, attitudes, perceptions, patients’ reports of their symptoms, and academic or clinical progress.
 

David J. Weiss, PhD, is Professor of Psychology at the University of Minnesota. Widely regarded as the father of computerized adaptive testing (CAT), he began his CAT research and founded the Quantitative and Psychometric Methods program at the University of Minnesota in 1970. His subsequent research provided the basis for the development of the CAT version of the Armed Services Vocational Aptitude Battery and a variety of other CAT implementations, including in education, nursing licensure, and personality measurement. Dr. Weiss has been active in CAT research continuously and has trained many leading CAT researchers and implementers. He was the founding editor of the journal Applied Psychological Measurement, and is co-founder and permanent President Emeritus of the International Association for Computerized Adaptive Testing (IACAT). He was the founding editor of IACAT’s official journal, the Journal of Computerized Adaptive Testing. Dr. Weiss also co-founded Assessment Systems Corporation and the Insurance Testing Corporation. He is a Fellow of the American Educational Research Association and the American Psychological Association.
 
Alper Şahin, PhD, is Chairperson of the Department of Basic English at Atilim University in Ankara, Türkiye, and Director of Education for the International Association for Computerized Adaptive Testing. He conducted his postdoctoral research on computerized adaptive testing (CAT) with David J. Weiss at the University of Minnesota. Dr. Sahin’s research interests include sample size requirements in item parameter estimation using item response theory, measurement accuracy in foreign language tests, use of asymmetric rubrics in performance assessment, and item-bank design features in CAT applications.
 

I. The Why, What, and How of CAT
1. Why CAT?
- What is CAT?
- Benefits of CAT
- Challenges of CAT
- When is CAT Feasible and When is it Not?
2. The Birth of Adaptive Testing and Its Current Status
- Binet’s Adaptive Test
- The Evolution of Adaptive Testing into CAT
- Early Approaches to CAT
- The Current Status of Applications of CAT
3. Item Response Theory Used in CAT
- Item Response Functions for Dichotomously Scored Items
- Option Response Functions for Polytomously Scored Items
- Information Functions
- Measuring People with IRT
- Estimating IRT Item Parameters
- Summary
4. Using CAT to Measure Individual Differences
- The Components of a Measurement CAT
- Putting it All Together: How CAT Works
- Some Properties of Fully Adaptive IRT-Based CATs
- Advanced Issues
5. CAT for Classification and Classification Plus Measurement
- Setting the Cutscore
- Types of Classification Applications
- Advanced Issues
II. Getting Your CAT Operational
6. Feasibility and Planning
- Developing a Business Case
- The Item Bank
- Costs of Test Delivery
- Potential Risks
- Non-Psychometric Risks
- Additional Considerations
- Conclusions
7. Developing a CAT Item Bank: Writing and Evaluating Items
- How Many Items Do I Need in My Bank?
- What Kinds of Items Work for CAT?
- Developing High-Quality Items and Item Banks
- Characteristics of a Good Item Bank
- Further Information
8. Pretesting and Linking Your Items, and Adding New Items to an Existing Bank
- Pretesting Your Items
- Creating Your Data Matrix
- IRT Item (and Person) Parameter Linking
- Additional Considerations in Item Bank Development
- Advanced Topic: Automated Item Generation
9. Using Simulations to Design a CAT
- Post-Hoc or Real-Data Simulation
- Designing a CAT from a CAT Item Bank
- Hybrid Simulations
- Conclusions
10. Implementing Your CAT
- CAT Delivery Software
- How Can I Get CAT Operational for My Organization?
III. Applications, Special Types of CATs, Special Issues, and a Potential Future for CAT
11. Examples of Operational CAT Programs
- CAT in Education
- CAT for Military Accessioning
- CAT for Pre-Employment Testing
- CAT for Certification and Licensure
- CAT in Healthcare
- Additional Applications of CAT
12. Other Types of CATs
- Multistage Tests
- CAT for Cognitive Diagnosis
- CAT for Measuring Multiple Variables
13. Special Issues
- Constrained CATs
- CAT Using Items with Multiple Questions Per Stimulus
- CAT Administration Issues
- The Examinee Experience
14. Adaptive Measurement of Change: Evaluating the Psychometric Significance of Measured Individual Change
- Identifying Significant Individual Change Using an Early Adaptive Measurement of Change (AMC) Method
- The Likelihood Ratio Index (LRI)
- Example AMC Applications
- Multivariate (Profile) Change
- AMC Research
- Discussion and Conclusions
15. Some Pathways Toward Future CATs and CAT Research
- Item Banking
- Test Delivery
- Supplementing and Improving Scores
- Integration of Testing and Instruction
- A Path for Future CAT Research: Does CAT Improve Validity?
- Leveraging Current and Future Technology to Improve Testing
Software Appendices
1. Software Packages for Estimating IRT Parameters
2. Software Packages for CAT Simulations
3. Software Platforms for CAT Delivery
References
Index