The generalizability theory as an approach to estimating the reliability and dependability of the academic language placement test

Date of Publication

1997

Document Type

Dissertation

Degree Name

Doctor of Arts in Language and Literature Major in English

Subject Categories

English Language and Literature

College

College of Liberal Arts

Department/Unit

Literature

Thesis Adviser

Emma Castillo

Defense Panel Chair

Ma. Lourdes Bautista

Defense Panel Member

Richard Gonzales
Glenda E. Fortez
Gundelina A. Velasco
Alexa P. Abrenica

Abstract/Summary

A particular approach that brings about more informative results than classical test theory is the Generalizability theory. Generalizability theory (referred to in this dissertation as G theory), an approach developed by Cronbach, Gleser, Nanda, and Ranjaratnam (1972), uses multiple factorial ANOVA as its basic formula to 1) identify the likely sources of error variance, 2) determine the magnitude of each source, and 3) provide information on the relationship these sources have with each other. This study aims to find differences between the power of Classical True Score Measurement Theory and Generalizability Theory in estimating the reliability of the Academic Language Placement Test (ALPT).Around 1,000 freshmen students from De La Salle University participated in the study by taking the ALPT. ALPT was a researcher-designed/adapted battery of tests which consisted of three timed-subtests: Reading, Listening, and Writing. Test results were presented with descriptive statistics, and analyzed using classical test theory and G theory approaches to test reliability.The analysis showed that G theory was more powerful than classical test theory in that it was able to provide estimates of the a) magnitude of multiple sources of variance, b) relationship between the different sources of variance, and c) scores through an analysis of combination of facets. Possible modifications in the number and combinations of facets that may bring about more dependable scores was also determined through G-study analyses which, in turn, provided insights on alternative designs for collecting large-scale assessment data for a D-study.

Findings in the ALPT showed that sources of error variance can be determined through various G-study designs. For the purpose of this study, Reading and Listening Tests were analyzed using 'px[i:s]' design, where 'p' is person facet, 'i' for items, and 's' for subtest. Results showed that the largest component of the variance for Reading and Listening test was due to the subtest facet. For the writing test a 'pxtxr' G-study design was used to analyze test results. The largest component of the error variance was found to be due to the interaction effect between rater and task. Further, the study showed the relationships between the facets and their interaction with each other in the G-study designs. Such relationships led to further analyses of possible combinations of facets which were illustrated to have marked effects on the magnitude of the sources of error of variance and to the G-coefficient as a whole. The study also showed how G-study results could help estimate the likely results of Decision-study of D-study using the same test results.In conclusion, after analyzing ALPT test results through both Classical Test theory and Generalizability theory, the G theory was able to provide the researcher with more information about the likely sources of error variance in the test. The results helped determine the contributions made by each facet and their combined interaction which, when appropriately manipulated, could obtain variance component magnitudes and G-coefficients that may be acceptable at a particular test situation. Finally, the G-study designs allowed to obtain insight as to how the ALPT couldb improved for future administration or D-study purposes.

Abstract Format

html

Language

English

Format

Print

Accession Number

TG02674

Shelf Location

Archives, The Learning Commons, 12F Henry Sy Sr. Hall

Physical Description

124 numb. leaves

Keywords

Examinations -- Design and construction; Educational tests and measurements; Language and languages -- Examinations

This document is currently not available here.

Share

COinS