Domain 2: Psychometric Theory & Methods
General Description
Both the discovery and application of knowledge to help improve the education and quality of life for individuals in our society require precise definitions of important constructs and accurate measurements of the extent to which individuals differ in terms of those constructs. Domain 2 is concerned with the theory and methods used to measure variables that are important in psychological and educational research, practice, and evaluation. Such variables include aptitudes, achievements, attitudes, personality traits and characteristics, and other cognitive and non-cognitive characteristics.
Psychologists must be able to develop and use measuring instruments that are appropriate for specified educational and psychological purposes. To be able to do so, they must develop an understanding of measurement principles and knowledge of available measuring instruments and procedures, as well as a good working knowledge of related statistical procedures and methods of experimental design and data analysis. Auxiliary skills that also must be developed include the use of computers and related mathematics (such as statistics and matrix algebra).
Courses
Currently, the only primary course that will satisfy the Domain 2 requirement is Psychometrics: Theory and Methods. Required for enrollment in this course is knowledge equivalent to that obtained in a one-semester introduction to statistics course (EDP 371 or EDP 380E.1) and a one-semester introduction to educational and psychological testing course (such as EDP 362T or EDP 380P.1).
Numerous secondary courses satisfy the rest of the Domain 2 requirement. (It should be noted that the three courses identified as “Secondary—1 or 2” will satisfy either the Domain 1 or the Domain 2 requirement, but not both.)
Detailed statements of the contents, objectives, and assigned reading for each course can be examined in the departmental office (SZB 504) or on the course descriptions webpage. Additional information can also be obtained from the instructor who teaches the course.
| P/S | EDP # | Topic | Usual Instructor |
|---|---|---|---|
| Primary | 380P.2 | Theory and Methods | Dodd, Koch |
| Secondary 1 or 2 | 380P.4 | Evaluation Models & Techniques | Borich |
| Secondary | 380P.6 | Item Response Theory | Dodd |
| Secondary | 380P.7 | Multidimensional Scaling | Koch |
| Secondary | 380P.8 | Test and Scale Construction | Koch |
| Secondary | 380P.14 | Applied Psychometrics | Dodd |
| Secondary 1 or 2 |
382K.3 | Factor Analysis | Beretvas |
| Secondary 1 or 2 | 384.4 | Introduction to Survey Research |
Topics
Brief descriptions of the topics covered in Domain 2 courses are given below. Although redundancies occur among the topics listed in the descriptions, they have been retained in order to allow students to gain a better understanding of the relationships among some of the topics.
Theory and Methods. Topics include: Mental test theory and methods (including classical true-score model, item response theory models, test construction methods), introduction to factor analysis (including purposes and processes involved in factor analysis studies), and psychological scaling theory and procedures (including principal types of unidimensional attitude scales and multidimensional scaling procedures).
Evaluation Models and Techniques. This course traces the contemporary development of evaluation in education and human service and identifies milestones and existing directions. The relationship between research and evaluation, as well as identification of distinguishing and overlapping characteristics, is also included. Students are required to present and synthesize evaluation strategies and concepts; identify practical and theoretical models for evaluation; provide a description of evaluation methodologies; identify and apply quantitative and qualitative tools for evaluation; present critical considerations on evaluation design, criteria for judging evaluations, and measurement problems related to evaluation; and distinguish adequate from inadequate evaluation designs. Topics include: parallels and contrasts between research and evaluation, evaluation models and approaches, behavioral objectives, needs assessment techniques, program modeling and decomposition, evaluation methodology, outcome evaluation, and evaluation instruments.
Item Response Theory. Content includes background and theory (classical test theory, objectivity in mental measurement, dichotomous and polytomous models, conventional item analysis versus the item response theory approach, item parameter and ability estimation, information functions) and applications (relative efficiency of tests, flexilevel and two-stage tests, computerized adaptive tests and mastery tests, test equating, differential item functioning, and attitude and personality measurement).
Multidimensional Scaling. Course content includes background [basic scaling theory: dimensionality problems, historical antecedents of Multidimensional Scaling (MDS) (Torgerson’s and Messick’s metric procedures, Coombs’ unfolding procedure), relationship of MDS to factor and cluster analysis, monotonic relations and transformations, distance measures and coefficients, brief example of the MDS algorithm], classifications of types of MDS analyses [Shepard’s taxonomy: Young’s taxonomy (ALSCAL package)], data collection and procedures, use of ALSCAL computer program, survey of popular methods of MDS [Shepard-Kruskal’s proximities scaling: Guttman-Lingoes’ smallest space analysis (SSA), Multidimensional unfolding (MDU), Carroll-Chang’s individual differences scaling (INDSCAL)]; detailed computational procedures, dimensionality and fit (stress, alienation, etc.), interpreting the stimulus space [axes of variation: clusters of similarities, regions and facets, vector and ideal points models, external criteria (regression, canonical correlations, etc.)], and interpreting the subject space (subject weights).
Test and Scale Construction. Content includes design of affective instruments [determining objectives; instrument design and construction; issues in question writing; special item response formats (Likert-type items: Semantic Differential items); developing the sampling plan; procedures for administration; return rates; and analysis, interpretation, and reporting of results] and cognitive tests [definition of the construct and its indicants: content specifications and test blueprint (setting objectives, using appropriate taxonomic levels, and determining proper emphasis), design and construction issues, facet theory for scale construction, measurement of opinions and beliefs, tryout of experimental forms and item analysis, use of factor analysis and multidimensional scaling, selecting the standardization sample, norms and scaled scores, reliability and validity assessment, and test equating methods].
Applied Psychometrics. This course provides students an in-depth examination of applied psychometric techniques. Students have the opportunity to apply the techniques covered in the course to real data sets. Topics include equating scores; setting standards; detecting differential item functioning; and selection, placement, and classification (including bias in selection and aptitude—treatment interaction designs).
Factor Analysis. Course content includes: General introduction, factor analysis models, elementary matrix algebra, rotation of axes, vector representation; basic formulas of common factor analysis and principal components analysis; test for significance of correlation matrix, criteria for number of factors, principal-axis, image analysis, and alpha methods of factoring; rotation models, simple structure, orthogonal and oblique axes, transformation matrix, and graphical rotation; criteria for analytical orthogonal and oblique rotation (including Procrustes); interpretation of factors, factor scores, and computer programs; and confirmatory factor analysis, including higher order and hierarchical models, and tests of model fit.
Introduction to Survey Research. This course is an overview of the survey research process, addressing various topics in survey research within the context of the development of a survey study. Accordingly, topics are sequenced to follow the creation of a survey study, proceeding from conceptualization, to measurement, to administration, to data management and analysis, and, finally, to writing the research report. Issues in quasi-experimental design, sampling, questionnaire construction, administration, and the use of computers in survey research are covered within this framework. The relative advantages, disadvantages, and appropriateness of the different modes of survey administration (i.e., face-to-face interviews, telephone interviews, mail, internet surveys, and other self-administered questionnaires) for different research questions are also examined.
Domain Exam
For students who elect this avenue for fulfilling the Domain 2 requirement, the three-hour domain examination consists of two parts of equal length. The first 90-minute part is required of all students. It covers the topics contained in the primary course in Domain 2, EDP 380P.2 Psychometrics: Theory and Methods, and consists of 60 objective items (approximately 70 minutes) and one essay item (approximately 20 minutes).
In the second half of the exam, the student chooses to answer questions concerning the topics covered in one of the seven secondary courses in Domain 2. The student must specify in advance which secondary course he/she selects for the second half of the exam. The second half of the exam will consists of essay items and may or may not include objective items as well.
Each essay response is evaluated independently by two readers. Each reader then considers the scores on both the objective items and the essay items and recommends either Pass or Fail. If the Pass/Fail recommendations of the two readers differ, a third reader will evaluate the student’s responses and resolve the discrepancy. Students may not retake the exam if they do not pass: a student who fails the examination must satisfy the domain requirement by coursework.
Readings
The following listing includes texts required or recommended for the Domain 2 primary and secondary courses listed above:
Babbie, E. (1990). Survey research methods (2nd ed.). Belmont, CA: Wadsworth.
Borich, G. (1982). Programs and systems: An evaluation perspective. New York: Academic Press.
Converse, J.M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Beverly Hills, CA: Sage
Crocker, L. & Algina, J. (1986). Introduction to classical & modern test theory. Holt, Rinehart, and Winston.
Davison, M. L. (1983). Multidimensional scaling. New York: John Wiley & Sons.
DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury, Park, CA: Sage.
Embretson, S. & Reise, S. (2000). Item response theory for psychologists. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). New Jersey: Lawrence Erlbaum Associates.
Jacobs, L. C., & Chase, C. I. (1992). Developing and using tests effectively. San Francisco, CA: Jossey-Bass.
Kalton, G. (1983). Introduction to survey sampling. Beverly Hills, CA: Sage.
McDonald, R. P. (1985). Factor analysis and related methods. Hillsdale, New Jersey: Erlbaum.
McIver, J. P., & Carmines, E. G. (1981). Unidimensional scaling. Beverly Hills, CA: Sage.
Schiffman, S. S., Reynolds, M. L., & Young, F. W. (1981). Introduction to multidimensional scaling. New York: Academic Press.
Sudman, S., & Bradburn, N. M. (1983). Asking questions. San Francisco, CA: Jossey-Bass.
Weber, R. P. (1990). Basic content analysis (2nd ed.). Beverly Hills, CA: Sage.
