The Program of Systematic Evaluation (PSE) 1965-2007

Francis Dwyer
College of Education
The Pennsylvanian State University
University Park, PA. 16802

Abstract

The Program of Systematic Evaluation (PSE) which began as an attempt to define the instructional characteristics of different types of visual illustrations has evolved into the most comprehensive set of experimental studies exploring the varied dimensions of cognitive load theory as they relate to the design of effective and efficient visual learning environments for different types of educational objectives. To date more than 160 experimental studies, 130 research associates, and over 100,000 participants have been involved in the PSE. Research studies conducted in the PSE have resulted in more than 175 refereed publications in scholarly journals, 100 doctoral dissertations, and more than 300 refereed presentations at the state, national and international levels. It should be noted that more than 95% of the doctoral dissertation research has been published in refereed journals at the national and international levels. The unique feature of the PSE is that all studies utilized the same instructional unit and four individual criterion measures while systematically examining the effectiveness of different independent variables.

Introduction

We live in a visually oriented society; consequently, visualization has become an integral part of our communication and educational processes. The use of visualized materials integrated with regular instruction has become a common instructional strategy at all levels of education extending from preschool activities through graduate school and also into adult in-service training and development programs. The purpose of the Program of Systematic Evaluation (PSE) initially was to examine the instructional effectiveness of different types of visualization used to complement print oral instruction in facilitating student achievement of different types of educational objectives. However, over time its purpose was expanded to measure the effects of visualization (a) presented via different delivery systems (text, slide-audiotape, programmed instruction, television, computer based, web based, etc.), (b) when employed to examine their effect on different types of personological variables (I.Q., prior knowledge, learning styles, reading ability, etc.), and (c) when different types of learner-content interaction strategies are used to complement print/oral instruction (cueing, visual/verbal feedback, scaffolding, questions, animation, etc.).

Overview

Since its inception in 1965 at The Pennsylvania State University the PSE has initiated over 160 experimental studies involving more than 130 research associates and 100,000 participants. Research studies implemented in the PSE have resulted in more than 175 scholarly publications in national and international journals and more than 300 refereed presentations at the state, regional, national and international levels. Additionally, over 100 doctoral dissertations have been completed at The Pennsylvania State University, Ohio State University, Texas A&M, University of Washington, University of Pittsburgh, Virginia Polytechnic Institute and State University, etc., utilizing the experimental materials developed in the PSE. Unique to each of the studies is the fact that each study used the same instructional module and four individual criterion measures designed to measure different educational objectives. The instructional/evaluation materials developed in the PSE have been translated into Arabic, Spanish, Chinese, and Korean and experimental research has been conducted in four countries.

The PSE was initiated at a time when there was widespread acceptance and use of visual materials for instructional purposes. Although there is research and theory supporting the use of visualization as a viable instructional variable in the learning process, the prevalent notion at that time seemed to indicate that one type of visualization was as good as another in facilitating learner achievement of different educational objectives. This fact was evidenced by the large number of experimental studies reviewed by Stickell (1963), Chu & Schramm (1967), and MacLennan & Reid (1967), that indicated that the use of visually mediated instruction in many cases resulted in no significant increases in achievement when compared with conventional types of instruction. Early in the PSE a comprehensive examination of the visually based research revealed that much of the research, in addition to suffering from the many threats to internal validity identified by Campbell and Stanley (1963), had additional faults. These problems tended to further complicate data interpretation and frustrated any attempts to derive broad generalizations useful to practitioners in the classroom. Following is a sampling of the types of problems found in many of the experimental studies:

lack of hypotheses or predictions based on theory
use of content that contained only one type of learning objective
use of content far removed from what is typically taught in the classroom
use of one global test when a variety of different types of learning objective were being assessed
failure to describe specifically the type of educational objectives to be achieved by learners
failure to describe specifically the type of visualization used in the study or how it was used
failure to specify how the visualization (and other independent variables) were positioned into the instructional sequence
failure to implement pilot studies
lack of a control treatment
inadequate experimental designs
lack of reported reliability coefficients for the assessment instruments
failure to utilize item analyses to identify acquisition difficulties
failure to match objective level with type of instruction and evaluation (instructional congruency/consistency)

Program of Systematic Evaluation

In an attempt to alleviate many of the problems associated with prior research, a generic instructional unit focusing of the physiology and functions of the human heart was developed. Development of this unit and the four individual criterion tests was based on the principles inherent in the instructional consistency/congruency paradigm which facilitate, through pilot testing and item analyses, the identification of locations within the instruction where students were having trouble acquiring information from the verbal/audio instruction. These locations provided the rationale for positioning redundant visualization in early studies, and the same procedure was employed for positioning interactive independent variables to improve the effectiveness of visualized instruction in later studies. The four criterion measures were designed to assess learner achievement of different types of educational objectives. Achievement obtained on the four 20-item individual tests were combined into a total criterion measure. Average reliability coefficients (KR-20) derived from a sampling of studies yielded consistency of results: .81 drawing test, .79 identification test, .82 terminology test, .76 comprehension test and .91 total criterion test. Each experimental study in the PSE utilized a theoretical justification, hypotheses, research questions, a pilot study utilizing item analyses for the positioning of independent variables, and a valid experimental design with appropriate numbers of participants.

In general, the research conducted in the PSE examining the varied dimensions of cognitive load theory as it relates to information acquisition, retention, and retrieval has indicated that effectiveness and efficiency in visualized instruction are primarily dependent upon (a) the amount of realistic detail contained in the visualization, (b) the method by which the visualization is presented to learners (eternally paced versus self paced), (c) learner characteristics, i.e., intelligence, learning style, prior knowledge in the content area, reading and/or oral comprehension level, etc., (d) the type of educational objectives to be achieved by the learners, (e) the technique(s) used to focus learner attention on the designated instructional content in the visualized materials, e.g., cues such as questions, arrows, animation, verbal/visual feedback, overt/covert rehearsal, etc., and (f) the type of test format employed (visual/verbal) to assess learner information acquisition. Following is a sampling of specific conclusions obtained in the PSE.

Research has shown that systematically designed and developed visualization properly positioned into instructional sequences can significantly improve learner information acquisition, retention and retrieval.
The use of specifically designed visualization to complement oral and printed instruction does not automatically improve achievement. The design of the visualization has to be consistent with the type of objective to be achieved, positioned properly in the instructional sequence, and assessed by the appropriate test format.
Identical types of visualizations are not equally effective when used for externally paced and self-paced instruction. The effectiveness of a particular type of visual in promoting student achievement depends on the amount of time students are permitted to interact with the visualized instruction.
Specific types of visualization properly designed and positioned can significantly improve learner achievement from verbal/oral instruction. However, to maximize learner achievement of higher order learning objectives, additional independent variables need to be used to complement the visualization.
For students in different grade levels, the same types of visualization are not always effective. Students’ ability to profit from visualized instruction is related to their intelligence, reading comprehension level, learning style, and background knowledge in the content area.
For specific types of students and for specific educational objectives, the use of color in certain types of visuals facilitates student achievement.
Student perceptions of the instructional value of different types of visualization are not valid assessments of their instructional effectiveness; that is, esthetically pleasing visuals may not be of significant instructional value.
The realism continuum for visual illustrations is not an effective predictor of learning. An increase in the amount of realistic detail contained in an illustration will not necessarily produce a corresponding increase in the amount of information assimilated.
Merely increasing the size of instructional illustrations by projecting them on larger viewing areas does not automatically improve their effectiveness.
The effectiveness of animated instruction can be significantly improved with the use of specific types of interaction strategies used to complement the animation.

Conclusion

The results of studies conducted in the PSE have contributed significantly to the knowledge base of instructional systems and communication research. Additionally, research results have yielded important findings regarding the psychological factors contributing to learning such as motivation, memory, cognitive learning strategies and testing/evaluation strategies. These materials have been very effective in evaluating specific internal and external conditions affecting learning behavior, instructional design strategies, and presentation methodologies. The studies to date conducted in the PSE represent the most comprehensive and systematic attempt to identify those variables associated with visualized instruction. A complete listing of the research associates of the PSE along with a listing of dissertations, publications, and presentations generated in the PSE along with a more complete description of the PSE can be found at http://www.ed.psu.edu/dwyer/index.htm

Reference

Campbell, Donald and Stanley, Julian. Experimental and Quasi-Experimental Designs for Research. In N.L. Gage (Eds.), Handbook of Research on Teaching. Chicago: Rand McNally and Company, 1963, pp 171-246.

Chu, George and Schramm, William. Learning From Television: What the Research Says. Washington D.C. National Association of Educational Broadcasters, 1967.

MacLennan, Donald and Reid, John. Research in Instructional Television and Film. Washington, D.C. U.S. Office of Education, 1967.

Stickell, David. A Critical Review of the Methodology and Results of Research Comparing Televised and Face-to Face Instruction. Doctoral Dissertation. The Pennsylvania State University.