The Program of Systematic Evaluation (PSE) 1965-2007

Francis Dwyer
College of Education
The Pennsylvanian State University
University Park, PA. 16802


The Program of Systematic Evaluation (PSE) which began as an attempt to define the instructional characteristics of different types of visual illustrations has evolved into the most comprehensive set of experimental studies exploring the varied dimensions of cognitive load theory as they relate to the design of effective and efficient visual learning environments for different types of educational objectives. To date more than 160 experimental studies, 130 research associates, and over 100,000 participants have been involved in the PSE. Research studies conducted in the PSE have resulted in more than 175 refereed publications in scholarly journals, 100 doctoral dissertations, and more than 300 refereed presentations at the state, national and international levels. It should be noted that more than 95% of the doctoral dissertation research has been published in refereed journals at the national and international levels. The unique feature of the PSE is that all studies utilized the same instructional unit and four individual criterion measures while systematically examining the effectiveness of different independent variables.


We live in a visually oriented society; consequently, visualization has become an integral part of our communication and educational processes. The use of visualized materials integrated with regular instruction has become a common instructional strategy at all levels of education extending from preschool activities through graduate school and also into adult in-service training and development programs. The purpose of the Program of Systematic Evaluation (PSE) initially was to examine the instructional effectiveness of different types of visualization used to complement print oral instruction in facilitating student achievement of different types of educational objectives. However, over time its purpose was expanded to measure the effects of visualization (a) presented via different delivery systems (text, slide-audiotape, programmed instruction, television, computer based, web based, etc.), (b) when employed to examine their effect on different types of personological variables (I.Q., prior knowledge, learning styles, reading ability, etc.), and (c) when different types of learner-content interaction strategies are used to complement print/oral instruction (cueing, visual/verbal feedback, scaffolding, questions, animation, etc.).


Since its inception in 1965 at The Pennsylvania State University the PSE has initiated over 160 experimental studies involving more than 130 research associates and 100,000 participants. Research studies implemented in the PSE have resulted in more than 175 scholarly publications in national and international journals and more than 300 refereed presentations at the state, regional, national and international levels. Additionally, over 100 doctoral dissertations have been completed at The Pennsylvania State University, Ohio State University, Texas A&M, University of Washington, University of Pittsburgh, Virginia Polytechnic Institute and State University, etc., utilizing the experimental materials developed in the PSE. Unique to each of the studies is the fact that each study used the same instructional module and four individual criterion measures designed to measure different educational objectives. The instructional/evaluation materials developed in the PSE have been translated into Arabic, Spanish, Chinese, and Korean and experimental research has been conducted in four countries.

The PSE was initiated at a time when there was widespread acceptance and use of visual materials for instructional purposes. Although there is research and theory supporting the use of visualization as a viable instructional variable in the learning process, the prevalent notion at that time seemed to indicate that one type of visualization was as good as another in facilitating learner achievement of different educational objectives. This fact was evidenced by the large number of experimental studies reviewed by Stickell (1963), Chu & Schramm (1967), and MacLennan & Reid (1967), that indicated that the use of visually mediated instruction in many cases resulted in no significant increases in achievement when compared with conventional types of instruction. Early in the PSE a comprehensive examination of the visually based research revealed that much of the research, in addition to suffering from the many threats to internal validity identified by Campbell and Stanley (1963), had additional faults. These problems tended to further complicate data interpretation and frustrated any attempts to derive broad generalizations useful to practitioners in the classroom. Following is a sampling of the types of problems found in many of the experimental studies:

back to top

Program of Systematic Evaluation

In an attempt to alleviate many of the problems associated with prior research, a generic instructional unit focusing of the physiology and functions of the human heart was developed. Development of this unit and the four individual criterion tests was based on the principles inherent in the instructional consistency/congruency paradigm which facilitate, through pilot testing and item analyses, the identification of locations within the instruction where students were having trouble acquiring information from the verbal/audio instruction. These locations provided the rationale for positioning redundant visualization in early studies, and the same procedure was employed for positioning interactive independent variables to improve the effectiveness of visualized instruction in later studies. The four criterion measures were designed to assess learner achievement of different types of educational objectives. Achievement obtained on the four 20-item individual tests were combined into a total criterion measure. Average reliability coefficients (KR-20) derived from a sampling of studies yielded consistency of results: .81 drawing test, .79 identification test, .82 terminology test, .76 comprehension test and .91 total criterion test. Each experimental study in the PSE utilized a theoretical justification, hypotheses, research questions, a pilot study utilizing item analyses for the positioning of independent variables, and a valid experimental design with appropriate numbers of participants.

In general, the research conducted in the PSE examining the varied dimensions of cognitive load theory as it relates to information acquisition, retention, and retrieval has indicated that effectiveness and efficiency in visualized instruction are primarily dependent upon (a) the amount of realistic detail contained in the visualization, (b) the method by which the visualization is presented to learners (eternally paced versus self paced), (c) learner characteristics, i.e., intelligence, learning style, prior knowledge in the content area, reading and/or oral comprehension level, etc., (d) the type of educational objectives to be achieved by the learners, (e) the technique(s) used to focus learner attention on the designated instructional content in the visualized materials, e.g., cues such as questions, arrows, animation, verbal/visual feedback, overt/covert rehearsal, etc., and (f) the type of test format employed (visual/verbal) to assess learner information acquisition. Following is a sampling of specific conclusions obtained in the PSE.

back to top


The results of studies conducted in the PSE have contributed significantly to the knowledge base of instructional systems and communication research. Additionally, research results have yielded important findings regarding the psychological factors contributing to learning such as motivation, memory, cognitive learning strategies and testing/evaluation strategies. These materials have been very effective in evaluating specific internal and external conditions affecting learning behavior, instructional design strategies, and presentation methodologies. The studies to date conducted in the PSE represent the most comprehensive and systematic attempt to identify those variables associated with visualized instruction. A complete listing of the research associates of the PSE along with a listing of dissertations, publications, and presentations generated in the PSE along with a more complete description of the PSE can be found at


Campbell, Donald and Stanley, Julian. Experimental and Quasi-Experimental Designs for Research. In N.L. Gage (Eds.), Handbook of Research on Teaching. Chicago: Rand McNally and Company, 1963, pp 171-246.

Chu, George and Schramm, William. Learning From Television: What the Research Says. Washington D.C. National Association of Educational Broadcasters, 1967.

MacLennan, Donald and Reid, John. Research in Instructional Television and Film. Washington, D.C. U.S. Office of Education, 1967.

Stickell, David. A Critical Review of the Methodology and Results of Research Comparing Televised and Face-to Face Instruction. Doctoral Dissertation. The Pennsylvania State University.

back to top