Evidence of gains in inquiry, reasoning, and problem-solving skills.

Experimenting with Model Rockets Research Studies: There is considerable evidence that the activities in the GEMS guide Experimenting with Model Rockets significantly improve students’ conceptual understanding of controlled experimentation, as well as their abilities to design, conduct, and critique their own experiments and the experiments of others. The findings of this research have been applied to a number of other GEMS units that include a major focus on investigation and experimentation. The body of research evidence is summarized in the article "Learning to Control Variables with Model Rockets: A Neo-Piagetian Study of Learning in Field Settings" by Cary Sneider, Kevin Kurlich, Steven Pulos, and Alan Friedman. Dr. Sneider and others worked on the initial research in the late 1970s and early 1980s. With the help of a National Science Foundation grant (SED79-18976), a series of studies with students in diverse learning situations, including schools, scout organizations, and summer camps was conducted. A workshop for teachers and other youth leaders was held, to train them in presenting the activities. Both individual interviews and paper-and-pencil questionnaires were utilized before and after students took the class from their newly trained teachers. The research showed that children from 9 to 15 years old could significantly improve in their abilities to design and critique controlled experiments as a result of the model rocketry course. An article about these findings was published in 1984 in the journal Science Education.

The article includes an analysis of about 40 studies showing that positive results could be obtained in a wide variety of settings where students were given opportunities to perform controlled experiments, and summarizes this body of previous research. The experimental treatment was the model rocket activities developed by Sneider. The study involved 275 children and adolescents, 9–15 years old. Four criterion tasks were developed and a scoring system developed. Subjects were also administered cognitive tests. The details and results of Experiment 1 and Experiment 2 (designed see if the results of the first experiment would be replicated and to rule out alternative explanations for positive results, with both an experimental and a control group—who did not do the rocketry activities until after the study was over). Both experiments showed a that subjects did much better on the post-test than the pre-test. In Experiment 1, among many statistical analyses, multiple t-tests were performed to compare those who took part in the instructional program with those who had not, within appropriate age level groups. The results showed that all ages learned significantly, and that the model rocketry program was effective in teaching controlled experimentation for children and adolescents between the ages of 9 and 15, in schools and non-school groups, and for both boys and girls. The results for Experiment 2 were similarly positive. Interestingly, ANCOVA analysis of group task results showed a significant difference between pre-test and post-test, but no significant difference between the experimental and control groups. When results were analyzed separately for boys and girls, these anomalous results were seen to stem from the fact that girls in both the experimental and control groups showed improvement in both pre and post-tests. The paper speculates on possible reasons for this result, including possible communication between girls who had taken part in the activities and those who had not.

The overall positive results and other research-based information gained during the model rocket study formed the basis for the development of the GEMS unit Experimenting with Model Rockets, first published in 1989, and revised in 1991 and 1997. Lessons gained from the study, as well as the subsequent thorough local and national testing process to which all GEMS units are subjected, were crucial in creating and refining the GEMS unit. The same progression of activities validated in the study is retained in the GEMS unit. The GEMS assessment handbook, Insights and Outcomes, includes an updated and revised version of two of the pre-post pencil and paper tests used in the original study, "Experimenting with Cars," and "Experimenting with Plants," with instructions for how to use them both as an assessment for the GEMS rocketry unit (Insights and Outcomes, pages 224-227, 1995).

This research provides clear and compelling evidence that the GEMS unit Experimenting with Model Rockets helps students understand the concept of a controlled experiment, and improve their abilities to design, conduct, and criticize controlled experiments. Acquiring this key capability and understanding improves students’ comprehension of what scientists do, and equips them with an important ability for living and working in the modern world. With the advent of the National Science Education Standards and their strong emphasis on both the ability to do and the understanding of scientific inquiry, this research and the instructional units that grew out of it can help make a significant contribution to the scientific literacy of students who experience these units. This research has helped guide the development of many other GEMS units. For example, the following GEMS units (in addition to Experimenting with Model Rockets) help students learn controlled experimentation and/or other related aspects of scientific inquiry and investigation:

• In Hot Water and Warm Homes from Sunlight, students are introduced to the concept of controlled experimentation through an activity sheet in which they reason about some plants that were given different amounts of fertilizer. They then perform pre-designed experiments, discuss why it is important to keep all of the possible variables constant, and can go on to design their own experiments.
• In Bubble-ology, students are introduced to a technique for measuring the size of bubbles so they can determine which of three soap solutions is best. They need conduct the tests so all variables are controlled except for the kind of soap solution.
• In Paper Towel Testing, students are challenged to design their own experiments to determine which brand of paper towel has greater wet strength, and which is more absorbent. They must identify the variables and design controlled experiments.
• Both Acid Rain and Global Warming and the Greenhouse Effect have key components involving student experimentation.
• In River Cutters students learn the distinction between systematic observations and controlled experiments, using their river cutting models to conduct a controlled experiment involving slope, and then designing their own systematic observation or experiment.
• In Dry Ice Investigations the intriguing behavior of dry ice provides a compelling way to systematically guide students through the entire process of investigation, from exploration through systematic observation and experimentation, with strong emphasis on areas that research has shown to be difficult for students, such as coming up with investigable questions and planning. River Cutters, Dry Ice Investigations, and a number of other GEMS units provide excellent platforms for the "full investigations" recommended in the National Science Education Standards.
  

C. Sneider, K. Kurlich, S. Pulos, A. Friedman. "Learning to Control Variables with Model Rockets: A Neo-Piagetian Study of Learning in Field Settings." Science Education 68 (4) (1984): 463-484.

Sneider, Cary I., Experimenting with Model Rockets, Great Explorations in Math and Science (GEMS) teacher’s guide, Lawrence Hall of Science, 1989, 1991, 1997.

Barber, J et al, Insights and Outcomes: Assessments for Great Explorations in Math and Science, " Selected Learning Outcomes for Experimenting with Model Rockets, pages 224–227, 1995.

Barber, J., Bergman, L, Sneider, C: "The Educational Effectiveness of GEMS activities" and Sneider, C. "GEMS and Research: Three Case Studies," GEMS Leaders Handbook, pages 19–32, 1988, 1994, 1997.

The GALAXY Classroom Science Project (for Grades 3-5): Evaluation of this project was conducted by Dr. Gloria Guth of Far West Laboratory (now WestEd). Galaxy is a package of integrated curricular and instructional approaches, supported by the nation’s first interactive satellite communications network designed to facilitate the introduction of innovative curricula to improve student learning in elementary schools. The Galaxy Classroom Science for Grades 3-5 features the organization of instruction around themes presented through television broadcasts and classroom hands-on activities that are facilitated by fax technology and ongoing teacher support. Classroom curriculum included 8 units from the GEMS program—Earth, Moon, and Stars, Bubble-ology, Oobleck, Investigating Artifacts, Crime Lab Chemistry, Fingerprinting, Chemical Reactions, and Of Cabbages and Chemistry. The evaluation found that GALAXY science for grades 3-5 is a highly successful initiative. For example:

• On measures of classification processes, GALAXY students had a statistically significant gain that was more than double the gain of non-GALAXY comparison students.
• Scores on curriculum-based performance assessments indicate that the majority of GALAXY students across all three grades were able to demonstrate that they understood the "big ideas" or core science concepts of the GALAXY curriculum.
• In general, when comparison non-GALAXY students were evaluated on some of the same measures, GALAXY students outperformed them in almost every case.
• In addition, GALAXY teachers displayed significantly more positive attitudes than they had initially regarding their own comfort with and preparation for teaching science and the adequacy of their science materials.
• Participating in GALAXY Classroom Science led to statistically significant positive change in attitudes among GALAXY students, when compared to their non-GALAXY peers, toward participating in science class and engaging in activities to which they did not know the right answer.

The Evaluation Approach: The evaluation gathered quantitative data on GALAXY impact by testing student learning through performance-based assessments, surveying student and teacher attitudes and teacher practices, and asking teachers to record their use of the GALAXY Classroom Science curriculum. Administration of four of the performance-based assessments and the attitude surveys followed a pre/post design. Four other assessments were more closely linked to the curriculum and activities, and they were administered during the course of GALAXY science.

Developing Scientific Thinking Processes and Results From Performance-Based Assessments: Researchers adapted four performance-based assessments from the California Learning Assessment System (CLAS) to test GALAXY and comparison students’ progress in several crucial areas. Researchers measured classification and organization with two hands-on assessments using fossils in the pre-test and leaves in the post-test. Skills related to experimentation were measured by two other pre/post performance-based assessments using rocks and soils, administered in a crossover design. Additionally, students took a multiple-choice test of science process skills.

Results from the Classification Pre/Post Assessments: The evidence shows that participation in GALAXY had a statistically significant positive effect on students’ classification abilities. These results are based on testing 600 GALAXY and 610 comparison students in the same grades at twelve GALAXY schools. Each of the two assessments had three tasks that were scored from 0 (no attempt) to 5 (accurate and informative). See Figure 1 in the Executive Summary to view the average (mean) scores for GALAXY and comparison students in each of the three grades, both before GALAXY science started (pre) and after it was completed (post).

Results from the Use of Scientific Thinking Assessments: Another assessment evaluated how students reason about and investigate the causes of unexplained phenomena. Overall, GALAXY students significantly more often chose a scientific explanation than a supernatural explanation than did comparison students. When asked to invent ways to prove their explanations, those who had participated in GALAXY were significantly more likely than comparison students to design an experimental approach. In sum, this evidence shows GALAXY achieved its goal of helping students understand the world through observation and experimentation and seek rational explanations for the way the world works.

Results from a General Test of Scientific Reasoning: An additional measure of GALAXY’s effectiveness was a multiple-choice test designed to measure specific scientific thinking processes. It was given twice to GALAXY and comparison fourth and fifth graders, once when classes were one third of the way into the GALAXY sequence and once at GALAXY’s conclusion. While fifth grade results were somewhat ambiguous, the fourth grade results showed advantages for GALAXY students in items testing skills such as experimenting and formulating correct hypotheses.

Curriculum-Embedded Assessments: In addition to comparing the performance of GALAXY students with similar non-GALAXY students, researchers assessed the GALAXY students on their level of mastery of the primary themes of the GALAXY curriculum. Curriculum-embedded assessments were used, together with videotaped performance assessments of small samples, to establish the degree to which the curriculum was achieving its goal of helping students learn about the core science concepts in each theme.

• Curriculum Theme 1: Using Patterns as Evidence: Approximately seventy-five percent of the 1,678 GALAXY students who completed the embedded performance task demonstrated a satisfactory capacity to use patterns as evidence.
  
• Curriculum Theme 2: Doing Experiments: GALAXY students performed five tasks associated with the experimental mixing of chemicals and recorded their findings for later scoring. Overall, between 65% and 70% of the 1,256 GALAXY students understood the concept of experimentation, could manipulate variables, and could predict probable outcome. Some students, however, had difficulty explicitly stating the cause and effect relationships among specific variables, a more demanding analytic task. The videotaped assessment provides strong evidence that GALAXY students approach unknown substances quite ready to experiment and with the understanding that they can systematically and collaboratively compare the properties of substances. (This was in contrast to comparison students, who had difficulty organizing themselves to work together and to begin to explore the problem systematically.) Taken together, the evidence suggests that the GALAXY experience made a positive contribution to student mastery of experimental methods.
• Curriculum Theme 3: Building Models to Explain and Invent Ideas: A paper-and-pencil performance test involving several common machines assessed students’ ability to draw and explain models that showed how something worked or that could be modified to serve a new purpose. The 1,503 GALAXY students performed very well, with more than two-thirds scoring in the top two performance categories. The videotaped performance task, however, indicated that for both GALAXY and non-GALAXY students, the mastery of the process of going from design to testing was limited, with no apparent advantage to GALAXY students.

Return to Educational Effectiveness > GEMS Home

Top