Persistence In Learning (MCWeb): David Licata -- Homepage

Abstract: This study of pre- and post test scores for 40 students taking first-year and second-year chemistry (Advanced Placement) compares the results for students in two traditional classes with the students in a class using computer technology and small-group instructional assignments. The tests focus on the skills of visualization and proportional reasoning. All teachers followed the district-prescribed pacing using similar labs and individual worksheets (often shared). The traditional teachers primarily used lectures (often highlighted by multimedia displays and demonstrations) followed by guided-practice. The MCWeb group teacher used lecture/demonstration (with and without media) to introduce topics, followed by individual or small-group computer assignments using the Mastering Chemistry program and by small-group instructional activities when appropriate. Although the number of students in the study is small, resulting in only tentative conclusions in some cases, the data point to the conclusion that the Mastering Chemistry group developed and retained better visualization and proportional reasoning skills than did the control students. When the control students, were exposed to the same teaching method, all students attained the same high level of achievement.

Several online websites are available to help students learn chemistry and practice techniques important to success. Older sites use simple drill and practice. More sophisticated sites have developed methods for context-sensitive help and hints. Some sites are static and aligned with particular textbooks or teaching programs. Other sites are more flexible and allow a range of choices in topics, level of questions, and extent of coverage. Most allow for flexible assignments, some collect and display performance data. Some sites include instructional material and other enhancements.

Since the overall point of these websites is simply to assign and enforce doing homework, is there an advantage to using an online site compared with the more traditional paper and pencil worksheet of end-of-chapter problems? Worksheets are flexible, can be prepared with a selected level of questions or problems, contain whatever number of items is deemed appropriate, and the textbook provides random access, context-sensitive help. Unfortunately, as most instructors know, worksheets also give students opportunities for collaboration which may or may not be beneficial. In simple terms: students can cheat. Further, instructors may put in a great deal of effort to assess the work, and students do not always pay attention to the corrections.

This paper will describe the use of the Mastering Chemistry online program used in a high school chemistry class. It will describe the apparent advantages of the online system compared with traditional teaching and show that the online system appears to increase learning, and retention, and performance.

Mastering Chemistry is an online learning system incorporating a homework/quiz system, animated tutorials and explorations, and online access to written guided instructional assignments (with answers). The quizzes feature immediate feedback, context-sensitive "ChemHelp" with text describing the concepts and sample problems, and extensive scoring and reporting. The homework/quiz system includes more than 55 units covering more than 175 topics. Teachers have flexibility to assign entire units or individual topics. The system, developed by Dr. Patrick Wegner, (California State University, Fullerton) can be accessed at http://titanium.fullerton.edu/mcweb. Teachers may use the system without creating a class using the link at http://treefrog.fullerton.edu.

Mastering Chemistry on the Web (MCWeb, pronounced em-see-web, not like an offering of a certain popular hamburger chain) was developed beginning in 1997 by Dr. Patrick Wegner (California State University, Fullerton). Grants from the National Science Foundation and the Fund for Improvement of Post-Secondary Education (US Dept. of Education) helped fund the development and dissemination of the program. Dr. Wegner and his staff continue to work on refinements. MCWeb can be accessed at http://titanium.fullerton.edu/mcweb. Teachers may use the system without creating a class using the link at http://treefrog.fullerton.edu.

MCWeb was developed in conjunction with the Molecular Science Project, one of the NSF Systemic Initiative Projects for chemistry. The central portion of MCWeb is the quizzing database. More than 55 units are available in nine broad categories of general chemistry. Each unit includes three to six topics. Teachers may assign each topic individually, or they may choose one of the pre-designed units. Some units are available in multiple configurations. Each topic quiz (or the summary "unit quizzes") requires students to answer from six to 15 questions. MCWeb includes three types of questions: proportional reasoning, visualization, and application of nomenclatures or formulas. Questions are posed in a variety of formats (see below) so that students do not become dependent on a single formulation terminology. The specific values, pictures, or formulas used in a question reside in a database. Each question and its accompanying items are selected randomly form the database resulting in as many as 50,000 different questions for each topic. It is nearly impossible for any two students to receive the same set of questions.

The topic "MOLE APPLICATIONS: Mass and Number" contains the following three formulations of questions (among others). The boldface items are selected from the database of values for this topic.

Sample 1: Determine the mass of one molecule of a covalent compound whole molar mass is 108.64 g

Sample 2: An 0.150 mol sample of a metal has a mass of 9.81 g. Determine the mass of one atom of the compound.

Sample 3: Determine the number of molecules in a 7.02 g sample of a covalent compound whose molar mass is 18.01 g.

In this topic Avogadro's number is given. Interested teachers may explore other units and topics at the "treefrog" URL above. Each question is marked correct or incorrect immediately so the student knows if he or she understands the problems. Because of the number of questions involved, the huge number of different "quizzes" which each student may view, and the requirement that students earn a minimum score for credit there is little incentive for students to have their work done by someone else. Though sharing computers may require students to work together, they are most often observed talking about how to do the problems and sharing the actual workload - students are teaching each other!

The other portions of MCWeb include tutorials, ChemHelp, animations, and guided instructional activities. Each MCWeb topic includes access to the ChemHelp function. When students click the ChemHelp link they may read an overview of the topic, read a brief textbook style exposition of the topic with worked examples, or view additional worked examples similar to the problems they are doing. ChemHelp is especially useful to students who get answers wrong, though many students read through the sections before they begin work.

The guided instructional activities (referred to as GIAs) are designed to get students to work cooperatively, most often using partnered-discussions, to explore new concepts and practice new skills. More than 200 GIAs are available to be assigned through the MCWeb scheduling function. The paper and pencil GIA may be used as an online assignment or photocopied as a handout Teachers may assign a GIA appropriate to any topic and give students in-class time to work through it, or require the cooperative assignment to be done outside of class. Answers to each GIA may also be posted so students can check their work.

Students using MCWeb receive immediate feedback from the computer, have ready access to ChemHelp, and must achieve a teacher-defined mastery score. Thus the teacher can be assured they are spending more "time on task" than with traditional assignments. In addition, the GIAs force students to be accountable to a partner, further enforcing the "time on task" and making it more likely that they will do the assignment correctly. As a result, the instructor can quickly assess each student's level of achievement, and can work closely with individual students who are struggling. The teacher now uses time more efficiently and students who need help the most get help, while those who are doing well can be assured that they are performing as expected.

Dr. Andrew Montana prepared more than 100 Flash animations to teach Lewis structure, valence bond and molecular orbital Theories, and mechanisms of organic reactions. The Lewis animations teach the procedure for drawing and confirming Lewis structures, and then extend to applying valence bond theory and illustrating the molecular orbitals of nearly 100 molecules. Students may be quizzes on each of the bonding topics. Additional animations are also accessible from MCWeb to illustrate other significant topics in chemistry.

Students in the first-year chemistry classes at Pacifica High School took an 128 question assessment of visualization and proportional reasoning skills in chemistry (the "OCCA") at the beginning of September 2003 (pretest 1 or "pre1"). After the first semester ended (February 2004), they took the OCCA again as a post test (post test 1, or "post1"). One teacher used the MCWeb system in place of many worksheets and textbook assignments. The other two teachers used traditional instruction and homework assignments. All teachers followed the same course of study, used similar labs, and gave similar (sometimes the same) tests. The students in the traditional classes are referred to as the "control." The students using MCWeb are referred to as the "MCWeb" group.

Advanced Placement chemistry at Pacifica is a second year class. It attracted 46 students, of these 24 were in the MCWeb group, and 22 in the control. They all took the OCCA as pretest 2 ("pre2") on the first day of class (September 2004). They took it once more at the end of the year (June 2005) as post test 2 ("post2"). Instruction in the AP class included MCWeb for all students. To assure that students in both groups had similar ability, test score percentiles on the California Achievement Test (sixth edition) for Mathematics were compared. The only significant difference found was that the MCWeb group had (apparently) lower mathematics ability.

Scores and gains on the OCCA were compared between groups, with adjustments for math ability. Both groups had similar scores on pretest 1. The MCWeb group outperformed the control group on post1 and pre2 during the differentiated instruction. Both groups had similar performance on post2, following the use of MCWeb by both groups.

Advanced Placement Chemistry at Pacifica High School, is a second year class. Three teachers share eight to ten sections of first year chemistry each year. There are usually two AP classes. The three teachers cooperate in planning the sequence and pace of instruction, share lab exercises, and often share quizzes and tests in the first year classes. This situation provided an excellent opportunity to assess the potential benefits of the MCWeb program. One of the three chemistry teachers was part of the MCWeb "Web-Assisted Tools for Chemistry, Developers Online Group" (WATChDOGs) and used the MCWeb tools regularly in class. The other two teachers used traditional paper and pencil homework and some self-developed online quizzing and other online research. Students in the group using the Mastering Chemistry program are called the "MCWeb group." Students in the traditional classrooms are the "control."

During the first week of school in 2003 all students enrolled in a Pacifica chemistry class completed the "OCCA" (1) assessment tool developed and validated by the WATChDOG team (2). This will be designated "pretest 1," or simply "pre1.". The OCCA includes 18 paired multiple choice questions testing students' visualization (VIZ) and proportional reasoning (PR) skills. Each question has six possible answer choices. Ten PR questions and eight VIZ questions comprise the test.

The Pacifica course of study covers moles, writing and interpreting formulas, balancing equations, and the gas laws during the first semester. Second semester includes basic quantum ideas and electron configuration, details about the periodic table, shapes of molecules, kinetics, calorimetry, and acid-base reactions. Since the primary instruction in molecular and reaction visualization and chemical proportional reasoning (stoichiometry) occur in first semester, teachers gave the OCCA as a post test at the beginning of the second semester (first week of February 2004). This is designated "post test 2," or "post2." Students in California take the California Achievement Tests, sixth edition (CAT-6) in late April or early May of each year. Student percentile scores on the math portion of this test were collected and are reported as "CAT6M."

Students in the MCWeb class completed 19 MCWeb units (approximately 55 individual topics) between pre1 and post1. They also did 15 GIA assignments in the classroom, besides their more traditional assignments. They did 15 labs during the first semester. The MCWeb group had fewer and shorter traditional assignments compared with the control group due to the work they did with the MCWeb program. During the second semester, after post1, students did another nine MCWeb units, eight GIAs, and 15 labs. The MCWeb classroom has 14 computers for 25 to 30 students. I encourage students to work together on MCWeb assignments. Thus, the online work also fosters cooperative learning as students often work in pairs.

Forty -six students continued from first year chemistry into AP Chemistry in the fall of 2004. Of these 22 students were part of the control classes and 24 students were in the MCWeb group during first-year chemistry. Each student took the OCCA again on the first day of class. This administration is designated "pretest 2" or "pre2." Students did not receive instruction during the summer and had no summer review assignments. At the end of 2005, after the AP Chemistry exam, students took the OCCA for a final time. This is "post test 2," or "post2."

All students took the California Achievement Test (sixth edition), known as CAT6 in late April of 2004. This test assesses reading, English/language arts, math and science in grades 9-11. These questions are all multiple-choice. Percentile scores for the reading, English, and math portions of the test were compared for the two groups of students. The science portion of the test was not compared, as that is the subject of this paper. The two groups of students had comparable scores that were not significantly different at the 95% confidence level for reading or English. However, the control group did have measurably better math ability than the MCWeb group, based on the CAT6. As a result, ANCOVA analysis was used to adjust the average OCCA scores for the differential math ability.

Mean Scores and Gains on the Assessment Tools
PRE and POST scores adjusted by ANCOVA for differential math ability

MEASURE / GROUP	CAT6M	PRE1	POST1	PRE1 to POST1 GAIN	PRE2	POST1 to PRE2 GAIN	POST2
Control	95.4	12.1	13.7	1.5	14.2	0.5	16.6
MCWeb	90.4	11.3	14.8	3.5	15.7	0.9	16.9
P-value	0.013	0.348	0.181	0.043	0.041	0.14	0.259

As Figure 1 shows, the MCWeb group has lower mathematical ability than the control group. This is significant, since 50% of the OCCA test used to compare visualization and proportional reasoning skills focuses on the mathematics used in chemistry. As expected, the MCWeb group did score lower on the first OCCA pretest, but the difference when corrected for math ability was not significant, and amounted to less than one question out of 18. Although the pots 1 scores are also not significantly different, the performance is now reversed - the MCWeb group average score is now 1.1 questions better. Overall, the control group gained only 1.5 questions (8%) compared with a gain more than twice as large (19%) for the MCWeb study group. This difference is statistically significant.

Both groups had 12 additional weeks of chemistry instruction before summer break. Neither group had any assigned study or chemistry practice during the 11 weeks of vacation. Teachers know from experience that students forget what they have learned over the vacation period, so a decline in scores would not be a surprise. The additional instruction and practice accounts for the fact that both groups of students showed a gain in score after the time off. However, the MCWeb group again showed nearly twice the gain of the control group. While the gains, since the are small, are not significantly different, the mean scores now favor the MCWeb group by 1.5 questions and this is a significant difference.

Both groups achieved similar scores after a year using MCWeb in the AP Chemistry class.

A comparison of students with negative gain (regression) shows that far fewer MCWeb students "forgot" what they learned compared with the control. A relatively large number of control students decreased performance during the time of instruction compared with only a single student in the MCWeb group. Many students regressed following the break, as expected. But the number of control students regressing was nearly twice as large as in the MCWeb group and the regression was nearly 50% larger than that from students using MCWeb.

Scores on several sections of the CAT6 test were compared for the two groups of students. The language and reading abilities of the two groups of students were similar. However, the control group had a significantly higher average math score than the MCWeb group. This difference in mathematics ability was reflected in the pre1 scores of the two groups, as was expected. The control students average score was 12.5 compared with the MCWeb average of 10.9. This difference is significant at the 95% confidence level. The two groups were compared using an analysis of covariance (ANCOVA) to control for math ability, and the adjusted averages were 12.1 for the control and 11.3 for the MCWeb group. The adjusted averages are not significantly different.

Following the first semester of VIZ and PR instruction, the MCWeb group had not only caught, but passed the OCCA score of the control group. The MCWeb students averaged 14.8 compared with the 13.7 (adjusted for CAT6M scores) on post test 1. While the difference in means is not significant, there is a real difference in gains made by each group. The MCWeb group improved their score by an average of 3.5 questions (19%) compared with only a 1.5 question improvement (8%) by the control students. MCWeb students averaged more than twice the improvement of the control group. This result corresponds to the average improvements seen for all groups of students using the MCWeb program (high school through university) compared with control groups (3). Statistical comparison of the control group pre1 to post1 scores shows that the difference in scores was not significant, suggesting that the apparent gain may or may not have been real. A similar comparison for the MCWeb group does show a statistically significant growth.

It is also worth noting that the variance in averages was near 8.5 for both sets of students on pretest 1, and for the control group on post test 1. However, the variance for the MCWeb group on post test 1 narrowed to only 5.7. This may show that not only had the group achieved greater improvement, but they had greater uniformity of the post test results. Further, six of the 22 control students regressed in score from pre1 to post1. The average loss for these six students was 2.2 questions. The single MCWeb student scoring lower missed only one more question on post1 compared with pre1.

Students continued to receive instruction from February through the middle of June (approximately 12 weeks not including finals, testing, and spring break). They were on summer break for 11 weeks during which they had no program of study for chemistry, nor any planned review or assignments. Teachers are familiar with the lack of retention, and the likelihood of regression during the summer break. However, the 12 weeks of instruction apparently benefitted both groups of students. The chemistry students who returned for a second year of chemistry took the OCCA test a third time on the first day of class (pre2). The MCWeb group improved to an adjusted average of 15.7 while the control group improved their mean score to 14.2. This is a statistically significant difference in means. The average improvement in score was 0.9 questions for the MCWeb group but only 0.5 questions for the control. Again, the MCWeb group outperformed the control by nearly double, though statistical analysis does not show that either gain is significant.

As expected, the control group had many students regressing. Ten of the control students did worse on pre2, losing an average of 2.4 questions. Only six of the MCWeb group regressed, by an average of only 1.5 questions. Again, the variance in scores was near 8.5 for the control group, but less than six for MCWeb students suggesting that these students maintained the greater uniformity of results.

The overall gain over the entire year from pre1 to pre2 was 2.1 questions (12%) for the control group, but more than twice that much (4.4 questions, 25%) for the students using MCWeb. During the AP year students used 36 MCWeb units (100 or more topics) and approximately 20 GIAs. When the OCCA was given for a final time (post2) in June 2005 the control group achieved a score of 16.6 and the MCWeb group 16.9. There is no statistical difference between these final scores. Both groups had reached the same level of achievement on the OCCA measures of VIZ and PR by the end of the AP term.

The results of the series of OCCA tests suggest that an online program (like MCWeb) which has immediate feedback, and enforces completion of assignments (particularly when combined with cooperative learning techniques) is more effective in teaching the important skills of visualization and proportional reasoning in a chemistry context than are traditional homework. Further, students who have used MCWeb have greater retention of material after 13 weeks without instruction or practice than their counterparts who did not use the program. Although the MCWeb students had lower mathematics ability than the control, they showed greater gains in performance (which presumably means learning) and surpassed their more able counterparts in the mathematics-related proportional reasoning skills. Therefore, the author concludes that an online system such as MCWeb, which enforces doing assignments and achieving correct answers, particularly when combined with cooperative learning techniques, is more beneficial that the usual types of homework.

I would like to express my appreciation to Dr. Wegner, the developer of the MCWeb program. His enthusiasm, support, and advice on using MCWeb have revolutionized my teaching methods. Dr. Barbara Gonzales's help in preparing and analyzing the data have been invaluable. Dr. William Sandoval, University of California, Los Angeles, provided much useful insight and discussion of results. The ANCOVA analysis was done using the applets created by Dr. Richard Lowry, Vassar College and was accessed at his website, http://faculty.vassar.edu/lowry/VassarStats.html (accessed during July 2005).

(1) The test was originally designed to be given via the Internet and was called the "Online Chemistry Concept Assessment." When it was found that there was too much variability in access, administration supervision, and lack of effort by students the test was converted to a more standard multiple choice format. The reference to "OCCA" as the test designation remains.

(2) A refereed journal article by Dr. Barbara Gonzales, California State University of Fullerton, will be published this fall with details on the validation study.

MEASURE / GROUP	PRE1 to POST 1			POST1 to PRE2
MEASURE / GROUP	Number of students	% of students	No. questions lost	Number of students	% of students	No. questions lost
Control	6	26%	-2.2	10	44%	-2.4
MCWeb	1	4%	-1.0	6	26%	-1.7