Synopsis of ChemConnections Evaluation Studies

From the Fall of 1995 through the Fall of 1999, the ChemConnections evaluation teams have conducted numerous types of studies in a wide variety of institutions (see Table 1).

Table 1. Institutions involved in ChemConnections Student Evaluation Studies.

Type of study

N (students)

Institutions

Early interviews & focus groups

345

Washington University, Colorado College, Contra Costa College, Spelman College, U.C. San Diego, Wooster College, Kalamazoo College, Miramar Community College, Mesa Community College, Cal State Los Angeles, U.C. Berkeley

Descriptive

797

Cal State Hayward, Grinnell College, U.C. Berkeley

Comparative

1,259

Grinnell College, U.C. Berkeley

Case

9

Cañada Community College

Late interviews & focus groups

323

American River, Beloit College, Carleton College, Evergreen College, New Mexico State, Randolph-Macon College, St. Thomas University, U. Michigan Dearborn, U.W. Rock County

First Round of Student Interviews: 1996-1998

As part of the formative evaluation of newly developed modules, interviews were conducted to document the teaching students had experienced and the learning gains that they had made, using matched pairs of introductory chemistry classes. Individual and focus group interviews were conducted with 345 students at a sample of 11 institutions representing research and comprehensive universities, liberal arts and community colleges, and HBCUs. At each site, students were randomly drawn from two matched classes–one in which a new module had recently been taught, and a second, more traditionally-taught class or section. Students interviewed were selected to allow analysis for differences in gender, ethnicity, and intended major.

Second Round of Student Interviews: 1999-2000

As modules were refined, a second round of interviews was conducted to clarify the nature and relative importance of learning issues raised for students in different institutional contexts and class sizes by the experience of well-developed, well-taught modules. By comparison with first-round findings, we could also discern changes over time in student responses to modular learning. Again, individual and focus group interviews were conducted, this time with 323 "modular" students at 10 institutions where beta-version modules were being taught by seasoned modular teachers. The classes again represented a range of institutional types and also variety in how modules were used: 3 classes experienced an all-modular year, 6 an all-modular semester, and 1 a single module as a capstone.

Processes of Change

We are also interested in how teachers and institutions change. Data were gathered by recorded live and telephone interviews with individuals and a few small groups, and solicited electronic accounts of developers and adapters’ observations on module teaching. Participants represented these groups:

(a) coalition participants, including module developers (interviewed over time), and key coalition leaders (PIs, working group chairs, workshop organizers, media specialist);

(b) module users: individual and group adapters, including "adapt and adopt" groups, workshop attendees, unusual adapters (engineers, high school teachers), and "converts;"

(c) the "unconvinced": former workshop attendees, colleagues of developers/adapters;

(d) institutional commentators: departmental chairs, lab staff, deans, provosts, presidents;

(e) teaching assistants in modular classes: graduate and undergraduate; at UC Berkeley (interview study of TA development); student interview data about TA responses.

Analysis

Analysis of ethnographic text data is time-consuming and cannot be done on partial data sets. However, the following analyses are well-advanced and will be among the first studies published.

1. TAs–their responses to classroom innovation, training issues;

2. Sources of student resistance to classroom change;

3. Why some faculty choose to teach with modules; why other faculty exposed to modules may remain unconvinced; why other colleagues oppose their use.

4. Comparison of 1st and 2nd round student interview data on student responses to small group learning.

Further analyses are underway, including analysis of the 2nd-round interview data for courses using beta modules and taught by seasoned modular teachers. Several publications are in preparation.

Emergent Findings and Targeted Areas of Analysis

1. In the early classroom use of modules (round 1), students showed gains in the following areas:

• appreciating the relevance of chemistry and its real world applications

• acquiring an increased interest in/enthusiasm for chemistry

• greater appreciation of the nature and methods of chemistry

• expectation of retaining more of what they learned from modules than from traditional classes

• appreciating the contribution made by class discussion to their learning

• feeling more comfort with complexity

• increased confidence in their ability to do chemistry

• appreciation of the value of small group learning (gains made by white males especially)

2. Early classroom use of modules also highlighted some areas of pedagogical difficulty:

• coherence or "fit" between various elements of the course, such as lab, readings and assignments

• framework and direction (e.g. need for "signposting" course direction, explaining use of texts as resources)

• pace and coverage

• labs (e.g. class-lab relationship, structure, buy-in of lab coworkers)

• assessments (for new learning objectives, mental "stretch" required in tests)

• "But is it real chemistry?" Some students think that if they understand and enjoy what is being taught, the chemistry content cannot be rigorous, generalizable, or meet their career preparation goals.

3. We learned that it is more relevant and productive to ask students about what they have gained from specific aspects of the class than what they liked or disliked about the class or teacher. This finding led to the development of the SALG instrument, or Student Assessment of Learning Gains.

In particular, analysis of 12,993 student observations from round-1 interviews showed that students gave both positive and negative ratings to specific aspects of the class and to their teacher’s classroom performance. However, total student evaluation of faculty teaching performance was broadly 50% positive, and 50% negative, for both modular and comparative classes. Thus neither group of faculty got a clear picture of the overall perceived utility of their classroom work.

However, in both modular and comparative classes, students gave clear indications about what they themselves had "gained" both from specific aspects of their classes and overall. When all specific gain-related observations were totaled, 55%of the observations referenced "things gained" (for both types of class); 33% (modular) and 32% (comparative) referenced "things not gained;" and 11% (modular) and 13%(comparative) were "mixed" or qualified gains.

4. Certain areas of risk were identified for those teaching with modules:

• Consequences of disrupting a dysfunctional system

• Departmental climate (non-tenured faculty, collegial responses, rewards, marginalization)

• Institutional rewards structure

• TA teaching norms

• Student resistance to change

5. However, faculty benefited from the formation of learning communities:

• Curriculum, pedagogy, assessment–shared professional exploration, education, and promotion

• Creative collaboration, live and electronic networks

• Active forms of dissemination (personal connection, workshops)

• Collaborations–across disciplines, institutions, consortia

Classroom Learning Assessment Initiatives

1. FLAG: the Field-Tested Learning Assessment Guide (in collaboration with Susan Millar and the New Traditions Coalition, and NISE). See link at http://www.wcer.wisc.edu/nise/cl1/

2. SALG: the Student Assessment of their Learning Gains On-line Classroom Evaluation Instrument (web development by Sue Daffinrud, LEAD Center, University of Wisconsin-Madison). See link at http://www.wcer.wisc.edu/nise/cl1/

3. "Carry Forward/Carry Away" Experiments: collegial faculty experiments on what students remember from class to class, over time.

4. "Pairs" Experiments: collegial faculty experiments to compare student achievements in modular and non-modular classes.

Comparative Design Studies

The early comparative studies were conducted at two institutions, a small college (Grinnell) and a large university (U.C. Berkeley). The experimental design at each school compared students in a course section taught with modules to those in a section that used a textbook and lecture format.

The assessments in both studies included pre/post conceptual tests, pre/post attitudinal surveys, post-test problem-solving interviews, a post-test ACS exam (1996 Brief version), and in-class exams or quizzes.

The modules used in these studies were draft versions that contained errors, did not always employ the final format of modules, and had fewer graphics than the final versions. Moreover, at the time of the studies, few faculty had used modules. One faculty member at Grinnell had never before taught with modules; the Berkeley study marked the first occasion anyone had attempted to employ modules in a large classroom (over 200 students). Because both studies were conducted early in the production and use of modules, the results of these studies must be taken as preliminary. As the modules evolved and faculty use of them matured, the student outcomes undoubtedly changed.

Given these caveats, the results of the early comparative studies are as follows. At both schools, students in the modular section outperformed the control group on conceptual problems in chemistry and on scientific thinking problems. No difference was seen in their performance on the standard ACS exam. Modular section students at the large university also outperformed their peers on the first midterm exam in the subsequent organic chemistry course. See Table 2 for student means on these assessments.

The post-course attitude survey showed that the Modular section students were more positive about chemistry and the course than their peers in the control section at Grinnell College. However, at U.C. Berkeley, the opposite attitudinal pattern was found. See Table 3 for a selected sample of post-course survey questions that showed statistically significant differences at both institutions.

An analysis of informal focus group data and data from a modular course held the subsequent semester suggests that much of the negative response at U.C. Berkeley was due to the embryonic nature of the course; modules had never before been employed in a large course. In the subsequent semester, students’ attitudes became more positive, roughly matching attitudes held by students in the control group section (Table 3).

A strong effect of Graduate Student Instructor (teaching assistant) was also found on the attitude of students in the modular section. Students’ attitudes toward the modular course paralleled those of the GSI for their section, though GSI and student attitudes for the non-modular section were uncorrelated. This shows the importance of good training for all teaching staff when new instructional methods are introduced. A separate study underway at Berkeley is designed to address the issue of TA professional development.

Table 2. Assessment results for two comparative studies.

 

Grinnell ‡

U.C.Berkeley ‡‡

 

fall 1997, comparative

spring 1998, comparative

fall 1998, mod only

Test

Modular Mean

Non-Mod Mean

Modular Mean

Non-Mod Mean

Modular Mean

Conceptual Pretest

7.80

7.24

6.58

6.36

7.35

Conceptual Posttest

9.69

9.59

10.98*

9.36

10.36*

In-class Exams

28.69*

26.12

297.54

297.91

N/A

ACS Exam

20.24

21.38

21.08

21.64

21.22

Interview — Gas Laws

8.20*

7.21

9.25

9.05

N/A

Interview — Exptl. Design

8.53

8.64

N/A

N/A

N/A

Interview — Sci. Thinking

N/A

N/A

7.70*

4.85

N/A

Follow-up Exam

N/A

N/A

53.72*

50.18

N/A

* Modular vs. non-modular difference within institution is statistically significant at p<0.05 level. Differences between sections at institutions are not comparable.

‡ All written assessments employed SAT scores as a covariate in the ANOVA to remove the effect of differences in SAT scores.

‡‡ All written assessments were analyzed using a General Linear Model with SAT, Section (Modular or Non-Modular) and Graduate Student Instructor (nested within Section) as independent variables to remove effect of difference in SAT and GSI .

 

Table 3. Mean responses to a selected sample of attitude survey questions at both institutions.

Grinnell

U.C. Berkeley

Survey Statement

Fall 1997

Spring 1998

Fall 1998

(1= strongly disagree; 5 = strongly agree)

Non-Mod Mean

Modular Mean

Non-Mod Mean

Modular Mean

Modular Mean

I understood most of the ideas presented in this course.

3.7

4.5

3.7

3.2

3.5

By the end of this course, I knew where and how to look up information.

2.7

3.5

3.4

2.6

2.9

Taking this class has helped me understand newspaper and other articles I read about science.

3.2

3.9

3.1

2.4

2.8

I enjoyed taking this chemistry course

2.7

3.4

3.2

2.1

2.7

I would recommend this chemistry course to my friends

2.3

3.3

3.0

2.0

2.6

Note: All group differences within an institution are significant at the 0.05 level. Differences between institutions are not comparable.