Is It Better to Be Feared Than Loved? Investigating the Efficacy of Different Teaching Methods on the Learning of Specific Content

Abstract

The use and efficacy of problem-based learning (PBL) exercises during single class periods, especially in political science courses, have largely been unstudied. This article explores the ability of a PBL exercise to provide better learning outcomes than a standard lecture-discussion presentation on a particular topic in an introductory political science course. The author finds that the PBL approach, as a one-time pedagogical exercise, is not more effective at producing learning. In exploring the data, however, the author finds that differences in course structure and methods of learning assessment can have a larger impact on learning outcomes than specific teaching techniques.

When first designing a group exercise in my introduction to political science course, my sole aim was to provide some variety for my students. A typical 50-minute class consisted of a mix of discussion and oral quiz questions, in which I posed a content-based, analytical, or more provocative question to the class—based on the daily reading assignment—and then randomly chose a student to answer it. I used that initial Q&A to further explore the students’ understanding of the concept, explaining the finer points and relating the content to the day’s news. Although this technique certainly kept students engaged, previous student evaluations indicated that they found it rather monotonous. They enjoyed something different now and then to “spice up” the class.

Hence, in considering how to make a chapter on constitutions more interesting, I devised a way to seize their imagination and encourage the students to apply the chapter content to a fictional problem: they would develop a constitution for the spaceship Aurora, which would take an international group of colonists on a 200-year-long journey to another planet.

Because my introductory classes typically have 20 to 25 students, it was obvious to me that smaller groups would be required. Therefore, I divided the students into four groups so they could tackle the problem and collaborate on a report at the end of the class session. That report would count as a quiz grade for all of the students involved in the project.

At this point, I had not read any scholarship on the use of simulations, problem-based learning (PBL), or group-based learning and therefore was operating solely on instinct. With a minimum of instructions, I left the groups to their own devices to outline a basic constitution, taking into account the assigned chapter’s discussion of successful constitution-building characteristics. Students seemed energized and engaged during the exercise, and they demonstrated a solid understanding of the main points of the chapter in their reports. I judged the exercise to be a success and used it in future sections of the course.

After attending my first APSA Teaching and Learning Conference and reading a study presented there about a similar exercise (Centellas and Love 2011), I realized that much more was at stake. Beyond simply providing variety in the classroom, this technique could be more effective in helping students learn course content. Moreover, it might help them acquire skills that would be useful in other classes and after graduation, such as the abilities to collaborate, problem solve, and present as a group. If one technique could do all of this, then should I be using it more often? Should I incorporate more of these exercises into my courses?

BACKGROUND: TWO APPROACHES TO GROUP-BASED LEARNING

Even a cursory survey of pedagogical techniques shows that there are many reasons and many ways to incorporate groups into active learning. The techniques vary in the specificity of the roles given to students, degree of situational detail provided, type and degree of the instructor’s involvement, and pedagogical goals of the exercise.

In general, the group-learning approach has been shown to be very effective, as found by a meta-analysis of 168 studies (Johnson et al. 1998). Group-learning pedagogies are classified generally as either cooperative team learning or collaborative learning (Bruffee 1995; Matthews et al. 1995; Smith 1995). Cooperative team learning has the clearest evidence of its efficacy in the educational literature (Johnson et al. 1998). In cooperative team learning, the instructor groups students, assigns roles, provides all background information necessary, and outlines specific procedures to be followed (Bruffee 1995). Most political science simulations are in this category, with students playing national leaders, United Nations negotiators, members of parliament, or senators. They interact with one another in well-defined ways, and the instructor interacts with the players and provides information, feedback, and guidance.

Collaborative learning is much less structured. This PBL approach presents a group of students with a goal to reach or a problem to solve; however, it does not provide all of the information needed or set procedures to follow in solving the problem (Bruffee 1995). Although these types of exercises sometimes are labeled as simulations, the lack of set roles and elaborate procedures makes them more akin to classic PBL exercises. ¹ (Project-based learning, a similar pedagogy, emphasizes the final product rather than the problem or process [Bransford and Stein 1993].)

The use and efficacy of these types of exercises during single class periods, especially in political science courses, have largely been unstudied. The quasi-experiment reported in this article is a small step in the exploration of the limits of PBL collaborative group learning.

In the literature, and for both types of learning, it is almost always recommended that the exercise last for long periods—multiple class periods, for example, or even several weeks. The use and efficacy of these types of exercises during single class periods, especially in political science courses, have largely been unstudied. The quasi-experiment reported in this article is a small step in the exploration of the limits of PBL collaborative group learning.

GROUP-BASED LEARNING IN POLITICAL SCIENCE

Evidence on the efficacy of PBL collaborative group learning in political science courses is mixed. Recent studies using this technique to enhance student learning in introductory-level political science courses attempted to measure its effect on student learning in different ways. Raymond (2010), for example, examined the effect of a simulation exercise on exam scores; Williamson and Gregory (2010) sought evidence that multiple PBL exercises lead to better critical thinking, use of sources, and understanding of political institutions in student papers. In a simpler approach, Centellas and Love (2012) looked for improved performance on a battery of multiple-choice questions testing specific knowledge that should have been reinforced in a four-week PBL exercise. The results of these studies are mixed: Raymond (2010) found no direct evidence of improved learning, whereas Williamson and Gregory (2010) and Centellas and Love (2012) did.

THE ORIGINAL STUDY: DOES PROBLEM-BASED LEARNING REALLY TEACH BETTER THAN LECTURES?

This research was originally modeled on an earlier study (Centellas and Love 2011). In that study, Miguel Centellas and Gregory J. Love evaluated their attempt to use a PBL approach to enhance student learning in an introductory-level political science course. After administering a pretest, the authors randomly grouped students to write a new constitution for the Land of Oz, a project that lasted several weeks. The authors then analyzed the students’ performance on a multiple-choice battery embedded in the final exam. They found that students who had taken part in the PBL exercise scored significantly better than the control group on the battery of items related to constitution building.

However, a careful reading of Centellas and Love’s methodology suggested an alternate explanation for the treatment group’s higher scores: the amount of time spent working with the tested content. In Centellas and Love’s quasi-experiment, the treatment groups spent additional time working on the project for about four weeks, whereas the control group did not have any extra assigned work outside of the classroom. I wondered if it was the extra time spent by the treatment groups—rather than the PBL approach—that led to their better learning of specific content than the control group.

The question I intended to answer, then, was whether a collaborative group exercise is more efficient in improving student recall of specific content than a lecture-discussion session taking the same amount of time. In other words, is it PBL or the time spent working with the material that improves the learning and retention of specific content?

RESEARCH DESIGN

Seven sections of introduction to political science during two semesters, with a total of 137 students, were included in this study, which was approved by the institution’s Committee for Human Subject Research (on file; available on request). Three of the classes were taught by Professor A and four by Professor B. Both instructors used the same textbook for all their sections: Power and Choice by W. Phillips Shively. On the day that each instructor was scheduled to discuss constitutional design, students were administered a short, three-item, multiple-choice pretest (see the appendix for this and other study documents) to ascertain how much they understood of that day’s reading assignment. Students were then assigned to groups using a count-off technique (to separate seatmates); two groups were sent to other rooms where they would do a PBL exercise, and the remainder stayed in their seats. After giving brief oral instructions to each PBL group, the instructor returned to the main classroom and used the standard lecture-discussion format to cover material on constitutional design.

In addition to the brief oral instructions, PBL groups were given a one-page written description of their task that encouraged them to use their notes and textbook to design a “good” constitution for the space colony. The PBL groups spent the next 35 minutes brainstorming and outlining a constitution, and then submitted their work at the end of the class period.

In the next class period, all students were tested—using a reordered version of the pretest instrument—to measure any differences in learning effects. Two weeks later, a different reordered version of the instrument was administered to measure any differences in retention. Students’ scores ranged from zero (none correct) to 3 (all correct) on each instrument. ²

FINDINGS OF THE ORIGINAL STUDY

Overall, the PBL exercise appeared to have no significant effect on the learning of content. ³ For the 67 students who were exposed to the standard lecture-discussion format, their scores increased an average of 0.60 points at the first retesting; the PBL students averaged a gain of 0.64 points. Differences between treatment groups were more pronounced when retention was measured two weeks later. While the results again did not reach statistical significance, this time they ran counter to the expected direction: students exposed to the lecture-discussion format averaged 0.68 points higher than their pretest scores, whereas those in the PBL group achieved a mean that was only 0.58 points higher (table 1).

Table 1 Results by Treatment

A linear-regression model that incorporates students’ final grades as a control variable confirmed these findings: the results yielded very low adjusted R-squares and no significant effects for the PBL dummy variable on posttest or retention-test scores (table 2).

Table 2 Linear-Regression Model: Treatment and Final Grade

** p<0.005

These results suggest that PBL exercises during a single class period are not more effective than the lecture-discussion format in helping students learn specific content. Moreover, the PBL exercises may be less effective than the lecture-discussion format in promoting the retention of knowledge during a period of several weeks.

The differences were not between the treatment and control groups but rather appeared to be related to the principal instructor.

IS IT BETTER TO BE FEARED? THE IMPORTANCE OF COURSE STRUCTURE AND METHODS OF LEARNING ASSESSMENT

Although these results provided a straightforward answer to the original research question, further examination of the data revealed significant differences between groups of students in the amount of learning produced. The differences were not between the treatment and control groups but rather appeared to be related to the principal instructor.

There were two major differences in how the two instructors structured this phase of the course. The first difference involved expectations of students’ preparation for the class. In Professor A’s classes, student learning was assessed through the use of scheduled quizzes and exams, none of which took place during the study period. Professor B, however, administered daily random oral quizzes on the assigned readings throughout the semester, with each quiz worth 2% of the final grade. ⁴ Students who were not prepared for class risked a lower grade and embarrassment in front of their peers.

The second difference between the two instructors entailed the requirement of further work with the tested content. The students in Professor A’s classes moved on to other topics, whereas Professor B’s students—after the posttest was administered—were assigned a version of the group exercise as an individual paper. This assignment was due one week before the retention test and was worth 10% of the final grade.

There are specific reasons why these two distinctive approaches could lead to significant differences in the amount of learning produced in each instructor’s class. First, the possibility of a daily quiz likely encouraged Professor B’s students to increase the time they spent on the assigned reading before class, resulting in higher pretest scores. Second, the assignment of a substantial paper on constitutional design due before the retention test likely reinforced Professor B’s students’ learning of the material, thereby yielding much higher retention scores.

FINDINGS: COURSE STRUCTURE AND METHODS OF LEARNING ASSESSMENT MATTER

As expected, Professor B’s students were significantly better prepared for the initial assessment instrument, scoring 0.86 points higher on the pretest (i.e., p <0.005) than Professor A’s students (table 3)—a difference that held steady in the posttest assessment. This is an important point: whereas the scores of Professor B’s students were higher on both instruments, there was no difference in pretest or posttest gain between the two instructors’ students. Any advantage enjoyed by Professor B’s students was a result of their being better prepared, not from any difference in the instructor’s efficacy in the classroom.

Table 3 Results by Instructor

Also as expected, there was a significant difference in retention-test scores between the two instructors’ students (i.e., 1.26 points, p<0.005). In addition, the gain in retention scores of Professor B’s students was much higher than that of Professor A’s students: 0.89 versus 0.49 points, a near-significant difference of 0.40 points (i.e., p <0.10). ⁵

…this study simply confirms the obvious: when students are forced to read, think, and write about a topic, they learn it better than if they only listen to lectures about it.

These findings suggest that differences in course structure and methods of learning assessment contributed to how much students were learning and retaining during the study. A second linear-regression model using a Professor B dummy variable to account for the difference in pedagogical approaches, as well as a control for the final grade, supported these findings. The results are shown in table 4.

Table 4 Linear-Regression Model: Instructor and Final Grade

⁺ p <0.10, **p <0.005

DISCUSSION AND CONCLUSION

These findings suggest several lessons that can be applied to introductory political science courses in general. First, one-time PBL collaborative group exercises should not be expected to be more effective than the standard lecture-discussion format in helping students to learn specific content. The PBL exercises may be useful in varying the classroom routine and they may help to develop skills or encourage other types of learning. However, the findings in this study suggest that they do not produce more learning of specific content—and may even produce less.

This does not mean that PBL as a technique is ineffective. As discussed in this article, other studies (i.e., Centellas and Love 2012; Vaidyanathan and Rochford 1998; Williamson and Gregory 2010) reported higher rates of learning when PBL group-based techniques were used during multiple class periods or when students were required to spend significant time working together outside of class. This study suggests, however, that those higher rates of learning are not due to the use of the PBL technique but rather to the requirement that students spend extra time and energy working with the content the mastery of which was being assessed in the study.

Other findings from this study, although not surprising, support this assertion. By establishing that students who face daily quizzes and are assigned a substantial paper learn specific content better, this study simply confirms the obvious: when students are forced to read, think, and write about a topic, they learn it better than if they only listen to lectures about it.

The main result of this study is that when we, as educators, can convince students to spend more time reading and working with specific content, the students apparently learn that content better. Finding the best way to motivate our specific student populations to do the extra work is perhaps the most important task we can do. PBL group learning can be an excellent way to accomplish this goal, but many other techniques may work just as well. v

Notes