Lisa Karosas
2016 Differentiated Project
Results By Student Group
Results By Unit / Standard
The results of 25 total assessment items were considered in this study. When individual student "mastery" is defined as answering 80% (20) of those questions correctly, it was found that 70% of students in the Honors level section were able to demonstrate mastery, while only 60% of the Comp level were able to do so. Though these results are not disappointing, they do not immediately seem outstanding.
However, if we define "mastery" differently, wherein only 70% (18) of those questions are answered correctly, the results indicate that "Teaching with Analogy" is a highly effective instructional strategy; 90% of students in the Honors section achieved mastery of the standards connected to the analogies provided, while 95% of students in the comp section achieved mastery over those same assessments. Those students who did not demonstrate mastery over these standard-based assessments answered only approximately 55% of the standards-based assessment questions correctly.
Analogies were presented to both of these groups in the same manner, over the same period of time. The academic behavior of these two groups historically varies, however. Honors students typically seem to exhibit ideal study behaviors which may include, for example, regular note-taking and seeking help as needed. They also tend to have a documented history of success in related courses. Students enrolled in the Comp section may not have an equivalently strong background in science comprehension and may or may not consistently practice positive academic behaviors. Clearly, the results of this study suggest that Comp level student achievement at various levels of Bloom's Taxonomy is not highly dependant upon prior science knowledge or study behaviors.
The Pennsylvania standards outlined for secondary chemistry instruction are limited in their detail of actual content and tasks, but wide in their scope. This is, in part, likely due to the nature of chemistry as a largely foundational and cumulative science in which simple principles need to be applied before one is able to fully comprehend those that are more complex. Six units presented, practiced, and assessed over 27 weeks are represented by only three content standards (indicated with the letter "C" in the reference code) and four grade-specific standards (indicated with the number "10" in the reference code). Identifying a technique which has the demonstrated ability to enhance students' memory of facts and comprehension of principles against the same few standards over long periods of time could certainly prepare students to access a greater depth of knowledge upon which they may refer when more complex principles are presented.
Here, student assessment data was compared to related standards to determine how effective each analogy, or group of analogies, may have been. In most cases, approximately three-to-four assessment items were evaluated per unit and standard. The exception to this is evident for the content contained in Unit 4, Chemical Bonding, where nine assessment items were considered. This unit is particularly long and includes the basis for both ionic and covalent bonding; the assessment items related to each corresponding standard are divided nearly evenly. The percentage of students who answered individual standard-based assessment items correctly was determined for each assessment item included in this study. Where that percentage exceeded 70% or 80%, that assessment item/standard was deemed "mastered".
Notably, we can observe that it doesn't matter how "mastery" is defined for the assessment items related to the content of Unit 2, Atomic Structure, and the simple analogies, "Hotels and Electron Behavior" and "Hotels and Electron Configurations"; 100% (all three) of the assessment items related to this content and these analogies were answered correctly 80% of the time. Likewise, where only one assessment item was analyzed for the Unit 6, Stoichiometry, content to evaluate the effectiveness of the "Reaction Stoichiometry and Cooking Recipes" extended analogy, 80% of the time students answered this question correctly. Similarly, 100% (all four) of the assessment items analyzed for the Unit 5A, Chemical Reactions, content and the extended analogies, "Legos & Synthesis/Decomposition Reactions", "Vampire Love Triangles and SIngle-Displacement Reactions", and "Square Dancing & Double-Displacement Reactions", were answered correctly 70% of the time by the entire body of students. Since the simple "hotel" analogies proved so successful, this research cannot support the suggestion that complete, descriptive mapping is the most critical component necessary to effectively communicate the translation of an analog to its target. Due to the exceptional performance demonstrated for these standards, we might suggest that students found these analogies to be the most interesting, the most memorable, or the easiest to map similarities and differences between the analog and target.
In this manner, students clearly indicated an overwhelmingly positive sentiment for this particular instructional strategy. In every area, scores above eight were earned. The most telling information that we can glean from this data is that students may have felt that the use of analogy helped them to recall and understand information more than it may have made the content more interesting to learn. Of course, being able to "remember" and "understand" ideas are the foundation for building depth of knowledge according to Bloom's Taxonomy. Their ability to do so is supported by the mastery data for all assessment items as well as individual standards.
The content standards resulting in the highest degree of mastery for this study pertained to writing electron configurations and describing electron behavior, classifying types of chemical reactions, and identifying the key components describing the stoichiometry of a reaction. The survey results relative to these topics and related analogies are highlighted above with stars.
The assessment data and survey data agree and support the effectiveness of the "Hotel" analogy used to describe electron behavior and electron configurations in Unit 2, Atomic Structure. One might also discern that the students' perceived ability to remember the way in which reaction stoichiometry was similar to cooking recipes may have enabled then to better master related assessment items. Much of the remainder of this data was insignificant. The data ranking which of the analogies most peaked students interest also yielded nothing notable.
Results By Depth of Knowledge Required
Since the delivery of this chemistry course was planned using backward design, assessment items were not written expressly to complement the standards and analogies taught in class. Rather, those assessment items pre-existed in a database; their selection for inclusion in this study was based on how closely they matched the nature of the standards that each analogy attempted to deconstruct.
That said, it wasn't until this study was well underway that the decision was made to select assessment items which spanned the levels of mastery described in Bloom's Taxonomy. It stands to reason that if simple or extended analogies could enhance the low-level processes, remembering and understanding, then, perhaps, depth of knowledge would be naturally enhanced and students would show mastery at higher levels of the pedagogy. However, the opposite rationale doesn't necessarily apply; if students didn't respond well to the mapped analogies, simple or extended, they might still employ other strategies to learn the content independently and excel at low levels and high levels. As a means of testing a hypothesis to determine its potential for further exploration, assessment data was collected for all levels of Bloom's Taxonomy.
Every assessment item chosen for this study was assigned a level of Bloom's taxonomy according to the task required of the student responding to the item. All those items requiring only the recollection of a definition or clear description were assigned the "REMEMBER" classification. Those items which required a student to interpret the meaning of a definition or basic description were assigned the "UNDERSTAND" classification. Items which required students to use their acquired knowledge to solve a problem were classified as "APPLY" items. Finally, some items presented options to students, prompting them to choose the most appropriate or most correct option as it related to a special scenario. These items were classified as "EVALUATE" items. The percentage of these questions answered with mastery, either 70% or 80% correct by all students, were noted according to classification.
Which Analogies Best Helped Students Remember?
Which Analogies Best Helped Students Understand?
Overview
Students were asked to rank how they felt their interest, ability to remember ideas, and ability to understand ideas were impacted by the use of analogy in their chemistry instruction. Responding with a "10" indicated that the technique made a unique impact in an area, while responding with a "1" indicated that the technique did not.
In an effort to ascertain student perception with regard to the use of analogies in chemistry instruction, a survey was issued to the entire student population selected for the study. 19 of 30 surveys were returned. The comprehensive, raw survey data can be found by clicking HERE.
Survey Results
Bloom's Taxonomy describes the natural evolution of learning from the introduction of basic facts (remembering), through the ability to solve problems (applying, analyzing, and evaluating) and, finally, to the generation of new ideas of methods (creating).
It is in the APPLY, ANALYZE, and EVALUATE levels where students are expected to do the following:
-
use gained knowledge and understanding in a new context
-
distinguish between different parts and understand how they are all connected
-
justify a decision, explaining which options are better than others and why
To earn high scores on assessments in any chemistry class, students must work through and successfully solve sometimes complex, multi-step problems which require access to these higher order thought processes.
As we have seen throughout this study, when "mastery" of an assessment item or a standard is defined at only 70%, the data support the effectiveness of this technique. An alarming 80% of assessment items in nearly every category were answered correctly by this population. Here, there is no room to argue that while analogies facilitate low-level learning, their utility doesn't extend to problems which require higher-order thinking and access to greater depths of knowledge. Quite the contrary! This data suggests that students were able to answer higher order questions correctly as often as they were able to answer lower order questions.
When the definition of "mastery" applied is reflective of 80% positive responses, a different conclusion must be considered. Here, it's obvious that many of the students participating in the study were able to easily recall, remember, and interpret (understand) the technical chemistry content concepts related to each analogy (approximately 70%), but the ability to apply those principles using problem-solving strategies fell markedly to 60%. Then, when prompted to access a greater depth of knowledge in the evaluation of specific scenarios, mastery of this level plummeted to 40%. It may be notable to consider and further investigate the disparaging difference between the percent of items answered with mastery for the APPLY and EVALUATE classifications when "mastery" is defined differently.