Chapter 7 Response March 23, 2007
Posted by flapp55 in Uncategorized.trackback
Robert A. Blackman
EDT 646
Response #11
Evaluation
When it comes to evaluation of online learning, there seems to be some controversy regarding what information should be included. R.E. Clark states that one should choose to evaluate either the instruction or the method of delivery while Kozma is of the opinion that the two are interwoven and need to be evaluated together. Of the two theories, I feel that Kozma’s views of evaluating online learning is more appropriate because it affords the evaluator more in-depth information as to what works and what doesn’t in the instructional process. In addition, evaluating both the instruction and the delivery method may yield valuable insights to how students utilize technology as a learning tool. To properly evaluate the online learning activity that I have set forth in paper #2: Saving the Rain Forest, it will be necessary to evaluate both my instruction and the method of deliver for the purposed unit.
To create a good formative evaluation I will first need to define the purpose of the evaluation. This purpose is three-fold; 1) to determine if my instructional design is causing an increase in student achievement in their core curriculum classes, 2) to see if the students can transfer the skills taught to new and different problems, and 3) if tappedin.org helps to facilitates this outcome. In order to guide my evaluation I would then need to draft an evaluation statement. This statement provides a mechanism with which I can begin to target the methods and measurement approaches that best align with an evaluation effort (Dabbagh, Ritland, 2005). To conduct a balanced evaluation that takes into account all factors, I will need to craft activities that address levels 1, 2, and 3 of Kirkpatrick’s model which are respectively reaction, learning, and behavior.
Since increasing my students’ achievement levels in their core curriculum classes is my first goal, my evaluation must include activities that help uncover this information. To achieve this, my first activity will be to discuss my design with colleagues to determine if the information that I plan on presenting to the students will be useful. The feedback that they provide me with will help me to determine if the information will be valuable when planning for my supplementary materials and give me some guidance for the direction I want my students to take in their learning. I would also ask them to help me design a pre and post test to use in determining their prior knowledge and their achievement after the unit has been delivered. Furthermore, by making my colleagues aware of my design, I may be able to use data taken from their assessments to further supplement my evaluation.
Informal feedback will also be gathered from the students. This feedback will come in the form of weekly reflections, student postings, and final projects. In their reflections students are to sum up what new information they have learned and how they might apply this information to a new problem. Additional data pertaining to the students’ postings and the frequency of their activity will also be collected from the MUD as well as the assessment of their finial project. Students will also give input to help evaluate the method of delivery. This information will be gathered by survey and informal interviews regarding any difficulties they might have or suggestions for improvement. These informal feedback mechanisms provides information for the instructor to review so that they can think about changes for the course or can be used as a data source for more rigorous qualitative analysis (Dabbagh, Ritland, 2005).
To achieve improvement in the lesson design, the data from many of these activities will be collected and repeated throughout the instruction. Using this method will afford me the opportunity to correct unforeseen complications that may have a negative effect on my desired outcomes for student achievement. Evaluation of the students post test and their final projects will also give me insight to the overall success of my design.
Hi Robert,
I am always impressed by your thorough and excellent writing style.
The data collection and review methods you describe sound like an effective way to evaluate and improve your rain forest learning tool.
One worry I would have about this approach is the time required to review, analyze and follow through on such extensive data. It can be a challenge to determine how much data is enough/necessary. One way to filter the set, which I think you have already pointed at, would be to focus only on data that supports your established learning goals. Perhaps there would be ways to present the collected data that provide clear and easy to digest metrics that support your established goals.
Again, nice post/plan. Makes me think a lot.
Robert,
This sounds like a great project. I think you have sufficiently covered the evaluation process here, and I like that you also plan to revise your online learning tool using learner and teacher feedback. This is important.
Andy,
Your statement about applying what the students have learned from the online learning experience to a new problem is a design step that should be included in the learning process. This is a step that I will try to remember to incorporate in my online learning designs.
Your evaluation also included planning for unforeseen complications, which I also believe is very important in designing online learning in the K-12 school environment.
Andy,
The fact that you included your colleagues as a “first step” in your evaluation process is excellent. I have a bad habit of wanting to work out the kinks in my plans before discussing them, and I am learning that this can be a time consuming mistake. Teamwork, although stressful at times can be the best first step. Thanks for reminding us of that!
Cindy