This is where we’ll talk about Susan Katz and Lee Odell, “Making the Implicit Explicit in Assessing Multimodal Composition: Continuing the Conversation.” (PDF) This is the introduction to their special issue of Technical Communication Quarterly that explored issues with assessment in multimedia and I think it speaks in part to where the field is now on multimedia. I think many have accepted the premise of Anderson’s argument, that multimedia projects have a place in writing classrooms. But a question remains on how to assess them beyond “I know what I like and what I don’t.”
Two other thoughts for now. First, this might be a special issue of TCQ that some of you might want to track down for your own projects since there are a number of articles here that speak to both the “teaching of writing” and “professional writing” side of the program. Second, keep the issues that Katz and Odell raise about the challenges of assessing multimedia projects in relation to the “Digital Artefact” project that is due this week for “E-Learning and Digital Cultures.” More on that in a different post.
After reading this, I think I probably will want to track down the full issue and read into this a little more at some point…
I think Katz and Odell make a valid point in arguing that instructors need to make the implicit explicit. That is, to define their criteria for assessment, discuss it (not only with colleagues, but with students too), and rethink it. They caution that “If we fail to engage in the ongoing process of defining, testing, adding to, refining, and making explicit the bases for our judgments, we limit ourselves as teachers and assessors. We penalize our students and we undermine the maturing of our profession.” (4)
I’ve never taught, but I always thought that assessment and grading would be one of the hardest parts of the job if I did–and for many of the same reasons that the authors discuss here. I think assessment would be difficult enough with an assignment composed of all written (or typed) text, but would get even more complicated when images, audio, video (or some combination of these) are incorporated into the finished product.
Personally, it’s a lot easier for me to assess an argument written out in plain text than it is to assess something that incorporates many other forms. For example, I find it easier to let my emotions pull me one way or the other when looking at an image or listening to a song. I wonder why this is.
http://www.daniellebreann.wordpress.com
Danielle, haven’t read the article yet, but your comment about songs/images and emotions caught my attention. I think maybe part of this could be that, for both images and songs, emotions and emotional appeals are almost expected. For many of them, they’re kind of supposed to move you, whereas with written arguments, especially academic arguments, they are meant to persuade you. Moving you versus persuading you–I think those are definitely related, but have some key differences.
I agree that it would be much easier (for me at least) to assess a written text than a visual one. Visual texts can potentially consist of much more implicit messages that may be easily missed by the assessor. And as we’ve heard again and again in numerous classes, writing instructors should write themselves to be effective teachers of writing, which should then also hold true for visual rhetoric–instructors evaluating visual rhetoric should be trained in and practitioners of it themselves.
The thought of teachers arguing over the definition of organization struck me as something I had never thought of. In my head, this is something teachers should know since they are the ones who’s job it is to assess it. I had never really considered the fact that it wasn’t something that fell into a universal rubric (as it appears to do so from the ‘grading rubrics’ that often accompany the essay prompt). I realize this is pretty naive, but considering my absolute refusal to teach, this is what happened.
All that being said, I think Katz & Odell absolutely have it right. To reiterate Danielle, assessors/teachers need to go beyond “cuz I like it” into a reflective stage that will lead to active evaluating, revising, etc.
I also thought it was interesting how they noted that new media present several challenges to assessing writing, when writing on the page is hard enough to assess. They brought up a good point that “After all, wikis and instructional videos rely heavily on qualities of written and spoken language
that we are accustomed to dealing with” (3), meaning that new media or not, these terms for assessment need to be defined and made explicit.
http://msyapin.wordpress.com
I had a similar moment, I think, where I thought…wait! They don’t have a universal definition for something as fundamental as organization?! I calmed down, considering there are many forms of acceptable organization depending on the rhetorical situation, and besides, aren’t they assessing how well the student followed the prompt?
I came to another conclusion, which is that all too often (and I don’t see how this will be different with multimodal texts), students are not necessarily learning a specific skill or body of knowledge as they are figuring out what the professor wants to see/hear. I’m not saying this is all bad, as dealing with people and determining what they are after from you is really an essential life skill. However, it makes sense to try to come up with some more concrete method of assessing multimodal projects, as they are sure to appear with increasing regularity.
http://visiblycynical.wordpress.com
I agree that lots of times in K-12 it is about writing how your teacher wants or doing what you know they like. I wonder if this is something that online classes overcome? If you have a “virtual” teacher, are their grading preferences as obvious?
http://msyapin.wordpress.com
Students at all levels often try to write what the teacher wants, but at all levels, that shouldn’t be the goal. You’re not going to find any contemporary pedagogical theorists that suggests it’s a “good idea” for this to happen. Rather, the goal is to get students to write for an audience beyond the teacher and to try to see some connection between what’s happening with a particular assignment beyond that class. So I would say that it is “all bad,” and your comment here Sarah explains why it’s bad– “dealing with people and determining what they are after” is an example of writing beyond the teacher to me.
I’m kind of surprised that anyone would be surprised that there isn’t a universal definition of organization, frankly. What would that be? Please tell me it isn’t the 5 paragraph essay….
The argument about organization didn’t strike me as so out of the norm really. As Professor Baker mentioned in another class, all evaluation is a comparison of some form or another; that is to say this is good or bad in comparison to that, which on one level or another is a subjective judgment.
It didn’t strike me either (I remember that discussion in Dr. Baker’s class, too) but I think the fact that there *are* so many different ways to define evaluative terms isn’t talked about much…
In fact, this was something I started thinking about more after I took Dr. Baker’s class last semester. In the Writing Center, I’ll usually ask the students that I tutor what their main concerns with are with what they’re working on. A lot of times, I’ll get responses like “my paper is really unorganized,” or “it doesn’t flow.”
But when I start asking students questions like “why do you think it’s unorganized?” they seem to have a hard time answering these types of questions. I think it’s impossible to define these terms in a broad context anyway, but I wonder if these students have had teachers in the past who have neglected to define what “organization” means in particular assignments. If so, then what do they have to work from?
Yet still, a lot of grading rubrics include these evaluative terms (organization, creativity, etc). I think Anderson is arguing that, in failing to define these, instructors are failing their students.
http://www.daniellebreann.wordpress.com
^And by “Anderson,” I meant “Katz and Odell.” Getting my authors mixed up.
http://www.daniellebreann.wordpress.com
I agree that defining what these commonly used evaluative terms mean explicitly, especially as to how they are rooted in the individual student’s paper, but at the same time I think it may be also partly due to the difficulty of speaking about writing. That is to say, the invisible processes that make it hard for writers to bring things to the surface and talk about them, the “I can’t explain it, but I know good writing when I see it,” when writers are unfamiliar with their own writing processes, and when they are just plain confused as to what organization means. There are so many facets of the issue. I think some teachers may also be afraid of being too specific in defining these evaluative terms because students may then just start jumping through the hoops they lay out for them. Does that make sense? But I think you’ve again touched on something as far as the rubrics are concerned, as well as how can you “punish” someone for not giving you what you want, when you haven’t explained to them what that is?
I agree with Katz and Odell that teachers should define the criteria they intend for the students to follow and then assess those students accordingly. I feel like every teacher should make grading more of a conversation between themselves and the students they are grading as well as discussing assignments wit other teachers.
It seemed like Katz and Odell also discussed techniques for assessing digital texts as well such as the Wikis. I wasn’t sure whether Katz and Odell feel that assessing digital text is much different than assessing print text. I know that they acknowledge that teachers didn’t used to have to consider how to assess a student’s project for it’s audio/visual qualities like the ones online tools make possible for us to use.
I agree that the authors’ point on instructors explicitly defining assignment criteria expected of students is only fair. How can you fault someone for not doing what you want them to, when you don’t explain what it is that you want? That’s tantamount to setting them up for failure.
I think this introduction did a fine job of both acknowledging some of the stumbling blocks that get in the way of defining assessment criteria and asserting that educators should not use these stumbling blocks as crutches. As others have said, just because a quality is difficult to define does not mean that “I’ll know it when I see it” is an acceptable substitute. There needs to be a balance here. “We are not well served by dogmatic overgeneralization, but we—and our students—will be well served by clarity and forthrightness” (4). I would definitely be interested in searching out the rest of this issue; I’d like to read what the various authors have to say about assessing multimedia compositions and wikis.
I appreciated that this introduction to the issue recognized, not just on a general level, but on a content-specific level, the ambiguity of conceptual assessment criteria. Coming up with tangible, explicit criteria for assignments is difficult – I feel like I’m either too simplistic (ie: assignment is double spaced, 12 pt font, etc.) or too abstract (demonstrates understanding of genre). This last example – genre – isn’t too bad, but you get my point. It was good to see examples, and the conversation sparked over what to do next. I also appreciated that the authors pointed out the value of having multiple definitions of organization. I think about students who come to the UWC who are concerned with grammar, and my gut instinct is to frown at the teacher who grades only on grammar. I realized, though, after reading this, that different writing situations call for different expectations and demands – even if they don’t make sense universally, it is useful for students to learn to recognize that different instructors will emphasize different qualities. Again, though, it comes back to being explicit and transparent about even this – an honest approach to rubric-making, maybe.
I’m interested, like the others, in tracking down the rest of this issue. Is there a way to order a print copy, or is it a matter of trying to find the individual articles?
http://parablematernal.wordpress.com
You can get the whole thing from the EMU library more or less from the same link I’m giving you here for this article. Click where it says “Volume 21″ and then “Issue 1″ and you’ll see what I mean.
I don’t know if this will work, but here’s a link that might work:
http://www.tandfonline.com.ezproxy.emich.edu/toc/htcq20/21/1