â€œObservable Objectsâ€: Assessing a Study of Instructorsâ€™ Grading by Ellen Quandahl
Iâ€™m going to use a small, under-theorized assessment project undertaken in the Lower Division Writing Program (which I direct) at San Diego State as an anecdote for thinking about practices of documentation within assessment.Â Â Our project–to gather information about how teachers grade student writing–was what the assessment literature calls a â€œformative evaluation,â€ undertaken to help us answer the question â€œHow can we improve our program?â€ (Erwin 7).Â Â It was driven partly by suggestions from TAs that they needed more clarity about what constituted the A, B, and C paper in the first semester course and partly by the institutional mandate to present an assessment plan, guided by principles articulated by the American Association for Higher Education, as part of the departmental self-study prepared for external review.Â Â (And I should add that this work antedates a system-wide call, this year, for an assessment driven curriculum, in response to which our campus is, I think wisely, looking into what we think constitutes G.E. through a series of faculty work-groups, even as departments develop learning outcomes and assessment instruments.)
My title, of course, refers to Foucaultâ€™s idea that todayâ€™s people, in order to be taught and brought into productive life in the society, are assessed, described, and made individual through an analytical pedagogy built upon schemas of observable, measurable or classifiable objects (218).Â Â (That was an idea that had the writing profession shaken up a number of years ago, but which, oddly, seems to have receded as administrators and faculty embrace a language of evidence about outcomes and service to stakeholders.)Â Â Â Iâ€™m interested in what happened when the text summarizing data in our study became an â€œobservable objectâ€ in this sense, a text reinforced by institutional demands and read back to us in the course of a departmental review.Â Â But I want also to refer to Jim Slevinâ€™s fine recentÂ CEÂ article, in which he argues that the language of assessment, with its attention to evidence of learning, also obscures theintellectual workÂ of students and faculty, which is difficult to make visible, because it seems â€œprivate and inaccessibleâ€ (294).Â Â Slevin calls for bringing this work into the picture.Â Â He writes: â€œOnce this intellectual work can be seen. . . it can be studied and reviewed with rigor, according to norms generally recognized in the academyâ€ (298).Â Â Iâ€™m not sure how much heâ€™s aware of the echo of Foucault, turned upside down, in that sentence, but what heâ€™s calling for is a making visible of the things that tend to defy normative documentation: like the labor of producing and testing knowledge.
For our project, we asked TAs who were using a common assignment sequence to turn in student papers responding to a prompt which asked for the analysis of information in a piece by Clifford Geertz.Â Â We invited departmental instructors to read four unmarked papers and to grade themÂ using the citeria for evaluation that had been given to the students and used by their instructors. These criteria were customized for the assignment from a one-page list of course criteria, not unlike the â€œoutcomesâ€ document recently published by the WPA.Â Â Our idea was simply to see the grading by TAs, lecturers and tenured faculty.Â Â Â We put the grades on a chart, which showed that there was not perfect consistency of grading for any one paper.Â Â Some were very close, but some papers received a wide array of grades.Â Â The departmental review took place just after we had collected these data, and we shared with the reviewers this interpretive but uninterpreted document.
Our own follow-up was to select several papers, both the consistently marked and the anomalous, for discussion in departmental meetings.Â Â I suggest that this ought to have been the key moment of data collecting, for the discussions revealed these things: that in addition to our different inclinations in reading Geertzâ€™ argument, which were substantial, and our tendencies to weigh the six criteria differently, instructors also brought to their reading additional criteria, mostly having to do with matters of voice, correctness, and their sense of evidence of ESL writing.Â Â These frequently overtook the focus on discussing whatÂ studentsÂ had to say about Geertz, to which, in fact, there was a quite high resistance.Â Â I believe, in other words, that a transcript of the discussions would reveal what one might call, following Christine Ross, the latent force of approaches to writing and reading quite different from those in our programmatic documents.Â Â Such a transcript would make visible a range of discourses about literacy among our teachers, including: the discourses of past and recent schooling, of what one might term popular or public notions of what first-year writing ought to do, of the culture of other institutions where our instructors also teach, of programmatic documents, and of rhetorical and practical training offered in our TA program and graduate courses.
Our reviewers, however (who were selected by us and who wrote a powerfully positive response to the program) wrote three lines of text interpreting the project that Iâ€™ve outlined.Â Â These were bullets, noting (despite our written criteria both for the course and the assignment) â€œinconsistency of grading standards across sections in GE courses,â€ and â€œthe absence of agreed upon common standards of student writing in GE courses,â€ and urging us to â€œinvestigate a common final assessment for the course to establish common standards for student outcomes.â€
My point is not to dispute these conclusions (for consistency across multiple sections is a real issue), but rather to suggest that they could not have been otherwise.Â Â Our chart of grades looked like a kind of â€œtrue discourse,â€ providing a set of facts, arrayed and made visible, about difference in teachersâ€™ grading.Â The array seemed at once to speak a question, to suggest that in this project what we sought to know is notÂ howÂ are we grading, but how can we normalize, and the answer came rushing in–by establishing common standards for outcomes, or rather clearer, more consistently gradable outcomes than those I spoke of a moment ago, which would in effect normalize our diversity of both students and teachers.
This makes very pointed the most important issue for assessment, and that is that one must carefully develop the question for what one seeks to know.Â Â Out of this attempt, an intriguing set of questions emerges:Â Â How, in rhetorical terms, is our faculty reading student texts?Â Â What are the interpretive models that allow us to produce a reading of a student paper?Â Â What are the markers that indicate ESL writing and what difference do those features make to teachers?Â Â How do the elements listed in our evaluative criteria square with instructorsâ€™ schemas, and how do those criteria become discrete tasks, which, to quote Christine Ross, â€œtake on specific force and produce a readingâ€?Â Â These are formative questions.Â To make such answers visible might be what Slevin is getting at: whether we see composition as preparatory to or enacting the intellectual work of the university, whether and how we expect writing students to participate in making knowledge and testing its truthfulness.Â Â I suggest that a better study would help us get at â€œthe relationship between assessment and the intellectual purposesâ€ (Slevin 289) of our particular institution.
Ewrin, T. Dary.Â Â Assessing Student Learning and Development.Â Â San Francisco: Jossey-Bass, 1991.
Foucault, Michel.Â Â The Archeology of Knowledge.Â Â New York, Pantheon, 1969.
Ross, Christine.Â Â â€œIn Pursuit of the â€˜Clearâ€™ and the â€˜Fairâ€™: Education Reform and the Limits of Institutional Discourse.â€Â Â Unpublished paper.
Slevin, James.Â Â â€œEngaging Intellectual Work: The Facultyâ€™s Role in Assessment.â€Â Â College EnglishÂ 63.3 (January 2001): 288-305.