“Observable Objects”: Assessing a Study of Instructors’ Grading by Ellen Quandahl

I’m going to use a small, under-theorized assessment project undertaken in the Lower Division Writing Program (which I direct) at San Diego State as an anecdote for thinking about practices of documentation within assessment.  Our project–to gather information about how teachers grade student writing–was what the assessment literature calls a “formative evaluation,” undertaken to help us answer the question “How can we improve our program?” (Erwin 7).  It was driven partly by suggestions from TAs that they needed more clarity about what constituted the A, B, and C paper in the first semester course and partly by the institutional mandate to present an assessment plan, guided by principles articulated by the American Association for Higher Education, as part of the departmental self-study prepared for external review.  (And I should add that this work antedates a system-wide call, this year, for an assessment driven curriculum, in response to which our campus is, I think wisely, looking into what we think constitutes G.E. through a series of faculty work-groups, even as departments develop learning outcomes and assessment instruments.)

My title, of course, refers to Foucault’s idea that today’s people, in order to be taught and brought into productive life in the society, are assessed, described, and made individual through an analytical pedagogy built upon schemas of observable, measurable or classifiable objects (218).  (That was an idea that had the writing profession shaken up a number of years ago, but which, oddly, seems to have receded as administrators and faculty embrace a language of evidence about outcomes and service to stakeholders.)   I’m interested in what happened when the text summarizing data in our study became an “observable object” in this sense, a text reinforced by institutional demands and read back to us in the course of a departmental review.  But I want also to refer to Jim Slevin’s fine recent CE article, in which he argues that the language of assessment, with its attention to evidence of learning, also obscures theintellectual work of students and faculty, which is difficult to make visible, because it seems “private and inaccessible” (294).  Slevin calls for bringing this work into the picture.  He writes: “Once this intellectual work can be seen. . . it can be studied and reviewed with rigor, according to norms generally recognized in the academy” (298).  I’m not sure how much he’s aware of the echo of Foucault, turned upside down, in that sentence, but what he’s calling for is a making visible of the things that tend to defy normative documentation: like the labor of producing and testing knowledge.

For our project, we asked TAs who were using a common assignment sequence to turn in student papers responding to a prompt which asked for the analysis of information in a piece by Clifford Geertz.  We invited departmental instructors to read four unmarked papers and to grade them using the citeria for evaluation that had been given to the students and used by their instructors. These criteria were customized for the assignment from a one-page list of course criteria, not unlike the “outcomes” document recently published by the WPA.  Our idea was simply to see the grading by TAs, lecturers and tenured faculty.   We put the grades on a chart, which showed that there was not perfect consistency of grading for any one paper.  Some were very close, but some papers received a wide array of grades.  The departmental review took place just after we had collected these data, and we shared with the reviewers this interpretive but uninterpreted document.

Our own follow-up was to select several papers, both the consistently marked and the anomalous, for discussion in departmental meetings.  I suggest that this ought to have been the key moment of data collecting, for the discussions revealed these things: that in addition to our different inclinations in reading Geertz’ argument, which were substantial, and our tendencies to weigh the six criteria differently, instructors also brought to their reading additional criteria, mostly having to do with matters of voice, correctness, and their sense of evidence of ESL writing.  These frequently overtook the focus on discussing what students had to say about Geertz, to which, in fact, there was a quite high resistance.  I believe, in other words, that a transcript of the discussions would reveal what one might call, following Christine Ross, the latent force of approaches to writing and reading quite different from those in our programmatic documents.  Such a transcript would make visible a range of discourses about literacy among our teachers, including: the discourses of past and recent schooling, of what one might term popular or public notions of what first-year writing ought to do, of the culture of other institutions where our instructors also teach, of programmatic documents, and of rhetorical and practical training offered in our TA program and graduate courses.

Our reviewers, however (who were selected by us and who wrote a powerfully positive response to the program) wrote three lines of text interpreting the project that I’ve outlined.  These were bullets, noting (despite our written criteria both for the course and the assignment) “inconsistency of grading standards across sections in GE courses,” and “the absence of agreed upon common standards of student writing in GE courses,” and urging us to “investigate a common final assessment for the course to establish common standards for student outcomes.”

My point is not to dispute these conclusions (for consistency across multiple sections is a real issue), but rather to suggest that they could not have been otherwise.  Our chart of grades looked like a kind of “true discourse,” providing a set of facts, arrayed and made visible, about difference in teachers’ grading. The array seemed at once to speak a question, to suggest that in this project what we sought to know is not how are we grading, but how can we normalize, and the answer came rushing in–by establishing common standards for outcomes, or rather clearer, more consistently gradable outcomes than those I spoke of a moment ago, which would in effect normalize our diversity of both students and teachers.

This makes very pointed the most important issue for assessment, and that is that one must carefully develop the question for what one seeks to know.  Out of this attempt, an intriguing set of questions emerges:  How, in rhetorical terms, is our faculty reading student texts?  What are the interpretive models that allow us to produce a reading of a student paper?  What are the markers that indicate ESL writing and what difference do those features make to teachers?  How do the elements listed in our evaluative criteria square with instructors’ schemas, and how do those criteria become discrete tasks, which, to quote Christine Ross, “take on specific force and produce a reading”?  These are formative questions. To make such answers visible might be what Slevin is getting at: whether we see composition as preparatory to or enacting the intellectual work of the university, whether and how we expect writing students to participate in making knowledge and testing its truthfulness.  I suggest that a better study would help us get at “the relationship between assessment and the intellectual purposes” (Slevin 289) of our particular institution.
Works Cited

Ewrin, T. Dary.  Assessing Student Learning and Development.  San Francisco: Jossey-Bass, 1991.

Foucault, Michel.  The Archeology of Knowledge.  New York, Pantheon, 1969.

Ross, Christine.  “In Pursuit of the ‘Clear’ and the ‘Fair’: Education Reform and the Limits of Institutional Discourse.”  Unpublished paper.

Slevin, James.  “Engaging Intellectual Work: The Faculty’s Role in Assessment.”  College English 63.3 (January 2001): 288-305.