First published: https://www.linkedin.com/pulse/response-aacus-solid-ground-report-ug-performance-andrew-mccann
Congrats to all the schools, faculty, & rubric-writers…plus AAC&U’s leadership and tireless, long-sighted work in completing “On Solid Ground,” a report on the massive assessment work completed through the VALUE project (Valid Assessment of Learning in Undergraduate Education). Inside Higher Ed just published a summary of the report here.
I haven’t thought about the VALUE project in a long time…but a few aspects to this positive treatment by IHE stand out:
Levels, along with language, are always the most debatable aspects of a rubric. From memory, VALUE intended a ‘4’ (called ‘Capstone’) to be equivalent to the abilities of a graduating senior. A ‘1’, or ‘Benchmark,’ was roughly equivalent to an incoming freshman. The rubrics have a ‘0’ option, so they are 5-level rubrics.
I just checked VALUE’s FAQ and the rubric description and didn’t find any discussion of the value judgment inherent in these levels. The report itself does include a thoughtful discussion of how to interpret ‘Capstone’ and that the rubrics are not intended as a literal scale, plus guidance on how to interpret the data.
At 90 credit hours we have two dynamics:
- The students who we really struggle to successfully retain and graduate are already gone.
- 90 credits is 75% of an UG degree…so shouldn’t we expect a much higher percentage of students to be at a ‘4’? After all, the data is for students above 90 credits, so it is fair to assume some good percentage of students assessed are close to graduation.
Fully a third of students in most measures are at a ‘2’…which doesn’t translate to the sort of outcomes I would aspire to for near-graduates.
For instance, the weakest criteria in the Critical Thinking rubric was ‘Evidence.’ Here is the description for the Capstone level:
Information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis. Viewpoints of experts are questioned thoroughly.
Vs. the first (lowest) Milestone:
Information is taken from source(s) with some interpretation/evaluation, but not enough to develop a coherent analysis or synthesis. Viewpoints of experts are taken as mostly fact, with little questioning.
So okay…maybe I’m just questioning the authority of IHE and AAC&U in declaring that we’re “On Solid Ground.”
41% of 4 year students were at the Milestone level described above; only 5% were at Capstone.
The report, and IHE article, call out Critical Thinking as the weakest area. But “Written Communication,” while showing double the number of students in the Capstone category still has more than a third in the first Milestone, with its rather dismal description of capabilities.
Perhaps college presidents should be forced to read out loud, at graduation, a statement praising the graduating class, “of whom fully 31% demonstrate an attempt to use credible and/or relevant sources to support ideas that are appropriate for the discipline and genre of the writing.” Kudos for the attempt!
Finally, the largest question for me has always been, “what’s the delta.” After teaching for a couple of years, and looking at my own data, I realized I was basically rubber stamping students. That is, “assessing” students as high performing who were ALREADY high performing. They didn’t get that way through the brilliance of our curriculum or my teaching.
So I began looking at the change over an entire academic year, and that data (though tougher to consider) was much more valuable. And it led to change.
It would not seem so difficult to perform the same VALUE analysis on assignments completed in the first year of academic work at these same institutions, even if a different population of students. We could then compare <30 credits performance to >90 credit performance.
So perhaps the question isn’t, “are we on solid ground” but “are we moving the needle?”
