The Brave New World of Teacher Evaluation, Part 2

By:  Dr. Mitchell Robinson

Originally posted on his blog http://www.mitchellrobinson.net/2016/01/20/the-brave-new-world-of-teacher-evaluation-part-2/

[A quick update on my experiences with the National Observational Teaching Exam, or NOTE, a new product from the Educational Testing Service, intended as a competitor to Pearson's edTPA...I've written about NOTE previously, but recently attended a presentation on the exam from ETS personnel, and thought it might be worth sharing some observations--no pun intended.]

I was invited to attend an informational session on NOTE, the National Observational Teaching Exam, yesterday. Full disclosure: I have serious reservations as to the wisdom of building a high-stakes teacher evaluation exam around the notion of student teachers "teaching" avatars in a virtual reality environment as a reasonable simulation of classroom teaching--and expressed these reservations to my colleagues before the meeting--but attempted to enter the session with an open mind. This proved harder than expected...

After some introductory remarks from the ETS employee sent to brief us on NOTE, we saw a video demonstration of the exam. A few details:

• the exam is being offered only for elementary ELA and math; other grade levels and subject areas may be released later
• modules consist of relatively short (7-12 minutes) "on demand" teaching episodes (more on this later)
• the exam is designed as a "high leverage" teacher entrance exam--pass and you receive licensure; fail and you may retake the exam--at $400+ per attempt per student
• the system can generate no more than 5 avatars at a time, so teaching episodes are limited to "classes" of 5 students or less
• emphasis is placed on student teachers being able to "elicit correct responses" from the simulated students (more on this as well)
• the avatars are controlled by "interactors," who are located at a remote site and interact with teacher candidates in real time via video hookups
• interactors control the avatars via headsets and joysticks; each interactor operates up to 5 avatars per episode, using voice modulation software to mimic the voices of children

After the demonstration there was an opportunity for some Q&A. The presenter had mentioned that one of the challenges involved in their work on the exam was the "tension" between authenticity and standardization. She also mentioned that the interactors were not teachers, but rather, actors who were recruited from the DisneyWorld theme park near their Orlando offices. These actors were given scripts from ETS, and told to try to elicit "at least 2 from a possible list of 5 right answers" for each teaching prompt.

I asked the presenter how this tension was influenced by lessons or subjects that didn't place a high priority on eliciting "right answers," but instead sought to promote divergent thinking skills, problem solving, and critical thinking that might value multiple "appropriate responses" to a given prompt. She didn't seem to have considered this possibility, which generated a healthy discussion around the room about the views of teaching and learning that provided the philosophical foundation for the NOTE. I mentioned that as a music teacher educator, my students were often engaged in helping their students form their own interpretations and critical judgments about music, their performance, or the performances of others--or were engaged in teaching creative tasks such as composing and improvising; acts and actions that are not all about "right answers." Some of my colleagues in other teaching areas agreed that getting students to provide "right answers" was a pretty simplistic approach to teaching, and that they believed this approach might prove to be somewhat limiting in the NOTE. There was not really a satisfactory answer provided to this question, so we moved on.

A few people expressed concern that the NOTE consisted of a series of very short teaching episodes, and didn't provide a rich, contextual overview of a teacher candidate's work over time. The presenter responded by suggesting that portfolio exams like edTPA were easier to "cheat on," as students could redo sections until they were happy with them, and have multiple attempts at portfolio components over time. She told us that the "on demand" nature of NOTE made it harder for students to "cheat" like this, and suggested that this factor increased the validity of the exam as a high stakes measure of teaching competence.

Leaving aside the notion that teaching avatars can hardly be considered a valid teaching practice under most conditions, I question the thinking behind an evaluation model that privileges "snapshots" and "one shot exams" over "movies" of a student's practice as a teacher over the course of a semester. The goal of assessment is not to "catch" test takers in tricks and traps--it should be to allow participants to show their work, their thinking, and their ability to think reflectively about their practice as teachers. The NOTE approach to evaluation seems more focused on accountability measures than on helping teacher candidates improve their teaching practice.

Finally, as the Q&A was drawing to a close, I raised my hand to share one last observation. "I think that it's interesting, in a tragic way," I said, "that the very skills that you seem to value in the actors that you hire to control the avatars--the ability to think on one's feet, to respond in an improvisatory way to unpredictable actions by the avatars, and the ability to tolerate ambiguity in the responses from teacher candidates--are exactly the skills that are found in music, art, drama and the fine and performing arts...subjects that are being cut from school curricula across the country. And this curriculum narrowing is a direct result of the obsession with standardized testing, regressive measurement and high-stakes exams that your NOTE test represents."

There was a stunned silence in the room. After a moment, she replied:

"I never thought of that before."

"I think about it every day," I said.