PRESENTER: It is my great pleasure to introduce to you our presenter, Dr. Grant Wiggins. He is the president of -- president of Authentic Education in Hopewell, New Jersey. Dr. Wiggins and his
Authentic Education colleagues consult with schools, districts, state and national education departments on a variety of reform matters. Authentic Education also organizes conference and
workshops and develops print and web resources on key reform issues. Dr. Wiggins is perhaps best known for being co-author with Jay McTighe of Understanding by Design, the award-winning and
highly successful program and set of materials on curriculum design used all over the world, and of Schooling By Design. He is also co-author for Pearson Publishing on more than a dozen
textbook programs in which Understanding by Design is infused. His work has been supported by the Pew Charitable Trust, the Geraldine R. Dodge Foundation, and the National Science Foundation.
Over the past 25 years, Dr. Wiggins has worked on some of the most influential reform initiatives in the world, including the Ted Sizer's Coalition of Essential Schools, the International
Baccalaureate Program, the Advanced Placement Program, state reform initiatives in New Jersey, New York, and Delaware, and national reforms in China, the Philippines, and Thailand. Dr.
Wiggins is widely known for his work in assessment reform. He is the author of Education -- I'm sorry, Educative Assessment and Assessing Student Performance, both published by Jossey-Bass.
Would you please join me in warmly welcoming Dr. Wiggins? GRANT WIGGINS: Thank you, Wendy. Hello everyone, so good to have you here on this miserable rainy day. Nobody wants to play hooky. I
was actually here yesterday. I was in Harrisburg. As many of you may know, missing from the bio is I have been working for the state of Pennsylvania for the last year as part of the SAS
Model District Project, which is nearing conclusion of year one, and we just met yesterday to think about year two, expanding the effort. So that goes on apace too. I'll talk a little bit
about that at the keynote. Our subject here today is one near and dear to my heart as a teacher, as a coach, as a parent, as a peer reviewer in UbD units, and that's feedback. And I'm
deliberately using the word feedback and not the FA phrase, formative assessment, because I think formative assessment is actually a phrase that's come to be badly interpreted and misused,
and I want to really spend a little bit of time with you today sort of explaining that by returning to the core term of feedback. Along the way, I'm going to ask you to have some pauses for
conversation and indeed a little formative assessment, in fact, for you. But the focus is on this term feedback, and I want to give you a heads up about it. Not everything we call
feedback is feedback. What's the most common two-word phrase heard in American classrooms? Good job! That's not feedback. That's praise. And they're not the same. So one of the things that we
want to think about is, well, what is feedback? And why does it matter? And in fact, why does it matter that we have true feedback as opposed to all these other loose things that fall under
the heading? Because once you get that right, once you really understand what feedback is and isn't and how it differs from evaluation, how it differs from advice, we'll begin to get the
formative assessment thing right. Because right now it's a mess. Any old interim benchmark assessment is feedback? I don't think so. Letter grades on a report card is feedback? I don't
think so. In other words, there's a pretty basic misunderstanding. So that's what we're going to do. We're going to revisit the term itself. Let me get my little clicker thing out and we'll
go. Once we get this distinction clear, we'll begin to realize something, that feedback is not only different from teaching and advice, but we tend to give too much advice and not enough
feedback. You know what you need to do? You're already in trouble. That's not feedback; that's advice. And it's pretty aggressive advice. And it doesn't matter whether we're talking about a
little kid or a supervisory conversation between adults. There's a tendency to give too damn much advice. The beauty of feedback done right is that you can minimize the advice and maximize
the empowerment of the learner to figure out what they need to do themselves based on the feedback. We know how this works because every video game in America is built off this idea.
There's no direct instruction. Not once in any successful video game, whether it's Angry Birds, Rock Band, Halo. It is a feedback system, not a teaching system. And yet, miraculously, people
learn. But it's not so miraculous when you understand it. So we want to better understand it. We want to understand how it's possible to do less teaching, have more feedback, and not only
cause more learning, but cause more autonomy in the learner, a huge issue in special needs populations. Because if you're dependent always on advice, you're less autonomous than you need to
be. Because if you're only dependent on advice and praise and blame, you're always insecure. Do you like my work? Is this right? That's not what we're after. We're after autonomous performance
based on a feedback system. Now you know the research. We've been swimming in the research. And it predates Black and William by a good amount, but Black and William certainly brought it
to everybody's attention. That in the exhaustive meta-analysis they did plus their own work, picked up by Marzano, picked up by John Hattie and advanced by John Hattie in his work, Visible
Learning, we know that one of the most powerful things that advances achievement is a robust feedback system. Now here's one of the interesting challenges that we face as educators. If I say
what I just said, you nod and many of you actually nodded and you looked at the slide and you go like this. But here's the problem. It means you have to teach less and teachers don't like that.
That's not what we were trained to do is teach less, but that's what you have to do if you want to honor the research. You have to teach less and make sure the student gets more feedback if,
in the end, you want learning, high levels of performance, and autonomy. So I hope this is a bit of a provocation. At one level, this is common sense. At another level, it's
counterintuitive, and that's what's interesting about it as a topic. Now I mentioned Hattie's work. How many of you are familiar with John Hattie's book, Visible Learning? Oh, not enough of
you. You got to read that book. I think it's one of the most important books of the last 25 years. John Hattie is a New Zealander who spent his entire career doing what Marzano basically
dabbled in by comparison. Massive meta-analysis of everything that's out there. In fact, Marzano used Hattie's work as part of what he did in his highly successful series on What Works. You
know how meta-analysis works. You don't actually do the result directly; you look at all the research and you make sense of it. Hattie has done it for every conceivable educational intervention.
Guy needs to get a life. But it's really cool. Again, with the caveat that it's meta-analysis. We won't bore you with the details of what an effect size is, but that's the number on the
right. These are the top. And to give you an idea of how powerful the effect size is, it means in terms of the first three that you can advance the student's performance an entire extra year
of achievement. You know what's really interesting about his analysis as well? All of the things on this slide are more significant than socioeconomic status of the family. So this is pretty
important stuff. You want to get this right. But again, if you've played video games, you get this. You know how much you grow just by getting the feedback and using it. As I said, many of
these ideas are not new. In fact, they're quite old. About 20 years ago, Harvard University did a self-study. It got published into something called the Harvard Assessment Seminar report and
that ultimately became a little book that Dick Light and his colleagues at the School of Ed did for the whole university. When Harvard students and alums were asked what were the most
effective courses at Harvard and what made them so effective, that was the answer. Feedback and opportunities to use it. Notice the little caveat at the end. Feedback depends on models.
There's no feedback system without models that you're comparing your performance to. This is true whether you're a Harvard undergraduate or a severely special needs student. There has to be
clarity about where you stand against model performance and where you are in closing the gap. Not only was that a good thing, said the Harvard students, but some of their most memorable and
empowering learning experiences came from being in that feedback loop. So we want to think more carefully than we have up to this point about the phrase formative assessment. A so-called
interim benchmark assessment is not a formative assessment if the questions will never be seen again, if it's just a quiz, if it's just a test of a conventional kind. So that's -- this is pretty
basic. Feedback means there's a loop. There's an opportunity to keep getting the feedback in relationship to the goal and charting your progress against the goal. A one-shot test is not a
feedback system. Yes, I know this has consequences for how you view PSSA results. I get that. An aside, PSSA results can provide perfectly fine accountability while not being a very good
feedback system. That's not a contradiction in terms. Unfortunately, for PSSA to be a good feedback system, you'd have to be tested every week and nobody wants that. So we want to think about
this idea of a feedback loop, a feedback system. I've already hinted at some of its components. There has to be a far off goal that you track to see if you're closing the gap between where
you are versus where you want to be. The goal is embodied by something called models or exemplars or specifications. I'm proud as a parent to report that my daughter ran the indoor state
championships at Penn State last week and came in eighth in the mile. Not too shabby. However, that's not what interested my daughter and it's not what interests me. She's working
diligently, heading to this spring's outdoor season, culminating a four-year running career and trying to break her five-minute mile. And we know that this is how it works, regardless of
whether you run a five-minute mile or a seven-minute mile. We track your times, we try to make them go down. And when you're racing, especially in distances, you have something called splits.
What are the splits? True formative assessment. Given the 400 you just ran, are you on pace to break five-minute miles? That's how my daughter and her coach are thinking. But it's true
whether you're trying to run seven-minute miles. Are you on pace to run a seven-minute mile? It doesn't matter how good you are. If you want to be better, you have to pace yourself formatively
against the goal. We don't do that in school enough. We just give you a grade and move on. We just give you your place of finish and not your time. That's often what a grading system is.
It's your place of finish compared to other people. How would you get better in track if all we did was keep your place of finish? 12Th, 26th, 129th, third, fifth. No wonder people don't
improve. Grading is not a good feedback system. It doesn't give you usable information about how to improve the next time. So all of these comments are intended to sort of prick your
thinking and your conscious a little bit, to say, you know what? Kids get lots of information, but not of it's really useful for self-improvement. Because that's what we're talking about, a
self-improvement scheme. Here's my view. I really believe this. This is not a provocation. The problem is not kids. The problem is it's a lousy feedback system. That if they had better
feedback, they'd all improve. Because when they're in really good feedback systems, they improve regardless of who they are, whether they're special needs students or honor students, whether
they're running five-minute miles or nine-minute miles. Without good feedback, you begin to lose heart and lose faith and you have fatalistic assumptions. Well, once a C+ student, always a C+
student. So it gets into your psyche that you can't improve much. And we know from Dweck's research that this is crucial, that you have to believe you can improve. You have to believe it.
You're only going to believe it if you have evidence that it's occurring. And most of the way we give information to kids is a too much good job, too much letter grades, and not enough
incremental measure of the progress that you're making. Even better, the student can see the information for him or herself. My daughter doesn't need her times shouted out. There's a clock
that she runs by every four laps. It becomes internalized for good athletes. They can give and get the feedback and the advice for themselves and that's how you build autonomy. So we want to
think about all these elements: feedback, autonomy, precise enough information that you can get better, less and less reliance on teacher coaches, more and more gradual release of responsibility
through a good feedback system. Does that make sense? That's what's going to improve learning for anybody, regardless of who they are. What do you think? Based on what I've said so far, just
react to it at your table with the person next to you. What's Grant saying? What does it make me think? He's offered a few zingers. He's offered a few common sense things. He's offered a few
simple analogies. What do you think so far? Just talk it over with the person next to you. As I said, we'll pause a few times and you'll have further opportunity to discuss and give some
feedback as well. It helps, of course, to have some models. And in a few minutes, I'm going to ask you to come up with a model and share it at your table because part of what I'm saying is
pretty straightforward. We don't have good examples of robust feedback systems in schooling at large. Some of us have had great teachers. Some of us have been fortunate outside of school to
be in good feedback systems. But in general, I'm proposing that the average student experience is not a happy one in terms of robust feedback systems. And so we need to think of models, and
we need to learn from the models and apply them to school. That's really the gist of what I'm saying is the next step in thinking and action research. So let's talk about some simple examples
from my end, and then you'll talk about some from your end, and we'll see where we are. My son Justin is attempting to be a musician out in Hollywood now, but when he was four, a number of
years ago, he wrote his letters, as you can see here. And you know how this goes as an educator. Your kids become your experimental subjects, and I saw a great opportunity here. I know, some of
you are going to say I'm cruel. I know that. I've heard that feedback, but I don't think it's cruel. And I would ask you to suspend that view as best you can and just pay attention to it as
a story about feedback because it's actually, I think, pretty interesting how it played out. So I see Justin writing his letters, and so I nonchalantly but with a little bit of enthusiasm
in my voice say, hey, J, what's that? He says, well, that's my name. I said, really? Show me. And he goes, J-U-S-T-I-N, Justin. I go, oh. I said, what's this? And I drew that. He said, well,
that's an N. Drew an H. What's that? Well, that's an H. Now let's just freeze frame it. He's either going to scream and cry and run out of the room, which he didn't, or he's going to say
something like, aw, you didn't like it, which he didn't. Not what I wanted. And so I want to propose, given his response, which is what makes it what it is, I want to propose this is a
really good feedback system. And I want you to try that idea on for size and I want you to talk a little bit with the person next to you about why it's a feedback system and why, since it
obviously worked because I want to propose by what he said, not what I wanted, that that makes it a feedback system and a good example of one. So would you do that? Would you say to the
person next to you, what's going on in this example? What happened that made it work as a feedback? You know, when all was said and done, I praised him and did all the good things that you
do as a parent of a four-year-old, but I didn't want to mess it up at that point. I wanted to just see it work as a feedback and indeed it worked like a charm. So how did it work? What did I do?
What was going on? Talk it through. I'll show the slides again just so you're aware of what I did, but talk it over. With this example and some of the other examples and the example that I'm
going to ask you to talk about from your own experience, again, what we're looking for is a transferable idea. We're not interested in the particulars so much as what they suggest are the
generalizable takeaways. So let me just call attention to two. There has to be a model. I had to draw the N and the H, and I had to confirm that he knew the difference between an N and an H
if the system was to work. To say it in a slightly different way, he had to have the goal of writing the letters accurately and know the difference between an N and an H. Without it, it doesn't
work. And that's why I had to probe because maybe he didn't know the difference. He was only four, but he did know the difference. And in fact, his last words indicate something
generalizable. There was an effect against an intent, and it's either the effect you wanted or it's the effect you didn't want. It has nothing to do with me. It has to do with his goal and
the feedback against his goal. It's interesting, fast forward to the present. We were talking about this just the other night. He played a gig at USC with his band and we talked about it,
you know, what worked, what didn't. He said, I'm so glad you sent me that article, this video. It was a great video by Ira Glass, the great NPR essayist, and Ira Glass talks about how hard
it is to get good because we get to this crucial stage where we recognize what good is and we know we're not there. And that that's when you have to persist, and that's where he is now as a
musician. He's in the same spot as a musician that he was with his letters. And that's the challenge, and it's especially the challenge with special needs students. You have to
believe and want the goal, you have to be somewhat irritated by the gap between where you are and the goal, and you have to persist in a feedback system that makes it a positive experience,
that makes it seem like, yeah, if I persist at this, I see movement, I see hope, I see progress. So those are some of the generalizable takeaways that I see in the story. Let's just hear a
couple of shout outs. What did you see that made it work as a feedback system beyond what I just said? AUDIENCE MEMBER: It wasn't judgmental. GRANT WIGGINS: Yeah, there's absolutely no
judgment of any kind. And it's sort of reflected in his words. There was no good, there was no bad, there was no how disappointing, there was no -- not even how frustrating, it just was. And
that's what happens if you've ever had a really good coach. It just is. You're just pulling your head out when you swing. It just is. There's no judgment, it just is. And that turns out to be a
very important weakness in our school feedback system. There's an overlay of judgment in a lot of what we call feedback that screws it up. Excellent. Some others? AUDIENCE MEMBER:
Self-evaluation. GRANT WIGGINS: We want to try to put the student, regardless of the goal, regardless of the issue, in a position to accurately self-assess and self-adjust based on clear
goals and clear feedback. Let's turn your point around. If the student is incapable of accurate self-assessment, we got problems. This is why you should make student self-assessment a
required feature of all assignments and assessments. That just follows from what we're saying because we get feedback. If the student judges their performance to be more successful or less
successful than it really was, we need to know that. And by the way, it works in both directions. Many students are incredibly harsh on themselves to too great an extent because they're
confusing their disappointment and embarrassment with how they actually did. Good. One or two more? What else did you notice? AUDIENCE MEMBER: [inaudible]. GRANT WIGGINS: There was a
choice. The models were just presented, there was no heavy-handed teaching, there was no advice. There was just here it is. Where are you against it? Yeah, good. So I want to propose to you
that this example and the things we just heard are all vital takeaways about robust feedback systems. Again, quickly think about video games now that we have them, now that they're
ubiquitous. Video games trade off of this ability. Self-assessment, self-adjustment. No judgment, just where do you stand against models. Video games do something else. They break it down into
hundreds of levels and give you incremental information. Didn't need it here because it was really kind of a binary system, but with a complex goal, you need layers of leveled
information. So would you just, one to another in pairs, define feedback. This'll be your working definition. I'm going to ask you to reconsider this definition after a few more interventions
on my part. But just how would you define feedback, mindful of the fact that we're trying to talk now about pure feedback, not some of this other stuff? So how would you define it? Just try
it out in pairs to one another. How would you define it? Okay, you got a working definition and you'll have an opportunity to reconsider your definition, either confirming it or feedback that
suggests that you might want to revise it as we move forward. The other thing you might also think about is applying your definition to something that we're all familiar with on -- sometimes
on both ends: supervision of teaching. Because most supervisors are not very good pure feedback-givers. They tend to jump to advice way too soon in the process. That's why clinical
supervision was invented 40 years ago, but I've done a lot of walkthroughs and overheard a lot of supervisory situations. And I'm really sort of surprised at how heavy-handed the advice is.
You know what you might want to consider? You might want to do -- why you telling me that? That should be your immediate response. Why you telling me that? Is that because it's your pet
issue? Is that because you don't like my style? I have no idea. And that's a tip-off as to why this distinction is so important about feedback versus advice versus evaluation. Just think about
your own experience in that situation. All right, let's test out your understanding and refine it, perhaps, with a little concept attainment test. Concept attainment is not only a good
strategy for learning, for understanding, but it's also a feedback system. Concept attainment basically says I'm going to share with you some examples, I'm going to share with you some
non-examples. If you think about what I've done so far, that's what I've been doing somewhat informally. Now we're going to do it a little more formally, and then you get to draw some inferences
and then you get to test out your thinking on some new examples where I don't tell you what they are. You tell yourself and it becomes its own little feedback system. So here we
go. Here are some examples of nonverbal feedback because one of the other bad habits that educators get into is they think that feedback has to be given by people speaking. Video games show
that that's not true, but there are any number of instances that show that that's not true. Now as you look at these, think about your definition, think about the Justin story. In a minute,
I'm going to ask you for some feedback about these examples. So here are the nonverbal ones. So you give me some feedback. Raise your hand if you agree that these are examples of feedback
given what we've said so far. Do you agree that those are feedback? If you don't, raise your hand or make a comment because we have to agree on the models. Information in relationship to a
goal. We're not saying it's the best possible feedback, perhaps. We're saying, is it feedback? Yeah. In fact, learning to improve feedback is an important skill. And would you agree,
similarly, that this is feedback? Okay. None of these are feedback then. Discuss any uncertainty at your table. And if you still have uncertainty, I want to hear about it because that's the
feedback that I need to get. How we doing? Here's my guess. You're not sure what to do about the question. Right? Now think about it. Why is a question not feedback? It's a question. How can
it be feedback given your definition? Never mind that I might have a good reason for asking the question. Is the question feedback? It can't be. I don't know what the hell to do with your
question. I don't know why you're asking it. Where are you going with your question? Wherever that is is probably where the feedback is. And that was true in the Justin story. Do you know
what this is? That's not feedback. It's actually feedback to me. Does he know the difference between an N and an H? See, that's the interesting thing. The questioner should be asking the
question to get more feedback themselves as to whether they understand the performance. Why did you do that? I overheard your phrase. Very good phrase. Seems a little loaded. Feedback is
never loaded. Feedback just is. The game isn't sitting thinking evil thoughts. It's not asking leading questions. It just is. Feedback just is. The question is going somewhere. I don't know
where it's going. Imagine you're lost, this is before GPS, makes a better story before GPS. You're lost, you stop at the gas station, and you get Socrates. So why are you driving there? What
do you hope to find when you -- I'm just trying to get there. Am I on the right road? So the question may be useful, but it's not feedback. Does that make sense? If anything, it's useful to you
as feedback, what the response is, because it determines what your next move is. And that's what happened in my case. Any other question about any of these others as not being feedback?
Yes? AUDIENCE MEMBER: [Inaudible] table question here. The table question was when you asked Justin questions. Was that feedback? GRANT WIGGINS: Based on what I just said 30 seconds ago,
no, it's a probe to find out if he knows the difference between an N and an H. It's not giving him any information. How can it be feedback? I am asking a question. How can that be feedback
to the learner? The feedback is information that the learner gets about whether they're on track, right? So the question is just a setup for some other task or issue. Really it's going to
be my feedback, and that's often how it works in practice. Does that make sense? AUDIENCE MEMBER: Yes. The question here was, but ultimately you circled that and you asked him, what's
this? GRANT WIGGINS: So where's the feedback? The feedback is in the comparison between the N and the H, not the question. I'm calling his attention to a discrepancy between the drawing
and the model. That's the feedback. And that's why he says, not what I wanted. Does that make sense? One way of distinguishing the examples from the non-examples is to use language that we've
already used. Feedback is different from praise. I like that! I like [inaudible]. By the way, that has a really nasty consequence. The student starts to try to please the teacher. The
teacher tries to please the supervisor. It's not about pleasing you, really. It's not. It's about getting good at something worth getting good at. Bringing in what you like as a
teacher-coach is a bad idea. It has corrosive consequences down the line. So there's liking and praising. That's different from feedback. Doesn't mean you can't praise, but you got to keep the
learner in the feedback world. We also said it's not a question. A question is just a question. A question is what it is. It's an interrogatory, it's a probe. That's different than
actionable information that I can use as a performer. If anything, the question is asked to give me an answer that gives me feedback if you know what the hell we're doing. And it's not
advice. So we want to be clear on those distinctions. If you're clear on those distinctions by the time we're done, then that's a good thing. I was going to say that would please me, but that
would be a mistake. All right, the next part of concept attainment is we give you a formative assessment. So on this first round, we'll let you talk it over, sort of like Who Wants to be
a Millionaire? You can have some friends, crowd- source it. What do you think? Which ones are feedback, which ones are not? 30 seconds. Okay, your 30 seconds is up. So first one. Show of
hands, how many think the first one is feedback? I agree with you, but let's now say why. There's an implied goal. What's the goal? I want to be heard. I'm speaking and I want to be heard.
Nobody speaks to not be heard, but it's important to state the goal, to make sure that the feedback is fully understood. So if I say I can't hear you, I immediately know that there's a
disconnect between my goal and an effect that I don't want. So it is feedback. Second one. No, that's praise. Third one. Raise your hand if you think the third one is also feedback. Okay,
everybody at the table tell the people who just raised their hands why they are, unfortunately, mistaken. As an aside, what we're doing is what Eric Mazur has done in his physics classes at
Harvard for the last dozen years, in which his students outperform all the other physics students at Harvard: less teaching, more problems with feedback, use of the clickers, instant graphs
of all the responses in the room. You just did it. And you saw the need to do it because, to the teacher, I can't believe that there is still a misunderstanding, but there always is. That's
why there's a misconception literature in physics. That's what woke Mazur up. I visited with him last month. It turns out he's a big fan of my work, which is great because I'm a big fan of his.
And I said, what really was the thing that convinced you that you had to not lecture anymore? He said, I gave a problem once and a very bright girl in one of my physics classes said, do you
want us to give the answer that we think makes sense or the answer you taught us? And that's what just happened. The third one is advice. Advice follows feedback. If you only give advice,
think about it this way, my immediate response should be, why are they telling me that? You may say, well, it's obvious. No, no, no, that's the road to hell paved with good intentions. It's
always obvious to the teacher. It's never obvious to the learner who's still processing it. That's why you do this. Yes, you had a question? AUDIENCE MEMBER: Does the first one, I can't hear
you, depend on your environment setting? Like if it's just you and the student, then that would be more like you were telling those, as opposed to class saying, I can't hear you. Because if
the teacher says it [inaudible]. GRANT WIGGINS: But what you're getting at, it seems to me, is the goal could vary in the situation and the meaning and intonation of the words could vary in
the situation, but it still would be feedback. If I meant by, I can't hear you, and I said it that way deliberately, then you know that I'm having difficulty processing, I'm in denial, I'm
resisting, I just -- it's too much at this moment. I'm not having auditory difficulties. And you pick that up from my tone. So I can't hear you could be completely different pieces of feedback.
So when my wife says, you're not listening, and I say, I'm listening, then it's a bad feedback system. And that's how men and women get in trouble because it's not a good feedback system
yet, because we're working under different definitions of the goal, listen. Does that make sense? The advice certainly comes right after the feedback. But in general, why would you give the
advice without the feedback? And going back to the answer about self-assessment, if I get good feedback, I can find the advice and I'm empowered by doing so. In fact, in the ideal system, I
just adjust based on the feedback. That's the difference between a good teacher and a bad teacher. The students don't say, timeout, you're doing a bad job of teaching. No, it's my job to
pick up on the cues that people are nodding. That's what we want. We want a self-correcting system based on using the feedback. Again, we give too much advice. I'm not saying don't give any
advice. We give too much advice too quickly. That's the caution. All right, let's try another one. Which is feedback, which isn't? Do this one yourself. This one's a little harder. The first
one is feedback. Some people might have said, well, it seems judgmental or it seems like it's just your opinion. No, I'm giving the same kind of information in terms of your goal that you
got when somebody said I can't hear you. Nobody wants to disengage or bore or confuse the reader. I'm telling you where I was confused or bored or lost interest. It's just a fact. The
coolest example of this I ever saw was about 20 years ago, a sixth grade teacher. He taught his students a peer review process and it went like this. The student will write the goal statement for
the piece of writing, so it goes something like this: I was trying to write a scary story like Stephen King's stories. Purpose. Audience: my peers. Great, peer reviewers. Just feedback
against the goal. Where were you scared, where were you not? These kids write better than any kids I ever saw in sixth grade. They are totally in the feedback loop of audience, purpose,
results, make the adjustments. Mark the places where you were not scared. That's feedback because that was the goal. It's no different than trying to tell the joke, no different than trying to be
heard. So it's not a judgment about the value of the performance. It's a fact that your goal is not being achieved. It's just like the N and the H. How about the second one? Yeah, that's
feedback because we don't want that. And by now you should know that asking the question is not feedback, even if it's an appropriate question to ask as part of the interchange. So what's
feedback? Little discussion at your table. Are we getting clearer? Did you -- do you have to modify your definition? Are you beginning to realize where it's easy to go wrong? Just sort of talk
that over at your table. All right, let me ask you to pause one more time. The challenge is to begin to realize -- and there was a great exchange at this table. The challenge is to begin
to realize that we just move too fast. And we assume the feedback is obvious or we assume the goal is obvious, but what you don't grasp is by making it more explicit about here's the goal,
here's the feedback, and then here's the advice, the learner is so much more willing to hear it and able to get it. So here was the little discussion they were having at this table. Well,
what do you do about the somebody who says speak softer? And someone said, well, wait a minute, maybe they're just -- it's not about loudness, it's really about expressiveness and using your
voice and not being raspy and being better heard in that sense. The point is, if you just get a piece of advice, you don't know why it's being offered. It's only when you get, ah, because
you're interrupting the class next door, you're talking so loud, and my ears hurt. Nobody wants that goal, so speak -- oh, okay. But if I just walk into your classroom and say, speak softer. Who
is she? So you can predict defensiveness or confusion when you just jump to advice prematurely. So let's flip it around. If you hear yourself saying things like good job, your immediate
next step is to develop the habit of, colon, describe what was done that met the criteria. Never say good job by itself, ever. Don't even point and say good job because I don't know what
you're pointing at. We know this about feedback. Sometimes it's important to know that you did the right thing, but you need to know exactly what you did. We've all had this with kids. Oh,
that was great, you did da da, and the kids goes, I did? Cool. Yeah, do more of that. What'd I do? That's why this matters. If you hear yourself giving advice, immediately say because,
which is what we talked about over here. In other words, that's how you begin to develop the right habits. It's not that you're going to be suddenly free of giving advice forever and giving
no praise or blame forever. You're not going to do that. But you can get better at improving the chain of are we clear on the goal, are we clear on the feedback, are we clear therefore on
the advice that follows from the goal and the feedback? And the corollary is, if your experience is like mine, the more you do this, the student can give themselves the advice if they have enough
prior learning. Oh, I see what I have to do, okay, okay, yeah. All right, now it's your turn to transfer this idea to your own situation, both to better understand it and to kind of
solidify the concept. I want you to think of a feedback system that you've been in that was really robust, really powerful. And we're talking the pure feedback now. We're not talking everything
else. We're not talking great teacher, wonderful classmates, good environment. We're just talking about the feedback in the sense that we've been talking about it. Maybe it's a video game.
Maybe it's the person who taught you to dance. Maybe it's the person who taught you to ride horses. It doesn't matter if it's non-school. What it has to be is a model of what we're talking
about to add to the pool of models that are worth thinking about so that we can transfer it to what is inadequate as a feedback system in school. Because we don't have that much time, I'm
going to ask you to just do it in pairs. It's more fun to do it as a table and to share out as an entire group, and you can do that back in your school, it's a really good exercise because
then you get a really powerful common sense about all the characteristics. But just in the interest of time, do it in pairs. So one person tell a story, the second person tell a story, and then
say, great, regardless of the particulars, what's common to both stories about good feedback? Okay, so now we're talking about good feedback, not just feedback. Let's just
hear a few words or phrases that grew out of your quick story comparisons to say, yeah, this is what good feedback is all about. What are some of the criteria that you just identified? Came out
of the story? AUDIENCE MEMBER: [inaudible]. GRANT WIGGINS: That is common and that is feedback to the world of education. AUDIENCE MEMBER: I think one that stood out for me was that
both me and the person giving the feedback were really, really clear about the goals. GRANT WIGGINS: That's really, to me, the number one criteria. If there is neither clarity about the goal
nor sharing of the same goal, you're going to be in a bind. That's the you're not listening to me problem. What do we mean by listen? Excellent. And one of the things that's notable about
good coaches is they really make sure we're clear about the goal and the means and on the same page about it. Good, so clarity about the goal. Non- academic examples are powerful. What else?
Yeah? AUDIENCE MEMBER: [inaudible]. GRANT WIGGINS: Yeah, that's the corollary of you don't find them very often in school. Sometimes it's even hard to think of any powerful feedback
system that you've ever been in. And then we're in a bind because we have to have best practice to draw upon. And so in those cases, where we really don't believe that we have personal
experience with powerful robust feedback systems, then you got to do some action research. You got to go find the coaches and the video games and the systems that really are models and then study
them together as an action research project. I highly recommend that you go back to your school, school district, if you're from a school or a school district, and you go to who are reputed
to be the best coaches in the district and just chat with them for an afternoon. It's very eye-opening. It really is, particularly because of the first comment. Okay, one more. AUDIENCE
MEMBER: [Inaudible] special education students that we're working on [inaudible]. GRANT WIGGINS: When there is clarity of goal, when there is clarity of feedback, when the student is in a
transparent environment where they can chart it, it becomes empowering and engaging for the learner far more than we might have managed. Yeah, I agree. All right, so these are all good
criteria and there are probably some others. Really good feedback is ongoing, it's consistent, it's based on expertise, it's not just random. One of the things that's been very striking to
me in watching outstanding baseball and soccer coaches in my coaching days and going to clinics and having good fortune to hear some professional coaches, is the professional coaches don't make a
mistake that we, as educators, often make. They're amazing for what they don't say. They target one change that has high leverage. And I can see a bunch of you nodding. I mean, that really
resonates when you see a good coach. So one of the things that follows from what we've just said, and one of you said it, is we can't leave the performance goals implicit. They have to be
explicit. So here's a really simple example. Guys, I've asked you to do this multimedia project. This was something I saw a fourth grade teacher do years ago. She said, I'm going to show you
on a big archery bulls-eye target which ones hit the mark and which ones didn't, though I'm not going to tell you which ones. You go see the ones that hit the target and which ones didn't and
let's talk about why. In other words, she was doing concept attainment with them. Why is this a better project than that given the rubric? Any of you who have ever worked with rubrics,
any of you who have ever been trained in Advanced Placement scoring, any of you who ever done statewide writing, it's not the rubric that's important. It's the samples. It's the anchors. The
kid needs the anchors, man, not the rubric. If I had to make a choice, and I've been lobbying for rubrics for 30 years, but if I had to make a choice, I'd ditch the rubric and show them lots of
samples, some of which were better than others. And indeed that's what I did in the Justin story. That's how you clarify the performance goals. Well, wait a minute, Grant, then they'll
cheat. Wait a minute, didn't you just say that that's what all the good coaches do? Well, if what you mean by it is we don't want students to be an unthinking copycat thing, then give them a
slightly different project or last year's project or last year's writing prompt. But they need to get inside the performance goal through samples. So that's another way to make it very
explicit. The other way to make it explicit is to say, just like that sixth grade teacher, okay, if the goal is scary stories, then scary stories is what we want. So let's talk about the
difference between scary and not scary. Let's do a little T-chart. Okay, now go read. Now we're crystal clear. Because it may turn out that the writer and the reader have different views of
what's scary. Some people don't think Alfred Hitchcock movies are scary. I think they're mind-bogglingly scary, and I think the Chainsaw Massacre movies are just stupid and they're not scary.
There are other people who feel the exact opposite way. So again, you got to be clear about the goals. You got to make it explicit, the goals and the criteria. What are we looking for? Are
we agreed about what the goal is? There's always stuff that's implicit. and the good coaches, the good supervisors, the good teachers make it explicit. One of the things that you do in
clinical supervision is you track details scrupulously with no value judgment. You count the number of ums and ahs. You count the number of hands that went up to which the teacher responded.
In other words, that's how clinical supervision works. And in part, it works because we're scrupulous about not judging anything. You said you want to engage, here's what I saw. You wanted
inquiry, here's what I heard. Seven questions that were lower order on Bloom's Taxonomy. So this is what we have to be more disciplined about. We have to be more explicit. So one of the
things that we can always improve on is to play back to the performer an if-statement. Well, if you were trying to engage me, if you were trying to write a scary story, that's a good feedback
check for you to -- yeah, that's what I was trying to do. Okay, well, then if you were, here's what worked and here's what didn't. And that's the key phrase: did it work given the goal?
Again, we're not saying that praise is bad. We're not saying that advice is bad. We're saying what we're trying to do is make utterly transparent and clear first that we agree on the goal,
that we understand the goal, and have you gotten helpful information about how you did against the goal. Then we can praise it or blame it or give advice. I think praise is fine, but you
should really delay giving it is the key, just as you should delay giving advice. And in fact, I did give Justin advice, and this is really the art of teaching. You wait until the person is
ready for advice. There's nothing more annoying than premature advice. Just ask any adolescent. Ask any teacher, from a supervisor. You'll know when people are ready for advice. That's a
real discipline to learn is to give advice when it's time, when you get from the verbal and the nonverbal cues that it's time. The praise thing, obviously, is something that has to be
handled very carefully. Praise does not get you better and too much of it will make you worse. Too much of it will make you dependent on the praiser. So again, if you hear yourself praise,
make sure that the underscore is on the feedback and that the praise is just in response to the feedback. Okay? Another way to say this is you should make clear to the student, as best you
can, that the questions you're asking are not leading. There's no issue. There's no agenda. They're questions that you're asking to get clear on what the student was doing and why they were
doing it. And the sooner that happens, the more the student is made more confident and less uncertain about where you're going with the questions. So I'm going to define feedback this way:
it's actionable information in terms of a goal. It doesn't matter whether it comes from the game or another person, but it's actionable information about what I did in light of a goal. Why
do I need feedback? Because I may not have known what I did. The little leaguer doesn't know that they're pulling their head out. And what is the time-honored piece of advice that coaches and
parents yell from afar? Keep your eye on the ball. Do you now see why that's a dumb thing to keep saying? Because the student doesn't believe they're not watching the ball. And that's their
immediate first response in little league. I was, Mr. Wiggins! I'm watching the ball! That's why you have videotape. That's why you have coaches, because you don't realize that you're
pulling your head out. You're convinced because you're trying so hard that you're watching the ball. And that example is generalizable to every educational interaction. The kid really
thinks they're doing what you asked. So does the adult. That's why adult pros have coaches, because you need extra ears and eyes sometimes. If you've ever done team teaching, it's amazing
what's going on that you don't see. That's why you have supervisors. That's why you have videotape. Advice is information about what to do differently in light of the feedback. Now as I
said earlier, I think that this is a critical issue in school. That actually is my daughter running, and what's interesting about this picture and why I put it in here is not that she is winning
the race, which she is, but you'll see her coach in the sweatshirt? See, he's not watching her. He's watching our number two runner, who is about to run a personal best, and he's yelling out
the split time to her, to say stay on that pace, you're going to beat your personal best. That's why you need coaches. If we had time, I would show you a 10-minute piece of video clip of an
ineffective teacher, but we've all seen ineffective teachers. I want you to imagine the post-conference conversation as what's the feedback you're going to give that person. Just imagine a
boring lecturer and imagine the goal is I want you to know about history or math or science, whatever it is. We won't challenge the goal. That was their goal. So you're scrambling, you don't
have much time. It's immediately after the class is over that you're going to have the post-ops conversation. So what's the feedback you're going to give him? Just sort of do a quick think-aloud
together. What kind of information would you try to pull together to put in front of the person as pure feedback? No advice, no praise, no blame. What sort of things given their goal? Okay,
talk it over. So let's have another little formative assessment. Feedback? No, what is it? Advice. If that's the advice, what was the likely feedback? What would you guess? See, that's the
way to start thinking about this. What's the sentence that has to precede that sentence? And that's what you want to say first. And the interesting thing is you can't easily figure it out
which is the position that the writer is in when they get that advice, so that's a good tip. Second one? No, that's -- in effect, that's a question the answer to which is going to be useful
to you to give feedback. Well, if that's the audience, then it seems to me you're talking way over the head of the audience, and that's my feedback. But if that's the audience, well, that
seems audience-appropriate to me. What about the last one? Yes, that's feedback. I'm giving you, in a sense, the commentary as a reader on what was working as I was reading. Peter Elbow who
wrote brilliantly about this 30 years ago in Writing with Power and Writing without Teachers. Said, you know, the goal is not evaluation. The goal is commentary about what's going on in your
head as a reader, that that's what you should strive for as a teacher of English or ELA. But this is a confusion because as the Eric Mazur case and common sense and video games show, this is
an egocentric delusion. If I taught you, you now get it, so let's go on. No, that's why we did the formative assessment and we showed that some of you didn't get it. And that's what Mazur
does repeatedly. See, this is the revolution. The revolution is to understand that if you go on when people are confused, you've lost all those people. And the longer you go without
feedback, the worse it is. Suppose I had just talked about feedback for the last hour and 40 minutes. You wouldn't get it as much as you do now. You just wouldn't. That's the point. It's
really cool what Mazur does, and don't tell me you don't have a small enough class. He does it with 200 every day. So does the track coach. It's not about numbers. So does the band director.
Anybody who thinks that it's about we're going to lose time from teaching and I have too many kids, that's feedback to you that they don't get feedback. Really, they don't get it, or they
wouldn't say that. That's like saying I just need to keep talking, so let's go on. That's efficient. Yeah, but it's not effective, so you don't get it, so how can I make you get it?
Interestingly enough, there is, as I said, this science misconception literature, and that was the epiphany that Mazur had because Mazur worked at Arizona State for a year where they
developed the Force Concept Inventory in physics, which was the real first shot across the bow in all the student misconception literature 25 years ago. Here's one of the questions on
the Force Concept Inventory. After a year of college physics, the majority of students still get this question wrong. There are your choices. I'm not going to quiz you. This isn't a physics
class, but the point is this test was deliberately designed to get at whether the student has overcome misconceptions that we know students bring to the table, and the results are truly
horrifying. A majority of students at good colleges get this question wrong after a year of physics. And even if you don't know much physics, you know that they ought to be able to get this
right. And that was really what motivated Mazur. He began to realize that the results, even at Harvard, were not as good as they should have been. And when that girl said, do you want us to
give the answer we think makes sense or the answer you taught us, he knew he was on the right track because some of you know that questioning is good. You can't help but see the question as
feedback. That's a predictable misconception. That's why I have it in there. The powerful generalization is every time you teach, you want a formative assessment of whether they're
overcoming predictable misconceptions. So you got to know the predictable misconceptions. We were talking about mathematics over there. And if you look at the released PSSA items in recent
years, all the distractors are coded as to what the predictable misconceptions are. Why they ask the question the way they did? Especially in math. So it's simply crazy to say you don't have
time. That betrays a fundamental misunderstanding of the difference between the teaching and the learning. That's like saying we don't have time to play soccer, let me just give you more soccer
lectures. Really, it is. You would know less about this topic if we hadn't stopped and processed it, and you know that at some level. So it's crazy to say you don't have time to do that,
it's crazy. Not if you want to cause learning. Well, believe it or not, this relates to PSSA because the research is pretty clear. The more you test, the better the performance. I know
you don't want to hear that, but why does it follow logically from everything we've been saying? Talk it over with the person next to you. Why is that inevitably true? The more you test, the
better people get. Why must that be true given the entire afternoon presentation? One of the clients that I worked with a number of years ago who heard me talk about this ran a controlled
experiment in his district in Ohio, outside Cincinnati. And they taught one group of fourth grade students the writing process. The other group, they just looked at released scored papers
and taught the students how to score the papers. The second group outperformed the first group by a wide amount. That should start to make sense. Utter clarity about performance goal,
internalization of those goals by learning to score papers. The writing process people weren't given feedback against the state standards. They were just taught the writing process. Here's
the corollary. You should be scoring student work against state and national standards. If you're only scoring it against local grades, that's a terrible feedback system. That just follows from
what we're saying. That's why we need standards-based reporting. Otherwise, you're screwing your kids. They're set up for failure. So we really need to get out of these knee-jerk ways of
thinking. I'm not saying that the tests are great. I'm not saying that I want more dopey test-prep. I'm not saying that at all. I'm saying the more the student gets feedback against goals
and opportunities to learn from the feedback, the better the student will get. One of the virtues of having the SAS portal is that there's a lot of stuff in there that you can use to
operationalize this idea. The bible on what we know about learning is the book from the National Academy of Sciences that many of you are aware of called How People Learn. If you're
unfamiliar with that book, you really must make it a central part of your library and PLC and action research because it is a summary of everything we know about how people learn. And
everything that we've been talking about is summarized in there in terms of the teaching for transfer, meta-cognition, using feedback to cause performance. Here's a little rubric that Jay
McTighe and I put together for the latest UbD book that's coming out next month. And it harkens back to what we said about the importance of self-assessment. There are two boxes, one for the
student and one for the teacher. So the student self-assesses every piece of work themselves, then sees the teacher feedback, and then writes a comment about what they're going to do differently
next time. Some of you know that John Wooden was arguably one of the great teachers at UCLA, one of the all- time great basketball coaches. What many of you may not know is that he was so
widely regarded as a fantastic teacher that two professors at UCLA shadowed him for an entire basketball season and wrote a book about it. And that book has fallen out of print,
unfortunately, but there's a new book that was co-written by one of the authors and the backup center to Bill Walton, called Just Because You Taught Doesn't Mean They Learned It. And it's a
beautiful summary of many of Wooden's moves. And this was the running total of tagging every comment that John Wooden made over an entire season. They actually had to invent a phrase, and
they called it a Woodie or a Wooden for his particular locution. He would freeze the practice or the scrimmage, he would say, here's what you did, here's what you need to do, go do it, then
he'd put them back in the game. So feedback, advice, go do it immediately. That was what he did 75% of the time. At Alverno College, one of the first competency-based colleges in the United
States in '70s, still in action, and some of you may know the good work at Alverno, they believe so much in this that 15% of every student grade is the accuracy of student self-assessment.
When you do the oral presentation for the first time as part of your communication proficiency as a freshman, it's videotaped and the first grade is not how well you did on the presentation,
but the accuracy of your self-assessment and watching yourself on tape. Ain't that cool? This was given to me by a friend who heard me talk about this. Dear Ms. So-and-So, I disagree on
giving me a three. You're not supposed to grade different from the other teachers. And worst of all, you didn't tell me how to get a four. And further on, in the next six weeks, I want you
to grade like the other teachers. Tell everyone how to get a four and tell the kids that are new to this school how it works. Part of this should be in the conference with my mom. From
Taylor to Ms. So-and-So. Now what's cool, what's cooler still, is that the teacher was new to the district. I mentioned earlier some criteria beyond the ones that you mentioned for how to
separate robust from not so robust feedback and you'll see some of the ones that we cited. Again, in terms of studying models, Ken Bane at the college level, they had a massive research
study at Northwestern to study college professors in America, drilling down to the top 50 professors, whom they then viewed and videotaped and studied. And the analysis of the best teachers
reflected a few important things, but the -- including these points, but for our purposes, what the Harvard report said was confirmed in Bane's research. Another way to say it is that the
predominant feeling, and this is true not only in schools, but I sometimes sense it in workshops, is that people are afraid to say something that's stupid. That is a huge impediment to
improvement of performance. Indeed, it may be the key impediment. And so we have to make sure that the student will express or perform without fear of embarrassment or ridicule or failure. And we
know that they bring baggage and education brings baggage to the table about this. So some few final tips. You know, you want to depersonalize. It's nothing personal, but here's what I
saw. Here are some tips for how to be more efficient at it. I don't have to get personal every single time. Really good coaches with large groups, you know, call their moments where they
say, look, here's a big problem I'm seeing. People are doing this. That's incorrect. We need to do that. Let's work on it. So it doesn't have to always be one-to-one, mano-a-mano. The last
bullet is really important because we're now living in a world with clickers and the Internet where we can do it. We can build non-human feedback systems that we can have at the ready for
reinforcement. This is really a key advance in the history of education, that the student can get feedback when we're not around that keeps them going and moving. And you really want to give some
thought in instructional design to systems around the outside where the student can get reinforcing feedback to make sure that they get it. And then of course, what's good for the goose
is good for the gander. We need feedback. One of the shocking realities of American education is that teachers are afraid of feedback. They don't seek it very willingly. They don't
particularly run out of their way to go get it if it's not provided to them, and as a result, American teachers are not as good as they could be by the very same reasoning. When I was a young
teacher, and I don't say this, you know, out of pride, I just think it's kind of unbelievable. I was the only teacher in my school who asked to be videotaped. This was in 1978 before there
were cassettes. This was those big Sony reel- to-reel decks for those of you who are old enough. And so that meant that the two AV guys had to come do it. That's cool. You know, to me, it was
a no-brainer as a coach. And it is a no-brainer. I don't know why we resist it. I really don't. It's weird to me that educators are so resistant to this when it's so obviously a good thing.
What, is the tape going to bite you? I mean, come on. Tape yourself. And if you don't have a video camera, audio yourself. You'll learn a lot just from listening to your words apropos the
distinctions we've been making all day. Finally, last but not least, I got a drive to get student feedback and last year I got 7,500 middle school and high school students from around
the country to give fantastic feedback based on a bunch of questions. And if you're interested in this survey and want to have it for free, you may do so. All you have to do is write me. And
the results were pretty fascinating and it grew out of an experience that I had as a young teacher. A colleague of mine, a veteran colleague, handed out a big card to his kids every Friday with
these two questions. Notice the language. Not what did you like. What worked and what didn't work? And there were two interesting results. No one ever agreed on what worked and no one ever
agreed on what didn't work. Boom, there's differentiation in 1973. Some of the student responses, you can read more about all the student responses if you go to my blog, which is
accessible from my website. And you have that information, it's the last slide. It was interesting that the students report that the work is not so hard if you just do your work. What was the
most interesting work you were asked to do in the last year? This is fascinating stuff. Almost all these things are really cool assignments and projects. So it's like anything else. Do more
of this and less of the other stuff. We also asked students, what is a very common teacher practice that you wish teachers would stop doing now? Number one, dumb PowerPoints, reading dumb
PowerPoints. Number two, boring lectures with no visuals. You can't tell me you can't learn from this feedback if you ask the question the right way. So you have some other examples in the
materials and they're pretty cool. And again, you can read more on my website. We also found out what students' pet peeves are about the subject matter. Interestingly enough, math was the
most liked and the most hated. But what was interesting is that there's a different profile as to why. English is more hated because it's boring. Math is more hated because it makes me feel
stupid. So you have all that. We'll skip over that. A little summary slide of some things you might try. And in addition to my website, you can also go to the site and get some more
resources and free downloads by just going to the resource page and typing in the little code there, PaTTAN 2012. Well, I hope this was helpful to you and I await your further feedback. Thank
you very much.
