PRESENTER: Okay, I'd like to introduce Dr. Pickering, please. Dr. Pickering is a private consultant working with schools and districts nationally and internationally. During more than 30 years in
education, she has gained practical experience as a classroom teacher, building leader, and district administrator. Dr. Pickering has used this experience to provide training and support to
K-12 teachers and administrators to seek to continually improve student learning. In addition to her work with schools, Dr. Pickering has coauthored, with Robert Marzano, educational
books and manuals, including Dimensions of Learning, and more recently, Classroom Instruction That Works, Classroom Management That Works, and Building Academic Vocabulary, all published
through ASCD. With a combination of theoretical grounding and over three decades of practical experience -- and I think that's what we especially enjoy, that practical experience -- in the
real world, Dr. Pickering has worked with educators to translate theory into practice. Her work continues to focus on the study of learning and on the development of resources for
curriculum, instruction, and assessment that can help all educators continue to meet the needs of students K-12. I'm sure you're back because you enjoyed her keynote session so very much,
and so let's welcome Dr. Pickering. DEBRA PICKERING: Thank you. Thanks. This afternoon, we'll slow down a little bit. [inaudible]. Sorry. We will slow down a little bit this afternoon. The
hard part about the folks in the back who are trying to follow what I'm doing, I'm just not very sequential, I apologize for that. I change my mind when I'm doing things. Based on your
questions, I will jump around a little bit. But I just -- you know, this is actually the whole formative assessment, standards-based grading topic is a Rubik's cube. I've worked with the
district for a long time. A couple of weeks ago, they gave me a Rubik's cube. They said, you keep saying that! I said, that's because there's no right place to start because just when you
think you get this part of the classroom kind of figured out, you've messed up this part. And then when you got kind of this part, then this part's messed up. And then none of it fits
together anymore. And when people start changing their assessment and grading practices, they go, well that, wait a minute, that means I can't do that anymore, and that means I need to do
more of that. And you know, it's just -- it's the nature of our business because we deal with human beings. I mean, it'd be great if they're all predictable, and you know, and it would be
really great if something you did one day actually worked the next day just as well, you know? Or as poorly. But it's just not that way. So I'll try to create a story here. That's what I did
in kind of laying this out in some topics here. And I will refer to the handout. Again, people who are trying to follow on the handout, I tried to explain, you have key slides you can go
back to. The main thing I want -- if I had my way, we would do no more handouts. People would just take it in, interact with your colleagues, take some key notes, and then go to the handout
or slides and go back and review on your own. I think that's a really better way to learn. So I apologize if it's a little random for you. Again, I have a sequence here and I'll try to
answer questions as we proceed because I threw a lot at you this morning. My guess is, however, the things that I was describing, you probably have some of that going on in your
school. There are so many schools around the country who are rethinking assessment, and they're rethinking grading practices and how you use the feedback from assessments. And there are --
they make great strides and then they struggle. Not everybody embraces it. I have scars from people who, some of the stuff I said this morning. You know, they're going, are you kidding me?
And we have lively, collegial conversations. I love to have those, so you don't have to agree with everything and you can say, what about, what about, what if? So I'll kind of continue
with the grading implications because I want to -- that brings up some other issues. And also here's what I've found. Some people say, well, let's just talk about assessment and really not
talk about giving grades. Here's what I've found is that teachers, rightfully so, have trouble wrapping their head around some of the assessment issues until they can picture what they're
going to do in their grade books, and on the test, and when they hand the paper back, and when they have to turn in grades. Because that's a reality. They have to do that. Now we could
wave a magic wand and say, let's get rid of grades and just give kids feedback. Well, you know what? I think that's an interesting conversation over a glass of wine, but I really do not
think it's going to happen, at least while I'm still in this business, very much. So let's at least make sure what we do and we must do, because we're kind of stuck there, has more meaning
for students. And that's the part that keeps me going. So as we look at grading implications, I'll talk about little baby steps. Before I start, are there any general questions that you have
that you just kind of got to get something cleared up before we can go on or have me clarify something from this morning? And I'll tell you what, take one minute, do this. Everyone raise
your right hand, point your index finger, when I count to three, point to your table leader. One, two, three! Okay. See, you didn't have time to say, no, don't pick me! Okay? So table
leaders, help me bring people back. Just take a minute and just see if you can, if there's anything you need to clear up, just going give you a minute to clear that up. So not very long,
you're not going to clear everything up this afternoon. But if there's a burning question, then I'll ask table leaders to ask the question when you come back, okay? So take one minute, say
what's clear, what's confusing at this point, share at your table, and then I'll bring you back. One minute, go. Thank you. Let's -- I'll take a few questions and we're going to move on, all
right? Table leaders or anyone, if you have a question that it can help to clarify from anything said a little while ago -- it wasn't this morning, it was this afternoon. I don't know what
day it is. Yes? Nice and loud. AUDIENCE MEMBER: Okay. I just wanted to know how your feeling is as far as students failing. I mean, you know, getting an F, or like, you know, an overall
F in the class and either lack of effort, you know, things like that. DEBRA PICKERING: My feeling about -- I'm sad when they do that. When my own child came home with F's, I was really sad. I
think we have to take responsibility for that. In that kind of -- that gets some hackles up, but the word responsibility means ability to respond. What a formative system tries to do is say,
we're here to try to maximize learning for every student. Usually when students are failing, there's a root cause that we can get to if we just sit and go after it. Sometimes people say they
don't care. I think they do. In fact, in fact I know they care very much. But students at a certain point sometimes learn pretend you don't care. Because it's a lot less painful to
pretend you don't care than to care and fail. So when we say you have an F, I mean, you know, we can call it something else because we don't like the fail word, if we don't like that, but
what we're saying is you haven't learned it yet. And that's for the whole formative system's geared to say. You haven't learned this yet, but we are not going to let it go. And lack of
effort? There's always a root cause for a lack of effort. Because they have learned that effort doesn't pay off. And you know what? Sometimes when I listen to students, I go, I get why you
believe that, I really do. Because you did put in effort and you didn't do it. So you weren't able to pass. So all of the reasons that kids fail, we can go after those, we can go after them.
And some of them we obviously can't fix. A whole bunch of them our system sets them up for that. And a whole bunch of those reasons I could bullet out right now. We can change that and we'll
have less failures. One of the things that, just so you know, the teachers have said who've been doing this for three or four years is, I've had so many fewer F's. Because contrary to what
people say, aren't you letting students off the hook by letting them retest? The opposite is true. We're saying if you didn't do well, we're going to find out why. You're not going to
disappear in the back of the room. You're not. And sometimes the kids say, whoa, the teacher cares a lot more about this F than I do. Good. Because we're saying, come on. You can show
success. So I think the bottom line is we have to take responsibility for that because we can respond to that. Make sure we find their gifts. Another question, yeah? AUDIENCE MEMBER:
[inaudible]. DEBRA PICKERING: Based on the idea that standards are good, yeah. Your science standard is graded as a D? I think any time you have a document or curriculum that is weak, it is
-- you're not going to like this, but it's just incumbent on the teachers and the school to get together and make them good and make them right. Because I think we can. I have never seen a
set of standards that say, you know -- there's some, I go, you're kidding me. I mean, some social studies standards, I go, whoa! They love their content way too much, you know? Lighten up a
little, you know? They don't need to know all of this. But what you do is put your heads together and make them right for kids. I think -- I just think we can. I really do. Some of the
documents that I was in the room, I don't know what they're thinking, but we can make them right at the school level. Okay? Other questions? Let me look at some of the grading
recommendations. This is the first baby step. I'm going to ask in a minute how many of you are already here. This is number one, probably the most common thing we've seen, a huge leap in
schools and assessment and grading. And that is keeping academic records separate from life skills and non-academics factors. So you have whatever, I have two in there: getting work in on
time, getting your homework completed. By the way, that homework score would not be the performance on the homework, it's a score for the extent to which they completed the homework. Some
people don't care whether kids do the homework, some care very much. So the thing is, I'm not telling you what to put under life skills, but keep them separate. So if a kid is a
four-level writer, knows four-level about cultures and cells and all those things, that's what the record shows. If they turned everything in late, they're still a four writer or a four
reader or a four understanding of cells. Likewise, if a kid is a good kid and you want to say, but I just don't want to give him a one, if they're a one reader, they're a one reader. Let the
records show that and let them also show that they hang in, they put in effort, they do the work, everything's on time. Anyone looking at the record would know this is a kid who writes well,
but doesn't like to do things on time. As opposed to this is a kid who will be there on time, but doesn't write well. Let the record be accurate, that's it. No 10% off, 10 points off for
every day it's late, or the silliest policies that we should get rid of tomorrow, which is I don't accept late work and I average in a zero. Stop it! It's not okay to do that. You read
your mission statement. It does not say, this school is set up to make sure the kids jump through the hoops the teacher puts in. This school, instead, is set up to make sure they learn as
much as possible. And it, you know, being hard-nosed about stuff -- again, can they have consequences? Yes. Because if you look at these, here's what you have to decide. See the far
right-hand side? Some people would argue to the death, would die on the hill, that you never put those two scores back together. Others would die on another hill that says, you must put
them together. So there isn't a right or wrong there. There's no research to tell you. That's philosophical. You kind of say, what's best for kids in order to do it? Probably right
now, today, the recommendation that I would make is put them back together, and here's why. Sometimes kids actually need that nudge to do some of the things that they know they should do,
and if they know it's going to have some effect on their grade, that might be a helpful nudge. Should it take them out, however? No. The most common percentage weight that the non-academic
life skills we're seeing have on the grade is about 15% of the grade. Is that right or wrong? I don't know. It's the most common that I've seen. However, in the non -- in the elective
classes like PE, the performing arts, you know, and some of those classes, I've been persuaded by teachers who say, I want them to be weighed more heavily. I want them to have a heavier
weight. Now is that right or wrong? There's not right or wrong here, it's a philosophical thing, you know? And I understand why they do. They say, it's elective, I want to give the kids a
lot of reinforcement in my art class or my music class, choir, to hang in there with me and not just be purely on their performance. I always think of my daughter, who is my A student, my
only one, my A student, came home one day from middle school. She was taking choir, they had a singing test and she got a C. Well, she gets straight A's, you know, so she's like, I got a C,
I can't believe it! And I said, honey, that was a gift. Let's not be complaining about that one. I say, you know, for one thing, the world's not going to come to an end if you don't get an A
in a class. Well, for her it was. She actually got on A in choir. You know why? Because the teacher very heavily weighted things like she showed up for the performances in the khaki pants
and the white blouse and the -- you know, did all the movements and was -- you know, she sang with major enthusiasm, it was just really bad. Really bad. Still is. She came by it honestly, by
the way. So I kind of see that and I've also been persuaded. Now is that -- again, is that right or wrong? No, you have to have collegial conversations about how you want to do that.
Here's what the research does show. You cannot move to a formative, standards- referenced or standards-based system unless you make sure your records are accurate. You can't have accurate
records unless you have a policy that is non-negotiable, that the grade book and the report card must show academic performance separate from the life skills. And that you, you know, keep
track of those separately. Now again, what you include in that is a personal, philosophical decision. But making sure the record is accurate is not philosophical, it's based on data. Whether
you put them back together for an overall, sometimes called an omnibus grade, is a philosophical decision. I can't give you data that makes it right or wrong either way, or more likely to
work either way. So that's the first baby step is just to keep those separate. Where are you in this journey? Is this something that you, if I came to your school, would everyone's
grade book and report card show accurate academic performance not confounded by non-academics, or do they kind of get mixed up in policies the teachers have about averaging in zeros for
missing homework and that kind of thing? Where are you? A four means it's absolutely clear, your records are good, and you're ready to move on. A one means you can't even talk about because
everybody gets mad and they have nasty arguments. As you're talking, please come up with what's the best argument you've heard for not doing this? I've never been persuaded by any argument
that you should not do this tomorrow, that makes sure this happens. Where are you on the journey and what would be arguments against it? Just take a minute at your table.
Questions or comments or your best shot at the argument you've heard -- not necessarily your argument, but the argument you've heard about not doing this? Table leaders,
what came up at your table? Questions, comments, challenges, good jokes? You're buying all of this? Yeah? AUDIENCE MEMBER: The challenge that I hear from upset parents is that it's not fair
because colleges -- DEBRA PICKERING: It's not fair because colleges don't do this. AUDIENCE MEMBE: [inaudible]. DEBRA PICKERING: Five tries to get it. Okay, but hold that argument.
That's for making sure that the topic academic score reflects where they've progressed to. But if you take that same argument, and I've heard it in terms of colleges don't do this, actually,
they kind of do. In a lot of colleges, it doesn't really matter what you're doing along the way, it's what you get on the assessment. So they're not checking off, you know, participation
and, you know, did your homework and when you're, you know, person in the study group. So you know, colleges actually are probably more like this. We include all of these other things in the
grade. Sometimes this is simple. I mean, you could do a traditional grade book. So you can see this is how the grade book looked when I first started. And you know, all you do is say, keep
your assessments separate from whether or not they did their homework, or whether or not they did this or that. But there's some questions we often hear. Here's one. But isn't it a
major responsibility for schools to teach these types of employability skills? That's one phrase used by one of the districts. Won't this approach make these skills less important? Actually,
what I think by doing this is you actually make them -- you shine a light on them. You shine a light on. They're not like hidden in the score, like 10 points off for every day it's late.
Well, you can't even find that later. But if the records show this, you're shining a light and saying, you know what? You're doing fine academically and your grade is actually suffering a
little bit because of these life skills. Work on that. Likewise, you can say, your life skills are so strong, that's going to serve you your whole life. You need to work on your
writing. And make sure you're clear. I had -- I raised three children and one, my oldest, had some learning disabilities. And one of the things that was a struggle for me because he was a
good kid and, you know, adorable -- and you know, a little biased there, but the teachers told me he was adorable. And sometimes I think they bumped up his scores and grades because he was so
adorable and likable and he was just so well-behaved. You know, and it's like I'd say, boy, we need to really work on reading this summer. My teacher thinks it's fine. I said, no, she doesn't
really think it's fine, because notice this little thing that's really hard to figure out, which means you're reading well below grade-level, which is really an oxymoron, but I get it. You
know, so it was confusing to have those -- let's be clear. And say, you know, your reading isn't strong, but we're here, we're going to get it strong. By the way, these life skills, that's
going to really serve you well by doing this. We have to be clear about that. Sometimes we put too much focus on them. Sometimes say, well, if they're not on time, you know, they're going to
be terrible employees. My second child went to -- he had an attitude problem. He went to an alternative high school and, you know, considered BD. He just kind of antsy, but anyway, he went
to -- I had friends who said, you're going to let him go to high school with those kids? I said, he is those kids. I mean, really. Did the three tattoos give you the clue that he was
already, you know? And I mean, wonderful on the ice, you know, was a hockey player, but not so good when he couldn't move in the classroom. So he went to the alternative school. And you
wanted to make sure that, you know, the records showed for him, as well, that in the academics, here's where he was, and here's where the problems were in the life skills. And I got to know
a lot of those kids at the alternative school. Some of them had really weak scores if you gave them scores on life skills, were the best employees in the world. Their bosses would say, this
kid shows up early, is responsible and wonderful, but not at school. Well I get that, because some of the rules in a traditional school setting don't work for them. I so get it when I watch
them focus -- function in a different environment. So we have to make sure that we don't, you know, go overboard on the life skills. Another thing I hear is the thing about homework is, but
you know what? If you don't count the homework in their grade, they won't do it. Well first of all, notice that if you really feel that strongly about homework, have a homework completion
grade. Don't put it as an academic score because you don't know how much influence they had when they did that homework. So just don't go there. Homework is not a good assessment of the
standard. It never will be. But you can say, at least they took the trouble to copy it on somebody else's on the bus. I mean, that's about all you know. You know, this whole game of trying
to figure out if they really did on their own, well good luck with that! But they're putting in the effort and if you have authentic conversations with them, I think we'll be good. I know
teachers who do a very good job of saying, if you don't do your homework, I don't know what you need next, so you've got to help me out by doing it. That's authentic conversation and kids
respect that. So those are the -- that's the baby step. Here's the next step. Track progress toward learning goals. That means setting up your grade book differently. That means we move
from grading assessments to using assessments to track specific learning goals and reporting and recording accurately. Let me blow this up a little bit. This is a typical kind of online
thing that you see now. Can you see? We only have one screen in here, don't we? The -- in the online, notice that you see, you know, there's chapter test and application quiz, and then
commentaries, whatever. You look at this and tell me what content area that is. It's hard to figure out what the learning goals were, correct? Some of you might know because micro-unit is
microeconomics, it's an economics class. But it's hard to tell. If we can't even tell what the standards were that were covered by looking at the student's record, you see the problem? We
don't even know that that is. Or if you look at -- I don't think I have a bigger version of this. It's a little harder to see. It's kind of an older computer grade book. But they have all
the assignments listed, how many points each one was worth, and how many points they got. And they included in that list the things like textbook cover, you know, bringing in Kleenex, and
all those things that we give points for. Really? Really? So you kind of say, wait a minute, if you want to do those things, okay. But let's make sure that we are clear about what it is
that -- seriously, let me go back to this. If you can't read it, let me just point out, it says Johnny Tremain quiz several times there. Johnny Tremain quiz. This is middle school. Johnny
Tremain quiz. What was the Johnny Tremain quiz assessing? Because I know the standards at this school. There's no standard for Johnny Tremain. So if you're using Johnny Tremain to quiz
something, tell me what it is and report that. Is it reading comprehension? Well I'm not so sure, I'll share that in a minute. In other words, academic comes to academic topics. And
again, I explained this earlier, the reason I'm using topic instead of the word standard is because teachers work with standards and say, this standard I need to split into three. This
standard here I need to combine with this one and make it this. Because they're written. You would say weak standards. You look at some of the documents and go, pfff. Don't try to set it up
standard by standard. It's weird. Try to set it up in a way that parents and students will understand it. That's why it says topics. So you make sure that you have a topic score, an ongoing
score in the academic area. So when you're -- then you put them together for an overall grade, overall score for academics. So those are put together and then, once again, you have to make
this decision about whether you put them all back together again. Let me reiterate this, that when you have to turn in grades, which means the summative score, it represents not an average,
but represents their level of achievement. Now this gets back to your question. But colleges need to know that. No, they don't. When people say that they, and I've heard that argument, you
say, really? You want to tell them how long it took them to learn it? So a kid who walks in knowing what's going to be taught already has the advantage, and now we give him another advantage
by saying, if you got it first, you can get the A, but two people can't get the A? How long did it take him? It doesn't matter. What we're going is preparing them by having them learn as
much as possible and then have our records show how much they learned. You would not want to be rated as a teacher based on the average of the last 10 years. So you want to be rated as a
teacher on how you are now. And so it's very, very, very common, you know, to do that. So it's what they can demonstrate in terms of learning. Yes? AUDIENCE MEMBER: Now in the example
that you showed earlier, where certain topics you keep covering if they haven't mastered it throughout the school year -- DEBRA PICKERING: Throughout the school year, yeah. AUDIENCE
MEMBER: That first mark in period three, they're still improving on the topic. Does that first marking period grade change? DEBRA PICKERING: Yes. No. Maybe. Next question. The reason is
because it depends on your software and the policies. Let me give you the ideal answer. You can turn in grades and your grade is a function of your performance on all the topics that have
been introduced so far, to that point. So if you have a one on a topic and you leave it as a one the whole year, let's say it's first quarter, that one is going to be averaged in to your
overall grade all year long. At any point, though, you can change it. So it's not that the first quarter grade changes, it's -- your grade is not a function of first quarter and second
quarter average. It's a function of the topics that have been introduced so far. Does that make sense? So the -- it changes some of the habits we have with grades. Another thing that
people are doing in the interim? Some school districts, the reason I said maybe, you can go back and change the grade at any time. That's their policy. Some places that would be like
blasphemy or asking them move the earth, if you did that. So if so, teachers have figured out a way to bump up the next grade so it ultimately does affect it, explain that to kids, and kids
get it like that. You know, it's just -- it's called a workaround. You say, I will honor you. But in an ideal world, if you want to give a grade, an overall, it's what's been assessed so
far, you know, how you're doing. So a kid knows that, you know what? It might pay off to go back and make that one a three. Because that's going to affect things for a long time, so why not
go do that? So that's the ideal answer. Does that help? So as you're looking at generating these scores, remember that the averages come in, when at a certain point you say, well I've
got to turn in a grade for math. Okay, now you can average across topics. There're other ways to do it, but that gets into some technical aspects. Most teachers average across topics, just
not average within the topic, and then we're good. So a standard might be a topic, a standard might be two topics, or it might be three topics. Or split it. It's whatever makes the most
sense to the teacher, that's the rule of thumb. So if I set up an old-fashioned grade book, here's what it might look like. Notice I have my topics across and the assessments are there, and
this is how students are doing. Notice that Josh and Jamal started at very different places, the first squares, and then -- but they ended up in the yellow square for the summative score for
the topic turned in at the same spot. So as you do this, a lot of things come up. Let me give you an example. Notice this is a cut-out from a teacher who's on this journey and started to
report out by topic. He was giving quizzes on the books that the kids were reading, Huck Finn, Bless Me, Ultima, Inherit The Wind. And as he was doing this record-keeping, this is just
trying it this way, he said, wait a minute! Could a kid's reading comprehension be just fine but still fail the Huck Finn test? Is that possible? Why would that happen? Because he didn't
read Huck Finn. So he said, wait a minute, kid's reading comprehension could be weak and they could actually pass my Huck Finn quiz. How could that happen? You know, SparkNotes online.
Someone gave, you know, whatever. He said, well wait a minute, especially could the reading comprehension be strong and not do well? That was the more common scenario. He said, well wait a
minute, then where do I put the Huck Finn test score? And I said, I don't know, what are you assessing? He said, Huck Finn. I said, that's the problem, you're going to have to create a new
topic called Huck Finn. He said, that's not in our curriculum. I know. So what should I do? I said, you tell me what you should do. What would you tell him to consider, reconsider? Solve the
problem at your table. He gives tests on the book, where's he going to put them in his grade book? Okay, come on back. What's your solution? He was really -- he was really frustrated. He
was a very, very thoughtful guy. We had a good conversation. What do you suggest? Anyone got any ideas? Yeah? AUDIENCE MEMBER: That his learning part would be what would be expected for
reading comprehension, like analyze plot. DEBRA PICKERING: Yeah, analyze plot and character interactions or symbolism within the, you know, whatever it happens to be. What if the kid
hasn't read Huck Finn? Can you still assess that? See, if you assess what the targets are, that means you can have alternative assessments and you won't ever hear, but that's not fair. No,
that's -- no, no, no, no, what we're saying is it might -- if it says with grade-level materials, you know, then you have some constraints. If it says with Huckleberry Finn, which it
doesn't, then, you know, you do that. So it might be grade-level materials, but very different grade-level materials. It kind of argues for giving kids choices. And you know, and it doesn't
mean you never teach everyone reading the same novel, especially if you're teaching and modeling something, but maybe they could do it in different ways. And say, but that's not fair if
they didn't have to read Huck Finn. You know, just that argument breaks down. There's no single book that every kid should read. You know, I've had people beat me up for saying that, and I'm
sorry. I had a teacher not long ago, she said, you're wrong about that. I said, well what's the book that every kid has to read? She said, Dante's Inferno. I said, in order to get through
college, they have to have read Dante's Inferno? She said, absolutely. I said, I've never read it. I have a PhD and I did very well, okay, because I was a very compliant student. So
it's just, you know -- just it all breaks down. All these arguments just break down. They say if you truly want to know, then there are all kinds of ways to assess it. If I want to know how
much they know about cultural conflicts throughout history, you can do that orally, you can do that with, you know, visually, you can do that with text, you can do that -- and it's all really
good assessment for the topic. So now you have many, many choices and kids can show that, demonstrate this, in many ways. Don't get me wrong. It's not just loose, you can do a collage
and say, now I understand, you know, that -- no. It's got to -- you have standards for your assessment, but it's what you're assessing that is the question, not whether or not everybody
does the same assessment. Now I know I'm preaching to the choir here. But it really is an argument that is worth having, but you better have collegiality going. Because people get pretty
riled up because it has major implications. To this guy, it did, and this was his own aha. This is his own personal aha. No one said this to him. He said, I'm just wondering why I'm doing
this. Well I said, one of the things you want to know is whether they read it. I get that. But I tell you what, the only solution I can give you is put their performance of that in your
non-academic life skills section of your grade book. Because now you're actually -- what you're doing is saying that they read what you told them to read. Now I would challenge that, but
that's a solution that you could have for right now. Because I would certainly challenge it. Because some kids are never going to read Huckleberry Finn. The reason I never read Huckleberry
Finn is I hate books, I struggle with books that have dialect, and so I would choose something else to show that I knew that there was symbolism and, you know, in a particular novel. So
people said they'd teach 10 novels, I go, I'm sad about that. As opposed to getting kids to read and going after your targets. Okay? Again, I know I'm preaching to the choir. Here's the
red flags. You'd better have collegiality because when you say no averaging in zeros for missing work. If you have sufficient assessments to know how well they do reading or writing or
understand cells or whatever, if you feel like you have good information but they have missing work, no averaging in zeros. You keep track of that separate. No decreasing in score on
assessments for late work, and it's just important to remember this. This is one that, you know, people are philosophic and say, I kind of get that, but it's going to be hard for them to
give it up. That's a real important thing in a formative system to do. The other set of red flags is no accumulation of points. The score represents what the student has achieved. It's
not how many points they've been able to gather through the quarter, and they're reassessed as often as they're needed and as long as it's feasible. Now let me make sure I'm clear about
that. In the journey, teachers go, I can't assess that many things endlessly. I say, well then set some parameters. You sometimes have to turn to kids right now -- now when your system
changes and there's assessment is one of a whole role of the school and reassessment is, that'll change, but right now just say, kids, see these topics? After next Friday, not going to do
them anymore. I can't, I can't manage it. So you got to, you know, show up. Because it's just manageable. It's real. And so -- or if you see this to be too much, choose three or four topics
where this is worth doing. Choose three or four topics, you go, you know what, if kids don't learn this this year, I'm setting them up for failure. And so with these topics, I'm going to
say, come back as often as you need to with resources that tell me that you've been trying to improve your performance and I'll reassess you as much as I can. Just start there. And then
validate that. Are you coddling kids? Absolutely not, you're not coddling kids. I'll go to the next point in a minute. What's clear, what's confusing about that second step on the journey?
First step, separate academic from non-academic. Second step, make sure your academic report's by academic topic that represents your standards, and that score represents what they've
achieved at the level they've achieved to. What's clear, what's confusing, validating about that? And then we're going to switch gears here a little bit. At your table, what's clear, what's
confusing, what's validating, what's upsetting to you about those first two steps? Just share at your table, then I'll take questions. Okay, come back, please. Questions? Questions or
comments? Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Subdividing into topics, yes. You're using the word retention to mean remember, yeah? So you want a score to see if they
remember what they knew the day before, day before? Many, many assessment scores are retention scores. I mean, what you recalled. All any assessment can do, any assessment, I don't care who
you're assessing, can do is say this is what they were able to demonstrate on the day I assessed. After that, you're done. Anyone who says, no, they'll remember it next year. Oh stop it. You
know, they're going to tell the teacher next year, no, I don't think we did that last year. And they go, fine, they don't teach them anything in middle school? Don't they teach them anything
before we get them? Don't hold me accountable for standards because they didn't teach their standards. Stop it. What you need to do in instruction is know that you have to kind of cycle
back for all of us. By the way, a bunch of you have forgotten what you learned in high school. A bunch of that stuff. Some of it you recall. Try to figure out why you recall certain things
and forgot other things. So that's all anything can do is say on the day I assessed and they knew it. And if you know a kid has -- my oldest son was this way. He really knew something really
well one day and it was gone on the next day. All you can say is he knew it. And I know for most kids, if you knew it once, it's a little easier to get back there. That's all I can tell you.
Knowing that he forgot it, you say, let's do it again. Okay, let's brush up again, let's -- you just have to do that. But for all kids you have to do that. It's just this no
accumulation of points is just so critical to do because, I mean, I know I shared probably when I was here before, I shared with you -- but I always still laugh. I laughed about this with my
daughter the other night. She's in college, getting ready to graduate, and how she accumulated points, you know, was having really cool poster board, you know, tri-fold poster board. And you
know, we cleaned out the attic and I talk about it in presentations. I said, there are some of the posters that she did. She got lots of points for those. You know, when new mothers ask me,
how do you make sure your kids are successful in school, you know what I say? Get a lot of really cool poster board, yeah. I take it to baby showers, I still do. Bob's daughter had a baby
and I actually did take tri-fold to Bob Marzano's -- she said, what's this? Bob said, don't ask. Yeah, you're going to need it, you know. Trust me, put it under your couch in the living
room. You'll need it. Don't write me a thank-you note. You will write me a thank-you note in about 10 years. It'll be midnight on a Sunday night. You'll bless the day that I gave you this
poster board, trust me on that. Wal-Mart will be closed, you'll be happy. Here's the second thing we do at the baby shower. We practice burning the edges of a paper without destroying
it. And the third thing is to do your family tree once and save it. That puppy will come up about five times before they graduate, yeah. For my kids, having three kids, oh gosh. My daughter
came to me, we were laughing, really, about this the other night. You see, she came in her sophomore year in high school, said, oh, I have the family tree assignment. And this was the last
time I was ever going to do this assignment. My youngest kid, sophomore, surely she doesn't have to do it again. So she got out this big sheet to do the family tree. This time, I made it up.
I figured, what are they going to do, go check? What was great-grandma's name? Helen Miller. Know how to spell that one, yep. When was she born? 1894 in Scotland. I mean, we were
accumulating points. We knew what we were doing when we did these assignments. So we're trying, we're trying hard to get that. And by the way, with technology now, if you do a video, you
will, you know, put it online and you'll get more points. And so let's, you know, make sure that we do this so we don't just continue to make the same kind of mistakes. So those things
are pretty straight forward, but they're painful sometimes to change our habits. This is a bigger leap, using a scale that represents learning. Here's what I want you to do. Pretend you have
given a test on a particular topic. There are three sections to the test. Here's the first section, section A. Read it, please. Section B. And section C. Here's what I want you to do, and
please do not let anybody at your table see your work if you jot it down. I want you to assign a certain number of points to each of these three sections, but you need to make it add up to
100 just so we can compare. So if you had 100 points to distribute on this test, how many would you assign to section A, how many to section B, and how many to section C? And don't tell
anybody what you're doing. Okay, everyone got it? Now I want you to grade my paper, okay? Here's how I did. Section A, I got them all correct, how many points did I get? Jot it down.
Section B, I got two of the four, half of them. Section C I couldn't do. Some of you are worried about me. I'll be fine. Now add up my total number of points. Okay? For how many people in
here, show of hands, did I get 50 or fewer points? 40 or fewer? 30 or fewer? 20 or fewer? How low did I go? 20? Anyone lower than 20? 18? All right, 15? 15's my low. For anyone, did I get
more than 50 points? 60 or more? 70 or more? 80 or more? How high did I go? 85? Anyone higher than 85? 90. Okay? On the 100-point scale, which people argue is more objective than any other
way to do it, my knowledge for you being exactly the same for everyone in this room, my score ranged from 15 to 85. What just happened? Share at your table. Why?
Not a trick question. Does that bother anybody? It should bother you. You always are going to get some disparity, but 15 to 85? And I think you're all right because you had a
rationale. I got 15 over here, hello? 15? Well, how did I get 15? What was your rationale? How many points did you assign to A? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Highest
obviously is C. Why? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Application, okay. All right, got it. AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Second highest was -- A
didn't weight much at all, that's why. Now who got me 85? How did you weight it? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Let's just put everybody in the same pot. AUDIENCE
MEMBER: [inaudible]. DEBRA PICKERING: Okay, okay. But that -- now you have to use a different scale for different levels of ability, which assumes we know. AUDIENCE MEMBER:
[inaudible]. DEBRA PICKERING: Okay. So A, you said, look, if everybody has that, we're good because it's important stuff. Who gave me 80? Someone else have 80? What's a different rationale
that you had? 75-80? In the back there. AUDIENCE MEMBER: I had -- I had 75. DEBRA PICKERING: Okay, 75. How did you get that? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING:
General knowledge, okay. So that's 70 points you gave? Yeah. So that would be like a C. Yeah, yeah, gotcha. AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: So B is at 20 points.
AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: That's okay, but I understand your rationale. You were saying if they got the A stuff, which you want everyone to know, that's a C, and so
that's going to be around 70. You know, and then I gave a certain amount and that will get me up to like the B pluses, at least, if they know the B. And then the A's, you know, you save for
that last 10%. That's because the 100-point scale is broken. The 100-point scale has never made any sense. There's so many holes in the 100-point scale. And the main argument people make,
but it's more objective. Oh stop. It's not. It depends on how many items you have in each section and how many points you started. See, I took a couple of variables out of that little
experiment. I took out -- I told you which ones were harder and which ones were easy, you know, the -- I told you what items were. I also told you whether I got it right or wrong. People
disagree about that. Now you even have more disparity and I'm going, whoa! You know, I want to be in your class instead of yours, okay? Because the same knowledge is going to get me a lot
better off. You could look at it. Sorry? Oh, it's great. But no, you guys, some of you were brutal. Yeah, it's okay. I don't care if everyone had decided on 65, I still say there's a
problem. Because if I wanted to put all the kids together who got 65, you see the problem? So you guys all got 65 on the test, all for different reasons, by the way, so putting you together
doesn't make any sense at all. Because 65 doesn't have any meaning except the way I assigned the points. And if I had skewed the points in a different way, I'd have a different group here at
the table. You see the problem? It's all over the place, it's silly. You can't rely on the 100-point scale. Now some people say, well you mean rubrics. I am not talking about rubrics the way
we did them in the 90's of those of you who were around and remember. Because we had some problems with rubrics back then. So let me show you a rubric that kind of represents what we
were doing when people use the word rubric. This is a rubric for a clean refrigerator. Get a picture in your head of your refrigerator at home. Now get a picture in your head of the
refrigerator at your school in the teachers' lounge. Okay? Now rate yourself. I'm going to give you the rubric. Here's a four. For a clean refrigerator, four is this. Entire refrigerator is
sparkling and smells clean, items are fresh in proper containers, original or Tupperware, with lids, and organized into categories. How many of you have a four- level refrigerator? I actually
met a gentleman who said, I have a four-plus. And I said, tell me how you did that. And he said, not only are they organized into categories, they're alphabetized within the category. Oh,
there are meds for that. Yeah, you got to get out more, okay? All right, yeah, yeah. All right. A three, some of you are waiting for the three. That means the refrigerator is generally
wiped clean, items are relatively fresh in some type of container, some Tupperware lids are missing or don't fit, and are sitting upright. Okay, some of you a three? How about the
twos? Some of the shelves are wiped clean, although there are some crusty spots. There are some suspicious smells. Items are in containers, but there seems to be some hairy green stuff
growing. And the one, which is the average faculty lounge refrigerator right before winter break, when the email goes out saying, if you want anything not thrown away, get it now, because
we're cleaning it out, okay? A one is items stick to the shelves when they are picked up, the smells linger long after the refrigerator doors close, several items need to be thrown out,
Tupperware and all. Nothing would make me open that lid. Now you can see we had fun with rubrics back then. We wrote wonderful rubrics, some I can't share with you, but for about everything
in your life. And we had some fun. And a lot of them were very holistic and they actually -- some of them were very basic, saying four means really, really good. You know, a three means kind
of good. Two is okay. One -- you know, so they weren't very helpful. There's some good rubrics out there, by the way, especially writing rubrics. They're pretty good. Look, when I use the
zero to four, I want to make sure that you understand I'm talking about something a little different than what I just showed you. It's called a curriculum scale. What I want you to see it
was zero to four, I want you to focus on the middle here. Notice, read the two and the three. This is the shortened form, you have sort of a longer version of it in your handout. That's the
heart of the scale. Instead of quality points or some description of every single thing, it's the knowledge that they need to learn. And here's the four. Here's the one and the zero.
Now a curriculum scale is a little different than a rubric because it actually states what it is they need to learn, and you either have learned it or you have not. And there are criteria for
it, and you have met those criteria or not. So you really do see the heart of it here. The complex targeted knowledge is in the three. The simpler foundational knowledge is in the two. By
the way, the two doesn't mean the less important stuff or lower level stuff. It's just as important, it's just simpler and foundational. If it's not important, don't assess it. Sometimes we
do assessments, we go -- I did it, by the way. Start to write an assessment, you know, well this will separate the men from the boys. You know, start to add, you know. Then you'd come up
with a certain number of points. Uh-uh. The scale is done with the knowledge populating the scale, and then you can have gradations in between that shows you can have a 0.5. Let me
give you an example. Oh, by the way, if I use that scale here, can you see in my assessment that I got all the simpler foundational stuff? That I got part of the complex targeted stuff and
none of the stuff that went beyond targeting the teaching? So what would my score be for this assessment using that scale? 2.5. Now not only would we all agree, which you can get agreement
in other ways, but that tells me, how would I increase my grade? If you tell me 2.5, how do I increase my score on this? By learning what I didn't show that I had learned in the three,
correct? If you tell me I'm 65%, what's going on in my head? Well, I hope I get more points on the next test so that that will then average out to be -- has nothing to do with what I
haven't learned, it has to do with how I'm going to get points later. And by the way, if I wash your car, can I get 10 points? You know, that kind of goal. If you use the scale, here is just
a fairly simple version, but a version of the scale. The green there is the three, the purple is the two. Once I as the teacher have the scale, then I design assessment items for three level
and I design assessment items for the two level knowledge. Here's another one. One more. We've been working with rubrics for probably 25 years. This approach, the scale, is the first
time we have been able to break down many of the barriers from other ways of scoring students. Now it has some implications. It means you say the two does not tell you what you don't have.
The two tells you what you have learned, and it's important stuff. And if the scale's written well, you say if every kid walks out with the knowledge that's in your two in all your scales,
that's a good education. That's very different than liking to tell kids how deep a hole they're in with the 100-point scale. One of the reasons I love the scale here is that students can
pursue things, you know, in different time frames and still say, I can now show you I've learned what's in the two. If someone shows me even late than they're a two, they're a two for the
topic. They're a two! It isn't they're 20%, it doesn't matter if I have 10 points for the green level and 20 points for the purple level or the other way around, which definitely affects
their grade. No. If they do the section and show me they have the twos, they're a two. Once they show me the knowledge in the three, they're a three. You give them a score for topics and
then those scores eventually average out to be grades. Why do we have threes and twos instead of just advanced or proficient or basic? Because at a certain point, you've got to calculate a
grade. Because that's real, that's reality. So make sure that we can do that. So you can -- but you give feedback this way. You say -- I know sometimes people talk about feedback, which
as long as you're one-on-one with students, you can give all that kind of feedback. This is feedback you say, this is what you have learned. This is what you still need to learn. And then it
focuses students. I'm telling you, I have seen more changes in the classroom from the switch to this type of curriculum and assessment coming together than any before in the 40 years I've
been in education. There are some -- there's a journey here, but it just changes it so the kids know what to learn. It re-focuses students on what they haven't learned and what they have
learned. Another reason I like it, when my eldest son had some struggles because he had some learning disabilities, I will tell you that in many cases he could have easily demonstrated the
twos. When I look at the scales, easily could have done that because his processing and output were some of the problems that he had in the classroom. He could demonstrate in lots of
different other ways everything in the two. So sometimes on a test, that might translate to 30% with a big, fat F on it. Or, well we're not going to put the F. Well, he still knows. As
opposed to you got the twos, you're good. I will also tell you, if it had been more focused on the knowledge, he would have done -- been able to do many of the threes. As a matter of
fact, many of the kids who have learning disabilities can demonstrate if they're clear about the knowledge and it's not about getting all the assignments in. One of the things that makes me
very sad is to go to resource rooms and have really highly skilled teachers working with students with certain disabilities help the students do an assignment that was not appropriate for
them in the first place, as opposed to helping them learn, be able to demonstrate what they needed to learn. So because the learning goals are clear, you say, let's start with the twos.
Can you do that? You go, yeah, it's only going to be 50%. 50% is an F. Who said 50% was an F? We made that up. We made it up a long time ago. It makes no sense, no sense. 50% isn't an F in
most places in the world. If a baseball player's batting .500, they're making a lot of money. Sometimes I look at a test and go, oh my gosh, this kid got 50% of this stuff right? I'm
impressed. I am. I had to say that a number of times to my kids, you know more about the Industrial Revolution than anybody knows, than I know, except some of the kids who've got A's. And
those were those things like could you actually tell whether the Spinning Jenny came before the steamboat. I mean, who cares? It's really okay. Well, there's still the big, fat, red F on
there because he only got 50%. 50% of incredible stuff going on. See, 50% has no meaning. 50% of this is different than 50% of this. So the percents, we've got to get rid of them. You got to
drink the Kool-Aid on this one. You got to get rid of the 100-point scale. Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Because it's not -- it's going to be hard without getting
into the assessments, but when you're ready to do level four, you usually taking the knowledge in level three and you're putting it into a new context, in a novel situation, a new situation
where, if they learned it well, the three, they should be able to do. But it's going to take a little bit more combining some things to do that. So it's not -- it's taking what was targeted
and putting it in a new context, and that's a little different in math than it is in social studies and science, which is a little hard to answer that. But it's not just -- by the way,
people, if they change the language, sometimes people say, you have to get a level four assessment to get an A. No, that's not how this translates. As a matter of fact, most teachers who are
doing this say that if kids nail the level three for all of their topics, that's an A because it's complex stuff. Level four is if you're going after and putting in new combinations. And
again, we need a longer period of time because it's a little different in math than it is in writing, than it is in reading, than it is in science in doing it, but it's not -- it's taking
what was targeted and putting it in a new context. By the way, it's really critical for all kids to have access to everything they need to demonstrate level four. Let me say that again.
It's important for every kid to have access to what they need in order to demonstrate four. It's not like I go home and we have a video camera at home and I can do this cool thing on the
editing equipment and get the four. It's using the knowledge in a new way, and everyone can do that. I know that's -- I'm not trying to sidestep, it's just a complex answer. So the scale
is one of the biggest kind of breakthroughs that we've had, but it really requires some switching in the way you think about how you're designing tests. It also is different in that you
can't say all this stuff that used to be the, you know, multiple choice on my test, that all goes into two. No. You don't separate your test into here is the multiple choice and here is the
construction response. You say, what's the level two knowledge and what's the multiple choice or construction response for that? What's the level three knowledge, which is more complex, and
what's the multiple choice or the construction response for that? And then demonstrate it. And if a student needs to then go back, they don't have to try to come up with points. They say,
learn what you haven't demonstrated yet, and we have resources and stations throughout the room for you to do it. Now, does that seem like too big of a departure? Yes, no, does that make
sense? Just share at your table your reaction. What are you thinking right now? Let me take questions, comments, challenges, confusing. This afternoon, these are
three kind of major implications for some of the recommendations I made earlier today when you're moving toward the grading, but you can kind of hopefully see the interaction of kind of the
first steps to the next steps. Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: How do you create the items for level 2 and level 3? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING:
One of the things -- the question is, how do you create the items for level two or level three and what about the -- what is that? Oh, Universal Design for -- UDL. The level - - the learning.
Can one person do it with multiple choice and another person do it with construction response? Generally with construction response, generally no. Generally, but there are exceptions to
that. Here's what I mean. When you look at level two, what's commonly there is the background knowledge, the academic vocabulary, and certain isolated terms. Those lend themselves very
nicely to selected response, short answer kinds of items. Generally, when you get to level three, it's more you got to demonstrate something either constructed response or over time. There
are exceptions, but multiple choice doesn't usually work as well for the level three just because the level of understanding and level of skill that you have to demonstrate. Then when
you get to level four, that's in kind of a new context and it usually takes -- in fact, sometimes it won't appear on a test, it'll be a task the students are doing over time that they're
doing. There are exceptions because you don't want to say multiple choice is easier than constructed response. Does anyone ever remember a test that you took where the multiple choice was a
lot harder than the constructed response? Yeah, yeah. So there aren't hard and fast rules. Here's what we have to do, and this has been a wonderful process. People who are teaching the
same thing and are working on a scale start to share assessment items that really work for the different levels. We're not big fans of common assessments because of the pacing that it
requires. I mean, it doesn't mean it's bad, but many common assessments, it's like, okay, guys, your common assessment for the first, you know, so many units is going to be October 30th.
Well, that flies in the face of a teacher making a decision about whether to move more quickly or move more slowly in the classroom. And then here's what happens. You get the common
assessment results back and I, as teacher, am already preparing students for the next common assessment. And now I'm getting the results from this one and I've got to go back and pick up
where, you know, I'm going -- how they're going to get this. As opposed to be able to regroup, group and regroup in your classroom along the way. What we do recommend is having common
assessment item banks so that when I determine it's time to assess you on this topic or reassess you on this topic, I've got a bank of items that have been vetted and have been, say, these
are strong items. When you see that happen -- like I said, level two you see a lot of multiple choice. Here's the thing, though. It opens up assessment into this much bigger topic as opposed
to everyone has to take this same test. If a kid has test anxiety or some kids, if they see a page of questions, panic -- I mean, there was a little experiment they did in Oklahoma where the
kids were taking exactly the same tests as everybody else, but these kids that struggled. All they did was get a piece of cardboard, cut out a little window the kid could put over the test
so they could only see one problem at a time, one passage at a time, just visually blocking out the rest. Scores went up. Exactly the same items and scores went up. Why not do that? Or
if a kid is having anxiety, I have one in my purse, you take out a little flip video, say go in there, the room, do these two problems, and videotape yourself doing it and talk it through as
you're doing it. I give that example because a teacher did that. A kid who was not doing well sitting in a classroom full of kids taking the test, you know how it is when someone next to you
finishes early? Really loud? Really? And your anxiety is going, oh, I'm only halfway done, I'm going to panic. So they say, get the kid out of here, especially those who are still working on
that anxiety. Put them in the room with the video and do it. And the thing is, the kid brought the video and came back and the teacher watched it and said, oh my gosh! He knows this stuff in
that setting. And you know, if someone says that's not fair, tell them to stop it. That is assessment, that is good assessment. And the teacher told me, she said, parent-teacher
conference and she took that video and showed it to his parents. They cried. This is a -- they said we know how much he really understands. He just, in the environment that he's in, can't
demonstrate it. Said, look, we've got 1,000 environments here. Our point is we want to see if he can do these. And by the way, just do this section. You're level two, congratulations.
You know who complained about that the most? High-achieving kids. I got them all right. Yes! You're level two. What? You're two. Well, if I give you a test and I just assess level 2, I say,
okay, we're going to get to three. You know, I can do that. They say, what do you mean I'm a two? I got 100%! Yes, you did, you got them all right. You're a two, yes! Now we're going after
three. So it's changing the mindset of the kids. It was really interesting, which is very different than saying, you're not a two yet, and with some help you can show me the two. And that's
what we're going to do, we're going after the two. That's very different than you got 15% of the points. It's a whole different mindset. So that bank of items so that you have good, strong
items is the key to that. And what you see is the construction response more at the three and four. Yes? AUDIENCE MEMBER: [inaudible] and I'm thinking about the child who's writing to an
essay. DEBRA PICKERING: Writing an essay, okay. AUDIENCE MEMBER: Okay. And I understand that there would be different learning targets for the quality of the essay separate from the
content of the social studies questions. DEBRA PICKERING: So what you're assessing is writing. AUDIENCE MEMBER: Well, you'd be assessing both things, I guess. DEBRA PICKERING: Oh, both
things. Well see, here's the problem. As soon as you look at the writing and say, the writing is weak so there's no way this type of assessment is an assessment that this kid can demonstrate
his level of social studies, you've got to switch to different assessment. And that's got to be a regular thing that you do. Whenever your assessment gets in the way of students
demonstrating what they know, you change the assessment. And the thing is, with technology it's just not a huge deal. Honey, go in here, take this little microphone, this little videotape.
It cost $39, by the way. And tell me everything you know about the Industrial Revolution and conflict and you know. And I'm sorry, I'll get on my soapbox about this because this is what
happened when my son would come home from school. He knew more about science and social studies than the average bear, and he did terrible in the regular classroom. He said, they make me
feel stupid. Whereas he knew stuff. If you just asked him, tell me everything you know about Finland and the Norse people, he's like, ah. Then try to get away because he's still talking. I
go, okay, got it, that's fine. No, no, no mom, I know -- no, that's fine, that's fine. Because he could just talk about it all day. It's called assessment. And if writing is a problem and
you know, it would be immoral to say, you have to show me in writing what you know and then to tell you that I did a good job of assessment. Because I didn't. Something got in the way of you
demonstrating how much you know. And I can give 1,000 examples of that. Sometimes teachers just need to know they have that ability to do. They have absolute autonomy to say, I can
assess different kids in different ways. They absolutely do. And there are tons of ways to do it without burying yourself. So that's the whole process of learning that assessment comes in
many different forms. Now again, if you're assessing writing, the kid might say, I can tell you right now it's going to be really bad. You know what's the highest correlation that John Hattie
found? Higher by -- and I think -- I actually think Grant had this in one of his slides I saw last night, but this is really important. The highest correlation he found was the correlation
between what kids said they had learned and what they demonstrated that they learned. He, tongue-in-cheek, would argue give the kids a test. The kid'll go, yeah, I know all of those. Some of
those, not all, yeah. No, I don't know that. And then hand it back. And that's their score because they know. And if you made them take the test, they would know all this, some of those, and
none of that. I mean, they just know. So if you give the test and they go, there's just no way that I can do this, you listen to them because it's an interaction that it's there. Because,
see, our whole point is trying to determine whether or not kids have learned what we want them to learn and what we can do next. Because that is the critical attribute of formative
assessment, is the behavior that happens, the behavior -- are the behaviors that happen after the results are in. That's the key. Yes? AUDIENCE MEMBER: [inaudible] design a sentence so the
kid doesn't have an emotional response, they have a cognitive response in our assessments, and that they go from a fixed mindset -- because you get 50% and you fail, just you're devastated.
[inaudible]. DEBRA PICKERING: Yeah. What she's saying is we've got to design assessments so we don't get this emotional thing caught up in it. How many times does a kid have to see an F on a
paper before they say, I don't care? How many times do they have to feel silly in front of their peers in order to say, I'm cooler with my peers if I say I don't care? And then the teacher
says, well you're lazy, you don't care, da da da da da. [inaudible] our assessment. I think my experience is people are having more honest conversations about what assessment means now
than ever before in my career. We're starting to say, you know what? They're better -- not everyone. Some of them, that's the test, that's it, I'm sorry, done. And I go, no, that's not what
this profession is. Read your mission statement. It says we're trying to get kids to learn as much as we possibly can before they leave here. And we do that by assessing and then deciding
what to do next. So the alternative assessment has just been a term out there where it's very, very real. And the whole notion that we have in our files the right way to assess every kid is
an arrogance we've got to let go of. Yes? AUDIENCE MEMBER: Do you have a recourse that would give some examples of the type of questions that would be [inaudible]. DEBRA PICKERING:
There's the book with the title of my presentation that has some. I think more are coming, more are coming. Classroom assessment has been a little shaky for many years and I think we're just
finding that out when people say, boy, I'm not sure how I assess level two and level three. Well, that says a lot that we need to learn about. AUDIENCE MEMBER: I'm more worried about level
four. And the reason I say that is because when you're implementing standards-based or standards-reference or whatever -- DEBRA PICKERING: Whichever way -- referenced or based. Thank you
very much, you get an A? AUDIENCE MEMBER: [inaudible]. In doing that, you have to politically make sure that you're ensuring your high fliers continue to be high fliers. DEBRA PICKERING:
Yes, yeah. By the way, some of the high fliers struggle. Yeah, yeah. AUDIENCE MEMBER: The hardest question for us is, what kinds of things really get to a four? DEBRA PICKERING: To a four,
yeah. You know, unfortunately, that's one of the longest sessions we do with people when we do stuff over time is what is a level four? Here's why. There are two common problems. One, let me
give you the number one common problem. Teachers start to write the fours that if you were getting a PhD in that topic you wouldn't want to do. Cure cancer and you get a four, you know? It's
like, whoa, whoa, really got -- so kids were saying, I like this approach, but it seems impossible to get an A. So if you write the fours so far out there and then you attach an A to the
four, which is an error, then you say we have just denied the A's to all of the students. What we want to do is make sure that they're challenging. But in answer to your question,
someone's question before, everyone has access to demonstrating the A. What we're going to eventually have is what one school who's struggling to get this to happen, but they're calling it
embedded honors. In other words, anyone can go after the level fours in the class. We don't have to say, well you're honors kids and you're regular kids. And as opposed to no, we teach and
we say if you want honor's credit, you need to pursue the fours. And you can either do the four assessments that I design or here's the coolest part. It's when a teacher says, you come to me
and tell me how you're going to demonstrate to me that for this topic, you're a level four. It's called student-generated assessments. Student-generated. Oh, that's one of your multiple
choice -- I just saw your multiple choice test for this topic. Sorry, I didn't make all those up. So, but student-generated is a good assessment to do, but you have to have standards. They
can't say, I'm going to do a tri-fold poster, can I get an A? And it's going to have glitter and paint and, you know, video, and you're going to have an A. Okay? Good questions and very good
points. What this is shining the light on is our need for assessments that assess what they say they do. And all the multiple ways that students can do it, and all the formats they can do it
in. Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Are we looking for the states to change? Yeah, I know you are. AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: All I can tell
you is we can't. We can only change what we can control, and we can't control that. There's an attempt by Park and Smarter Balanced, which is the two people that are doing the assessments
for the country on common core to improve the assessments. I think because of just broadly implemented tests and all the constraints, that they're going to struggle to get them much better.
So I think what you say to students is, you know what? The state's going to do some things that they have to do because they're assessing thousands and thousands of kids. I know that and
I'll prepare you for that not by boring test prep, but making sure you learn a lot, and you will do fine. I can't control that, you just need to know it. However, what I'm not going to
do is emulate the negative parts of what the states do by being boring. So I'm going to be a creative teacher, I'm going to give you really dynamic assessments, and I wish I could give
alternative assessments when you get to state tests, but I can't control that now. I don't know how to change that. It's like people say, are colleges teaching kids to do better assessments?
No, I can't change that, either. We can only change what we can control, and that's what this is. Good questions, guys. Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Yeah. In
general, this is not for every case, so don't beat me up. In general, pacing guides fly in the face of formative classrooms, in general. They're the enemy because they say teach this much by
here, and then teach this much by here, and this much by here. And then I say to leaders that how are the students, how are the teachers supposed to then react when they get the results back
from that first section and then they're already moving to the next and the pacing guide says keep going. And they've got to go back and they're going to have trouble getting this in.
I said, how do they do that? And the most common answer I get is, well, differentiate! I said, thank you, I never thought of that. And I go, could you describe differentiation? And they
just say it louder and slower. You know, individualize. Until we actually create formative systems like I described earlier today, that isn't going to change. Until we say, you and I teach
the same thing in our lab days, our lab stations, our lab centers' days, you take this group that's working on this, you take this group working on this, and we're going to keep moving along
and we're going to have that as part of the system. Until we do that, we won't have true differentiation to make a difference. So pacing guides, in general, they say learn formative
assessment plus do pacing guides. I go, you've got a conflict. So let's go after it and try to either clean up the language of what you're saying or look at it from a different lens. But so
you're right on, there's a conflict there. Okay? Yes? AUDIENCE MEMBER: [inaudible]. DEBRA PICKERING: Oh, final exams. There's several answers. I'll try to give you the short ones
and it'll sound like I'm sidestepping it because it's a long answer. It kind of depends on what the purpose of your final exam was in the first place. One answer is this. If you have a final
exam and it covers five topics, it'll have five scores that you then put in your grade book under the topic that it assessed. You don't have a single grade. It's interesting to me that it is
very, very common right now for teachers in high school, secondary, where the final exams usually happen, to actually not do a comprehensive final exam anymore because they have been -- you
know, it doesn't work very well. Because we're saying they learned it in September, but now we have to find out if they remember it in January. And say, they're going to forget! You know, so
now things pile up and they don't like them. So it's very common for teachers to say, your final is the unit test for the last unit we did or it's a project you turn in. That's going
to be your final. That's also very common at college. Or this last thing that we do that's going to count as your final because people -- the emperor's naked on final exams, has been for a
long time. Some are starting to say, I'm going to abandon them, I'm not going to do them anymore. Then there's some other alternatives to that, but it calls into question anyway, but that's
been called into question a long time. Okay? We're a little bit early, but I will stick around. I don't want to start a whole new topic here. I will stick around if people have individual
questions and -- because this is what happens at this point. Here's where we are, what do you think we should do next? So I will do that. But as far as an official, large-group presentation,
I know we're a little bit early, but I'll stick around, okay? Thank you very much. Hopefully you have some things to think about.
