When incentives are incentives to cheat.

Does anyone else see the irony in the fact that the NYC principal who was removed from her job for changing students’ grades and giving course credit to failing students was retained by the NYC public schools as an “Achievement Coach?” (See New York Times article.) That seems like a stretch. Apparently she will still be paid a principal’s salary “but will no longer be eligible for a $25,000 performance bonus like the one she earned in 2009.” Well, that’s a relief.

I’m not even going to bother being outraged (or even surprised) that this principal wasn’t fired, that the NY schools chancellor seemed to think this wasn’t a very big deal. Let the NYC parents get upset about that. I’m more concerned about the lesson to be learned by those who push financial incentives for higher test scores. This type of thing – tweaking scores or massaging grades or “misapplying grading policies” or, let’s be real, cheating – is going to become tempting as the stakes get higher and higher. Perhaps you’ve heard about the cheating scandal recently coming to light in some Pennsylvania public schools? Or what about Atlanta? Desperate people, in desperate times, will do desperate things to keep their jobs.

Of course, another irony is the fact that incentives like bonus pay have been shown to have ZERO effect on student performance. (Click here and here.) But lawmakers don’t want to hear that. The public doesn’t want to hear it. It’s easier for them if the answer is simply to reward educators for higher test scores.

Seems pretty clear to me that this is not going to end well…

Update: There's a new story today on the NY Times website, reporting that allegations of cheating and grade tampering in the NYC public schools have tripled since 2002. Yep, expect to see more and more stories like this...

Some thoughts on merit pay for teachers, tied to test scores.

Teacher quality is vital, but I’m concerned about increases in high-stakes standardized testing (and imperfect tests) to evaluate teachers, influence compensation, etc. Testing and data mining is going to cost a lot of money. Who will pay? What will be the effect on students?

How will the state develop standardized tests for Art? History? Theater? Health? You know there are no state tests for those right now. How long is that going to take? How much more learning will students lose in order to participate in testing? These are just a few of my concerns.

Then I read Dana Goldstein's story in the Daily Beast about high-stakes testing and something called "Campbell’s Law" — the social-science maxim that holds that the reliability of a decision-making tool is inversely proportional to the importance of the decision being made. That is, the more a test score is worth, the more it’s worth cheating on the test. (The story was about allegations of high erasure rates -- cheating -- on supposedly-improved standardized tests in Washington, D.C.)

So that gives me even more worry about placing all this emphasis on state-developed tests.

Did I mention how pretty much every 9th grader in New Jersey failed the state's first Biology test? So now they're back at the drawing board trying to come up with a better test.

And, finally, Diane Ravitch tweeted today, "Imagine putting fate of students and teachers in hands of the testing agencies" like those described in the book, Making the Grades: My Misadventures in the Standardized Testing Industry by Todd Farley, an eye-opening (and frightening) book that I read last year. Another good point, Diane!

What does this mean for Ridgewood? The political pressure in Trenton is forcing a rush to policies for evaluating teacher quality, based on extremely flawed state tests. Eventually, the requirements will hit Ridgewood, we'll be forced to comply (rather than rely on our own education leaders to evaluate our teachers), it will cost a lot of money, it will suck even more time away from our administrators and teachers...like I said, I'm worried.

Are we willing to re-think the "Race?"

A few of my thoughts in advance of the third Ridgewood showing of Race to Nowhere, tomorrow (Monday) night at the Ridgewood Public Library Auditorium (7:00 p.m.).

When I watched the movie the week before last, I noticed what I felt was a moment of irony. A parent in the film said that she has always wanted her kids to do better, to succeed, so that they will have "more choices in life." But when I listened to these pressured kids talk, it was clear to me clear that they actually end up having fewer choices. Or no choices. No real choice about where they go to college, or what they're going to study, or where they will find their passion. They are on a track and there is no getting off that track.

On the topic of homework, the film cited research that looked at the correlation between the amount of homework kids have and how they perform academically. (To be honest, I don't remember what they used to measure that performance...I think it may have been standardized test scores.)

According to the research cited in the film, in the elementary school years, there is no correlation between the amount of homework and achievement. In middle school, there is some correlation, but it falls off after one year. And in high school, there is some correlation, but not after an average of two hours of nightly homework. Any more than two hours of homework has no effect of achievement.

This actually dovetails nicely with the Parent/Guardian Survey results, which I wrote about earlier today. In some of the parent comments for elementary schools, parents wrote that they wanted more homework and more rigor in their children's education. There were parents who want more tests in kindergarten and first grade, to "prove" that kids are learning! Here's a little snapshot of the problem, right here in Ridgewood: how do we balance these parents' wish for "academic success" with these emerging conversations about the so-called "race to nowhere?" Are we really willing to make a change?

And on another related note, the principal of Ridgewood High School announced that this upcoming Winter Break will be "homework free." The high school administration wants to see what it's like -- for students and for teachers -- if kids are allowed to simply rest over the break. Of course, in our house, this has already created some confusion. My son has a large research/writing project that's currently underway. My son is wondering how he can just take a week off without falling behind on the project overall. My natural response is to hope he can get enough work done this week so that he can enjoy the experience of the "no homework break" next week. But something doesn't feel right to be "rushing" now in order to "rest" later. Jumping off the merry-go-round is a little scary. Sort of like...jumping off a merry-go-round.

What are your thoughts on the concept of a "race to nowhere?"

Reinventing standardized tests.

Apparently U.S Education Secretary Arne Duncan is publicly acknowledging the severe limitations of conventional standardized testing. In a recent speech, Duncan said, “Students, parents, and educators know there is more to a sound education than picking the right selection for a multiple choice question." I'm glad to hear him talking about "Assessment 2.0" or "next-generation assessments." (Click here for the New York Times article about the speech.)

Jonathan Martin, Principal of St. Gregory College Prep in Tucson, Arizona, wrote on the blog Connected Principals, "I want us to be held accountable for educational excellence, and I believe that by using external measurements, we are able to demonstrate that accountability. I think too that we can use this data for marketing our school as we seek to grow it. But most of all, I want to know how well our school is doing compared to others so I can receive the hard truth about where we are not doing well enough, and I can know where to focus for improvement."

Martin continues, "I do think we should evaluate ourselves by our school’s own standards, but not only by them; it is too easy for us to be seduced by our own biases. We know our schools, and we love our schools, and sometimes it is hard to see our blind spots or fully appreciate where we may be under-achieving. But that I seek and appreciate external measurements doesn’t mean I love or like scantron multiple choice bubble tests of basic skills that are administered once a year to “grade” a teacher or school. I don’t."

Mr. Martin says what he likes in “next-gen” assessments is, first, "that we can now evaluate higher order thinking skill development in tests that are not multiple choice, but authentic assessments where students write essays reviewing and responding critically to documents and offering thoughtful solutions to complex problems."

Second, Mr. Martin appreciates "new assessments which are computer adaptive, able to shape themselves to individual student proficiency levels, give immediate feedback to students, teachers and parents, and provide the information we need to better personalize instruction."

These “2.0″ approaches are what Secretary Duncan called for in his speech. As for the first, he says:

New assessments will better measure the higher-order thinking skills so vital to success in the global economy of the 21st century… To be on track today for college and careers, students need to show that they can analyze and solve complex problems, communicate clearly, synthesize information, apply knowledge, and generalize learning to other settings.

The PARCC consortium will test students’ ability to read complex text, complete research projects, excel at classroom speaking and listening assignments, and work with digital media. Problems can be situated in real-world environments, where students perform tasks or include multi-stage scenarios and extended essays.

As for the second approach, Mr. Duncan said:

Most of the assessment done in schools today is after the fact and designed to indicate only whether students have learned. Not enough is being done to assess students’ thinking as they learn to boost and enrich learning, and track student growth. [new] assessments will make widespread use of smart technology. They will provide students immediate feedback, computer adaptive testing, and incorporate accommodations for a range of students.

The SMARTER consortium will test students by using computer adaptive technology that will ask students questions pitched to their skill level, based on their previous answers. And a series of interim evaluations during the school year will inform students, parents, and teachers about whether students are on track.

Better assessments, given earlier in the school year, can better measure what matters—growth in student learning. And teachers will be empowered to differentiate instruction in the classroom, propelling the continuous cycle of improvement in student learning that teachers treasure.

I agree with Principal Martin that no single assessment will ever be perfect. We need multiple approaches and multiple ways to assess how our students are learning and how well our teachers are performing. This fact, which seems so obvious, is what makes me worry when I hear about school rankings based on state tests, or teacher hiring or pay decisions based on value-added assessments (which use test scores to judge "value added").

I agree with Mr. Martin:

"We must not ever substitute data and “evidence” for an educator’s judgement. Data inform judgement, but data must never replace discretion and wise judgment."

On the other hand, "let’s not refuse to improve data collection because of the inappropriate abuse of poor data; let’s seek to improve it and use it appropriately."

As our students in Ridgewood continue to take the NJASK year after year, it's encouraging to think that the "powers that be" may be (quietly) acknowledging that we need a better way. Two that I am interested to check out are the College & Work Readiness Assessment (CWRA) and the NWEA’s Measurement of Academic Progress (MAP). Both of these offer hope that maybe we can finally move on from outdated techniques and use creative new tools to truly measure and improve learning.

Count on this: no easy solutions in the effort to evaluate teachers.

Not everything that can be counted counts, and not everything that counts can be counted.

This quote is usually attributed to Albert Einstein (although I couldn’t find any proof of his writing or saying it). It’s an interesting and valid thought. And it’s often mentioned in the nationwide conversation about teacher performance and test scores.

Last Wednesday’s New York Times contained the story, “When Does Holding Teachers Accountable Go Too Far?” I found it to be a good review of some of the challenges faced by educators who, with budget-driven urgency, are trying anew to determine the best way to evaluate teachers (and, it follows, to better connect performance with compensation). The story’s main focus was the concept of measuring “value-added data” for teachers. This is a statistical technique where student test scores are analyzed, measuring improvement from when a teacher started teaching the student to when he/she completed the year. According to the theory, the change in test scores is a measure of the “value” added by the teacher: more increase = more value added and, it follows, better teaching.

The Los Angeles Times recently analyzed seven years of elementary school test scores and created a ranking of 6,000 teachers. “The newspaper named a few teachers — both stars and laggards — and announced that it would release the approximate rankings for all teachers, along with their names.”

The Los Angeles Times articles got teachers riled up, the union called for a boycott of the paper, but...the union also said, more calmly, that they’re willing to discuss making such scores at least a part of teachers’ evaluations. That’s encouraging.

The critics of value-added data point to some real limitations. According to the NY Times, “scores can bounce around from year to year for any one teacher...so a single year of scores — which some states may use for evaluation — can be misleading. In addition, students are not randomly assigned to teachers; indeed, principals may deliberately assign slow learners to certain teachers, unfairly lowering their scores. As for the tests themselves, most do not even try to measure the social skills that are crucial to early learning.” And, thinking about Ridgewood, what about teachers who are already very successful or whose students begin at fairly high-achieving levels? If your students already score very high, and there’s not much higher for them to go, is it fair to judge that teacher as not “adding value?” This is exactly what happened to Ridgewood in last year’s state QSAC evaluation – the District received a lower score in student achievement because our test scores didn’t improve enough. But it was physically impossible for our scores to improve that much – they would have had to have been higher than 100% perfect.

Some educators feel that value-added data can probably help identify the best and the worst teachers, but probably won’t help much in truly evaluating teachers in the middle. Other educators have suggested using the value-added data on the school level, without tying the scores to individual teachers' names. I agree with the NY Times’ writer’s feeling that an important first step would be for us all to agree that no system will ever be perfect. No system can do it all. And, personally, I will never feel good about placing too much emphasis on standardized tests, especially when the tests we have are so imperfect.

The New Jersey Department of Education recently announced a special committee that will be working on teacher evaluation. The big deal, according to them, is that the committee contains teachers, administrators, school board members, legislators, "experts," etc. An apparently inclusive group that will calmly and rationally research and develop some solutions. When I heard they wanted school board members, I immediately tried to volunteer. Turns out, their idea of school board representation was that a staffer from the New Jersey School Board Association will participate. Darn.

Clearly, parents and communities want their schools to be accountable. I believe teachers want to be accountable. It’s in our human nature to rank things, to create order, to understand the value (relative or intrinsic) in everything we do. Test scores are such a seemingly easy solution – what could be more cut and dried than numbers? “The scores don’t lie,” sounds so simple. But aren’t there more things that “count?” Can’t we find a way to wrap our human brains around the concept of things that can’t be counted?

Something to think about as we begin the new school year – a school year in which a new contract will be negotiated with our Ridgewood teachers.

Have a great first day of school tomorrow!

Realizing the mistake that was NCLB.

Interview in US News & World Report with Diane Ravitch, former Assistant Secretary of Education under George W. Bush. Ms. Ravitch has gone from a conservative advocate of school choice, vouchers and high-stakes testing to being critical of same. Change of heart? Her book, The Death and Life of the Great American School System: How Testing and Choice are Undermining Education, is reportedly a strong indictment of No Child Left Behind and other Bush-era policies she helped create.

In the interview, Ms. Ravitch answers the question, What should parents do to ensure that their children are getting the best education? She answers:

There are two different questions. One is what should parents do, and the other is what should policymakers do. If policy makers simply say, "It's every family for itself," we're going back to the early 19th century, before we had public education. Some people had private tutors, and some people sent their kids to religious schools, and some people got together and had little schools that they created. Then at a certain point, there was an awareness that the public had a responsibility to educate the children of the community. If we're doing a bad job of that, we really should develop public policy that looks to improving the quality of those schools and not just close them down and hand them over to private entrepreneurs. Because then we're creating a marketplace, and markets have failures. Markets do not succeed in providing equal opportunity. They succeed in creating winners and losers. We saw that in the [economic collapse] of the fall of 2008, and that could happen to our schools as well.

Interesting and sure to incite protests from pro-school marketplace folks. I have to admit, the winners vs losers point is a good one...because when there are losers they will be children, and that's not acceptable.

Teaching teachers

I heard an interesting story on NPR's All Things Considered today (yes, I suppose you can tell a lot about a person by the radio she listens to, haha), about teacher training programs. It was titled, "Study Tries To Track Louisiana Teachers' Success" (click here to listen or read the transcript). Apparently, the study tracked student test scores and linked them to where those students' teachers had been educated. There were some interesting findings -- some teacher training programs appeared to result in higher test scores. But one of the problems was that when the teacher education was linked to lower test scores, there was no way to know WHY. There was no way to understand just what aspect of a college's teacher education program was lacking or might have contributed to the lower student achievement. In addition, while the results gave general info about teacher performance overall, the results were not tied to specific teachers -- there was no way to know which teachers needed some sort of intervention or remedial training.

Hmmmm, these are the same challenges that result from standardized testing of students...generalized scores vs. specific assessment of specific students, etc. Rather than improve our ways of assessing student learning, we're just pushing the same semi-useful tools further up the chain, to teachers.

I found the comments -- many from teachers and teacher educators -- on the NPR story as interesting as the story itself.

Part Two of the story will air tomorrow. I'l check it out and try to report back with a little more thought (sorry it's late and one of my resolutions is to try and get a LITTLE more sleep).

Standardized Tests and Referendum Recovery

So, I've been a little quiet over the past week...after the months of pre-Referendum activities (including 20+ meetings with the community), I guess I was a little burned out. I’ve been focusing on holiday stuff. And my “real” job, too, (for a change, haha).

Anyway, you can expect my posts to be a little less frequent through the end of this month. I wish everyone a happy, healthy, restful and peaceful holiday season and happy new year!

I’ve been meaning to write about the District-Wide State Testing Report, which Assistant Superintendent Regina Botsford presented at the December 7 meeting. But I’ve been dragging my feet a little because I don’t like the excessive and narrow focus on standardized tests, which I sometimes see. And then the other day I heard an interview with a man who wrote an insider’s “tell-all” book on the standardized test business. Definitely got me feeling even more ambivalent on the topic.

I understand why people can get excited about test results. It’s human nature. We need to order things in the world, and we need to “place” ourselves somewhere in that order. (Bear with me on my amateur psycho-evaluation...this is just my opinion.) And of course we want to place ourselves as high as possible in the food chain. Standardized test scores come out and we immediately want to see how our students compare.

The good news is, overall our scores look good. At the elementary level, the percentage of our students who are “advanced proficient” was better than our DFG* average in every category -- Language Arts, Math and Science. At the high school, same thing. We beat our DFG average in all subjects. At the middle school, our students are basically equal to our DFG in most categories – and just below our DFG in some. It’s important to note that the gap between our scores and the DFG average is narrowing over the past few years, so that’s a good trend. When we discussed the results at the Board meeting on December 8, the general feeling was that middle school presents special challenges and even though our scores dip in grades 6-8, they “catch up” again in high school. That is one way to look at it, but I wonder why our scores dip more than the average scores in our DFG. At a future meeting, the District will present goals for every grade/school, including the middle schools, so I’ll have a chance to ask about that. Again, it does look like our middle school scores are improving relative to the average scores in our DFG, and that’s a good thing.

I've written about standardized test issues in the past. I think there's a place for them, but they need to be part of a complete assessment program. You can read a little more about summative vs. formative assessment here.

(*DFG = District Factor Group. It's the group of districts the state puts us in, based primarily on population density and community affluence. We are a "J" district and other districts in this group are Glen Rock, HoHoKus, Northern Highlands, Saddle River, Upper Saddle River, Woodcliff Lake, Millburn, West Windsor-Plainsboro, Rumson, Chatham, Mountain Lakes.)

Now about that book I mentioned. It’s called Making the Grades: My Misadventures in the Standardized Testing Industry, by Todd Farley. The author was really interesting on WNYC the other day. I’ve ordered the book and I’ll let you know what I find out. I can tell I’m not going to like it.

What is Learnia and why are students taking more tests?

As you may know, I’m not a big fan of standardized tests as the be-all, end-all of determining success in education or learning. (Visit FairTest.org for more info on why.) So you may wonder why I support the use of Learnia in our schools, which at first glance looks like more testing.

First, all testing is a form of “assessment.” We assess what students have learned. Of course we do. Learnia is a type of “formative assessment.” What is that?

Two definitions, according to FairTest.org:

“Summative assessment is the attempt to summarize student learning at some point in time, say the end of a course. Most standardized tests are summative. They are not designed to provide the immediate, contextualized feedback useful for helping teacher and student during the learning process. High quality summative information can, of course, shape how teachers organize their courses or what schools offer their students.

By contrast, formative assessment occurs when teachers feed information back to students in ways that enable the student to learn better, or when students can engage in a similar, self-reflective process. If the primary purpose of assessment is to support high-quality learning, then formative assessment ought to be understood as the most important assessment practice.”

In simpler terms, Learnia allows teachers to see how students are doing during teaching/learning. It gives feedback on specific areas within subjects – showing what needs more attention and what is being mastered. And it shows this on an individual student basis, collecting learning data over time. So instead of just getting a score of 000 or “Proficient” on a test, the teacher can see that Johnny is doing better with decimals but not with fractions. (That’s a super-oversimplication -- in reality the results are much more precise and detailed.)

NJASK or those other summative assessments may be great for drawing lines in the sand and evaluating districts or schools from 30,000 feet, in order to assign funding or populate magazine rankings, but formative assessments such as Learnia are much more useful in terms of teaching and learning…which is what we’re here for, right?

Blogged with the Flock Browser

SAT tutors offer scant help, study finds

From today's Record...as the parent of someone whose SAT scores went DOWN after doing a weekly Kaplan test prep course, this does not surprise me!

BY KATHLEEN CARROLL
NorthJersey.com

Recession-minded families wondering if they should ditch that $100-an-hour SAT tutor got a murky answer this week from the National Association of College Admissions Counselors.

Coaching and prep classes for college-admissions exams are helpful, but don’t usually change students’ scores all that much. But any increase could be significant because some colleges use very small score differences to choose between similar candidates for admission, an NACAC report said.

Some test-prep providers promise score gains of up to 100 points on the SAT, but students usually gain about 30 points, according to the report. On the math section of the exam, average gains are between 10 and 20 points; on the reading section, students gain an average of 5-10 points after being coached, the report said.

Short-term preparation classes result in "very, very small gains," said Laurence Bunin, general manager of the SAT and senior vice president for operations at the College Board, which developed and administers the test.

"The test is testing what you learned in school, so unless students are learning more material, they are not going to do a lot better," he said. "There’s no trick. Tricks don’t work."

Recession-minded families wondering if they should ditch that $100-an-hour SAT tutor got a murky answer this week from the National Association of College Admissions Counselors.

Coaching and prep classes for college-admissions exams are helpful, but don’t usually change students’ scores all that much. But any increase could be significant because some colleges use very small score differences to choose between similar candidates for admission, an NACAC report said.

Some test-prep providers promise score gains of up to 100 points on the SAT, but students usually gain about 30 points, according to the report. On the math section of the exam, average gains are between 10 and 20 points; on the reading section, students gain an average of 5-10 points after being coached, the report said.

Short-term preparation classes result in "very, very small gains," said Laurence Bunin, general manager of the SAT and senior vice president for operations at the College Board, which developed and administers the test.

"The test is testing what you learned in school, so unless students are learning more material, they are not going to do a lot better," he said. "There’s no trick. Tricks don’t work."

Better ways to test (and teach) reading/language arts

Thought-provoking op-ed by E.D. Hirsch, Jr. in the online NY Times today. Here are some excerpts:

IN his recent education speech, President Obama asked the states to raise their standards and develop “assessments that don’t simply measure whether students can fill in a bubble on a test.” With the No Child Left Behind law up for reauthorization this year, the onus is now on lawmakers and educators to find a way to maintain accountability while mitigating the current tendency to reduce schooling to a joyless grind of practice exams and empty instruction in “reading strategies.”

Before we throw away bubble tests, though, we should institute a relatively simple change that would lessen the worst effects of the test-prep culture and improve education in the bargain.

These much maligned, fill-in-the-bubble reading tests are technically among the most reliable and valid tests available. The problem is that the reading passages used in these tests are random. They are not aligned with explicit grade-by-grade content standards. Children are asked to read and then answer multiple-choice questions about such topics as taking a hike in the Appalachians even though they’ve never left the sidewalks of New York, nor studied the Appalachians in school.

Later, it continued:

Let’s imagine a different situation. Students now must take annual reading tests from third grade through eighth. If the reading passages on each test were culled from each grade’s specific curricular content in literature, science, history, geography and the arts, the tests would exhibit what researchers call “consequential validity” — meaning that the tests would actually help improve education. Test preparation would focus on the content of the tests, rather than continue the fruitless attempt to teach test taking.

A 1988 study indicated why this improvement in testing should be instituted. Experimenters separated seventh- and eighth-grade students into two groups — strong and weak readers as measured by standard reading tests. The students in each group were subdivided according to their baseball knowledge. Then they were all given a reading test with passages about baseball. Low-level readers with high baseball knowledge significantly outperformed strong readers with little background knowledge.

The experiment confirmed what language researchers have long maintained: the key to comprehension is familiarity with the relevant subject. For a student with a basic ability to decode print, a reading-comprehension test is not chiefly a test of formal techniques but a test of background knowledge.

And later he he wraps up:

Better-defined standards in history, science, literature and the arts combined with knowledge-based reading tests would encourage the schools to conceive the whole course of study as a reading curriculum — exactly what a good knowledge-based curriculum should be. Schools would also begin to use classroom time more productively, which is important for all students and critically so for disadvantaged ones.

Reform of standards and tests needs to begin in the earliest grades. Knowledge and vocabulary are plants of slow organic growth. By eighth grade, after the cumulative benefits of a more coherent curriculum and more productive tests, students would begin to score much better on all reading exams, including those that aren’t based on a school curriculum. More important, they would be prepared to be capable, productive citizens.

We do not need to abandon either the principle of accountability or the fill-in-the-bubble format. Rather we need to move from teaching to the test to tests that are worth teaching to.

As the parent of a student who is really smart yet freezes on standardized tests and performs horribly even though she knows the material cold, I find this extremely interesting. (Click here to read the entire article.)

Grades for Effort or Achievement?

The New York Times recently had an article about college kids complaining about grades. (Student Expectations Seen as Causing Grade Disputes, 2/18/09). One professor said, "Students often confuse the level of effort with the quality of work. There is a mentality in students that ‘if I work hard, I deserve a high grade.’" Another professor said, “I think that it stems from their K-12 experiences. They have become ultra-efficient in test preparation. And this hyper-efficiency has led them to look for a magic formula to get high scores.”

Some of the subsequent letters to the editor to the Times reinforced the discrepency. Students who wrote in felt that effort is what grades should signify. Professors or deans felt grades were for achievement.

Hmmm...interesting. I wonder what students, parents and teachers feel here in Ridgewood?