Unusual pattern found in Janesville school test results; officials satisfied no wrongdoing took place
JANESVILLE—It might be a great irony: A Janesville school's staff worked so hard to improve students' scores that the district was flagged for possible cheating.
That's what district officials say about an analysis of state tests that Franklin Middle School students took last fall: No one did anything wrong, but efforts to get students to take the tests seriously resulted in an abnormally high number of erasures leading to correct answers.
“Inordinate numbers of wrong-to-right answer changes … may indicate inappropriate intervention on students' answer documents by an educator,” the state Department of Public Instruction said in a May 15 letter to Franklin's principal.
The chance of so many wrong-to-right erasures is one in 10,000, according to McGraw-Hill, the company that provides the Wisconsin Knowledge and Concepts Exams, known as the WKCE. McGraw-Hill also does the erasure analyses.
The Gazette learned of the analyses by requesting documents from the state and school district.
Similar erasure analyses in Washington, D.C., and Altanta public schools in recent years led to educators losing their jobs and continuing controversies, including criminal charges in Atlanta's case. In both those cases, the cheating produced surprisingly higher test scores, something that officials said didn't happen at Franklin.
Franklin was among 29 schools out of more than 2,000 statewide which were “flagged” for having an “abnormally” high number of erasures, according to notices sent to those schools this spring.
The state suggested that Franklin Principal Charlie Urness “explore test-administration practices ...to determine the cause of your school's high erasure rate” and submit a summary of his findings.
Urness replied in a one-page letter June 24, saying all staff members who proctored the tests had viewed the state's PowerPoint presentation on test security and that test booklets were collected at the end of each testing day and locked in the school's main office.
District testing coordinator Amy Sheridan said one person at Franklin was responsible for delivering test booklets to classrooms each day and collecting them when the tests were done. More than 500 test booklets were locked in a conference room inside the office each day and then redistributed on subsequent testing days.
Teachers had no access to the tests between testing times, Sheridan said.
When completed, the tests were picked up and prepared for shipment at the district's central office.
At each step, the tests were inventoried so that they were all accounted for, Sheridan said.
“The only thing we can attribute to the high number of erasures is the fact that we instruct students to take the test very seriously and to review their answers again if they have time remaining …” Urness wrote.
Urness concluded that no one did anything wrong.
Franklin was flagged for high erasure levels in seventh-grade reading and math tests. Reading and math are the only tests students take in that grade.
The state accepted Urness' explanation and did not investigate further.
The state has never conducted an investigation into any district based on the erasure analysis, said Patrick Gasper, spokesman for the state Department of Public Instruction.
"What we have done is contacted several districts to follow up on the erasure analysis, requesting that they provide notice to us that they have reviewed the matter and that they have refreshed testing security training with district personnel," Gasper wrote.
But experts contacted by The Gazette said the erasure data call for more investigation.
“The fact that it was all at one school, to me that sounds suspicious, I would at least want to look into it further,” said Timothy Shanahan, a professor at the University of Illinois at Chicago and a former director of reading in the Chicago Public Schools.
Kim Ehrhardt, the district's director of curriculum, instruction and assessment, said the state wanted to know if the school followed testing protocols, the answer is “yes,” and the state is satisfied with that.
Teachers know, for example that they should not give students hints of any kind during test taking, Ehrhardt said.
That's one of the possibilities if cheating is involved. Another is that students cheated.
Ehrhardt said students have little reason to cheat, as the tests do not affect their grades.
A third possibility is school officials making post-test changes on test booklets.
Janesville teachers' pay does not depend on test scores, but principals and other administrators are evaluated, in part, on test scores.
Shanahan said if the problem is those who oversee the test taking, the solution is to hire independent proctors.
“People get tempted. They think they will fix something by doing this,” Shanahan said. “Obviously, it undermines all educators with the public.”
But local and state school officials appear satisfied in this case.
The state director of the Office of Student Assessment “has accepted our findings regarding the DPI inquiry regarding erasure marks at Franklin Middle School. There is no expectation for additional information. They are satisfied with Dr. Urness' response,” Ehrhardt said in an email.
In an interview, Ehrhardt and assessment coordinator Amy Sheridan said students are taught to use their time wisely, check their work and correct mistakes. Officials went so far as to emphasize proper erasing, because in previous years, students did not erase thoroughly enough. The result was the machines scoring the tests marked answers wrong because when two answers are marked, the question is scored as incorrect.
“We've even bought good erasers,” Ehrhardt said.
Sheridan said some students tend to spend all their energy on one answer, leaving little time to complete the rest of the test.
“So one of the strategies is, do your best guess, move on, make sure that you have ample time for all the questions and when you get to the end, then you go back to those questions that you got stuck on,” Sheridan said.
“So, it really wouldn't be unusual for a student to go back, if they were using that strategy,” she said.
The state told the district that only 12 of the 191 Franklin seventh-graders tested were responsible for all those erasures that led to the state's warning letter, Ehrhardt said.
“I think what you have is a kid who has a perfectionist personality; they have really taken heed to this, and they probably have done more erasing than they had done before because they are looking to do well and are probably following the directives and our emphasis on making sure you do your best on all assessments, not just the WKCE,” Ehrhardt said.
Another indicator that backs up the district's position is that Franklin's test results did not spike up, which is another thing the state checks, Ehrhardt said.
Brad Thiessen of St. Ambrose University, who investigated test cheating for his doctoral thesis, said schools that teach students to quickly answer each question and then check their answers with the remaining time are much more likely to have a high number of erasures.
Or, large numbers of erasures could be pure coincidence.
“People win the lottery all the time. People do get struck by lightning,” Thiessen said.
“But in all honesty, the odds point to that it wasn't something at all innocent,” Thiessen said. “Chances are probably that it was something unusual. We just don't know what that was.”
Thiessen said further analysis of the patterns of the erasures, might help. For example, it's suspicious if students got the right answers on difficult questions while missing easier questions.
But the state does not do those types of analyses, according to a nationwide survey by the Atlanta Journal Constitution last year.
“If they're not doing those analyses, then you can't say anything more than unusual things happened at this district,” Thiessen said.
Shanahan agreed more investigation is called for.
“If all you know is that you have these aberrant patterns of test corrections, then that suggests the possibility of cheating, but it is not considered to be adequate evidence,” Shanahan said.
Another thing to check is whether the erasures were for the same questions on different students' tests, experts said. Ehrhardt said the information provided by the state did not include that level of information.
Students in grades 3-8 and 10 will fill in bubbles on state tests again this fall. But the following round of testing, in spring 2015, is planned to be online—no more erasing.
“I think it'll be kind of a non-issue when we go to the new tests,” Ehrhardt said.
The state could, however, incorporate a keystroke analysis in its online tests, which would provide data similar to erasure analyses.
Gasper said the state still is developing test-security measures for the new tests in grades 3-8.
ACT will provide tests for grades 9-11 starting in 2014-15. ACT “already has strict procedures for their tests, and we will follow their guidelines,” Gasper said.
An ACT spokesman declined to reveal any of its test-security procedures.
The state has been sending written notices of erasure analyses for the past three years. Previously, the state made phone calls, Gasper said. This is the second year in a row that Franklin received such a notice. Last year's notice also cited the seventh grade for high erasures.
Janesville's Marshall Middle School received a similar notice noting high erasure counts on its 2010 state tests.