Abstract. Reports on the action taken by a number of U.S. mathematicians to study a judging method developed by the International Skating Union, as of January 2003. Mathematicians involved in the effort; Incident that led to the change in the judging method; Findings of the study.
WHEN JUDGING GETS SLIPPERY: The 2002 Winter Olympics served up unintended drama with its "scandal on ice" show, starring a crooked French judge and two aggrieved Canadian figure skaters, primped to telegenic perfection. In the aftermath, officials of the International Skating Union, figuring that they had an image problem, developed a system to protect judges from what the organization terms "external influence." But three mathematicians who analyzed the new procedure conclude that the method doesn't add up.
The skating group "hoped that the system would eliminate bias," says Elyn K. Rykken, an assistant professor at Muhlenberg College. "But if you look at the results, it doesn't seem to actually do that. And the two previous methods that were in place were better." Ms. Rykken, with Maureen T. Carroll, of the University of Scranton, and Jody M. Sorensen, of Grand Valley State University, will publish a paper, "The Canadians Should Have Won!?" in the February issue of Math Horizons, a magazine put out by the Mathematical Association of America.
The details of the figure-skating affair make multivariable calculus seem simple by comparison. According to an investigation by the skating group, the French judge, Marie Reine Le Gougne, gave better marks to the Russian pair, Elena Berezhnaya and Anton Sikharulidze, even though she felt that they were outskated by the Canadian pair, Jamie Salé and David Pelletier. The investigation found that Ms. Le Gougne had succumbed to pressure from Didier Gailhaguet, president of the French federation of ice sports. Widely reported allegations spread the scandal further, claiming that one of the French officials had agreed to swap votes with a Russian judge in the ice-dancing competition.
In June 2002, the International Skating Union Congress voted to overhaul the judging methods, building in anonymity and randomness to thwart potential schemers. In the new system, a computer counts the marks of only a subset of the judges present, without revealing whose marks go into the decision. For example, at the championship level, 14 judges watch the women's short program, and the computer uses the marks of only 9 of them. During the women's long program, the computer selects a different set of 9 judges.
Judges have tested the new system in several competitions, and "the results so far obtained are beyond the most optimistic expectations," the skating group announced last month.
The American mathematicians, however, give far different marks. "It appears to be a hasty and ill-planned change," they conclude.
The scholars took an interest in studying the judging method after the scandal and subsequent changes to the rules. They conducted their own test by using the new system to adjudicate the women's figure-skating competition in the 2002 Games. Because that event had only 9 judges and the analysis required 14 sets of marks, the researchers randomly duplicated the marks from 5 of the judges. They then calculated all the possible results of picking 9 out of the 14 judges. If such a system had been in place during the 2002 Olympic Games, Sarah Hughes--the 16-year-old American underdog who won the real competition--would most likely have left Salt Lake City with a silver. Only 491 out of 2,002 possible combinations of judges would have put her on top. The rest would have awarded the gold to Irina Slutskaya--the Russian skater who took the silver medal in the actual competition.
Far from making the judging fairer, the changes have the opposite effect, say the mathematicians. "This lack of reproducibility is one of the most troubling aspects of the newly proposed system. It leaves the final outcome up to chance. For the competitors, this method is especially unfair and capricious."
Their criticism goes even further. Because of other, planned changes in the way the new system will tabulate marks, say the mathematicians, skaters' scores will be more vulnerable if judges engage in "block voting"--colluding to give low scores to particular skaters. The current method guards against that practice.
The skating group, however, stands behind the new system. Rowland Jack, a spokesperson, says the random process "is a disincentive for anybody to try to put pressure on a judge because you have no way of knowing whether a judge has caved in to your pressure. It gives the judge more freedom to judge without fear of reprisals."
The plan also gets positive marks from Patricia M. Benoit, a mathematician who also happens to be a skating judge in Canada. Ms. Benoit, who is director of strategic and program analysis for Skate Canada, the country's governing body for figure skating, is helping the International Skating Union develop the new system. She brushes aside complaints about selecting 9 out of 14 judges because "the randomness was always there and will always be there." In the past, officials slashed a pool of 200 potential judges down to a panel of 9, but the public did not witness the process. Now the selection happens in two stages--from 200 down to 14 and then down to 9. The real difference is that the new system makes the judging anonymous.
The skating group is planning other changes that will further guard against manipulation and bias, she says. At present, judges compare each skater's overall performance with those of other skaters, giving only two broad marks for each person, which enables them to favor a particular athlete. The new system will force judges to score each element of a performance by comparing it with a written standard, which did not exist in the past. Such changes, Ms. Benoit says, make it both harder for judges to skew the results and more obvious when they do.
~~~~~~~~
By Richard Monastersky