Researchers have found flaws in some of the data that track and field officials used to formulate regulations for the complicated cases of Caster Semenya of South Africa, the two-time Olympic champion at 800 metres, and other female athletes with naturally elevated testosterone levels.
Three independent researchers said they believed the mistakes called into question the validity of a 2017 study commissioned by track and field’s world governing body, the International Federation of Athletics Associations (IAAF), according to interviews and a paper written by the researchers and provided to The New York Times.
The 2017 study was used to help devise regulations that could require some runners to undergo medical treatment to lower their hormone levels to remain eligible for the sport’s most prominent international competitions, such as the Olympics.
The researchers have called for a retraction of the study, published last year in the British Journal of Sports Medicine. The study served as an underpinning for rules, scheduled to be enacted in November, which would establish permitted testosterone levels for athletes participating in women’s events, from 400 metres to the mile.
“They cannot use this study as an excuse or a reason for setting a testosterone level because the data they have presented is not solid,” one of the independent researchers, Erik Boye of Norway, said on Thursday.
The IAAF has updated its research, which was published last week, again in the British Journal of Sports Medicine. “The IAAF will not be seeking a retraction of the 2017 study,” the governing body said in a statement on Thursday. “The conclusions remain the same.”
Caster Semenya, right, races against Laura Muir, left, and others earlier this month.ALAIN GROSCLAUDE/Getty Images
But the statement did little to dampen criticism by the independent researchers. The IAAF seems “bound to lose” an intended challenge by Semenya to the Court of Arbitration for Sport, a kind of Supreme Court for international athletics, said Boye, a cancer researcher and anti-doping expert.
The IAAF has argued for years that rules governing testosterone levels are needed to level the playing field and to reduce an unfair advantage gained in some women’s events by athletes with so-called differences of sexual development. The 2017 study was only one facet of 15 years’ worth of field study, the IAAF said.
Dr. Stephane Bermon, the IAAF’s senior medical and scientific consultant and a co-author of the 2017 study, last week acknowledged some errors in the data in an e-mail sent to one of the independent researchers. But Bermon added in the e-mail, obtained by the Times, that the mistakes “do not have significant impact on the final outcomes and conclusions of our study.”
Karim Khan, editor-in-chief of the British Journal of Sports Medicine, did not respond to e-mails seeking comment.
The disputed 2017 study in the journal examined results from the 2011 and 2013 world track and field championships. It found that women with the highest testosterone levels significantly outperformed women with the lowest testosterone levels in events such as the 400 metres, the 400-metre hurdles and the 800 metres, which distill speed and endurance.
But, in examining the study’s results from those three races, plus the 1,500 meters, the three independent researchers said they found that the performance data used in the study’s analysis were anomalous or inaccurate 17-33 percent of the time.
The errors included more than one time recorded for the same athlete; repeated use of the same time for individual athletes; and phantom times, when no athlete could be found to have run a reported time. Also included were times for athletes who were disqualified for doping.
“I think everyone can understand that if your data set is contaminated by as much as one-third bad data, it’s kind of a garbage-in, garbage-out situation,” said one of the independent researchers, Roger Pielke Jr., the director of the Sports Governance Center at the University of Colorado.
Referring to the IAAF, Pielke said: “I really see no option for them other than to retract the paper. If they retract the paper, then the regulations don’t have a scientific basis.”
“This is an effort at what I would call a do-over, and it’s embarrassing and it’s not how science is expected to be done,” Pielke said. “I think this adds considerably more weight to our call for the original paper to be retracted. This is everything but putting up a billboard saying, ‘We really screwed up the data in the original study.’”
Another of the independent researchers, Ross Tucker, an exercise physiologist who specializes in sports performance at the University of Cape Town in South Africa, agreed that the first study should still be retracted. The reanalysis, he said, included “too much uncertainty to trust.”
A reanalysis of the original study might, in fact, make even a stronger case for the IAAF’s position on the need to regulate testosterone levels, the independent researchers said in interviews. But any new analysis should be conducted only with a full independent audit and with publicly available performance data that could be replicated by independent scholars, the researchers said.
The IAAF has lacked transparency in providing and presenting its data, said Boye, the Norwegian researcher, who described the governing body as “doing everything with their hands over the data.”
Given the data errors, the original study is “entirely untrustworthy,” and “an impossible position” for the IAAF to defend, said Tucker, the South African researcher.
He added, “If I was on Semenya’s team, this would be among the best news I could receive.“
The pending rules would affect women with testosterone levels of five nanomoles per litre and above. Most women, including elite athletes, have natural testosterone levels of 0.12-1.79 nanomoles per litre, the IAAF said, while the typical male range after puberty is much higher, at 7.7-29.4 nanomoles per litre.
No female athlete would have natural testosterone levels at 5 nanomoles per litre or higher without so-called differences in sex development or tumours, the IAAF said. In effect, it has said that athletes with such elevated levels are biologically male.
But the scientific discussion has now been somewhat sidetracked by questions about the independence and validity of the IAAF’s research.
“You don’t have drug companies doing their own studies that no one else can see,” Pielke, the Colorado researcher, said.