With no explanation, an app told large numbers of foreign residents in Beijing that they could not leave their homes this weekend, a digital quarantine order that was rescinded just hours later after causing widespread confusion in the Chinese capital.
The Beijing Health Kit app has been used by millions of people, part of a nationwide strategy of deploying digital tools in the hope that computerized decision-making can succeed in containing the deadly new coronavirus where mere human oversight might fail.
China’s experience offers one example of how to manage large populations during a pandemic, as governments and tech companies around the world search for solutions.
But the brief home quarantine order in Beijing – whose appearance and disappearance have yet to be explained – offers a vivid example of the risk of entrusting software with decisions about human freedom.
The isolation notice appeared Sunday morning, when many foreign residents living in Beijing saw that their status on the app had changed from a green code – indicating a person is free to move about – to a yellow one, with the instruction “home observation.” In China, home observation is a strict form of isolation in which people are told to wear masks indoors and barred from going outside for 14 days. Officials in some areas have installed alarms and even security tape on front doors to ensure they are not opened.
The Health Kit app bases its determination on what city authorities have called “comprehensive multi-source data research and judgment,” which blends community information, disease control orders and data from highway, rail and air travel. It requires users to upload identification document images and pass a facial recognition screening before producing a green, yellow or red code: green is all clear, yellow signifies home isolation, while red mandates quarantine in a government-approved facility.
Other health check services have been created by mobile service providers, a military contractor and the country’s cabinetlike State Council.
But by March 30, nine million people in Beijing had used the Health Kit service, which is offered by both Tencent-owned WeChat, China’s near-universal messaging app, and Alipay, a widely used digital payment platform.
Anyone unable to show a green code can be barred from office buildings, residential compounds, public transit, restaurants and shopping centres. The code has become a central arbiter of daily life – but authorities have disclosed little about how it is generated. “It is impossible for humans to intervene or modify the results,” the Beijing central information hotline wrote on April 9 in response to questions posed by the Beijing Evening News.
Numerous problems have been reported with the health apps. In Hangzhou, code labelled “reportInfoAndLocationToPolice” in the Alipay app immediately sent the user’s location to a server, along with an identifying code number, according to an analysis by The New York Times.
In cities around China, people have reported erroneous codes that arbitrarily brought normal life to a halt. In Beijing, the app has taken days to release people from quarantine, even when they possess the necessary documents from local authorities. Others have been barred from going to work because they spent time in the mountains west of the city, the Beijing Evening News reported. Still others have been prevented from entering their own communities by such errors.
Experts have acknowledged but downplayed the problems.
Systems such as Health Kit “enable governments to make wiser and more scientific decisions in their work,” said Cui Xiaohui, a scholar at the School of Cyber Science and Engineering at Wuhan University. He called it “a social management system that’s backed by science.”
But the opacity of the system means no one knows, for example, what weight is given to each data point. In addition, “each city and province has different codes, and we have no idea what additions or omissions each has made in their version,” Prof. Cui said. As a result, “once a problem appears, people will feel confused and begin to complain. I think people would have fewer complaints if there was more transparency.”
It’s time for the companies involved to release their source code, he said.
But the digital quarantine apps are “emergency tools,” and as such some problems can be overlooked in the short term, said Wu Fei, the director of the artificial intelligence lab at Zhejiang University. In an urgent health situation, it comes down to “human rights or human lives,” he said. “When it’s a matter of survival, human rights should be ranked less important.”
Governments and corporations around the world have embraced AI systems for their ability to better allocate resources, manage traffic, plan future development, provide proper skills training and improve health care.
But China has begun to use computerized governance in more ambitious ways. It’s being used to assess individual credibility in social credit systems. In the northwestern Xinjiang region, state-run systems target people considered at risk of criminal conduct before police are dispatched to investigate.
“China is, in that sense, ahead of us technologically,” said Simon Joss, a University of Glasgow scholar and associate director of the Urban Big Data Centre in the U.K. But “I don’t think China’s approach to public monitoring and tracking would be acceptable to the public in many other countries, for obvious data protection and civil liberties reasons.”
Still, the pandemic has created a sense of urgency among authoritarian and democratic governments alike to find technological solutions to contain the virus.
”It’s seen as the potential way out of the current lockdown – so people are hoping that these various apps might allow us to get back to normal life,” Prof. Joss said. The challenge for Western governments will be to conduct “responsible innovation.”
It’s a matter of creating something driven not only by government or corporate appetites for data but in a way that prioritizes “civic interest,” he said.
With reporting by Alexandra Li
Sign up for the Coronavirus Update newsletter to read the day’s essential coronavirus news, features and explainers written by Globe.