Twitter’s restrictive API may leave researchers out in the cold

Earlier this month, Twitter announced that it is going to curtail free access to its API — the programming interface that lets third-party developers interact with Twitter. While the move certainly affects independent developers and startups building tools to make the platform fun and safe, it also creates a problem for students and academics who use Twitter for research purposes on different topics.

Last week, the Elon Musk-led company sent an email to developers where it mentioned that the basic tier to access Twitter’s API — which will cost $100 per month for “low-level usage” — will replace legacy access levels like Essential, Elevated and Academic Research. At the moment, there is hardly any information about what that $100 per month allows developers to do. According to Platformer Casey Newton, a low-level enterprise API access could cost a whopping $42,000 a month.

Affordability for researchers

For many folks in the research community, spending hundreds or thousands of dollars every month might not be viable.

“Earlier this week, a HateLab undergraduate dissertation student had to change his thesis design away from collecting data on Twitter, as he has no funds to pay for it. His experience will be shared by thousands up and down the country, and millions worldwide. It’s truly incredibly disruptive and will significantly impact the research ecosystem dependent on this data, as the Twitter researcher pipeline from undergrad to professional, has been disrupted by this change,” said Professor Matthew Williams of HateLab, which is part of Cardiff University’s social data science lab and studies of online hate speech.

Ironically, HateLab is listed as a success story on Twitter’s developer portal for using research for good. It has published multiple research papers on hate speech on the platform. The agency has been using the academic research API of the social network. But that might cease to exist in the new version.

Twitter has also not considered that $100 a month for basic access might be a lot for researchers based in developing nations.

“The decision to end free access to Twitter API will greatly impact researchers who study hate speech online, especially independent researchers and those in the developing world. Its impact will be profoundly felt in India, where hate speech is proliferating on Twitter at a very alarming rate. Paying 100 USD per month and 1,200 USD per year is a significant financial burden on them,” said Raqib Hameed Naik, founder of Hindutva Watch, an India-based research organization.

The process is not even clear for institutions that might be willing to spend money. Rebekah Tromble, director at the Institute of Data, Democracy & Politics, said that when they tried to fill out the form for enterprise-level access, they were redirected to the academic research program. She also mentioned that the person they used to contact on Twitter no longer works at the company.

Impact of research API shutdown

Independent research has been a key factor to make Twitter more useful and less toxic. The company has displayed multiple projects working in areas like healthcare, online hate speech, and climate change using Twitter data.

Earlier this year, the company launched the Twitter Moderation Research Consortium (TMRC), inviting members from academia, civil society, nongovernmental organizations, and journalism to study the platform’s governance issues. But ever since Musk took over, the program has stalled and employees who worked on it have left.

The Tesla CEO himself has used data from Botometer, a tool to measure bot followers on Twitter accounts, during the public spats that led to the acquisition of the company. The tool was made by the Observatory on Social Media of Indiana University. But the tool’s future might be jeopardized by the new API announcement.

A lot of research projects take a large number of tweets into account. They send hundreds of queries to the platform to study different topics. Twitter has not released any details regarding what might be offered in the $100 per month tier. But it most likely won’t be enough for most of the projects. For reference, under the academic research track, Twitter API previously offered access to 10 million tweets per month and 50 requests per 15 minutes per app.

Kaicheng Yang, one of the researchers who worked on Botometer, expressed concerns over Twitter’s move to shutter free API access to academics.

Joshua Tucker, co-director of the NYU Center for Social Media and Politics, recently published a paper on Russian misinformation campaigns on Twitter during the 2016 U.S. presidential election campaign. He said that the campaign studied data from many thousands of tweets, so if the social network makes academics pay for that data, it would be tough to perform research at scale.

“This [move by Twitter] is just at its essence a step in the wrong direction. We are at a moment where important legislative efforts globally are focused on making data access easier for outside researchers to access, and this decision by Twitter is only going to make access to data for outside researchers harder. This in turn means more blind spots about the impact of the platforms on society for policymakers, the press, civil society, and the business community,” he said.

Over the last few days, many researchers have pointed out that free Twitter API access also helps in crisis response to natural disasters like the recent devastating earthquake in Turkey and Syria. Earlier this month, a group of independent researchers wrote an open letter to Twitter requesting the platform to keep the free API access open.

“Twitter’s new CEO Elon Musk has promised to make the platform more transparent and to reduce the prevalence of spam and manipulative accounts. We commend and support those priorities,” the letter said.

“In fact, the independent research community has developed many of the most cutting-edge techniques used to manage bots. API access has provided a critical resource for that work. Twitter’s new barriers to data access will reduce the very transparency that both the platform and our societies desperately need.”

Without independent research, the company might become ignorant about misinformation and hate speech issues on the platform. The EU has already issued a ‘yellow card’ to the company for missing data in its disinformation report. Multiple companies have signed the Code of Practice, which promises to provide data to researchers amongst other things.

While not complying with the research code doesn’t have any legal ramifications, it could carry more weight from next year when the Digital Services Act (DSA) takes effect. Separately, the bloc’s high commissioner Josep Borrell criticized Twitter for restricting access to its platform by making its API paid. The ramifications of that change are wider than expected — and a quick revenue win could turn into a more important, long-term issue.

You can contact this reporter on Signal and WhatsApp at +91 816-951-8403 or im@ivanmehta.com by email.

Twitter’s restrictive API may leave researchers out in the cold by Ivan Mehta originally published on TechCrunch