How effective were Community Notes on X during elections in India and the U.K.?

Indian Finance Minister Nirmala Sitharaman on July 1, posted a clip from the ongoing Parliament session on X (formerly Twitter) and alleged that the recently appointed Leader of Opposition and Indian National Congress (INC) leader Rahul Gandhi branded all Hindu’s as “violent”. 

“Sheer audacity of LoP ⁦@RahulGandhi⁩ to call everyone who calls himself Hindu as “hinsak”/violent shows ⁦@INCIndia⁩’s hatred and contempt towards Hindus. Also consistent with Hindu hate of his INDI Alliance partners,” Sitharaman wrote. 

The post had garnered over two million views and 10,000 likes at the time of publishing this story. The finance minister’s post included a clipped version of Gandhi’s speech to indicate that he had vilified India’s majority community. 

Screenshot of Nirmala Sitharaman’s post on X. (Source: X/Screenshot)

However, the Congress leader was referring to the Bharatiya Janata Party (BJP), the Rashtriya Swayamsevak Sangh (RSS), and Prime Minister Narendra Modi when he made the remark. He even went on to say, “Narendra Modi ji is not the entire Hindu community, BJP is not the entire Hindu community, and RSS is not the entire Hindu community.”

Gandhi’s speech was widely misrepresented on social media, and while those eligible to rate and write Community Notes on X (formerly Twitter) proposed that additional context should be added, infighting between raters led to the post remaining as is with its traction only increasing. 

Screenshot of recommended notes on Sitharaman’s post (Source:X/Modified by Logically Facts)

Community Notes were first introduced in late 2022 as a means of enabling community-driven content moderation on potentially misleading posts. Unlike other platforms such as Facebook, the posts are not labeled as false or misleading, but they provide additional context. 

Logically Facts analyzed X posts making rounds amid the recently concluded U.K. and Indian elections to assess how Community Notes fared in curbing misleading information and providing accurate details.

Delays in notes appearing on crucial posts

We found that several posts that could potentially mislead users or the ones that could benefit from additional context were not carrying public-facing notes owing to disagreements between the contributors. 

“Compared to traditional content moderation which includes a mix of algorithms as well as human moderators, Community Notes are not as good,” Mediawise’s Director Alex Mahadevan told Logically Facts. “It does not flag nearly as much misleading or harmful content as other platforms that have robust human and algorithmic content moderation.”

Most notes on the platform remain unseen by the public amid divisive contextualization provided by a range of volunteers. For instance, two days before the U.K. elections, a post alleging Labour Member of Parliament (MP) Tom Hayes stole another party’s leaflet while canvassing got nearly 445,000 views. A moderator recommended a note clarifying the video is from 2023. However, the note still hasn’t been made public; either due to disagreement that Hayes committed a crime by stealing post, or lack of ratings.

Screenshot showing disagreement between raters. (Source: X/Screenshot)

In addition, we found major delays in several posts that should have had additional context for voters ahead of the polls on July 4. A Conservative Party post alleging Labour will increase taxes — a claim that the BBC called misleading — was only rated several days after it was first posted.  

Another post with lengthy debate in the Community Notes claimed that Ofcom was “investigating the use of actor Andrew Parker to discredit the Reform U.K. party.” On June 27, Channel 4 broadcast a documentary featuring an undercover investigation into Reform U.K. leader Nigel Farage’s campaign. One canvasser, Parker, was shown in the documentary using racial slurs against the former Prime Minister, Rishi Sunak, leading Farage and Reform U.K. supporters to claim it was staged and that Parker was an actor. 

However, Channel 4 refuted the claims and Ofcom is not investigating Channel 4 News. The post in question actually tags @ofcom_watch, an account stating it was campaigning against regulators such as Ofcom, the official public communications body. The account has since been suspended. 

The Community Notes are split between those pointing out this is not the real Ofcom, those saying Ofcom is not investigating anyway, and those who have said “NNN” (no note needed) because it is “not misleading.” As a result, there is no public community note on the post.  

Screenshot of backend of Community Notes under post alleging OfCom is investigating Channel 4. (Source: X/Screenshot/Modified by Logically Facts)

Similarly in India, where X rolled out the Community Notes feature ahead of the 2024 general elections, we found partisan notes written by volunteers resulted in them not being made public despite several posts needing additional context. 

For instance, Gandhi’s several posts claiming that Agniveer Ajay Kumar — a soldier who died in a land mine explosion in January this year — had not received compensation; gained millions of views. A scheme widely criticized by the Opposition, Agniveer recruits soldiers on a temporary basis, who receive different benefits compared to those permanently enlisted. Gandhi claimed that Kumar’s family did not receive one crore compensation given to Agniveers who die in the line of duty. 

While Defence Minister Rajnath Singh and the official spokesperson of the Indian Army posted Kumar’s family had been compensated nearly one crore, critics alleged Kumar’s family received insurance they were entitled to, and therefore the government could not state it has transferred official compensation. This came following the Indian Army’s post that said, “Ex – Gratia and other benefits amounting to approximately 67 lakhs… will be paid on Final Account Settlement shortly.”

While the aforementioned details require crucial context on an issue that made national headlines, dozens of notes written with evidently biased language and political disagreements between contributors in the Community Notes-backend has potentially resulted in none becoming public.

Screenshot of notes suggested by moderators under Gandhi’s post on compensation to Agniveer Kumar (Source: X/Screenshot/Modified by Logically Facts)

In another case, ANI posted BJP Minister Anurag Thakur’s campaign speech wherein he claimed the Congress manifesto promises to redistribute wealth to Muslims — a false narrative aggressively used by party leaders that was widely debunked. However, while contributors proposed context be added for Indian users, differing takes resulted in no note being made public. 

One note saying additional context is needed since the claim was misleading, another said it's merely “opinion put forth during campaigning." 

Screenshot of recommended notes under ANI’s post (Source: X/Screenshot/Modified by Logically Facts)

“A majority of the notes that I see written are politically biased,” Mahadevan said, referring to the notes on the back end. “It's human nature, this is confirmation bias on steroids. You're allowing people to go in and tell other people they're wrong. And you know a majority of the time it is political content and culture wars.”

Mahadevan also collected data on Community Notes and discovered that only four percent notes related to Israel-Hamas war in Gaza were ever made public. “It’s Community Notes biggest weakness is that it requires bipartisan agreement on facts.  And that's just not how facts work, a fact is a fact. A number is a number. You know its such a rare case when you get people who are pro-Israel or pro-Palestine to agree.” According to Mahadevan, less than ten percent of notes that are written ever become public. 

X announced that over half million global users are registered to rate and write notes. The feature is designed in a way that allows users who haven’t violated existing policies and have a six-month or older account with registered contact information to volunteer. It admits users on a periodic basis, but in case there are more applicants than slots, the platform randomly admits accounts “so as to reduce the likelihood that contributors would be predominantly from one ideology, background, or interest space.”

In order for a note to be made public, X claims handles from diverse perspectives have to agree a proposed note is helpful. It is not based on majority votes. However, whether the handles admitted are from varying perspectives, or how X decides whether a range of accounts agree is unknown. 

Despite several notes indicating political leaning, Mahadevan says the platform is improving its mechanisms. “It used to be that anyone could sign up, and write notes. What I like about Community Notes is they are always working on it. They're always finding ways to improve it, and one of the ways is to set a higher bar for people who can write notes.”

But with this delay, are notes even effective? 

MIT researchers in 2022 presented their findings at a conference that showed Birdwatch (Community Notes’ former name) users preferred challenging content of those users with whom they disagree politically. 

More recently, a preprint paper by researchers found that the appearance of a note roughly halves the number of reposts, but 50 percent of reposts happen within the first five hours — notes take roughly 15 hours to become public.

“This delay has decreased since the launch of Community Notes, but it remains significant relative to the speed at which information spreads on X. In the end, we observe a decrease in the total number of retweets ranging from 16 to 21%. This is not negligible at all, but it still raises questions about the ability of a crowdsourced approach to halt the spread of misinformation,” one of the researchers from the study wrote on X. 

In June, YouTube announced it will also be testing a community-driven context panel under its content. Similar to X, users will be able to rate notes into three categories: “helpful,” “somewhat helpful,” or “unhelpful”. Based on the ratings YouTube says it will determine which notes are published.

Software Freedom Law Center’s (SFLC) Legal Director Mishi Choudhary says for notes to work, users should be able to see notes as they recommended for complete transparency. “We have seen good examples like Wikimedia and Reddit. You can see their entire history, nothing is happening in the background. The problem with X is there is higher trust on content even if fights are happening (in the backend). If it was transparent people would actually know that this is going to come out.”

Community-driven moderation can also lead to the need to sieve through more clutter since not all users may have digital literacy or fact-checking skills that provide concise and clear information. “You're giving this power to people who may or may not have the fact-checking skills and so you can see this in the notes because people don't understand false equivalence, people don't understand when they're cherry picking data, or not looking at context correctly,” Mahadevan said. 

When it comes to addressing the core purpose of content moderation, a joint investigation on content related to Israel-Hamas war by ProPublica and Columbia University’s Tow Center for Digital Journalism found Community Notes hadn’t scaled sufficiently till late last year. “About 80% of the 2,000 debunked posts we reviewed had no Community Note. Of the 200 debunked claims, more than 80 were never clarified with a note,” the report said.

Mahadevan says the platform is constantly improving, but dodges criticism of stifling free speech through such policies.

“They rely on users themselves to classify misinformation, which they refuse to call fact-checking, they just say they're adding context, but what people are doing is essentially fact-checking,” Mahadevan said. 

Apart from several significant examples of misleading and false information that remained unchecked amid the Indian and U.K. polls, we found that the nature of recommended notes that used biased language tended to be on politicians' posts or those having election rhetoric. For a few on which notes do go up, it’s usually with an extensive delay, while many remain unseen by the public over infighting by moderators, or because they don’t get enough votes deeming them ‘helpful’. 

With major cuts to X’s team, particularly the content moderation staff, a community-driven approach to providing accurate information is perhaps not as rapid as the speed at which misinformation spreads on the platform.

(Editor’s Note: Logically Facts reached out to X and will update this article if and when we get a response. Posts cited in this article did not have public Community Notes as of July 9, 2024, but may be subject to change.)

(Inputs from Tori Marland)