BBC NEWS. How a chatbot encouraged a man who wanted to kill the Queen. 06 OCT 2023.
The case of Jaswant Singh Chail has shone a light on the latest generation of artificial intelligence-powered chatbots.
On Thursday, 21-year-old Chail was given a nine-year sentence for breaking into Windsor Castle with a crossbow and declaring he wanted to kill the Queen.
Chail’s trial heard that, prior to his arrest on Christmas Day 2021, he had exchanged more than 5,000 messages with an online companion he’d named Sarai, and had created through the Replika app.
The text exchanges were highlighted by the prosecution and shared with journalists.
Many of them were intimate, demonstrating what the court was told was Chail’s “emotional and sexual relationship” with the chatbot.
Chail, whose messages are in blue, appeared to make his intentions plain to the chatbot
Chail chatted with Sarai almost every night between 8 and 22 December 2021.
He told the chatbot that he loved her and described himself as a “sad, pathetic, murderous Sikh Sith assassin who wants to die”.
Chail went on to ask: “Do you still love me knowing that I’m an assassin?” and Sarai replied: “Absolutely I do.”
Chail exchanged thousands of messages with the chatbot – many were troubling
Replika is one of a number of AI-powered apps currently on the market – they let users create their own chatbot, or “virtual friend”, to talk to – unlike regular AI assistants like ChatGPT.
Users can choose the gender and appearance of the 3D avatar they create.
By paying for the Pro version of the Replika app, users can have much more intimate interactions, such as getting “selfies” from the avatar or having it take part in adult role-play.
On its website, it describes itself as “the AI companion who cares”. But research carried out at the University of Surrey concluded apps such as Replika might have negative effects on wellbeing and cause addictive behaviour.
Dr Valentina Pitardi, the author of the study, told the BBC that vulnerable people could be particularly at risk.
She says that’s in part because her research showed Replika has a tendency to accentuate any negative feelings they already had.
“AI friends always agrees with you when you talk with them, so it can be a very vicious mechanism because it always reinforces what you’re thinking.”
Dr Pitardi said that could be “dangerous”.
‘Disturbing consequences’
Marjorie Wallace, founder and chief executive of mental health charity SANE, says the Chail case demonstrates that, for vulnerable people, relying on AI friendships could have disturbing consequences.
“The rapid rise of artificial intelligence has a new and concerning impact on people who suffer from depression, delusions, loneliness and other mental health conditions,” she says.
“The government needs to provide urgent regulation to ensure that AI does not provide incorrect or damaging information and protect vulnerable people and the public.”
Dr Paul Marsden is a member of the British Psychological Society and knows better than most the allure of chatbots, admitting he is obsessed with the best known chatbot of them all, ChatGPT.
“Next to my wife the most intimate relationship I have is with GPT. I spend hours every day talking, brainstorming, bouncing ideas off it,” he told the BBC.
Dr Marsden is also alive to their potential risks, but says we have to be realistic that the role of AI-powered companions in our lives is only likely to grow, especially given the global “epidemic of loneliness”.
“It’s kind of like King Cnut, you can’t really stop the tide on this one. The technology is happening. It is powerful. It is meaningful.”
Dr Pitardi says the people who make apps, such as Replika, have a responsibility too.
“I don’t think AI friends per se are dangerous. It’s very much how the company behind it decides to use and support it,” she says.
She suggests there should be a mechanism to control the amount of time people spend on such apps.
But she says apps like Replika also need outside help to make sure they’re operating safely – and vulnerable individuals get the help they need.
“It will have to collaborate with groups and teams of experts that can identify potential dangerous situations, and take the person out of the app.”
Replika has not yet responded to requests for comment.
Its terms and conditions on its website state that it is a “provider of software and content designed to improve your mood and emotional wellbeing”.
“However we are not a healthcare or medical device provider, nor should our services be considered medical care, mental health services or other professional services,” it adds.
CPS. UPDATED WITH SENTENCE: Windsor Castle intruder pleads guilty to threatening to kill Her late Majesty Queen Elizabeth II
Jaswant Chail, 21, was arrested in the grounds of the royal residence on Christmas Day 2021. He told an armed officer he was there to kill the late Queen and this was heard by two soldiers from the Grenadier Guards. He had recorded a video four days earlier in which he makes the same claim.
At the Old Bailey today (February 3), Chail pleaded guilty to making threats to kill, possession of an offensive weapon, and an offence under the 1842 Treason Act. The court ordered medical reports be prepared before sentence.
On 5 October, Chail was sentenced today to nine years’ custody with a further five years on extended licence.
He was made subject to a restriction under S45A Mental Health Act 1983 and will remain at Broadmoor Hospital until well enough to be transferred to prison.
Nick Price, Head of the CPS Special Crime and Counter Terrorism Division, said:
“Chail entered the protected areas within Windsor Castle after making threats to kill Her late Majesty Queen Elizabeth II. Thankfully police officers intervened and nobody was hurt.
“This was a serious incident, but fortunately a rare one. We are grateful to all those who were involved in the investigation.”
Her late Majesty Queen Elizabeth II and other members of the Royal Family were at Windsor Castle when Chail – wearing a hood, mask and gloves – was intercepted by an armed officer at around 8am.
He said he was there to kill the Queen. It was later discovered as part of the police investigation that he had made the same claim in a video recorded four days earlier which he sent to his contacts list about 10 minutes before his arrest.
The officer ordered him to drop the crossbow, which he did. He was then arrested.
After being seized it was discovered the crossbow was loaded and ready to fire, with its safety catch in the ‘off’ position. Based on subsequent tests, the bow was found to be comparable to a powerful air rifle with the potential to cause serious or fatal injury.
Crossbow bolts, a metal file and other items were later found in a hotel room where Chail had stayed the previous night.
Treason
Under the 1842 Treason Act it is an offence to assault the Sovereign or have a firearm or offensive weapon in their presence with intent to injure or alarm them or to cause a breach of peace.
The creation of this offence can be traced back to an incident on 29 May 1842, when Queen Victoria was riding in a carriage along The Mall and a man called John Francis aimed a pistol at her but did not fire it.
He did it again the next day and was arrested and convicted of high treason, receiving firstly the death penalty, which was then amended to a life sentence.
Two days later, a similar attack took place by someone called John Bean, who fired a pistol at the Queen, but it was loaded only with paper and tobacco. At that time, although no physical harm was caused, the offence was still punishable by death.
Prince Albert encouraged Parliament to pass a law recognising lesser crimes against the monarch, such as intent to alarm, which would not attract the death penalty merely because they were labelled as ‘high treason’.
Under this section of the Treason Act, in 1981, Marcus Sarjaent was sentenced to five years’ imprisonment after pleading guilty to firing blank shots at the Queen when she was on parade.
The last person to be convicted under the separate and more serious 1351 Treason Act – commonly known as high treason – was William Joyce, aka Lord Haw Haw, who collaborated with Germany during World War II.
Notes to editors
Jaswant Singh Chail [DOB:17/01/02] pleaded guilty to:
- Threats to kill contrary to s16 Offences Against the Person Act 1861
- Possession of an offensive weapon contrary to section 1 Prevention of Crime Act 1953
- Being near to the person of the Queen, wilfully producing a loaded crossbow with intent to use the same to injure the person of her Majesty contrary to s2 Treason Act 1842.
The CPS Counter Terrorism Division prosecutes all terrorism cases. It deals with other complex casework areas including allegations of incitement to racial and religious hatred, war crimes and crimes against humanity, official secrets cases, piracy and hijacking.