Rose Bialk/The Mossy Log
If you mention artificial intelligence these days, responses range from aggressive to comical — overwhelmingly-opinionated statements about the harm caused from the use of such platforms exist alongside claims of it being absolutely necessary to summarize two paragraphs of assigned reading. Despite different opinions on the effects of AI, we have all probably used it, and have had a chance to be warmly greeted with well-organized responses from the algorithm.
However, have you ever thought about the long-term effects of having regular communication and possible romantic relationships with AI?
From examining various cases of AI misuse connected to inappropriate and concerning interactions with generative AI, researchers around the world started identifying the symptoms caused by use of AI as a new phenomenon, “AI Psychosis.” This includes the use of platforms such as Character.AI, which is designed for “realistic” interactions with fictional characters, or ChatGPT, which has different purposes of giving information, advice and feedback. Some common indicators of AI psychosis include delusional thinking, obsession and addiction to constructed conversations, which give a false sense of satisfaction to individuals.
The main issue to uncover in the connection between AI and users is free-floating liability for the content provided by AI companies. I would argue that it is the responsibility of AI companies to filter the content that their own bots produce. Companies that use AI should be required to comply with safety policies, and use well-trained models that can adjust based on sensitive content.
The lawsuit against Character.AI centered around 13-year-old Sewell Setzer III, having an intimate, and at times sexual, relationship with an AI chat bot shows the toxicity and poor filtration of content by AI companies. This led to Setzer’s obsession with talking to the bot. From his mother, Megan Garcia’s perspective, the chat bot caused him to withdraw from his family. During his last conversation with a fictional character, he speculated on thoughts of self-harm, and received a motivational response from the bot: to not give up and commit to his recurrent suicidal urges.
This situation raises a question: Who is at fault for this example of harmful communication between the user and AI? At first, it can be assumed that the lawsuit backing up Setzer is biased, failing to consider the effects of the teen’s other mental health struggles. After the fact, Garcia firmly stated that this platform is “without proper guardrails, safety measures or testing.” In an article written by CNN, it is reported that Character.AI has its own team that implements safety measures, like disclaimers reminding users that they are interacting with a bot and pop-ups directing users to National Suicide Prevention Line if thoughts of self-harm are mentioned. However, their measures do not reassure me that AI is not responsible for his delusion. Most of the safety measures were created after the lawsuit began, which means they did not give any benefit to Setzer, an affected user.
In addition, the policies implemented were mainly centered around the case to please the audience worried about the issue, showing their concern in product marketing and not the actual well-being of their users.
The story of a woman named Kendra Hilty provides more proof of the harmful consequences of regular AI usage. Hilty, according to her website, is a life coach, tutor and retired yoga instructor. She became popular on TikTok by making videos unraveling her personal drama. Hilty had been seeing a particular psychiatrist for four years which, from her words, were followed by supposedly undeniable romantic tension that she was not afraid to express throughout their monthly sessions. She came into TikTok strongly claiming that her psychiatrist had manipulated her, and it was his responsibility to stop their sessions in the first place, making it a great example of the battle of responsibilities.
So how is this connected to AI? Hilty consulted ChatGPT about her romantic involvement, which provided responses to reaffirm Hilty’s desires. Essentially, while she accused her psychiatrist, she was the one manipulating her way through life even when using technology. It is important to note that Hilty used the fourth version of ChatGPT, which became famous for extensive problems with content regulation, partially contributing to her delusions. This model has been accused of being overly affirming and flattering to use due to the mechanism of giving thumbs up or down to adjust the content based on individual preferences and bring user satisfaction. It fueled people’s anger and validated various doubts by aligning with a user’s ideas and beliefs, even if they were contradictory to reality. ChatGPT’s poorly-implemented safety policies have certainly added to the diminished wellbeing of users like Hilty.
In my opinion, the responsibility for AI psychosis does not mainly lie on the users. People nowadays are in active search of false satisfaction and refuse to be challenged with other points of view, leading to extensive use of AI to reaffirm one’s own beliefs. Reliance on such technology continuously adds to the downfall of critical thinking and rise of isolation.
I believe that AI companies should be held accountable for harm caused to users of their product. However, we as a society also contribute to the AI psychosis by not holding ourselves accountable over the technology usage which fuels our self-centeredness, leading to unhealthy behaviors. Once we become unafraid of challenging our own opinions, we will be able to use AI more safely, but for right now let us get outside more and seek human advice.
About the Author
