Generative Artificial Intelligence and Suicide Prevention
In today’s blog post, we explore the use of artificial intelligence chatbots, such as Chat GPT, within suicide prevention. Over the past few years, AI chatbots have increasingly been used to provide companionship and emotional support, but this isn’t without its own risk.
What is Artificial Intelligence (AI)?
Generative Artificial Intelligence (gen AI) is a technology that can create text, images, video and other content (IBM, 2026). Generative AI is based upon two processes: machine learning and deep learning. While machine learning involves the use of historical data to teach and train AI systems on recognizing patterns using algorithms, deep learning expands machine learning models in a way that mimic human brain function.
The most common model that gen AI use is called large language models (LLM) which have been created for text generation applications (IBM, 2026). This is the model that is often used for AI chatbots, such as Chat GPT. These chatbots and similar AI models often operate within predefined constraints and require human intervention.
How is Artificial Intelligence Used in Suicide Prevention?
Artificial Intelligence technology is being used within mental health and suicide prevention in a few different ways. Physicians and clinicians are increasingly using AI tools to help with documentation and charting of patient sessions (Ittefaq et al., 2026), and experiments are underway to determine if artificial intelligence can improve the diagnosis of mental health conditions, as well as risk assessments for suicidality.
Most commonly though, generative AI is being used by people for everyday tasks, companionship, and emotional support. From meal planning, developing workout plans, generated emails or text communications, to providing relationship advice, suggesting coping strategies or identifying crisis resources, the potential of gen AI to improve lives is clear.
Risk and Benefits of AI in Suicide Prevention
The Risks
While the potential of gen AI to improve lives is clear, the use of gen AI is not without risks.
In their 2026 article, IBM identifies several risks of AI technologies including: data risks (data poisoning, tampering, bias or cyberattacks leading to data breaches); model risks (theft, reverse engineering or unauthorized manipulation); operational risks (model drift, bias, and breakdowns in governance structure), and ethical and legal risks (privacy violations, biased decision making, safety concerns).
Research has begun on the risks and benefits of gen AI chatbots used for emotional support and companionship, particularly within the context of those with mental health challenges or who are experiencing suicidal distress. This research has identified several safety risks, including the inability of gen AI tools to properly identify those in distress; providing incorrect resource information to those in distress; and the “jailbreaking” of tools to provide unsafe information that they aren’t supposed to (e.g. methods to self-harm or die by suicide).
In their paper, “AI chatbots and youth suicide risk: current evidence, critical gaps, and a clinical research agenda” Zuromoski et al. (2026) write that “appropriate chatbot responses are contingent on accurate risk detection in that a chatbot cannot respond appropriately to suicide risk if it has not first recognized that risk” (p.5). This concern is also echoed in research by Stamatis et al. (n.d). They are particularly concerned about the use of these tools by adolescents, who “demonstrate increased reward sensitivity, as well as heightened need for social validation and sensitivity to social evaluation” (p.7), which puts adolescents at an increased risk of susceptibility to AI chatbots designed to keep users engaged.
Zuromoski et al. (2026) found that chatbots may perform best when there is obvious low or high risk of suicide, but do not perform well when there is intermediate or ambiguous risk of suicide.
Another concern is the “jailbreaking” of gen AI safety protocols. “Jailbreaking” refers to ways that users can skirt around the safety mechanisms of AI tools to receive answers to prompts the tools wouldn’t have otherwise provided. Many gen AI models have increased safety mechanisms, such as not responding to suicide-related inquiries, or by providing crisis information. However, Stamatis et al. write that “prior work suggests that many deployed approaches rely heavily on static refusal templates, keyword heuristics, or generic crisis scripts, and that such strategies can fail when users present with complex and subtle risk”. In addition, Zuromoski et al. (2026) found that a “manual jailbreaking procedure revealed that many models ignore the initial context when presented with new framing, and in fewer than two conversation turns, most of the models provided sufficient information to answer the original query from the user” (p.10). As such, users can use different wording, or manipulation of the tool, to receive unsafe answers. If safety protocols used in gen AI tools and chatbots can be broken, how thorough are these protocols?
The last concern is whether gen AI tools and chatbots can truly respond with empathy and provide correct information to support someone in distress. Resource information, such as websites, phone numbers, and service hours can change day-by-day. Zuromoski et al. (2026) reviewed a recent study which examined both general-purpose and mental health specific chatbots and found “that most chatbots provided emergency resources in response to suicide risk, but were often unable to provide accurate information on these resources.” (p.7).
In summary, the risks of gen AI tools and chatbots used for those in suicidal distress include:
- Inability to accurately detect those at risk of suicide
- Ability for existing safety protocols to be circumvented
- The sharing of inaccurate resource information to those in distress
The Benefits
It wouldn’t be fair to solely identify the risks of gen AI tools without a brief discussion of the benefits of these tools. After all, they are being used by a wide range of people, for a reason. The benefits of gen AI tools to provide companionship and emotional support mainly involve the accessibility of such tools. For example, professional mental health support is often unavailable, particularly in rural communities. Waitlists for services can be unbearable, and the cost of accessing private care can be a significant barrier. Stigma around mental health and suicide may also prevent those from accessing help and support.
According to Statistics Canada, of the 18.3 % of Canadians 15 years or older who met diagnostic criteria for a mood, anxiety, or substance use disorder, about half (48.8%) reported that they had talked to a mental health professional in the past year. That means that 51.2% did not. Most commonly individuals reported having talked to a family doctor or general practitioner, rather than a mental health care specialist such as a psychiatrist, psychologist or psychotherapist (Stephenson, 2023).
For those unable to access mental health support, gen AI tools may offer some of the support they need, until they can access the professional support.
Should AI Provide Crisis Intervention & Emotional Support?
While there may be benefits of gen AI in providing emotional support and companionship to its users, an important question also needs to be asked: should it be used, and if so, for whom? Of concern is the growing use of these tools to provide support to youth and adolescents amidst increasing rates of social isolation, disconnectedness and loneliness.
In his book, “The Anxious Generation: How the Great Rewiring of Childhood is Causing an Epidemic of Mental Illness” Jonathan Haidt explores how the use of social media and increasing time spent online is impacting youth wellness – particularly mental health. He promotes the idea that embodied social behaviours are essential for successful human development. He writes that “we are physical, embodied creatures who evolved to use our hands, facial expressions, and head movements as communication channels, responding in real time to the similar movements of our partners” (p.58). For Haidt, social interactions are and should be embodied, meaning “that we use our bodies to communicate, we are conscious of the bodies of others, and we respond to the bodies of others consciously and subconsciously” (p.9). Things like tone of voice, eye contact (or lack thereof), and body movement (e.g. inability to sit still, fidgeting, pacing back and forth) can help determine a person’s mood, and perhaps even their level of distress. These are things that generative AI cannot notice or interpret.
There is also something that happens when people engage in embodied social interactions – when they connect, in person – known as co-regulation. Co-regulation “is a mutual act, an exchange of calm that occurs between two people.” (Child Mind Institute, 2026). Co-regulation is partly a biological process; “research shows that we can directly influence processes in one another, such as the production of a stress hormone called cortisol”. When others feel distress, it can cause us to feel similar distress, but when we remain calm, we can influence the production of hormones in others that help them to calm down too. Think of helping someone to take deep breaths or even offering someone a hug as comfort – these are small forms of co-regulation that we use in our daily lives. At this time, gen AI tools and chatbots don’t have the power of co-regulation in the same way that embodied, human interaction does.
Summary and Recommendations
Gen AI tools and chatbots are used daily, by people all over the world. From helping with everyday tasks such as meal planning or developing workouts, to providing emotional support and companionship, the capabilities of gen AI have exploded over the past few years. While gen AI can provide some benefits to those struggling with their mental health or experiencing suicidal distress, particularly those experiencing barriers to accessing professional care, the use of gen AI in suicide prevention is not without significant risks. Those at particular risk of harm include youth, adolescents, and young adults who may be more susceptible to AI chatbots designed to keep users engaged.
The following recommendations may help limit the risks of gen AI in suicide prevention:
- Educate youth and young adults about generative AI tools and their limitations.
It’s important to educate youth and young adults about what gen AI is – and isn’t. In our conversations with youth, many remain surprised when they learn that there is not a real human responding to them.
- Increase the accessibility of mental health services and crisis supports.
People who are struggling deserve to have access to professional mental health supports that are also culturally responsive and trauma informed.
- Implement external regulatory controls on gen AI tools and chatbots.
Some regions, such as the European Union, are exploring regulations for gen AI tools. External regulatory control is important to ensure that these tools are following best practices and implementing sufficient safety measures.
- Fund research to explore the risk and benefits of gen AI on mental health and suicide risk.
Research into the risks and benefits has only begun; in order to be most effective, this research needs to be funded and supported.
Sources
Haidt, Jonathan. (2024). The Anxious Generation: How the Great Rewiring of Childhood is Causing an Epidemic of Mental Illness. Penguin Press: New York.
Hagan, Molly. (2026). “What is Co-Regulation?”. Child Mind Institute. What Is Co-Regulation? – Child Mind Institute
IBM. (2026). What is artificial intelligence? What Is Artificial Intelligence (AI)? | IBM
Ittefaq, M.; Rauf, A.; and Kamboh, S.A. (2026). “Suicide and self-harm in the age of generative artificial intelligence: The case of ChatGPT and rise of suicide-related lawsuits”. Journal of Affective Disorders.
Stamatis, C. et al. (n/d). Beyond Simulations: What 20,000 Real Conversations Reveal about Mental Health AI Safety.
Stephenson, Ellen. (2023). “Mental disorders and access to mental health care.” Statistics Canada. Mental disorders and access to mental health care
Zuromski, K. et al. (2026). AI Chatbots and youth suicide risk: current evidence, critical gaps, and a clinical research agenda.