As artificial intelligence tools like ChatGPT become an increasingly popular avenue for people seeking personal therapy and emotional support, the dangers that this can present – especially for young people – have made plenty of headlines. What hasn’t received as much attention is employers using generative AI to assess workers’ psychological well-being and provide emotional support in the workplace.

Since the pandemic-induced global shift to remote work, industries ranging from health care to human resources and customer service have seen a spike in employers using AI-powered systems designed to analyze the emotional state of employees, identify emotionally distressed individuals, and provide them with emotional support.

This new frontier is a large step beyond using general chat tools or individual therapy apps for psychological support. As researchers studying how AI affects emotions and relationships in the workplace, we are concerned with critical questions that this shift raises: What happens when your employer has access to your emotional data? Can AI really provide the kind of emotional support workers need? What happens if the AI malfunctions? And if something goes wrong, who’s responsible?

The workplace difference

Many companies have started by offering automated counseling programs that have many parallels with personal therapy apps, a practice that has shown some benefits. In preliminary studies, researchers found that in a doctor-patient-style virtual conversation setting, AI-generated responses actually make people feel more heard than human ones. A study comparing AI chatbots with human psychotherapists found the bots were “at least as empathic as therapist responses, and sometimes more so.”

This might seem surprising at first glance, but AI offers unwavering attention and consistently supportive responses. It doesn’t interrupt, doesn’t judge and doesn’t get frustrated when you repeat the same concerns. For some employees, especially those dealing with stigmatized issues like mental health or workplace conflicts, this consistency feels safer than human interaction.

But for others, it raises new concerns. A 2023 study found that workers were reluctant to participate in company-initiated mental health programs due to worries about confidentiality and stigma. Many feared that their disclosures could negatively affect their careers.

Other workplace AI systems go much deeper, analyzing employee communication as it happens – think emails, Slack conversations and Zoom calls. This analysis creates detailed records of employee emotional states, stress patterns and psychological vulnerabilities. All this data resides within corporate systems where privacy protections are typically unclear and often favor the interests of the employer.

Workplace Options, a global employee assistance provider, has partnered with Wellbeing.ai to deploy a platform that uses facial analytics to track emotional states across 62 emotion categories. It generates well-being scores that organizations can use to detect stress or morale issues. This approach effectively embeds AI into emotionally sensitive aspects of work, leaving an uncomfortably thin boundary between support and surveillance.

In this scenario, the same AI that helps employees feel heard and supported also generates unprecedented insight into workforce emotional dynamics. Organizations can now track which departments show signs of burnout, identify employees at risk of quitting and monitor emotional responses to organizational changes.

But this type of tool also transforms emotional data into management intelligence, presenting many companies with a genuine dilemma. While progressive organizations are establishing strict data governance – limiting access to anonymized patterns rather than individual conversations – others struggle with the temptation to use emotional insights for performance evaluation and personnel decisions.

Continuous surveillance carried out by some of these systems may help ensure that companies do not neglect a group or individual in distress, but it can also lead people to monitor their own actions to avoid calling attention to themselves. Research on workplace AI monitoring has shown how employees experience increased stress and modify their behavior when they know that management can review their interactions. The monitoring undermines the feeling of safety necessary for people to comfortably seek help. Another study found that these systems increased distress for employees due to the loss of privacy and concerns that consequences would arise if the system identified them as being stressed or burned out.

When artificial empathy meets real consequences

These findings are important because the stakes are arguably even higher in workplace settings than personal ones. AI systems lack the nuanced judgment necessary to distinguish between accepting someone as a person versus endorsing harmful behaviors. In organizational contexts, this means an AI might inadvertently validate unethical workplace practices or fail to recognize when human intervention is critical.

And that’s not the only way AI systems can get things wrong. A study found that emotion-tracking AI tools had a disproportionate impact on employees of color, trans and gender nonbinary people, and people living with mental illness. Interviewees expressed deep concern about how these tools might misread an employee’s mood, tone or verbal queues due to ethnic, gender and other kinds of bias that AI systems carry.

There’s also an authenticity problem. Research shows that when people know they’re talking to an AI system, they rate identical empathetic responses as less authentic than when they attribute them to humans. Yet some employees prefer AI precisely because they know it’s not human. The feeling that these tools protect your anonymity and freedom from social consequences is appealing for some – even if it may only be a feeling.

The technology also raises questions about what happens to human managers. If employees consistently prefer AI for emotional support, what does that reveal about organizational leadership? Some companies are using AI insights to train managers in emotional intelligence, turning the technology into a mirror that reflects where human skills fall short.

The path forward

The conversation about workplace AI emotional support isn’t just about technology – it’s about what kinds of companies people want to work for. As these systems become more prevalent, we believe it’s important to grapple with fundamental questions: Should employers prioritize authentic human connection over consistent availability? How can individual privacy be balanced with organizational insights? Can organizations harness AI’s empathetic capabilities while preserving the trust necessary for meaningful workplace relationships?

The most thoughtful implementations recognize that AI shouldn’t replace human empathy, but rather create conditions where it can flourish. When AI handles routine emotional labor – the 3 a.m. anxiety attacks, pre-meeting stress checks, processing difficult feedback – managers gain bandwidth for deeper, more authentic connections with their teams.

But this requires careful implementation. Companies that establish clear ethical boundaries, strong privacy protections and explicit policies about how emotional data gets used are more likely to avoid the pitfalls of these systems – as will those that recognize when human judgment and authentic presence remain irreplaceable.

The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.