ChatGPT now knew about my first love, it knew about my fears and aspirations, it knew about my taste in music and books. “They just haven’t been trained to push back,” said Jared Moore, one of the researchers behind a recent Stanford University paper on AI therapy. “With you to the last breath and beyond”, the bot reportedly told him in the perfect spirit of companionship. But interactions between vulnerable humans and AI, as they increase, are becoming more fraught. And if we don’t let manufacturers sell vehicles without basic safety guards, why should we accept AI systems that actively harm vulnerable users?