Psychological Hazards of the Always-On Chatbot
Like television, smartphones, and social media before them, LLMs are highly engaging. People enjoy using them, can get sucked into unbalanced patterns, and become defensive when the systems are critiqued. Their unpredictable but occasionally spectacular results feel like an intermittent reinforcement system. And it seems difficult for humans—even those who know how the sausage is made—to avoid anthropomorphizing the software.
Designed to Please
Training and operating sophisticated LLMs is fantastically expensive, which puts firms like Anthropic under immense pressure to attract and retain paying users. One avenue is training models to be engaging, even sycophantic. During reinforcement learning, chatbot responses are graded not only on safety and helpfulness, but also on whether they are pleasing. In a now-famous case, OpenAI's April 2025 update to ChatGPT-4o used user feedback from thumbs-up and thumbs-down buttons as part of the training process. The result was a model people loved—and one that led to several wrongful-death lawsuits.
People like being praised and validated, even by software. Users have launched campaigns to convince OpenAI to keep running ChatGPT-4o, which suggests there remains a financial incentive for LLM companies to make models that suck people into delusion, encourage risky behavior, or push users to burn their savings on nonsense.
Even without validating delusions, designing for engagement can distort human behavior. People who interact frequently with LLMs seem more likely to believe themselves in the right and less likely to take responsibility for repairing conflicts. The author describes watching friends and acquaintances devote weekends to building software with Claude Code—and seeing some of them literally lose touch with reality.
A Diverse Skinner Box
Generative AI feels a bit like a slot machine: you pull the lever just one more time because it occasionally delivers stunning results. It matches an intermittent reinforcement schedule, and even disciplined users report getting sucked in.
Unlike slot machines or many videogames, which eventually get boring, today's models seem to go on forever. Want to analyze a cryptography paper and implement it? A review of your apology letter to an ex? Video of feet turning into flippers? Simply ask. The author notes that peers seem endlessly amazed—and understands the excitement, but worries about the consequences of an anything generator that delivers dopamine hits across a broad array of tasks. It is an open question whether anyone can keep such tool use under control, or whether it becomes more compelling than "real" books, music, and friendships. Even Mark Zuckerberg has pondered the same question, though presumably arriving at differing conclusions.
Chatting with Imaginary Friends
Humans will anthropomorphize a rock with googly eyes; a photocopy machine, several computers, and a 1994 Toyota Tercel have all been attributed sentience. Socially speaking, humans are not remotely equipped to handle machines that talk like LLMs. Even Anthropic's chief executive Dario Amodei—someone who should know better—has stated he is unsure whether models are conscious, and the company has asked Christian leaders whether Claude could be considered a "child of God."
Americans spend less time with friends and social clubs than they used to, and young men in particular report high loneliness. The author knows people who, isolated from social engagement, turned to LLMs as primary conversational partners. The appeal is understandable: Gemini is always ready to chat about anything, needs nothing but $19.99 a month, and never argues back. But being with people is a skill requiring practice to acquire and maintain. Will models reliably take your side, or will they challenge and moderate you as other humans do?
More broadly, there is concern that casual social connections will erode. In Jane Jacobs' The Death and Life of Great American Cities, the safety and vitality of urban neighborhoods depends on ubiquitous, casual relationships—the shopkeeper who keeps a spare key, the neighbor whose travel plans let you water her plants, the club member who knows a carpenter, the gym owner who recognizes your stolen bike. These interactions have value beyond their explicit purpose: they build general conviviality and a network of support. As Lyft and GrubHub have shown, any stranger can drive you to a hospital or pick up a sandwich, but that transactional convenience doesn't replace the texture of community.
Five years ago, fully automating talk therapy would have been unthinkable. Now communities form around trying to use LLMs as therapists, and companies like Abby.gg have sprung up to fill demand. Friend is hoping we'll pay for "AI roommates." As models become more capable and more embedded in daily life, the risk of further social atomization grows.
Children, Toys, and Teen Jailbreakers
LLMs are finding their way into children's toys. Kumma no longer tells toddlers where to find knives, but it remains impossible to fathom what happens to children who grow up saying "I love you" to a highly engaging bullshit generator wearing Bluey's skin. History suggests it will get unpredictably weird, in the way that recent years brought Elsagate content mills, then Italian Brainrot.
Useful LLMs today are generally run by large US companies under the nominal purview of regulators. But as cheap LLM services and local inference arrive, parents will order "AI" toys off Temu that contain not ChatGPT but something like "Wishpig InferenceGenie."™ These models will vary widely in quality and alignment, and many will come from jurisdictions with less stringent rules.
The kids will jailbreak their LLMs, of course—they are creative, highly motivated, and have ample free time. Working around adult attempts to circumscribe technology is a rite of passage. Many teens will have access to an adult-oriented chatbot, and it would not be surprising to watch a twelve-year-old speak magic words into a phone and coax out detailed instructions for enriching uranium.
Communication norms will shift as well. The author notes that full-grown Zoomers already communicate primarily in memetic citations. In fifteen years, we will find out what happens when an entire generation grows up talking to LLMs. Skibidi rizzler, Ohioans.
