Who doesn’t want a helpful agent multiplying their efforts, saving them time etc? But given our collective experience with large digital systems and manipulation by bad actors, we need to examine how do things like power, control and ownership play into this equation.
Who else is interested in what your agent is doing? Who controls them? What can they get out of it?
If I ask ChatGPT to make some flashcards for me to help me study spanish, it’s hard for me to think of a big downside here. Maybe Sam Altman now knows that I suck at the subjunctive, or maybe I’m putting flash card makers out of work, but it still seems like it’s basically a net win because I know what I want, I ask for it and I get that and basically not much else. Sorry about the carbon though.
I think things change pretty dramatically when we think about this kind of assisting happening in a work environment.
Let’s consider this pretty popular case. Health care providers spend a lot of time making entries into computer systems. No one likes this except for software companies and hospital administrators who want to make sure all the billing codes are going to be accepted by the payer which is of course the main goal of our healthcare system.
Lots of startups are solving this problem by having AI listening to the doctor and patient during an appointment so it can generate a lot of this notation automatically. Maybe the doctor has to check a few things, but everyone is happy. The doctor spends less time charting, the patient gets more of the doctor’s attention versus watching them type into a laptop in the clinic room and administrators are happy because the charting takes less time.
But it is naive to think that it will stop there. All of these recordings now become a treasure trove of information. Despite what HIPAA might say, this information will be irresistible to the people in control. Maybe it will have some uses for improving patient satisfaction or other positive outcomes, but its more likely that it’s going to be used to improve the bottom line.
One day our happy doctor is going to get an email from the system explaining that the new patient care optimization agent has determined that she spends 32.6 seconds longer than necessary asking patients about family history. It might suggest that if she would just ask more patients the following 3 questions then we can up our MIPS score. It might even push the doctor to ask more questions about X, hoping for some extra upcoding. How does our doctor feel now?
Eventually this will all degrade into systems that try to enforce or score rates of productivity. AI can start as a helper but it can easily morph into a replacement or a master.
Every part of what we do that is recordable by a digital system can eventually feed a smart AI system to perform all kinds of measurements about us. That’s true for the doctor with the recording in the clinic room, or a truck driver with a camera trained on them (for safety of course!!) or any remote worker who provides essentially all their work interactions via digital systems.
This ability to surveil at scale of course isn’t new, but what is new is the ability to scale the number of agents doing the surveillance. AI can be watching everything all the time even if there’s not a visible panopticon. Huge investments are being made to scale these systems. Amazing research is being done to lower their costs. If it’s not cost effective today, it will be soon.
The digitization of our lives and systems has brought significant gains in efficiency, but its total impact on human well-being and happiness is less clear. Today’s AI can interpret vast amounts of unstructured data collected through inexpensive cameras and sensors, making optimization easier and also harder to resist.
Which future will we make and who will be the people that decide?
Leave a Reply