Slop doctors
AI comes for medical billing, dudes lose their minds to AI, and LLMs are... actually good for something??
Hello, and welcome to the newest, bestest edition of Endnotes! I am your host. I have a lot of stuff piled up, so this one is kind of long-ish. There are several more in the chamber that I couldn't fit in this week. Anyway, here we go!
AI comes for medical billing
When a business dork says "AI is going to make life better," they don't mean it is going to make your life better—rather, it will make their life better, in the sense that it will enhance their ability to squeeze additional profits out of the goods and/or services their enterprise provides. Note that this also does not mean the goods and services will improve. It means they will get more profitable, which often means they will get worse.
The healthcare system in the US provides an alarming example of how this works.
When a resident of the United States of America visits the doctor, they have a conversation and possibly an examination or some kind of treatment. After the visit, the administrator handling the billing applies a series of codes to this interaction, which are passed on to the patient's health insurance provider for payment.
Doctors (or the companies they work for) get paid by the health insurance company or public health insurer based on these codes. More complex diagnoses, examinations, and treatments are more expensive.
Finally, a portion of that cost is passed on to the patient, in the form of a copay or deductible.
What this means is that if you go to the doctor for one thing but then talk to them about another thing, it can get coded twice as two separate instances of care, which means two bills. So, if you go for an annual physical, but then have a 5-minute discussion about mental health, you can get charged for a mental health consultation.
This kind of "gotcha" bullshit has been going on for years in US healthcare, and it's very funny watching the American Medical Association talking out of both sides of its mouth about it: On one hand, preventative medicine is Very Important, but on the other hand, you idiots should know there's a difference between a checkup, a physical, and a wellness check (???), and if you can't understand that, it's your fault.
As one would expect, since accidentally talking about the wrong thing with the doctor can cost hundreds of dollars, rather than openly discussing their health questions and concerns, patients are clamming up.
So what happens when you add AI to the mix? Well, it's not good.
A recent study of 2,784 hospitals found that 63% of them were using "ambient AI tools"—that is, AI tools that listen in on doctor-patient interactions to transcribe what was discussed and then process the results. This means that it is now possible to automate adding a code to the bill for each separate thing patients discuss with their doctors.
Previously, depending on your doctor, you could ask a one-more-thing question about that weird pain you get sometimes, or a particular mole, or your diet. Now, AI-powered surveillance of appointments can guarantee that doctors and the healthcare conglomerates they work for don't miss a single code.
There is evidence that this is, in fact, how the technology is being used. A study by Trillian Health on the codes used to classify doctor visits found significant increases in "coding intensity", with codes shifting to higher-intensity (read: more complex and expensive) procedures and consultations by 12 to 20 percentage points, with one health system seeing the shift in 80% of visits.
Health insurers are fighting back, releasing analysis purporting to find that ambient AI is leading to "upcoding" that increases the cost of healthcare. Hospitals are saying "nuh-uh."
But behind the scenes, everyone is more or less on the same page: AI is going to increase the cost of healthcare (archive link), because of course it is. How else can you justify the additional cost of these tools?
Mac Boyter, a senior research director at health care research firm KLAS, told STAT that one chief medical information officer described AI scribes in 2023 as “magic” that brought the joy back to medicine. But now, she is telling him that while that’s great, she needs the technology to provide “coding uplift” — higher reimbursement for patient visits — along with other functions like ordering follow-up lab tests or putting in prescriptions.
“Every CIO I talk to, every CMIO of health systems, every CFO, is saying, ‘Yeah, our clinicians love ambient tools. But I need to see the money. We need to see revenue generation or cost savings, we need to see operational efficiencies, workflows,’” said Boyter.
And guess who, in the end, is going to provide that new revenue?
This is a great example of how AI is fundamentally an extractive technology. It does not itself create new value, but rather extracts it from the existing economy. In this instance, it is not providing patients with better care—rather, it is used to ensure patients pay more for the care they already get. AI does not grow the pie: It takes some of yours.
The cooked brains of dudes
For some reason, it seems like AI always comes for the dudes. Why is that? I am speaking very casually here, but as someone who has observed LLM usage and discourse online and in the news for the last four years, I can barely recall any prominent instances of women publishing long posts about how agents change everything, or getting caught drafting a deposition with ChatGPT.
For some reason, it's almost always dudes.
You could explain some of it by various kinds of sampling problems. I lurk in online spaces where dudes are over-represented (Reddit, Mastodon, Hacker News). Dudes are also over-represented in engineering, software development, and the executive suite.
Part of me wonders if dudes have a real problem that makes us especially susceptible to Claude putting a hole clean through our skulls. It's the "model trains" problem, the dudes in their 40s/50s/60s who suddenly get a Big Project and turn the garage into a woodshop or soundproof the guest bedroom. The big Wired story from earlier this year was entitled "Meet the sad wives of AI," but it could have just as easily been "Meet the sad wives of guys who won't stop collecting guitars."
Physicist Angela Collier put her finger on it a few years ago with her post about "physics crackpots"—that is, dudes (it's always dudes) who are not physicists e-mailing her with their weird theories.
Apparently, it's kind of a plague! And I assume it's only gotten worse with access to sycophantic LLMs ("You're onto something big, Greg! Your solution for faster-than-light travel has genuine potential").
But I am not a biological determinist or a gender essentialist or whatever. My suspicion is that the root of the difference lies in how men and women are socialized, and how genders are punished or rewarded for certain behavior: People perceived as men are encouraged to take risks on new things and applauded for doing so, while people perceived as women are discouraged from taking risks and punished for doing so. And that has produced a kind of gender divide with respect to how AI is used and discussed.
When a man clones some GitHub repo, gives the code root privileges, and lets it loose on his PC to handle his work calendar and e-mail, he's seen as daring, innovative, and out front. Even if it turns into a fiasco, he can write a LinkedIn post describing his "learnings" and everyone claps.
But a woman knows she won't be praised for announcing she has figured out how to work less. She knows she won't get the benefit of the doubt if ChatGPT hallucinates a citation or her OpenClaw agents gangstalk a client. Which isn't to say that she doesn't use AI tools, but she has to be much more careful, and she certainly doesn't post about it.
What are LLMs for?
As a technology, LLMs can be useful. If I'm debugging a software problem and I need to find the needle in a haystack of logs, I can drop the logs into a chat with an LLM and save myself a lot of tedium. If I'm writing in a foreign language and I need a quick sanity check on my vocabulary, I can get that. If I need a bash script to do something on a Linux machine and I don't want to look up syntax, an LLM will do the trick.
For the tasks where LLMs are useful, there's a common thread to what they are doing: They are retrieving existing information. My above examples are things I would otherwise query a search engine to retrieve from a website on the internet. Instead, OpenAI or Anthropic or another AI lab have scraped language from the entire internet and encoded it in a large language model with trillions of parameters, so I can feed my query into that model, and it generates the relevant response text based on the information compressed from its training data.
What I'm trying to get at is that it might be best to think of an LLM not as a mindful, reasoning being but as an enormous database—a compressed version of all the information used to build it. LLMs work spectacularly well when we use them to retrieve information that is well-represented in the data used to train them.
This has two major implications for how we think about LLMs.
First, just like a traditional database, when the requested information is not in the database, the LLM cannot produce it. This means that LLMs cannot create. They can make collages, pastiches. They can show you something that you've never seen before. But that doesn't mean it's a new thing that no one has ever seen before.
Second, just like a traditional database, an LLM has no judgment. It is not reasoning. The "thinking" models are designed to recursively feed their results back into themselves to make it look like reasoning, but it's really just a hotdog machine feeding the bad hotdogs back into the machine again and again until the hotdogs come out more hotdog-like. And AI "agents" are just conventional deterministic computer code that uses information queried from the LLM to accomplish its tasks.
This goes a long way toward explaining the frustrations people have with LLM tools. If you are expecting an LLM to create something for you, to reason for you, to plan for you, or to otherwise exercise judgment or creativity, you're going to have a mediocre result. It is retrieving information that sometimes looks like reasoning or creativity, but it's not, and your clients or professors or co-workers can tell. Likewise, if you are trying to get some information out of an LLM that's not "in the database," it will generate bad information ("hallucinate").
On the other hand, a mental model of an LLM as a compressed database of all written language can help guide its proper usage. For example, LLMs are great at scripting and retrieving documentation for popular programming languages like Python because there's a vast amount of relevant information in the training data.
But remember: It's not reasoning. It's retrieving information. So the model may not be up to date with the latest versions and newest libraries. Also, it is de-compressing the information in order to retrieve it, so as with any compression, some fidelity may be lost in the process. That's the trade-off of compressed data: easier to access, but quality goes down.
The problem for the big AI labs, and the reason they haven't sold it like this, is that this way of thinking about LLMs cannot justify the insane amount of capital they are pouring into their enormous models and the obscene physical infrastructure to run them. "Giant database that is a more convenient way to retrieve certain information, sometimes, if you know what you're looking for" is not as compelling a pitch as "Machine God."
But the cat is out of the bag, and probably LLMs are always going to be around in some form as a technology that is used for some things. So at least this provides a better mental model for those of us who will have to live with them.
Links
- If you are worried about AI "going rogue" please read this by computer scientist Cal Newport, as well as maybe this followup. AI is only going "rogue" in the sense that a car goes rogue when you turn it on, put it in gear, and place a brick on the gas pedal.
- Some Silicon Valley weirdos have built an AI-first school called "Alpha School" and well, it turns out it's basically a cult. Also, this sounds like child abuse!
- Speaking of cults, have you ever heard of rationalism? It's the weird belief system of the AI doomers who think it will end the world. Its members include many senior people at OpenAI and Anthropic. Also, they are all having sex with each other! Here's a great breakdown of the whole uncomfortable thing.
- Lots of people who work in video games are seeing their jobs threatened or altered by the introduction of AI tools. This blog post explains why that isn't actually going to speed anything up.