Opaque Systems Are the Problem, Not Algorithms
As AI tools spread into more areas of life, concern is growing over software making decisions that affect us. Algorithmic social media feeds choose what we see. Automated systems help set insurance premiums. Even legal processes are seeing software suggest sentencing guidelines to judges.
These worries have fueled calls for restrictions, such as a recent New York push to limit how social media networks generate feeds for children. But curbing algorithms misses the point. A regulation forcing a reverse-chronological feed in place of a "smart" one ignores that a reverse-chronological feed is itself an algorithm. Software can make consequential decisions badly without any AI in the mix.
The governing principle should be different: decisions made by software must be explainable.
Why Explanation Beats Restriction
When a decision affects someone’s life, that person needs to understand how it was reached. The underlying data may be wrong. The logic may have a flaw worth questioning or escalating. Or the individual may simply need to know what to change to get a better result next time.
A personal example illustrates the point. A rental car returned to the same airport it came from still triggered a one-way fee exceeding 150% of the rental cost. Fighting the charge went nowhere; customer service denied the appeal without any explanation. The opacity cost time and goodwill—and ultimately cost the company the customer, who only recovered the fee through a credit card dispute. A simple explanation likely would have surfaced the error long before the company racked up expenses beyond the fee itself. The failure mechanism was not software versus AI; it was the refusal to explain.
For social media, this principle translates into a simple requirement: platforms should be able to show a user why a given post appears in their feed, and why it appears where it does. A reverse-chronological feed meets this test trivially. Any more sophisticated feed should meet it too.
The Limits of Current Systems
This is where modern AI stumbles. Traditional explicit logic can, at least in principle, be explained by examining source code and data. Most current AI tools cannot offer that kind of account. The unavailability of explanation is therefore good grounds to restrict their use, at least until explainability research matures. Such limits would also create an incentive to develop AI that can account for itself.
None of this calls for exhaustive justification behind every software-driven choice. Nobody should need a full pricing breakdown for a hotel room. But where a dispute arises—say, when two people consistently see different prices for identical goods—explainability becomes essential.
Keeping Humans Accountable
One practical consequence is a division of labor: an AI can propose options, but the human making the final call must be able to explain their reasoning independently of the machine’s suggestion. There is a real risk that people simply rubber-stamp computer outputs, but the principle makes clear that "the computer said so" is not a valid justification. Overly frequent agreement with AI recommendations should itself be treated as a warning sign.
There is also a productive role for opaque models: as tools for understanding decision processes, potentially replacing them with clearer logic. Expert Go players already study computer play to refine their own strategies, and similar approaches could help untangle convoluted legacy systems. AI may threaten to make decision-making less transparent, but with the right incentives, it can instead serve as a stepping stone to deeper human insight.
