Accountability and recourse: The challenges of building fair systems

Published On Sep 12, 2024

In the last century, an age of automation and mechanization has allowed humanity to scale to eight billion people, walk on the moon, and double human lifespan. In 1990, the World Bank Poverty and Inequality Platform estimated that 38% of the world lived below the poverty line; in 2024, less than 9% do. Since the U.S. started keeping productivity statistics in 1947, that country has become roughly 1% more productive every year. We created banks, public health, and education, in ways unique to every country. We regulated them. Despite all the ills in the world, it is getting much, much better for many, many people.

Screenshot 2024 09 11 at 1.17.17 pm

(Productivity in the United States since 1947)

In the last quarter-century, the changes became far more radical. Today, much 5.45 billion humans use the Internet, and when they do, 96% of them on a mobile device. Many of those who use them think nothing of this, even as we check weather via satellite, send high definition videos to friends halfway around the planet instantly, learn about any topic we can think of, have personalized directions that factor in realtime traffic and puts it on a free map that's read to us, and much more. For free. We even chat with a digital idiot savant that can code, counsel, and create better than almost all humans, in every language, but still has trouble drawing fingers properly.
 
Dall·e 2024 09 11 13.31.18   a Highly Futuristic City With Shiny Glass and Metal Buildings, now Incorporating Vibrant Colors Like Turquoise, Red, and Navy Tones as Highlights Thro
(AI generated image)
 
One of the big ideas that got us here was mass production. Henry Ford's assembly line ended bespoke car manufacturing and put a car in every garage. As every industry adopted standard, replaceable parts and bulk manufacturing, making things got cheaper and faster. The Ford River Rouge complex took more than a decade to build, and opened in 1928. It's 2.4 km by 1.6 km long, and has 93 buildings.
 
That might seem big, but today's assembly lines span the world. Think about an iPhone, perhaps the peak of consumer technology at any given time.
  • It contains chips, and software, and screens, sourced from the work of millions of people—miners, developers, testers, factory workers, product managers.
  • It gets to your pocket thanks to millions of others, such as truckdrivers and retail workers.
  • And of course, millions more who extract the gas to drive those trucks, or cook food for the retail workers.
  • Not to mention governments filled with public servants to keep it all moving as equitably as possible while averting disaster.
Modern assembly lines are vast, incomprehensible, and dynamic. They are emergent properties of unthinkably complex systems. We aren't really in charge of them: We work for them as much as they serve us.
 
But while a Model T Ford might be a simple enough product, there's far more to these systems than physical output. You don't just travel on a plane: you book a ticket on a website, pay with a credit card, book a taxi to the airport, get a refund for a cancelled flight, and so on. It's the automation of these things, as much as the automation of a factory, that has made our world possible. In the same way that Henry Ford's assembly line ended bespoke car manufacturing and put a car in every garage, this system put significant upgrades to health, wealth, ability and justice in many people's lives.
 
While it's easy to use the term "assembly line" because most people understand that concept, it's harder to give a name to these second system. Distribution line perhaps, but it's about far more than distribution—it's about all of the processes. And more of a mesh of processes than a linear one. Process mesh, then.
 
Car
(An AI generated image of a Model T Ford)
 
If you've read this far, you might think I'm a huge fan of globalization, mechanization, and automation. Dear reader, read on.
 
Mass production of the automobile had its downsides. Instead of the varied, colorful, and often ingenious designs of the car industry's early years, we now have trucks, sedans, and SUVs that all look identical. They're probably safer. They're not good at innovating.
 
This is true of every industry automation touches. Small ISPs and hosting providers have been replaced by the big tech firms. Artists make a fraction of what they once did thanks to streaming. Selection plummeted, but everyone got some. The corner store shut down when Wal-Mart came to town. The groceries were cheaper, but everyone had to work there.
 
That's a pretty good critique of globalism itself: The groceries were cheaper, but everyone had to work there.
 
The process mesh has a similar set of problems.
 
The purchase of an airplane ticket requires that dozens of organizations, each of which trusts the other, agrees that it will change a credit card balance in return for transporting someone from one place to another. That in turn requires objective facts, trust, interfaces, rules, liability, and more. And these things must happen, whenever possible, without human intervention. After all, the goal is automation.
 
This presents two big problems:
  1. Processes cannot be designed to handle every eventuality. Just as the assembly line only works when physical products are standardized, so the process mesh only works when almost everyone's needs are satisfied by a fairly small set of options. "Any color the customer wants, as long as it's black."
  2. Processes must outlive their creators. If a process cannot function without its creator, it is not resilient. Important processes like the banking or healthcare system cannot have a single point of failure, and therefore cannot rely on a single person. Which means every person is replaceable—and people quit, retire, or die all the time.
The process mesh, then, marginalizes a few people for the good of the many, yet nobody is in charge. Which means:
  1. The processes are complicated if you're poor. If you're wealthy enough, you pay someone to do taxes that are unfathomable to you. Meanwhile a single parent is filling out a dozen unnecessary forms to get food stamps.
  2. When a bad actor finds a loophole, it takes the system too long to fix it. You can't get that scammer to stop calling, and wait on hold for three hours to cancel a subscription that should take a single click.
  3. When a good actor falls through the cracks, bad things happen. It's often the most marginalized that aren't well served by the system. Even when there are ways to resolve an issue, those without access to modern technology are often left behind.
 
Dan Davies' extraordinary book The Unaccountability Machine is a long, hard look at how we got here, and why "unaccountability sinks" are behind much of our world's frustration and outright anger at bureaucracy, regulatory overreach, and the entirety of the process mesh that has lifted us up.
 
A human must, as a basic right, have recourse. Accountability for our actions is the basis of our entire judicial system, and yet we have built unaccountable processes.
 
This is a hard problem because any system of recourse will simply become another process in the system. Imagine there's an automated bank loan website. If an application for a loan is refused, the user should have recourse. Clicking a button would allow them to speak with a human who could explain the decision. But by adding a button would mean that the legitimately aggrieved and those legitimately disqualified would call, flooding the lines.
 
How do we design systems of recourse that won't be consumed by processes, itself becoming riddled with loopholes and unaccountable? The incredible mesh of processes threatens to suffocate us beneath its Byzantine folds. If we want to continue the human journey up and to the right, we must make it accountable.
Notice at collection Your Privacy Choices