Most respondents to this canvassing wrote brief reactions to this research question. However, a number of them wrote multilayered responses in a longer essay format. This essay section of the report is quite lengthy, so first we offer a sampler of a some of these essayists\u2019 comments.<\/p>\n\n\n\n
What follows is the full set of essays submitted by a number of leading experts responding to this survey.<\/p>\n\n\n\n
Paul Saffo<\/strong>, longtime Silicon Valley foresight guru, observed, \u201cWe have already turned the keys to nearly everything over to technology. The most important systems in our lives aren\u2019t the ones we see, but the systems we never notice \u2013 until they fail. This is not new. Consider the failure of the Galaxy IV satellite a quarter century ago: puzzled consumers who never noticed the little dishes sprouting atop gas stations discovered they couldn\u2019t fill their tank, get cash from ATMs, or watch their favorite cable TV programs.<\/p>\n\n\n\n \u201cWe have experienced 16 Moore\u2019s Law doublings since then. Our everyday dependence on technology has grown with even greater exponentiality. We carry supercomputers in our pockets, our homes have more smarts than a carrier battle group, and connectivity has become like oxygen \u2013 lose it for more than a few moments and we slip into digital unconsciousness, unable to so much as buy a latte, post a tweet or text a selfie.<\/p>\n\n\n\n \u201cTechnologists are optimists. They promise that the next wave of technology will solve the failings of prior innovations and make glitches a thing of the past. Empowered by AI, Richard Brautigan\u2019s \u2018machines of loving grace\u2019<\/a> will keep omniscient watch over our lives in a harmonious cybernetic meadow. There is no reason why the next technological wave can\u2019t expand human agency, giving us greater satisfaction and control. It is just a matter of design. Or, rather, if it was just a matter of design, the now ubiquitous spell-checkers that so annoy us would actually be helpful \u2013 and come with an off switch to flick when they weren\u2019t. This is just a minor example, but if we can\u2019t make the small, simple stuff work for us, how will more complex systems ever live up to our expectations?<\/p>\n\n\n\n No matter how brilliant AIs, avatars and bots become, they will never be truly autonomous. They will always work for someone \u2013 and that someone will be their boss and not you, the hapless user.<\/p>Paul Saffo, longtime Silicon Valley foresight guru<\/cite><\/blockquote><\/figure>\n\n\n\n \u201cBut don\u2019t blame the machines. No matter how brilliant AIs, avatars and bots become, they will never be truly autonomous. They will always work for someone \u2013 and that someone will be their boss and not you, the hapless user. Consider Uber or any of the other mobility services: In theory, their ever more brilliant algorithms should be working tirelessly to enhance the customer experience and driver income. Instead, they answer to their corporate minders, coldly calculating how far fares can be boosted before the customer walks \u2013 and how much can be salami-sliced off the driver\u2019s margin before they refuse to drive.<\/p>\n\n\n\n \u201cNearly a century ago, Will Durant observed<\/a> that \u2018history reports that the men who can manage men manage the men who can manage only things, and the men who can manage money manage all.\u2019 If Durant were here today, he would surely recognize that those who manage our synthetic intelligences will inevitably become the ones who manage all. And they will instruct their intelligences to grant you just enough agency to keep you from noticing your captivity.\u201d<\/p>\n\n\n\n Raymond Perrault<\/strong>, a distinguished computer scientist at SRI International (he directed the AI Center there from 1988-2017), wrote, \u201cCurrent AI systems based on machine learning are showing continued improvement on tasks where large amounts of training data are available. However, they are still limited by their relative inability to incorporate and interact with symbolic information.<\/p>\n\n\n\n \u201cThe role of symbolic information and reasoning is one of the major outstanding questions in AI, and there are very different opinions as to whether and how integration should be achieved. I believe that robust, verifiable AI systems, needed for high-reliability systems such as self-driving cars, depend on progress in this area and that this technical problem will eventually be solved, though whether that will be sufficient to field high-reliability systems remains to be seen. I accept that it will, but I don\u2019t know when.<\/p>\n\n\n\n \u201cAI is and will continue to be used in two kinds of scenarios, those where the AI operates completely autonomously, as in recommender systems and those where humans are in ultimate control over the decisions suggested by the AI, and as in medical diagnostics and weapons. The higher the risk of the AI system being wrong and the higher the consequences of a bad decision, the more important it is for humans to be in control.<\/p>\n\n\n\n \u201cLet\u2019s look at a few of the main categories where that sorting will likely occur:<\/p>\n\n\n\n Jamais Cascio<\/strong>, distinguished fellow at the Institute for the Future, predicted, \u201cSeveral scenarios will likely coexist in the future of agency by 2035.<\/p>\n\n\n\n 1) \u201cHumans believe they are in control but they are not:<\/strong> The most commonly found scenario will be the one in which humans believe themselves to be in control of important decision-making in the year 2035, but they\u2019re wrong. This will (largely) not be due to nefarious action on the part of rogue AI or evil programmers, but simply due to the narrowing of choice that will be part of the still-fairly-simple AI systems in 2035. Humans will have full control over which option to take, but the array of available options will be limited to those provided by the relevant systems. Sometimes choices will be absent because they\u2019re \u2018obviously wrong.\u2019 Sometimes choices will be absent because they\u2019re not readily translated into computer code. Sometimes choices will be absent because the systems designed to gather up information to offer the most relevant and useful options are insufficient.<\/p>\n\n\n\n \u201cIn this scenario, as long as the systems allow for human override to do something off-menu, the impact to agency can be minor. If it\u2019s not clear (or not possible) that humans can do something else, partial agency may be little better than no agency at all.<\/p>\n\n\n\n 2) \u201cHumans know they are not in control and they\u2019re OK with that:<\/strong> Less common will be the scenario where humansdo NOT believethemselves to be in control of important decision-making in the year 2035 and they like it that way. Humans are, as a general rule, terrible at making complex or long-term decisions. The list of cognitive biases is long, as is the list of historical examples of how bad decision-making by human actors have led to outright disaster. If a society has sufficient trust and experience with machine decision-making, it may decide to give the choices made by AI and autonomous systems greater weight.<\/p>\n\n\n\n \u201cThis would not be advisable with current autonomous and AI systems, but much can happen in a decade or so. There may be examples of AI systems giving warnings that go unheeded due to human cognitive errors or biases, or controlled situations where the outcomes of human vs. machine decisions can be compared, in this case to the AI\u2019s benefit. Advocates of this scenario would argue that, in many ways, we already live in a world much like this \u2013 only the autonomous systems that make decisions for us are the emergent results of corporate rules, regulations and myriad minor choices that all add up to outcomes that do not reflect human agency. They just don\u2019t yet have a digital face.<\/p>\n\n\n\n 3) \u201cA limited number of AI-augmented humans have control:<\/strong> Last is a scenario that will somewhat muddy the story around human agency, as it\u2019s a scenario in which humans do have control over important decision-making in the year 2035, but it\u2019s a very small number of humans, likely with AI augmentations. Over the past few decades, technologies have vastly extended individuals\u2019 power. Although this typically means extended in scale, where human capabilities become essentially superhuman, it can also mean extended in scope, where a single or small number of humans can do what once took dozens, hundreds, or even thousands of people. By 2035, we\u2019ll likely see some development of wearable augmentations that work seamlessly in concert with their humans; whether or not we think of that person as a cyborg comes down to language fashion. Regardless, the number of people needed to make massive life-or-death decisions shrinks, and the humans who retain that power do so with significant machine backup.<\/p>\n\n\n\n \u201cThis may sound the most fantastical of the three, but we\u2019re already seeing signals pointing to it. Information and communication systems make it easy to run critical decisions up the chain of command, taking the yes-or-no choice out of the hands of a low-ranking person and giving it to the person tasked with that level of responsibility. Asking the president for authorization to fire a weapon is just a text message away. Whether or not we go as far as cyborg augmentation, the humans-plus-AI model (as described by Kevin Kelly<\/a> as \u2018centaurs,\u2019 his name for future people who use artificial intelligence to complement their thinking) will deeply enmesh decision-making processes. Advocates will say that it leads to better outcomes by taking the best parts of human and machine; critics will say that the reality is quite the opposite.<\/p>\n\n\n\n \u201cFor these scenarios, the canonical \u2018important decision-making\u2019 I\u2019ve had in my head regards military operations, as that is the topic that gets the most attention (and triggers the most unrest). All three of the scenarios play out differently.<\/p>\n\n\n\n \u201cOutside of these scenarios there are some key factors in common. The primary advantage to AI or autonomous decision-making is speed, with machines generally able to take action far faster than can a human (e.g., algorithmic trading). In competitive situations where first-mover advantage is overwhelming, there will be a continued bias toward AI taking charge, with likely diminishing amounts of human guidance over time.<\/p>\n\n\n\n \u201cAnother advantage of AI is an imperviousness to tedium, meaning that an AI can undertake the same repeated action indefinitely or pore over terabytes of low-content data to find patterns or anomalies, and give the final pass as much attention as the first. An amount or diversity of information that would be overwhelming to a human could easily be within the capacity of an intentionally designed AI. When decisions can be made more precisely or accurately with more information, machine systems will likely become the decision-makers.<\/p>\n\n\n\n \u201cThe most unusual advantage of AI is ubiquity. If an AI system can make better (or at least useful) decisions, it does not need to be limited to the bowels of the Pentagon. Arguably, a military where every human soldier has AI \u2018topsight\u2019 that can see the larger dimensions of the conflict is more effective than one that has to rely on a chain of command or potentially biased human decision-making in the field. More broadly, a decision-making system that proves the most insightful or nuanced or aggressive or whatever can be replicated across all of the distributed AIs. If they\u2019re learning systems, all the better \u2013 lessons learned by one can very rapidly become lessons learned by them all.<\/p>\n\n\n\n \u201cI suggested at the outset that the conditions of 2045 will likely differ significantly from the world of 2035. The world of mid-century would be an evolution of the world we made in the previous couple of decades. By 2045<\/em>, I suspect that our three scenarios would be the following:<\/p>\n\n\n\n \u201cOf course, there\u2019s always some aspects of the #1 scenario across issue areas \u2013 the Miserable Muddle. Stuff doesn\u2019t work exactly as we\u2019d like, but we can get enough done to carry on with it. People in power always change, but day-to-day demands (food, shelter, entertainment) don\u2019t. Humans just keep going, no matter what.\u201d<\/p>\n\n\n\n Andre Brock<\/strong>, associate professor of literature, media and communication at Georgia Tech and advisor to the Center for Critical Race Digital Studies, wrote, \u201cIn 2035, automated decision-making will largely resemble the robo-signing foreclosure courts of the 2020s, where algorithms tuned to the profit\/governance models of extraction and exploitation are integrated into legal mechanisms for enhancing the profits of large corporations.<\/p>\n\n\n\n \u201cMy grounds for this extraordinary claim draw upon my observations about how governments have been captured by private\/business entities, meaning that any pretense of equity based on the recognition of the \u2018human\u2019 has begun being supplanted by what Heidegger deemed humanity\u2019s future as a \u2018standing reserve\u2019 of technology.<\/p>\n\n\n\n \u201cMany decisions affecting everyday life for those in need of equity and justice already are framed through anti-blackness and extractive models; I\u2019m specifically focused on the United States, whose \u2018democratic\u2019 government was conceptualized by white men who worshiped property, owned Black folk, destroyed entire indigenous populations and denied women the vote.<\/p>\n\n\n\n \u201cDecision-making, from this perspective, largely benefits the political and economic interests of particular interests who fight savagely to retrench the gains made by Black folk, Asian folk, queer folk, women and the differently abled. There is no inherent democratic potential in information or algorithmic technologies designed to counter these interests, as the creators are themselves part of a propertied, monied, raced and sexualized elite.<\/p>\n\n\n\n \u201cIf anything, rolling out tech-abetted autonomous decision will further entrench the prevailing power structures, with possibilities for resistance or even equitable participation left to those who manage to construct alternate socialities and collectives on the margins.<\/p>\n\n\n\n \u201cI\u2019m intrigued by your question \u2018What key decisions will be mostly automated?\u2019 I feel that \u2018key decisions\u2019 is a phrase often understood as life-changing moments such as the purchase of a home, or what career one will pursue, or whether to become romantically involved with a possible life partner. Instead, I urge you to consider that key decisions are instead the banal choices made about<\/em> us as we navigate society:<\/p>\n\n\n\n \u201cThese decisions currently rely upon human input, but the human point of contact is often culturally apprehended by the institutions through which these decisions are framed. I\u2019m already uncomfortable with how these decisions are made; technology will not save us.\u201d<\/p>\n\n\n\n Alf Rehn<\/strong>, professor of innovation, design and management at the University of Southern Denmark, observed, \u201cWe need AIs that are less \u2018Minority Report\u2019 and more of a sage uncle, less decision-makers than they are reminders of what might be and what might go wrong.<\/p>\n\n\n\n \u201cI do believe that \u2013 yes \u2013 humans will still be making the big decisions, and if things play out well we may have algorithms that help us do that in more considered, ever-wise ways.<\/p>\n\n\n\n \u201cWhen it comes to the obvious issues \u2013 making decisions about immediate life or death, peace or war, and the most impactful laws \u2013 I think we humans will always insist on having our hand on the throttle or finger on the button. The trouble will more likely start brewing in smaller things, decisions we may think are best handled by algorithmic logics, and where we may lack an understanding of long-term consequences.<\/p>\n\n\n\n \u201cTake research funding and innovation projects, for instance. These may seem like things that are best handled \u2018objectively,\u2019 with data, and could be an area where we are fairly open to leaving some of our agency to, e.g., an AI system. At the same time, these are often things where the smart move is to fund longshots, things where you have to rely on intuition and imagination more than historical data.<\/p>\n\n\n\n \u201cOr consider things such as education. We have already made things such as school districts and university admittance partially automated, and there seems to be a desire to let assumedly rational systems make decisions about who goes where and who gets to study what. Whilst there might be benefits to this, e.g., lessening bias, these also represent decisions that can affect people for decades and have impacts generations into the future.<\/p>\n\n\n\n \u201cThe key issue, then, might be less one of any sort of change in what is perceived as agency, and more one about the short term versus the longer term. We might lose some agency when we let smart machines pick the soundtrack to our everyday life or do some of our shopping for us without asking too many questions beforehand.<\/p>\n\n\n\n \u201cSure, we might get some dud songs and some tofu when we wanted steak, but this will not have a long-term impact on us, and we can try to teach the algorithm better.<\/p>\n\n\n\n \u201cAllowing an algorithm to make choices where it might be impossible to tell what the long-term effects will be? This is an altogether different matter. We\u2019ve already seen how filter bubbles can create strange effects \u2013 polarized politics and conspiracy theories. As smart machines get more efficient there is the risk that we allow them to make decisions that may impact human life for far longer than we realize, and this needs to be addressed.<\/p>\n\n\n\n \u201cWe need to pay heed to injections of bad data into decision-making and the weaponization of filtering. That said, such effects are already seen quite often in the here and now. Our perspective needs to shift from the now to address what may come in years and decades.<\/p>\n\n\n\n \u201cWe don\u2019t need to fear the machines, but we need to become better at understanding the long-term implications of decisions. Here, in a fun twist, algorithms might be our best friends, if smartly deployed. Instead of using smart machines to make decisions for us, we need to utilize them to scan the likely future impact of decisions.\u201d<\/p>\n\n\n\n Barry Chudakov<\/strong>, founder and principal, Sertain Research, wrote, \u201cBefore we address concerns about turning the keys to nearly everything over to technology, including life-and-death decisions, it is worthwhile to consider that humanity evolved only recently to its current state after hundreds of thousands of years of existence.\u201d<\/p>\n\n\n\n \u201cThe Open Education Sociology Dictionary defines agency<\/a> as \u2018the capacity of an individual to actively and independently choose and to affect change; free will or self-determination.<\/em>\u2019For much of human history, individual agency was not the norm. David Wengrow and David Graeber asked in \u2018The Dawn of Everything<\/a>\u2019: \u2018How did we come to treat eminence and subservience not as temporary expedients, \u2026 but as inescapable elements of the human condition?\u2019 In a review of that book<\/a>, Timothy Burke argues, \u2018An association between small-scale foraging societies and egalitarian norms is robust. \u2026 If we are to understand human beings as active agents in shaping societies, then applying that concept to societies at any scale that have structures and practices of domination, hierarchy and aggression should be as important as noting that these societies are neither typical nor inevitable.\u2019 \u2026<\/p>\n\n\n\n \u201cWithin the context of limited liberal democracies, human agency took a quantum leap with the advent of computers and the smartphone. Today via podcast, YouTube, Snap, TikTok or an appearance on CNN, a Greta Thunberg or Felix Finkbeiner can step out of the shadows to fight for climate change or any other issue. Today humans have a host of tools, from cellphones to laptops to Alexa and Siri to digital twins. These tools are still primitive compared to what\u2019s coming. They don\u2019t only provide opportunities. They can also usurp agency, as when a person driving looks down at a text ping and crashes the car, even ending their life.<\/p>\n\n\n\n \u201cWe cannot fully grasp the recency of the agency we have gained, nor the encroachments to that agency that new tools represent. In concert with understanding this, we come to the startling realization \u2013 the acknowledgment \u2013 that today we are not alone; our agency is now shared with our tools<\/em>.<\/p>Barry Chudakov, founder and principal, Sertain Research<\/cite><\/blockquote><\/figure>\n\n\n\n \u201cWe cannot fully grasp the recency of the agency we have gained, nor the encroachments to that agency that new tools represent. In concert with understanding this, we come to the startling realization \u2013 the acknowledgment \u2013 that today we are not alone; our agency is now shared with our tools<\/em>. \u2026 Technology outpaced our awareness of the effects of technology gadgets and devices. For most of us who use these tools, agency today is impinged, compromised, usurped and ultimately blended with a host of tools. This is the new baseline.<\/p>\n\n\n\n \u201cSeeing agency as shared compels response and responsibility. If people are to remain in charge of the most relevant parts of their own lives and their own choices, it is imperative to realize that as we more deeply embrace new technologies to augment, improve and streamline our lives, we are not outsourcing some decision-making and autonomy to digital tools; we are using tools \u2013 as we always have done \u2013 to extend our senses, to share our thinking and responses with these tools. We have done this with alphabets and cameras, computers and videos, cellphones and Siri.<\/p>\n\n\n\n \u201cWe are facing a huge paradigm shift with the advent of new technologies and AI and machine learning. We need to reconfigure our education and learning to teach and incorporate tool logic. Anticipating tool consequences must become as basic and foundational as reading or numeracy. We can no longer thoughtlessly pick up and use, or hand over to children, devices and technologies that have the ability (potential or actual) to alter how we think and behave.<\/p>\n\n\n\n \u201cAgency has no meaning if we are unaware. There is no agency in blindness; agency entails seeing and understanding. From kindergarten to postgraduate studies, we need students and watchers who are monitoring surveillance capitalism, algorithm targeting, software tracking, user concentration and patterns and a host of other issues.\u201d<\/p>\n\n\n\n \u201cConsidering agency from this perspective requires a rethink and reexamination of our natures, our behaviors and the subliminal forces that are at work when we pick up technology gadgets and devices. As Daniel Kahneman<\/a> wrote, \u2018Conflict between an automatic reaction and an intention to control it is common in our lives.\u2019 We have little choice but to become more conscious of our reactions and responses when we engage with smart machines, bots and systems powered mostly by autonomous and artificial intelligence (AI).<\/p>\n\n\n\n Stephen Hawking said of AI and human agency<\/a>, \u2018The real risk with AI isn\u2019t malice, but competence. A super-intelligent AI will be extremely good at accomplishing its goals, and if those goals aren\u2019t aligned with ours, we\u2019re in trouble.\u2019 Our goals start with understanding how humans mostly unconsciously adopt the logic of tools and devices. \u2026 We are now \u2013 and we will be much more so in the future \u2013 co-creators with our tools; we think with our tools; we act with our tools; we are monitored by them; we entrain with their logic. This is a restatement of agency for those who claim the line from \u2018Invictus<\/a>,\u2019 \u2018I am the master of my fate: I am the captain of my soul.\u2019 Actually, our technologies are at the helm with us. \u2026<\/p>\n\n\n\n \u201cLike technology itself, agency is complicated. The short history of modern technology is the history of human agency encroached upon by tools that include ever greater \u2018intelligence. The Kodak Brownie camera, a cardboard tool released in 1900, had no computing power built into it; today\u2019s digital SLR has a raft of metadata that can \u2018take over\u2019 your camera, or simply inform you regarding many dimensions of light intensity, distance, aperture or shutter speed. In this instance, and in many others like it, humans choose the agency they wish to exert. That is true of computers, cellphones, video games or digital twins. We must now become more nuanced about that choice and shun simplistic encapsulations. As the website AI Myths<\/a> notes:<\/p>\n\n\n\n \u2018No AI system, no matter how complex or \u2018deep\u2019 its architecture may be, pulls its predictions and outputs out of thin air. All AI systems are designed by humans, are programmed and calibrated to achieve certain results, and the outputs they provide are therefore the result of multiple human decisions.\u2019<\/em><\/p>\n<\/div>\n\n\n\n \u201cBut how many of us are aware of that programming or calibration? Unless we acknowledge how our agency is affected by a host of newer tools \u2013 and will be affected to an even greater extent by tools now in the works \u2013 our sense of agency is misguided. Our thinking about and assumptions of agency will be erroneous unless we acknowledge that we share agency with these new tools. \u2026<\/p>\n\n\n\n \u201cThat\u2019s not all. We are capable of creating new beings. Yuval Noah Harari<\/a> says, \u2018We are breaking out of the organic realm and starting to create the first inorganic beings in the history of life.\u2019 These alt beings will further confound our sense of agency. Along with a question of our proprioception \u2013 where does our body start and end as we take ourselves into the metaverse or omniverse \u2013 inorganic beings will force us to ask, \u2018what is real?\u2019 and \u2018what does real mean anymore?\u2019 Will people opt for convenience, romanced by entertainment, and allow the gadgetry of technology to run roughshod over their intentions and eventually their rights?<\/p>\n\n\n\n \u201cThe answer to those questions becomes an issue of design informed by moral awareness. Technology must, at some level, be designed not to bypass human agency but to remind, encourage and reward it. Software and technology need to become self- and other-aware, to become consequence-aware.<\/p>\n\n\n\n \u201cTechnology seduction is a real issue; without engendering techno-nag, humans must confront AI with HI \u2013 human intelligence<\/em>. Humans must step up to embrace and realize the potential and consequences of living in a world where AI can enhance and assist. Partnering with artificial intelligence should be an expansion of human intelligence, not an abdication of it.\u201d<\/p>\n\n\n\n danah boyd<\/strong>, founder of the Data & Society Research Institute and principal researcher at Microsoft, complained, \u201cOf course there will be technologies that are designed to usurp human decision-making. This has already taken place. Many of the autopilot features utilized in aviation were designed for precisely this, starting in the 1970s; recent ones have presumed the pilot to be too stupid to take the system back. (See cultural anthropologist Madeleine Elish\u2019s<\/a> work on this.)<\/p>\n\n\n\n \u201cWe interface every day with systems that prevent us from making a range of decisions. Hell, the forced-choice, yes-no format of this survey question constrained my agency. Many tools in workplace contexts are designed to presume that managers should have power over workers; they exist to constrain human agency.<\/p>\n\n\n\n What matters in all of these systems is power. Who has power over whom? Who has the power to shape technologies to reinforce that structure of power? But this does not mean that ALL systems will be designed to override human agency in important decisions. <\/p>danah boyd, founder of the Data & Society Research Institute and principal researcher at Microsoft<\/cite><\/blockquote><\/figure>\n\n\n\n \u201cWhat matters in all of these systems is power. Who has power over whom? Who has the power to shape technologies to reinforce that structure of power? But this does not mean that ALL systems will be designed to override human agency in important decisions. Automated systems will not control my decision to love, for example. That doesn\u2019t mean that systems of power can\u2019t constrain that. The state has long asserted power over marriage, and families have long constrained love in key ways.<\/p>\n\n\n\n \u201cAny fantasy that all decisions will be determined by automated technologies is science fiction. To be clear, all decisions are shaped (not determined!) by social dynamics, including law, social norms, economics, politics, etc.<\/p>\n\n\n\n \u201cTechnologies are not deterministic. Technologies make certain futures easier and certain futures harder, but they do not determine those futures. Humans \u2013 especially humans with power \u2013 can leverage technology to increase or decrease the likelihood of certain futures by mixing technology and authority. But that does not eliminate resistance, even if it makes resistance more costly.<\/p>\n\n\n\n \u201cFrankly, focusing on which decisions are automated misses the point. The key issue is who has power within a society and how can they leverage these technologies to maximize the likelihood that the futures they seek will come to pass.<\/p>\n\n\n\n \u201cThe questions for all of us are: 1) How do we feel about the futures defined by the powerful, and 2) How do we respond to those mechanisms of power? And, more abstractly: 3) What structures of governance do we want to invest in to help shape that configuration?\u201d<\/p>\n\n\n\n Maggie Jackson<\/strong>, award-winning journalist, social critic and author, commented, \u201cUnless urgent steps are taken to protect human autonomy in our relations with AI, human agency in the future will be seriously limited by increasingly powerful intelligences other than our own. I see the danger arising from both humanity\u2019s innate weaknesses and from the unintended consequences of how AI is constructed.<\/p>\n\n\n\n \u201cOne point of vulnerability for human agency stems from how standard AI has been formulated. As AI pioneer Stuart Russell has brilliantly noted<\/a>, we have created AI systems that have one overarching goal: to fulfill the objectives that humans specify. Through reinforcement learning, the machine is given a goal and must solve this objective however it can. As AI becomes more powerful, its foundational motivation becomes dangerous for two reasons.<\/p>\n\n\n\n 1) People can\u2019t know completely and perfectly what a good objective is; AI doesn\u2019t account for a device or a person\u2019s interactions within an unpredictable world.<\/p>\n\n\n\n 2) A machine that seeks to fulfill a specific objective however it can\/will stop at nothing \u2013 even dismantling its off switch \u2013 in order to attain its goal, i.e., \u2018reward.\u2019 The implications are chilling.<\/p>\n\n\n\n \u201cConsider the case of using AI to replace human decision-making. AI is increasingly used to diagnose health problems such as tumors, to filter job candidates, and to filter and shape what people view on social media via recommender systems. While attention has rightly been drawn to the innate bias that is invested in AI, a larger danger is that AI has been created to solely to maximize click-through or other similarly narrow objectives.<\/p>\n\n\n\n \u201cIn order to maximize their goals, algorithms try to shape the world, i.e., the human user, to become more predictable and hence more willing to be shaped by the AI system. Social media and search engines, for instance, aren\u2019t giving people what they want as much as modifying users with every click to bend to the goals they were created to pursue. And the more capable AI becomes, the more it \u2018will be able to mess with the world\u2019 in order to pursue its goals, write Russell and colleagues in a recent paper on AI\u2019s future<\/a>. \u2018We are setting up a chess match between ourselves and the machines with the fate of the world as the prize. We don\u2019t want to be in that chess match.\u2019 The result may be a form of chilling human enfeeblement, a dependence on powerful devices coupled with an indifference to this imbalance of power. It\u2019s a mark of the seriousness of AI\u2019s perils that leading scientists are openly discussing the possibility of this enfeeblement or \u2018Wall-E problem\u2019 (the movie of that name that portrayed humans as unwittingly infantilized by their all-powerful devices).<\/p>\n\n\n\n \u201cA second point of vulnerability can be found in the rising use of caregiver robots. Simple robots are used mainly with vulnerable populations whose capacity to protect their cognitive and physical agency is already compromised. Robots now remind sick and elderly people to take their medicines; comfort sick children in hospitals; tutor autistic youth and provide companionship to seniors. Such \u2018care\u2019 seems like a promising use for what I call \u2018AI with a face.\u2019 But humanity\u2019s proven willingness to attribute agency to and to develop intense social feelings for simple robots and even for faceless AI such as Siri is perilous. People mourn \u2018sick\u2019 Roombas, name and dress their health care assistants, and see reciprocity of social emotions such as care where none exists. As well, patients\u2019 quick willingness to cede responsibility to a robot counters progress in creating patient-centered care.<\/p>\n\n\n\n \u201cWhile studies show that a majority of Americans don\u2019t want a robot caregiver, forces such as the for-profit model of the industry, the traditional myopia of designers, and the potential for people with less voice in health care to be coerced into accepting such care mean that public reservations likely will be ignored. In sum, human autonomy is threatened by rising calls to use caregiver robots for the people whose freedom and dignity may be most threatened by their use. I am heartened by the urgent discussions concerning ethical AI ongoing around the world and by rising public skepticism \u2013 at least compared with a decade or so \u2013 of technology in general. But I am concerned that the current rapid devaluation of human agency inherent in AI as it is used today is largely absent from public conversation.<\/p>\n\n\n\n \u201cWhen it comes to human agency and survival, we are already deeply in play in the chess match of our lives \u2013 and we must not cede the next move and the next and the next to powerful intelligences that we have created but are increasingly unable to control.\u201d<\/p>\n\n\n\n Maja Vujovic<\/strong>, owner\/director of Compass Communications and editor of the Majazine, based in Belgrade, Serbia, wrote, \u201cWhether we are ready or not, we must find ways to restore our control over our digital technology. If we don\u2019t build user interfaces with a large button, simple keyword or short voice command for clearly separating what we agree to give out willingly (that which can be used) and what we don\u2019t (which is off limits), then we\u2019re just dumb. And doomed.<\/p>\n\n\n\n \u201cLet\u2019s look at the larger picture. We don\u2019t need to wait for 2035 to automate our world. We can already patch a half a dozen applets, get our smart fridge to converse with our grocery app and link them both to our pay-enabled smart phone and a delivery service; they could restock our pantry on their own, every week. Yes, in the coming years, we will happily delegate such decisions in this interim period, when a sea of compute power will have to match an ocean of tiny parameters to just propose our next beach read or our late-night dinner-on-wheels.<\/p>\n\n\n\n \u201cBut wait! A nosy wearable will sound an alarm about that late-night meal intent and might even independently report it to our family doctor and to our health insurer. Our life insurance plan might also get \u2018upgraded\u2019 to a steeper premium, which our smart bank would automatically approve and honour every month. We might then also lose points on our gym score list, which could trigger a deserved bump of our next month\u2019s membership fee, as a lesson.<\/p>\n\n\n\n \u201cAnd just as we use our Lessons Learned app to proscribe late-night eating (because it makes us sick in more ways than one), we could see a popup flash before us, with a prompt: \u2018Over three million of your look-alike peers voted for this candidate in the last election. She fights to protect our privacy, empowers disadvantaged groups and leads a healthy life \u2013 no late-night meals in her house! Would you join your peers now and cast your vote, quickly and confidentially?\u2019<\/p>\n\n\n\n \u201cAll of this seems not implausible. The systems invoked above would work for each of us as users \u2013 we are their \u2018Player One.\u2019 Alas, there are also those systems that we are rarely aware of, where we are not users, but items. Any of those systems could \u2013 right now \u2013 be assessing our credit or dwelling application. Some applicant-tracking systems already blindly filter out certain job candidates or education seekers. Airbnb, hotels and casinos filter out unruly guests. In some countries of Europe, Middle East and Asia, authorities use facial recognition (de facto, though not always de jure) to keep tabs on their perceived opponents. It\u2019s chilling to see the U.S. on the brink beyond which a patronizing governmental body or a cautious medical facility could filter out and penalize people based on their personal life choices.<\/p>\n\n\n\n \u201cThe technology to generate all kinds of recommendations already exists and is in use, often in ways that are not best for us. What is conspicuously lacking is real utilities, built for our benefit. Perhaps we might have a say in evaluating those who work for us: professors, civil servants, police officers, politicians, presidents. In fact, electoral voting systems might be equipped with a shrewd AI layer, Tinder-style: swipe left for impeachment; swipe right for second term.<\/p>\n\n\n\n \u201cOne reason more useful public-input recommender systems are not widely available is that they haven\u2019t been successfully built and deployed. All other recommender systems have backers. We, the people, could try using Kickstarter to crowdfund our own.<\/p>\n\n\n\n \u201cWe can and will draft and pass laws that will limit the ability of technological solutions to decide too many things for us. In the coming decade, we will simply need to balance those two coding capacities of ours \u2013 one based on numbers, the other on letters. That\u2019s a level of \u2018programming\u2019 that non-techies are able to do to put technology (or any unbridled power, for that matter) on a short leash. That interface has existed for several millennia; in fact, it was our first coding experience: regulation.<\/p>\n\n\n\n \u201cThere are already initiatives. An example is California\u2019s \u2018Kids\u2019 Code\u2019<\/a> (an age-appropriate-design code) that incorporates youth voices and energy. It shows that legislators and users possess impressive maturity around human-computer interaction and its risks, though the tech industry may appear unfazed, for now.\u201d<\/p>\n\n\n\n Ben Shneiderman<\/strong>, widely respected human-computer interaction pioneer and author of \u201cHuman-Centered AI<\/a>,\u201d wrote, \u201cIncreasing automation does not necessitate less human control. The growing recognition is that designers can increase automation of certain subtasks so as to give humans greater control over the outcomes. Computers can be used when they are reliable, safe and trustworthy while preserving human control over essential decisions, clarifying human responsibility for outcomes and enabling creative use by humans. This is the lesson of digital cameras, navigation and thousands of other apps. While rapid performance is needed in some tasks, meaningful human control remains the governing doctrine for design. As automation increases, so does the need for audit trails for retrospective analysis of failures, independent oversight and open reporting of incidents.\u201d<\/p>\n\n\n\n Shneiderman agreed to also share for this report his following insights from his August 2022 interview<\/a> with the Fidelity Center for Applied Technology: \u201cThe hopeful future we can continue to work toward is one in which AI systems augment, amplify and enhance our lives. Humans have agency over key decisions made while using a vast number of AI tools in use today. Digital cameras rely on high levels of AI for setting the focus, shutter speed and color balance while giving users control over the composition, zoom and decisive moment when they take the photo. Similarly, navigation systems let users set the departure and destination, transportation mode and departure time, then the AI algorithms provide recommended routes for users to select from as well as the capacity to change routes and destinations at will. Query completion, text auto-completion, spelling checkers and grammar checkers all ensure human control while providing algorithmic support in graceful ways. <\/p>\n\n\n\n \u201cWe must respect and value the remarkable capabilities that humans have for individual insight, team coordination and community building and seek to build technologies that support human self-efficacy, creativity, responsibility and social connectedness. Some advocates of artificial intelligence promote the goal of human-like computers that match or exceed the full range of human abilities from thinking to consciousness. This vision attracts journalists who are eager to write about humanoid robots and contests between humans and computers. I consider these scenarios as misleading and counterproductive, diverting resources and effort from meaningful projects that amplify, augment, empower and enhance human performance. <\/p>\n\n\n\n \u201cThe past few years we have seen news stories about tragic failures of automated systems. The two Boeing 737 MAX crashes are a complex story, but one important aspect was the designers\u2019 belief that they could create a fully autonomous system that was so reliable that the pilots were not even informed of its presence or activation. There was no obvious visual display to inform the pilots of the status, nor was there a control panel that would guide them to turn off the autonomous system. The lesson is that the excessive belief in machine autonomy can lead to deadly outcomes. When rapid performance is needed, high levels of automation are appropriate, but so are high levels of human independent oversight to track performance over the long-term and investigate failures.<\/p>\n\n\n\n \u201cWe can accelerate the wider, successful adoption of human-centered AI. It will take a long time to produce the changes that I envision, but our collective goals should be to reduce the time from 50 to 15 years. We can all begin by changing the terms and metaphors we use. Fresh sets of guidelines for writing about AI are emerging from several sources, but here is my draft offering:<\/p>\n\n\n\n John Sniadowski<\/strong>, a systems architect based in the UK, said, \u201cOur lack of agency has arrived. I suggest that the bias toward never challenging the machines is inevitable. Decision systems are generally based on opaque formulas with targeted outcomes [that] usually serve only the best interests of the AIs\u2019 vendors. In most cases, the ultimate outcome from these automated, data-based decisions cannot be challenged and are, in fact, rarely challenged because the human belief is the system is correct often enough to be followed.<\/p>\n\n\n\n \u201cConsider the financial industry today, in 2022. Lending decisions are based on smart systems that are nearly impossible to challenge. In addition, AI is frequently trained on datasets that are biased and may contain hidden anomalies that significantly alter the decision process. The vast majority of the population will be convinced by marketing, propaganda or other opinion-bending messages that these systems are right and any individual\u2019s opinion is wrong. We already see that sort of behaviour in human-based systems operated by Big Pharma, where millions\/billions of revenue could be lost if a significant outcome of a product\/decision is successfully challenged.<\/p>\n\n\n\n \u201cLife-and-death decisions should always require responsible human input, and they should have a set of criteria that the AI system must present in arriving at its decision that is transparent and capable of human interpretation. This should be enshrined in legislation with punitive consequences for vendors that do not comply with decision transparency.<\/p>\n\n\n\n \u201cI would strongly suggest that this should be incorporated in a global human rights framework, that all humans have the right to challenge the outcome of an autonomous decision. This should be part of the UN charter and put in place as soon as possible.<\/p>\n\n\n\n \u201cGiven what we are experiencing on social media, where people can become captured by \u2018echo chambers,\u2019 there is a significant danger that AI and autonomous decision processes will exacerbate a broad range of societal inequalities. The vast array of data metrics now harvested from individuals\u2019 internet activities will continue to categorize each person more and more toward an inescapable stereotype without the individual even being aware of the label unfairly applied to them.<\/p>\n\n\n\n \u201cCompanies will harvest information from \u2018smart cities,\u2019 and AI will build dossiers on each citizen that will be applied for a wide variety of decisions about a person completely without their personal consent. This is very dangerous, and we are already seeing this capability being subverted by some governments to tighten their authoritarian grip on their population.\u201d<\/p>\n\n\n\n Clifford Lynch<\/strong>, executive director of the Coalition for Networked Information, wrote, \u201cAs I think about the prospects for human agency and how this compares to delegation to computational decision-making in 2035, I\u2019m struck by a number of issues.<\/p>\n\n\n\n 1) \u201cAs far as I know, we\u2019ve made little progress in genuine partnership and collaboration between computational\/AI systems and humans. This seems to be presented as a binary choice: either hand off to the AI, or the human retains control of everything. Some examples: AI systems don\u2019t seem to be able to continuously learn what you already know, information that you have already seen and evaluated, and how to integrate this knowledge into future recommendations it may offer. One really good example of this: Car navigation systems seem unable to learn navigational\/routing preferences of drivers in areas very close to their homes or offices.<\/p>\n\n\n\n \u201cAnother example: Recommender systems often seem unable to integrate prior history when suggesting things. As far as I can tell, stunningly little real work has been done on computational systems that co-evolve, co-mature or co-develop with humans; this has been largely left to science fiction writers. As an additional issue, some of the research here involves time horizons that don\u2019t fit conveniently with relatively short-term grant funding. Without a lot more progress here, we\u2019ll continue to tend to frame the issue as \u2018delegate or don\u2019t delegate agency to computational systems.\u2019<\/p>\n\n\n\n 2) \u201cI wonder about the commercial incentives that might exist in maintaining agency as a binary choice (retain or delegate) rather than seeking the cultivation of collaborations between humans and machines. There are many situations when delegation is the easy choice because making a human decision will take a lot of time and have to encompass a lot of complex data; combine this with opaque decision-making by the algorithms once delegation has been made, and this may well advance commercial (or governmental) objectives.<\/p>\n\n\n\n 3) \u201cThere are staggering commercial incentives to delegate decision-making to computational agents (including really stupid agents like chatbots) in areas such as customer service, billing, fraud detection and the like, and companies are already doing this at massive scale. Most of these systems are really, really bad. Bluntly, the companies deploying these could mostly care less about errors or misjudgments by these computational agents unless they result in a high-visibility public relations blowup. There\u2019s every reason to expect these trends to continue and to get worse rather than better. This represents a really huge abdication of human agency that\u2019s already far advanced.<\/p>\n\n\n\n 4) \u201cThere are situations where there\u2019s a very strong motivation to default to the machines. Human decision-makers may be overworked, overwhelmed and don\u2019t have time. Not delegating (or delegating and overriding or collaborating) may be risky. There are also still widespread general public beliefs that computational decisions are less biased or more accurate than human decision-making (though there\u2019s been a lot of good research suggesting this is frequently not true). Good examples here: judges going against sentencing or bail recommendations; doctors going against diagnostic\/treatment recommenders (often created by health care systems or insurers trying to minimize costs). These overrides can happen, but often only when someone is important enough or persuasive enough to demand and gain the human attention, risk-acceptance and commitment to override the easy default delegation to the AI. Put another way, when they want to, the wealthy and powerful (and perhaps the tech-savvy as well) will have a much better chance of appealing or overriding computational decision-making that\u2019s increasingly embedded in the processes of our society.<\/p>\n\n\n\n 5) \u201cAs an extension of the last point, there are situations where human decision-makers are legitimately overwhelmed or don\u2019t have time, when they cannot react quickly enough, and algorithmic triage and decision-making must be the norm. We do not understand how to define and agree on these situations. Relatively easy cases include emergency triage of various kinds, such as power grid failures or natural disasters. Computationally directed trading in financial markets might be a middle ground. More challenging cases might include response to minimal warning nuclear strikes (hypersonic vehicles, orbital strikes, close offshore cruise missiles, etc.) where there\u2019s a very short time window to launch a major \u2018use it or lose it\u2019 counterforce strike. One can also construct similar cyberwar strike scenarios.<\/p>\n\n\n\n 6) \u201cRelated to the previous point: As a society we need to agree on how to decide when agency delegation is high-stakes or low-stakes. Also, we need to try to agree on the extent to which we are comfortable delegating to computational entities. For example, can we identify domains where there is a high variance between human and computational predictions\/recommendations, hence we should probably be nervous about such delegation?<\/p>\n\n\n\n 7) \u201cWe haven\u2019t considered augmented humans (assuming that they exist in 2035 in a meaningful way) and how they fit into the picture of computational decision-making, humans and perhaps collaborative middle grounds. This could be important.<\/p>\n\n\n\n 8) \u201cI have been tracking the construction of systems that can support limited delegation of decision-making with great fascination. These may represent an important way forward in some domains. Good examples here are AI\/ML-based systems that can explore a parameter space (optimize a material for these requirements, running experiments as necessary and evaluating the resultant data); often these are coupled by robotics that allow the computational system to schedule and run the experiments. I think these are going to be very important for science and engineering, and perhaps other disciplines, in the coming years; they may also become important in commercial spheres. The key issue here is to track how specific the goals (and perhaps suggested or directed methodologies) need to be to make these arrangements successful. It\u2019s clear that there are similar systems being deployed in the financial markets, though it\u2019s more difficult to find information about experiences and plans for these. And it\u2019s anybody\u2019s guess how sectors like the intelligence community are using these approaches.\u201d<\/p>\n\n\n\n Amali De Silva-Mitchell<\/strong>, founding coordinator of the UN Internet Governance Forum Dynamic Coalition on Data-Driven Health Technologies, said, \u201cThe true, intuitive human decision-making capabilities of technologies are still in their infancy. By 2035 we will have hopefully opened most of the AI developers\u2019 minds to the issues of data quality, trojan data, data warps and oceans, ethics, standards, values, and so forth, that come in a variety of shapes and sizes across segments of society.<\/p>\n\n\n\n \u201cThe bias of using the data from one segment on another can be an issue for automated profiling. Using current statistical techniques does not make for strong foundations, for universal decision-making, it only allows for normalized decision-making or even group think.<\/p>\n\n\n\n \u201cLots of caveats have to be made, and these caveats must be made transparent to the user; however, there are some standardized, commonly identified processes that can be very well served by automated decision-making, for example, for repetitive practices that have good procedures or standards already in place. In some instances, automated decision-making may be the only available procedure available, say for a remote location \u2013 including outer space. What is critical is that human attention to detail, transparency and continuous betterment is ever-present every step of the way.<\/p>\n\n\n\n \u201cWe may be forced to enter into the use of an AI before an application is fully ready for service due to the need to service at speed, fill a gap, and so forth. In these cases, it is especially important that human oversight is ever-present and that members of the public \u2013 everyday users \u2013 have the opportunity to provide feedback or raise concerns without reprimand.<\/p>\n\n\n\n \u201cHumans must not feel helpless and hopeless with no opportunity for contact with a person when it is necessary. This is something that some developers of bots \u2013 for instance \u2013 have not taken into account. Humans must also have the opportunity to be the owners of their own identities and be able to check it if they wish to and get it corrected within a reasonable amount of time.<\/p>\n\n\n\n \u201cAssumptions must not be made of persons, and the \u2018reasonable person\u2019 concept<\/a> must always be maintained. Good Samaritans must also have a way to weigh in, as compassion for humans must be at the core of any technology.\u201d<\/p>\n\n\n\n David Weinberger<\/strong>, senior researcher at Harvard\u2019s Berkman Center for Internet and Society, commented, \u201cMachine learning models\u2019 interests can and should be regulated and held up for public debate. That could alter our idea of our own autonomy, potentially in very constructive ways, leading us to assume that our own interests likewise affect more than our own selves and our own will. But this assumes that regulators and the public will do their jobs of making machine learning models\u2019 interests \u2013 their objective functions \u2013 public objects subject to public control.<\/p>\n\n\n\n \u201cAutonomous selves have interests that they serve. Those interests have to be made entirely explicit and measurable when training a machine learning model; they are objects of discussion, debate and negotiation. That adds a layer of clarity that is often (usually?) absent from autonomous human agents.<\/p>\n\n\n\n \u201cThere is certainly a case for believing that humans will indeed be in control of making important decisions in the year 2035. I see humans easily retaining decision-making control things like who to marry, what career to pursue, whether to buy or rent a home, whether to have children, which college to go to (if any), and so forth. Each of those decisions may be aided by machine learning, but I see no reason to think that machine learning systems will actually make those decisions for us.<\/p>\n\n\n\n \u201cEven less-important personal decisions are unlikely to be made for us. For example, if an online dating app\u2019s ML models get good enough that the app racks up a truly impressive set of stats for dates that turn into marriages, when it suggests to you that so-and-so would be a good match, you\u2019ll still feel free to reject the suggestion. Or so I assume.<\/p>\n\n\n\n \u201cBut not all important<\/em> decisions are major<\/em> decisions. For example, many of us already drive cars that slam on the brakes when they detect an obstacle in the road. They do not ask us if that\u2019s OK; they just bring the car to a very rapid halt. That\u2019s a life-or-death \u2018decision\u2019 that most of us want our cars to make because the car\u2019s sensors plus algorithms can correct for human error and the slowness of our organic reactions. And once cars are networked while on the road, they may take actions based on information not available to their human drivers, and so long as those actions save lives, decrease travel times, and\/or lower environmental impacts, many if not most of us will be OK with giving up a human autonomy based on insufficient information.<\/p>\n\n\n\n \u201cBut an uninformed or capricious autonomy has long been understood to be a false autonomy: In such cases we are the puppets of ignorance or short-sighted will. Delegating autonomy can itself be a proper use of autonomy. In short, autonomy is overrated. The same sort of delegation of autonomy will likely occur far more broadly. If smart thermostats keep us warm, save us money and decrease our carbon footprints, we will delegate to them the task of setting our house\u2019s temperature. In a sense, we already do that when we set an old-fashioned thermostat, don\u2019t we?<\/p>\n\n\n\n \u201cBut there are more difficult cases. For example, machine learning models may well get better at diagnosing particular diseases than human doctors are. Some doctors well may want to occasionally override those diagnoses for reasons they cannot quite express: \u2018I\u2019ve been reading biopsy scans for 30 years, and I don\u2019t care what the machine says, that does not look cancerous to me!\u2019 As the machines get more and more accurate, however, \u2018rebellious\u2019 doctors will run the risk of being sued if they\u2019re wrong and the machine was right. This may well intimidate doctors, preventing them from using their experience to contradict the output from the machine learning system. Whether this abrogation of autonomy is overall a good thing or not remains to be seen.<\/p>\n\n\n\n \u201cFinally, but far from least important, is to ask what this will mean for people who lack the privileges required to exercise autonomy. We know already that machine learning models used to suggest jail sentences and conditions of bail are highly susceptible to bias. The decisions made by machine learning that affect the marginalized are likely to be a) less accurate because of the relative paucity of data about the marginalized most affected by them; b) less attuned to their needs because of their absence from the rooms where decisions about what constitutes a successful model are made; and c) are likely to have less power to get redress for bad decisions made by those models. Does this mean that the \u2018autonomy gap\u2019 will increase as machine learning\u2019s sway increases? Quite possibly. But it\u2019s hard to be certain, because while machine learning models can amplify societal biases, they can also remove some elements of those biases. Also, maybe by 2035 we will learn to be less uncaring about those whose lives are harder than our own. But that\u2019s a real longshot.<\/p>\n\n\n\n \u201cAs for less-direct impacts of this delegation of autonomy, on the one hand, we\u2019re used to delegating our autonomy to machines. I have been using cruise control for decades because it\u2019s better at maintaining a constant speed than I am. Now that it\u2019s using machine learning, I need to intervene less often. Yay!<\/p>\n\n\n\n \u201cBut as we delegate higher-order decisions to the machines, we may start to reassess the virtue of autonomy. This is both because we\u2019ll have more successful experience with that delegation, but also perhaps we\u2019ll come to reassess the concept of autonomy itself.<\/p>\n\n\n\n \u201cAutonomy posits an agent sitting astride a set of facts and functions. That agent formulates a desire and then implements. Go, autonomy! But this is a pretty corrupt concept. For one thing, we don\u2019t input information that (if we\u2019re rational) determines our decision. Rather, when in the process of making a decision we decide which information to credit and how to weigh it. That\u2019s exactly what machine learning algorithms do with data when constructing a model.\u201d<\/p>\n\n\n\n Kathryn Bouskill<\/strong>, anthropologist and AI expert at the Rand Corporation, said, \u201cLooking ahead, humanity will be challenged to redefine and reimagine itself. It must consider the unprecedented social and ethical responsibilities that the new speed of change is ushering into our lives \u2013 including crucial issues being raised by the spread of AI.<\/p>\n\n\n\n \u201cThe number of places in which individuals have agency and can take control in this era of swift technological speed is dwindling. Hitting the brakes is not an option. When life happens quickly, it can feel difficult to process change, create a purpose, hold our social ties together and feel a sense of place. This kind of uncertainty can induce anxiety, and anxiety can lead to isolationism, protectionism, fear, gridlock and lack of direction. \u2026<\/p>\n\n\n\n \u201cIs AI going to completely displace human autonomy? We may forget that humanity still has the opportunity to choose what is being developed. That can still be our decision to make. Most people are just passively watching the technology continue to rapidly roll out without being actively engaged as much as they should be with it. For now, I\u2019m leaning toward the optimistic view that human autonomy will prevail. However, this requires the public implementation of educational components, so the black-box aspects of AI are explored and understood by more people. And the public and Big Tech must learn how to build equity into AI and know what levers to pull to assure that it works for the good of humanity. Smart regulation and robust data protection are also critically important.<\/p>\n\n\n\n \u201cThe greatest resource in the human toolkit is our ability to cooperate and creatively adapt to or change our surroundings. It will take a concerted effort across multiple stakeholders \u2013 citizens, consumers, employers, voters, tech developers and policymakers \u2013 to collectively devote attention to vetting and safeguarding technologies of the future to make the world safer.\u201d<\/p>\n\n\n\n Claudia L\u2019Amoreaux<\/strong>, principal at Learning Conversations, a global internet consultancy, and former director of education programs at Linden Lab (developers of Second Life) wrote, \u201cThe two words that stand out in your top-level question are \u2018designed\u2019 and \u2018easily.\u2019 In designing for human agency and decision-making, we do<\/em> have a choice. Looking at how the EU handled data protection with the GDPR privacy legislation<\/a> vs. how the U.S. has pretty much continued business as usual shows that we do have a choice. \u2026<\/p>\n\n\n\n In designing for human agency and decision-making, we do<\/em> have a choice. …However, I am extremely skeptical that choices will be made in favor of human agency here in the U.S. Where\u2019s the incentive? <\/p>Claudia L\u2019Amoreaux, principal at Learning Conversations, a global internet consultancy<\/cite><\/blockquote><\/figure>\n\n\n\n \u201cHowever, I am extremely skeptical that choices will be made in favor of human agency here in the U.S. Where\u2019s the incentive? As long as tech companies\u2019 profits are based on separating users from as much of their personal data as possible \u2013 for ad targeting, self-serving recommendations that maximize engagement, and resale \u2013 this situation will not improve. Broader, more sophisticated applications of AI will only accelerate what is already happening today.<\/p>\n\n\n\n \u201cAnd as regulations around privacy and data extraction do tighten in the U.S., however slightly, companies in the AI industry are and will continue to exploit the data of people in the less-developed world, as Karen Hao lays out so well in the AI Colonialism series<\/a> in MIT Technology Review.<\/em><\/p>\n\n\n\n \u201cI\u2019ll share two examples that fuel my skepticism about human agency and decision-making. The first example regards the UK Biobank\u2019s transfer of genetic data of half a million UK citizens in a biomedical database to China (reported in The Guardian<\/a>). The sharing of sensitive genetic data in the UK Biobank project, launched as an \u2018open science project\u2019 in 2016, is based on a relationship of trust that is eroding as West\/China relations transform. Sharing is not reciprocal. Motives aren\u2019t parallel. The 500,000 humans with their DNA data in the Biobank are asked to trust that researchers will do a good job \u2018managing risk.\u2019 Is their agency and decision-making being prioritized in the conversations taking place? I don\u2019t think so.<\/p>\n\n\n\n \u201cA second example is the massive surveillance model employed by China that they are now exporting to countries that want to follow in their footsteps. With large infrastructure projects underway already through China\u2019s Belt and Road Initiative, surveillance tech has become an add-on.<\/p>\n\n\n\n \u201cRegarding the use of the term \u2018easily\u2019 in your question \u2013 will people \u2018easily be in control of most tech-aided decision-making that is relevant to their lives\u2019 \u2013 it\u2019s not looking good for 2035.<\/p>\n\n\n\n \u201cWhat key decisions will be mostly automated?To start to understand what key decisions will be mostly automated, we can look at what\u2019s mostly automated today (and how quickly this has occurred). Let\u2019s look at two related examples \u2013 higher education and hiring.<\/p>\n\n\n\n \u201cMany universities have moved to automating the college admissions process for a variety of reasons. Increasing revenue is an obvious one, but some schools claim the move helps reduce bias in the admissions process. The problem with this which has become very clear across many domains today is that it all depends on the datasets used. Introducing AI can have the opposite effect, amplifying bias and widening the equity gap. Students most at risk for paying the price of increased automation in the admissions process are lower-income and marginalized students. <\/p>\n\n\n\n \u201cOnce students do make it into a university, they are likely to encounter predictive analytics tools making academic decisions about their futures that can narrow their options. The education podcast by APM Reports did a good piece on this, titled \u2018Under a Watchful Eye<\/a>.\u2019 While most elite universities are keeping a hands-on approach for now, colleges that serve the majority of students are adopting predictive analytics to point students to what the universities deem as a successful path from their perspective: continued tuition and graduation. This approach benefits the schools but not necessarily the students. <\/p>\n\n\n\n \u201cDon\u2019t get me wrong \u2013 identifying students at risk for failing early on and offering support and options to help them graduate is a good thing. But if the school\u2019s priority is to ensure continuing tuition payments and maximize graduate rates, this can actually co-opt student agency. Once again, predictive analytics relies on historical data, and we know historical data can carry extensive baggage from long-term, systemic bias. Students of color and low-income students can find themselves pushed in different directions than they set out \u2026 when an alternative approach that prioritizes around student agency might help them actually succeed on the path of their choice. In this case, predictive analytics helps the schools maintain their rankings based on graduation rates but sacrifices student preferences.<\/p>\n\n\n\n \u201cThen there\u2019s hiring. Hiring is already so heavily automated that job seekers are advised to redesign their CVs to be read by the algorithms. Otherwise, their application will never even be seen by human eyes. These are just a few examples.<\/p>\n\n\n\n \u201cWhat key decisions should require direct human input? In the military, the use of autonomous lethal weapons systems should be banned. In the 2021 Reith Lecture series \u2018Living With Artificial Intelligence,\u2019 Lecture 2: AI in Warfare<\/a>, Stuart Russell, founder of the Center for Human-Compatible Artificial Intelligence at the University of California, Berkeley, explains, \u2018At the moment we find ourselves at an unstable impasse, unstable because the technology is accelerating \u2026 we have about 30 countries who are in favor of a ban, as well as the EU parliament, the United Nations, the non-aligned movement, hundreds of civil society organizations, and according to recent polls, the great majority of the public all over the world.\u2019 <\/p>\n\n\n\n \u201cI stand with Russell \u2013 who supports a ban \u2013 along with leaders in 30 countries and the majority of people around the world. But as Russell says in his Reith Lecture \u2026 <\/p>\n\n\n\n \u2018On the other side, we have the American and Russian governments, supported to some extent by Britain, Israel and Australia, arguing that a ban is unnecessary. \u2026 Diplomats from both the UK and Russia express grave concern that banning autonomous weapons would seriously restrict civilian AI research. \u2026 I\u2019ve not heard this concern among civilian AI researchers. Biology and chemistry seem to be humming along, despite bans on biological and chemical weapons.\u2019<\/em><\/p>\n\n\n\n \u201cThe U.S. and Russian positions do not speak well for the future of human agency and decision-making, although Russell said he is encouraged by decisions humanity has made in the past to ban biological and chemical weapons, and landmines. It is not impossible, but we have a long way to go to ban autonomous lethal weapon systems. Am I encouraged? No. Hopeful? Yes.<\/p>\n\n\n\n \u201cBecause of a long history of structural racism and its encoding in the major databases used for training AI systems (e.g., ImageNet<\/a>), the justice system, policing, hiring, banking (in particular, credit and loans), real estate and mortgages, and college applications and acceptance all involve life-changing decisions that should require direct human input. And in the medical domain, considering possible life-and-death decisions, we\u2019ve seen that the use of image identification for skin cancer that has been trained predominantly on white skin may misidentify skin cancers. This is just one example in health care. Until we rectify the inherent problems with the training sets that are central to AI solutions, key decisions about life and death should require direct human input.<\/p>\n\n\n\n \u201cHow might the broadening and accelerating rollout of tech-abetted, often autonomous decision-making change human society? Because this is already happening now, it\u2019s not hard to see how it will be and is changing human society. For one, it is creating two distinct classes with a huge gap in between \u2013 a techno-savvy class, and let\u2019s call it a techno-naive class. Techno-savvy humans understand the basics of AI, algorithms, etc. They have the knowledge and the ability to protect their privacy (as much as is possible), opt out, assess validity and sources of content, detect fakes or at least understand that fakes are proliferating, etc. Techno-naive humans are currently and will be easily duped and taken advantage of \u2013 for their data, for their eyeballs and engagement metrics and for political gain by the unscrupulous groups among the techno-savvy.<\/p>\n\n\n\n \u201cAnd whether savvy or naive, people (especially people of color) will find themselves at the mercy of and in the crosshairs of autonomous decision-making \u2013 e.g., misidentification, biases embedded in datasets, locked out of jobs, education opportunities, loans, digitally red-lined. <\/p>\n\n\n\n \u201cThe uses of AI are so vast already, with so little scrutiny. The public\u2019s knowledge of AI is so minimal, agency is already so eroded, people are too willing to trade agency for convenience, most not even realizing that they are making a trade. Sure, I can find out what data Facebook, etc., has on me, but how many people are going to 1) take the time to do it, 2) even know that they can and 3) understand how it all works.<\/p>\n\n\n\n \u201cI\u2019ve made it clear I think we have serious work to do at international and national levels to protect privacy, human agency, access and equity.<\/p>\n\n\n\n \u201cBut we also need to make serious efforts in 1) how we teach young people to regard these technologies and 2) in how we put these technologies to work in the pre-K-12 education systems and higher education. Education will play a major role in future outcomes around technology, decision-making and human agency.<\/p>\n\n\n\n \u201cI am encouraged by the efforts of organizations like UNICEF\u2019s AI for Children<\/a> project, the Harvard Berkman Klein Center\u2019s Youth and AI project<\/a>, MIT\u2019s Responsible AI for Social Empowerment and Education (RAISE<\/a>) project, to name a few. I think these projects are exemplary in prioritizing human agency and decision-making. I especially appreciate how they go out of their way to include youth voices.<\/p>\n\n\n\n \u201cThe next horizon is already upon us in education. The choices we make in AI-enabled teaching and learning will play a tremendous role in future outcomes around human agency and decision-making. China is currently pushing hard on AI-enabled teaching and adaptive learning with a focus toward helping students perform on standardized testing. And school systems in the U.S. are looking at their success.<\/p>\n\n\n\n \u201cI understand and appreciate the role for adaptive learning systems like Squirrel AI<\/a>, a dominant tutoring system in China today. But I lean in the direction of educators like Harvard professor Chris Dede, an early innovator of immersive learning, who emphasizes the necessity for an education system that prioritizes creativity, innovation, directed by student interest and passion. To become adults who value human agency and decision-making, young people need to experience an educational system that embodies and models those values. They need opportunities to develop AI literacy that presents a much wider lens than coding \u2013 offering opportunities to explore and engage algorithmic justice, biases, ethics, and especially building and testing AI models themselves, from a young age.<\/p>\n\n\n\n \u201cDespite my rather bleak answer of \u2018No\u2019 to the primary question, this is where I find encouragement and the possibility of \u2018Yes\u2019 for the year 2035. The children in kindergarten today who are training and building robots with constructivist platforms like Cognimates<\/a> will be entering college and\/or the workforce in 2035.<\/p>\n\n\n\n \u201cIn the 2019 post \u201cWill AI really transform education?<\/a>\u201d in The Hechinger Report, writer Caroline Preston reports on a conference on AI in education that she attended at Teachers College, Columbia University. Stephania Druga, who created the Cognimates platform, spoke at the conference, and Caroline summarized: \u2018In her evaluations of Cognimates, she found that students who gained the deepest understanding of AI weren\u2019t those who spent the most time coding; rather, they were the students who spent the most time talking about the process with their peers.\u2019\u201d<\/p>\n\n\n\n James Hendler<\/strong>, director of the Institute for Data Exploration and Applications and professor of computer, web and cognitive sciences at Rensselaer Polytechnic Institute, commented, \u201cWhile I would hope we will get better and better at keeping humans in control in many cases, there are three reasons I think we may not get there by 2035 \u2013 two positive and one negative:<\/p>\n\n\n\n \u201cPositive 1<\/strong> \u2013 There are a few cases where machines are superior and should be acknowledged as such. This is not something new; for example, there are very few businesses that do payrolls by hand-automated payroll systems (which don\u2019t need AI technology, I note). [Automated systems] have been around a long time and have become trustworthy and relied upon. There will be some cases with existing and emerging AI technologies where this is also true \u2013 the key will be a need to identify which these are and how to guarantee trustworthiness.<\/p>\n\n\n\n \u201cPositive 2<\/strong> \u2013 There will be cases where the lack of trained humans will require greater reliance on machine decision-making. As a case in point, there are some single hospitals in the U.S. that have more X-ray analysts than some entire nations in the global south. As machines get more reliable at this task, which is happening at a rapid rate, the potential deployment of such would not be as good as human-machine teaming (which will happen in the wealthier countries that do have the trained personnel to be in the loop) but will certainly be way better than nothing. A good solution that could improve health care worldwide, in certain cases, would be worth deploying (with care) even if it does require trusting machines in ways we otherwise might not.<\/p>\n\n\n\n \u201cThe Negative<\/strong> \u2013 The main thing holding back the deployment of autonomous technology in many cases has more to do with policy and litigation than technology. For example, many of the current autonomous driving systems can, in certain situations, drive better than humans \u2013 and with improvements to roads and such, at least certain kinds of vehicles could be turned over to autonomous systems part, if not all, of the time. However, deploying such has huge risk to the companies doing so, due to long-established rules of liability for automobile-related regulations \u2013 which will keep humans in the loop until the technology is provably superior and the rules of the road (if you\u2019ll pardon the pun) are more clearly developed. Thus, these companies opt to keep humans in the loop out of legal, rather than technical, reasons. On the other hand, there are many industries where the speed of technological change is far beyond the scope, scale and speed of regulatory change (face-recognition systems deployment vs. regulation is an example). The companies developing technologies in these less-regulated areas do not have the restrictions on taking humans out of the loop, whether it is a good idea or not, and the economic rewards are still, unfortunately, on the side of autonomous deployment. <\/p>\n\n\n\n \u201cAll of that said, in general, as Alice Mulvehill and I argue in the book \u2018Social Machines: The Coming Collision of Artificial Intelligence, Social Networking and Humanity<\/a>\u2019 and as other authors of similar recent books have argued, in most cases keeping humans in the loop (due to the differences between human and computer capabilities) is still necessary for the foreseeable future. I do consider my prediction a pessimistic one \u2013 it would be better to see a world where humans will remain in control of many areas where the speed of technical deployment coupled with the lack of regulation may hinder this happening. This could have potentially disastrous consequences if allowed in high-consequence systems (such as military weaponry, political influence, privacy control \u2013 or the lack thereof \u2013 etc.).<\/p>\n\n\n\n \u201cAlso, investment in human development would be a wonderful thing to see (for example, in the case of X-ray analysts, training more humans to work with automated systems would be preferable to simply deploying the systems), but right now that does not seem to be a political reality in most of the world.\u201d<\/p>\n\n\n\n Charles Ess<\/strong>, professor emeritus of digital ethics at the University of Oslo, Norway, wrote, \u201cThese past few years of an \u2018AI spring\u2019 have distinguished themselves from earlier ones, at least among some authors and projects, as they are accompanied by considerably greater modesty and recognition of the limits perhaps intrinsic to what AI and machine learning (ML) systems are capable of. Notably important resources along these lines include the work of Katharina Zweig, e.g., \u2018Awkward Intelligence Where AI Goes Wrong, Why It Matters, and What We Can Do about It<\/a>\u2019 (MIT Press, 2022).<\/p>\n\n\n\n \u201cOn the other hand, I still find that in most of the emerging literatures in these domains \u2013 both from the corporations that largely drive the development of AI\/ML systems as well as from research accounts in the more technical literature \u2013 there remains a fundamental failure to understand the complexities of human practices of decision-making and most especially judgment<\/em>. Some AI ethicists point out that a particular form of human judgment \u2013 what Aristotle called phronesis and what Kant called reflective judgment \u2013 are what come into play when we are faced with the difficult choices in grey areas. In particular, fine-grained contexts and especially novel ethical dilemmas usually implicate several possible ethical norms, values, principles, etc.<\/p>\n\n\n\n \u201cIn contrast with determinative judgments that proceed from a given norm in a deductive, if not algorithmic fashion to conclude with a largely unambiguous and more or less final ethical response \u2013 one of the first tasks of reflective judgment is to struggle to discern just which ethical norms, principles, values indeed are relevant to a specific case, and, in the event of (all but inevitable) conflict, which principles, norms, etc., override the others. As many of us argue, these reflective processes are not computationally tractable for a serious of reasons \u2013 starting as they draw from tacit, embodied forms of knowledge and experience over our lifetimes. As well, these processes are deeply relational \u2013 i.e., they draw on our collective experience, as exemplified in our usually having to talk these matters through with others in order to arrive at a judgment.<\/p>\n\n\n\n \u201cThere is, then, a fundamental difference between machine-based \u2018decision-making\u2019 and human-based ethical reflection \u2013 but this difference seems largely unknown in the larger communities involved here. In particular, there is much in engineering cultures that sets up \u2018decision-making\u2019 as more or less deductive problem-solving \u2013 but this approach simply cannot take on board the difficulties, ambiguities and uncertainties intrinsic to human reflective judgment.<\/p>\n\n\n\n \u201cFailing to recognize these fundamental differences then results in the engineering paradigm overriding the ethical one. Catastrophe is sure to result \u2013 as it already has, e.g., there is discussion that the financial crises of 2008 in part rested on leaving \u2018judgments\u2019 as to credit-worthiness increasingly to machine-based decision-making, which proved to be fatally flawed in too many cases.<\/p>\n\n\n\n \u201cAs many of us have argued (e.g., Mireille Hildebrandt, \u2018Smart Technologies and the End(s) of Law<\/a>,\u2019 2015, as an early example, along with more recent figures such as Virginia Dignum, who directs the large Wallenberg Foundation\u2019s projects on humanities and social science approaches to AI, as well as Zweig, among others), leaving ethical judgments in particular and similar sorts of judgments in the domains of law, credit-worthiness, parole considerations (i.e., the (in)famous COMPAS system), \u2018preemptive policing,\u2019 and so on to AI\/ML processes is to abdicate a central human capacity and responsibility \u2013 and this to systems that, no matter how further refined they may be with additional ML training, etc., are in principle incapable of such careful reflection.<\/p>\n\n\n\n \u201cWe are very often too prone to off-loading our tasks and responsibilities to our machineries \u2013 especially when the tasks are difficult, as reflective judgment always is. And in this case, failure to recognize in the first place just what it is that we\u2019re offloading to the machines makes the temptations and drives to do so doubly pernicious and likely.<\/p>\n\n\n\n \u201cLike the characters of \u2018Brave New World\u2019 who have forgotten what \u2018freedom\u2019 means, and so don\u2019t know what they have lost, failing to take on board the deep differences between reflective forms of judgment and AI\/ML decision-making techniques \u2013 i.e., forgetting about the former, if we were ever clear about it in the first place \u2013 likewise means we risk losing the practice and capacity of reflective judgment as we increasingly rely on AI \/ML techniques, and not knowing just what it is we have lost in the bargain.<\/p>\n\n\n\n \u201cWhat key decisions should require direct human input? This category would include any decision that directly affects the freedom and quality of life of a human being. I don\u2019t mind AI\/ML driving the advertising and recommendations that come across my channels \u2013 some of which is indeed useful and interesting. I am deeply concerned that offloading ethical and legal judgments to AI\/ML threatens to rob us \u2013 perhaps permanently \u2013 of capacities that are central to human freedom and modern law as well as modern democracy. The resulting dystopia may not be so harsh as we see unfolding in the Chinese Social Credit Systems<\/a>.<\/p>\n\n\n\n \u201c\u2018Westerners\u2019 may be more or less happy consumers, content with machine-driven options defining their work, lives, relationships, etc. But from the standpoint of Western traditions, starting with Antigone and then Socrates through the democratic and emancipatory movements that mark especially the 18th-20th centuries, that emphasize the central importance of human freedom over against superior force and authority \u2013 including the force and authority of unquestioned assumptions and rules that must always be obeyed, or else \u2013 such lives, however pleasant, would fail to realize our best and fullest possibilities as human beings, starting with self-determination.\u201d<\/p>\n\n\n\n Neil Davies<\/strong>, co-founder of Predictable Network Solutions and a pioneer of the committee that worked on the UK\u2019s initial networking developments, said, \u201cAs someone who has designed and taken to production large-scale systems, I am abundantly aware that feasibility of executing fully autonomous systems is, for all practical purposes, zero. The main reason is the ontological\/epistemological chasm: People forget that machines (and the systems they create) can only<\/em> \u2018know\u2019 what they have \u2018experienced\u2019 \u2013 the things they have been exposed to. They, by definition, cannot reach out to a wider information base and they can\u2019t create \u2018ontological\u2019 framing. And that framing is an essential way in which humans \u2013 and their societies \u2013 make decisions.<\/p>\n\n\n\n As someone who has designed and taken to production large-scale systems, I am abundantly aware that feasibility of executing fully autonomous systems is, for all practical purposes, zero.<\/p>Neil Davies, co-founder of Predictable Network Solutions and a pioneer of the committee that worked on the UK\u2019s initial networking developments<\/cite><\/blockquote><\/figure>\n\n\n\n \u201cI can see great use for machine-learning tools that look over the shoulders of experts and say, \u2018Have you considered X, Y or Z as a solution to your problem?\u2019 But fully autonomous systems cannot be made that detect problems automatically and deal with them automatically. You have to have human beings make certain types of decisions or penalize certain bad behavior. Often behaviour can only be considered bad when intent is included \u2013 machines can\u2019t deal in intent.<\/p>\n\n\n\n \u201cThe problem is that if you try to build a machine-based system with autonomy, you have to handle not only the sunny-day cases but also the edge cases. There will inevitably be adversarial actors endeavoring to attack individuals, groups or society by using the autonomous system to be nasty. It\u2019s very hard to account for all the dangerous things that might happen and all the misuses that might occur.<\/p>\n\n\n\n \u201cThe systems are not god-like, and they don\u2019t know the whole universe of possible uses of the systems. There\u2019s an incompleteness issue. That incompleteness makes these systems no longer autonomous. Humans have to get involved.<\/p>\n\n\n\n \u201cThe common problem we\u2019ve found is that is not feasible to automate everything. The system eventually has to say, \u2018I can\u2019t make a sensible\/reasoned decision\u2019 and it will need to seek guiding human input.<\/p>\n\n\n\n \u201cOne example: I work with companies trying to build blockchain-y systems. When designers start reasoning about what to build, they find that systems of formal rules can\u2019t handle the corner cases. Even when they build systems they believe to be stable \u2013 things they hope can\u2019t be gamed \u2013 they still find that runs on the bank can\u2019t be ruled out and can\u2019t easily be solved by creating more rules. Clever, bad actors can still collapse the system. Even if you build incentives to encourage people not to do that, true enemies of the system don\u2019t care about incentives and being \u2018rational actors.\u2019 They\u2019ll attack anyway. If they want to get rid of you, they\u2019ll do whatever it takes, no matter how irrational it seems.<\/p>\n\n\n\n \u201cThe more autonomous you make the system, the more you open it up to interactions with rogue actors who can drive the systems into bad places by hoodwinking the system. Bad actors can collude to make the system crash the stock market, cause you to be diagnosed with the wrong disease, make autonomous cars crash. Think of \u2018dieselgate<\/a>,\u2019 where people could collude by hacking a software system to allowing a company to cheat on reporting auto emissions. Sometimes, it doesn\u2019t take much to foul up the system. There are frightening examples of how few pixels you need to change to make the driverless car navigation system misread the stop sign or the speed-limit sign.<\/p>\n\n\n\n \u201cAnother example of a problem: Even if you build a system where the rules are working well by reading the same environment and making the same decisions, you can run into a \u2018thundering herd problem.\u2019 Say, everyone gets rerouted around a traffic problem to the same side streets. That doesn\u2019t help anyone.<\/p>\n\n\n\n \u201cIn the end, you don\u2019t want to give systems autonomy when it comes to life-and-death decisions. You want accountability. If a battlefield commander decides it\u2019s necessary to put troops at risk for a goal, you want to be able to court martial the commander if it\u2019s the wrong choice for the wrong reasons. If an algorithm has made that catastrophic command decision, where do you go to get justice?<\/p>\n\n\n\n \u201cFinally, I am pessimistic about the future of wide-scale, ubiquitous, autonomous systems because no one is learning from the collective mistakes. One of the enduring problems is that many big companies (as well as others, such as researchers and regulators) do not disclose what didn\u2019t work. Mistakes get buried and failures aren\u2019t shared, these things are prerequisites for people to learn from them.<\/p>\n\n\n\n \u201cIn the large, the same mistakes get made over and over as the collective experience and knowledge base is just not there (as, say, would be the case in the aircraft industry).<\/p>\n\n\n\n \u201cThere is a potential light at the end of this tunnel: the insurance system. They will have a lot to say about how autonomous decision-making rolls out. Will insurers underwrite any of these things? Clearly not, where an autonomous system that can be arbitrarily forced into a failure mode. Underwriting abhors correlations, the resulting claims are an existential risk to their business.<\/p>\n\n\n\n \u201cThe battle over who holds the residual risks extant in autonomous systems is already being played out between the judicial, commercial, insurance and political spheres. Beware the pressure for political expediency, either dismissing or capping the consequences of failure. It may well be that the insurance industry is your greatest ally. Their need to quantify the residual risks for them to underwrite could be the driver that forces the whole industry to face up to issues discussed here.\u201d<\/p>\n\n\n\n Gillian Hadfield<\/strong>, professor of law and chair of the University of Toronto\u2019s Institute for Technology and Society, said, \u201cBy 2035 I expect we will have exceedingly powerful AI systems available to us including some forms of artificial general intelligence. You asked for a \u2018yes-no\u2019 answer, although the accurate one is \u2018either is possible and what we do today will determine which it is.\u2019 If we succeed in developing the innovative regulatory regimes we will need \u2013 including new ideas about constitutions (power-controlling agreements), ownership of technology and access to technology by the public and regulators \u2013 then I believe we can build aligned AI that is responsive to human choice and agency. It is just a machine, after all, and we can decide how to build it. At the same time, it is important to recognize that we already live with powerful \u2018artificially intelligent\u2019 systems \u2013 markets, governments \u2013 and humans do not have abstract, ideal agency and choice within those systems.<\/p>\n\n\n\n \u201cWe live as collectives with collective decision-making and such highly decentralized decisions that constrain any individual\u2019s options and paths. I expect we\u2019ll see more automated decision-making in domains in which markets now make decisions \u2013 what to build, where to allocate resources and goods and services. Automated decision-making, assuming it is built to be respected and trusted by humans because it produces justifiable outcomes, could be used extensively in resolving claims and disputes. The major challenge is ensuring widespread support for decision-making; this is what democratic and rule-of-law processes are intended to do now. If machines become decision-makers, they need to be built in ways [that] earn that kind of respect and support from winners and losers in the decision.<\/p>\n\n\n\n \u201cThe version of the future in which decisions are automated on the basis of choices made by tech owners and developers alone (i.e., implementing the idea that a public services decision should be made solely on the basis of a calculation of an expert\u2019s assessment of costs and benefits) is one in which some humans are deciding for others and reducing the equal dignity and respect that is foundational to open and peaceful societies. That\u2019s a bleak future, and one on which the current tensions between democratic and autocratic governance shed light. I believe democracy is ultimately more stable, and that\u2019s why I think powerful machines in 2035 will be built to integrate into and reflect democratic principles, not destroy them.\u201d<\/p>\n\n\n\n Marcus Foth<\/strong>, professor of informatics at Queensland University of Technology, Australia, responded, \u201cThe question [of] whether humans will or will not be in control of important decision-making in the future is often judged on the basis of agency and control \u2013 with agency and control thought of as good and desirable. Compared to the individual realm of conventional decision-making, most humans come from a culture with a set of values where \u2018being in control\u2019 is a good thing. And there is merit in staying in control when it comes to the usual use cases and scenarios being described in tech utopias and dystopias.<\/p>\n\n\n\n \u201cHowever, I want to raise a scenario where relinquishing individual control and agency can be desirable. Perhaps this is a philosophical\/conceptual thought experiment and deemed unrealistic by many, but perhaps it is nonetheless useful as part of such futuring exercises. Arguably, the types of wicked problems humanity and the planet face are not a result of lacking scientific ingenuity and inventiveness but a lack of planetary governance that translates collective wisdom and insights into collective action. While we sometimes see positive examples such as with the rapid response to the COVID-19 pandemic, my overall assessment suggests there continue to be systemic failures in the systems of planetary governance. I argue that maintaining individual human agency and control as a value is partly to blame: Human comfort, safety, control and convenience always triumph over planetary well-being. Would relinquishing individual human control in favour of collective human control offer a more desirable future scenario of governance systems that serve not just the well-being of (some) humans but also forgotten humans \u2018othered\u2019 to the fringes of public attention, as well as more-than-humans and the planet?<\/p>\n\n\n\n \u201cIn essence, what I propose here is arguably nothing new: Many First Nations and indigenous peoples have learnt over millennia<\/a> to act as a strong collective rather than a loose amalgamation of strong-minded individuals. Relationality \u2013 or as Mary Graham calls it, the \u2018relational ethos<\/a>\u2019 \u2013 is a key feature of good governance, yet despite all the AI and tech progress we still have not been able to achieve a digitally supported system of governance that bestows adequate agency and control to those who have none: minority groups of both human and nonhuman\/more-than-human beings.<\/p>\n\n\n\n \u201cThis is why I think having the typical humans (who are in control now) not<\/em> being in (the same level of) control of important decision-making in the year 2035 \u2013 is absolutely a good thing that we should aspire toward. The alternative I envisage is not the black-and-white opposite of handing control over to the machines, but a future scenario where technology can aid in restoring the relational ethos in governance that serves all humans and more-than-humans on this planet.\u201d<\/p>\n\n\n\n Gary Grossman<\/strong>, senior vice president and global lead of the Edelman AI Center for Excellence, previously with Tektronix, Waggener Edstrom and Hewlett-Packard, observed, \u201cThe U.S. National Security Commission on Artificial Intelligence<\/a> concluded in a 2021 report to Congress that AI is \u2018world-altering.\u2019 AI is also mind-altering, as the AI-powered machine is now becoming the mind. This is an emerging reality of the 2020s. As a society, we are learning to lean on AI for so many things that we could become less inquisitive and more trusting of the information we are provided by AI-powered machines. In other words, we could already be in the process of outsourcing our thinking to machines and, as a result, losing a portion of our agency.<\/p>\n\n\n\nThe higher the risk of the AI system being wrong and the higher the consequences of a bad decision, the more humans should be in control<\/h3>\n\n\n\n
\n
Scenarios for 2035 and beyond are likely to range from humans benefiting from \u2018machines of loving grace\u2019 to being under the thumb of digital dictators<\/strong><\/h3>\n\n\n\n
\n
\n
In automated decisions relying on human data, the human point of contact is often culturally framed by institutions that do not represent everyone<\/h3>\n\n\n\n
\n
Better tech and better data have improved human decision-making; they are also whittling away at human agency \u2013 without us even realizing it<\/h3>\n\n\n\n
\u2018Today we are not alone; our agency is now shared with our tools\u2019<\/h3>\n\n\n\n
Focus on who has power over whom, and who has the power to shape technologies to reinforce that structure of power<\/h3>\n\n\n\n
We may face \u2018a form of chilling human enfeeblement, a dependence on powerful devices coupled with an indifference to this imbalance of power\u2019<\/h3>\n\n\n\n
\n
We need a large button \u2013 a warning mechanism to clearly tell the AI what we want to cede to it and what we want to control ourselves<\/strong><\/h3>\n\n\n\n
We must \u2018build technologies that support human self-efficacy, creativity, responsibility and social connectedness\u2019<\/strong><\/h3>\n\n\n\n
\n
Enshrine it in legislation: Everyone should have the right to challenge the outcome of an autonomous decision<\/strong><\/h3>\n\n\n\n
Computational systems should be designed to co-evolve, co-mature or co-develop with humans<\/strong><\/h3>\n\n\n\n
\u2018Humans must not be left to feel helpless and hopeless\u2019; they must be able to be owners of their own identities and correct errors in a timely fashion<\/h3>\n\n\n\n
\n
\u2018Delegating authority can itself be a proper use of autonomy\u2019<\/h3>\n\n\n\n
\u2018The public and Big Tech must learn how to build equity into AI and know what levers to pull to assure that it works for the good of humanity\u2019<\/h3>\n\n\n\n
\u2018Where\u2019s the incentive for tech companies to make design choices in favor of human agency?\u2019<\/strong><\/h3>\n\n\n\n
In many industries \u2018the scope of technological change is far beyond the scope, scale and speed of needed regulatory change\u2019<\/h3>\n\n\n\n
We are prone to offloading our tasks and responsibilities to machines, resulting in \u2018the engineering paradigm overriding the ethical one\u2019<\/strong><\/h3>\n\n\n\n
To build autonomous systems you must handle difficult edge cases and account for bad actors, and can\u2019t account for every possible danger<\/h3>\n\n\n\n
Democratic processes might determine how these systems make decisions<\/h3>\n\n\n\n
We could overcome human shortcomings by having individuals relinquish their control of some decisions<\/h3>\n\n\n\n
The risk is that we will increasingly live our lives on autopilot; it can make us less able to change and less conscious of who we are<\/h3>\n\n\n\n