Reinforcement learning (RL) techniques offer a versatile and powerful extension to the toolkit for computer scientists and marketplace designers for their use in online marketplaces. As the use of these techniques continues to expand, their application in online marketplaces raise questions of their appropriate use, particularly around issues of fairness and market transparency. I argue that the use of RL techniques, alongside similar calls in domains such as automated vehicle systems, is a problem of sociotechnical specification that faces a set of normative and regulatory challenges unique to marketplaces. I provide a selective overview of the RL literature as applied to markets to illustrate challenges associated with the use of RL techniques in online marketplaces. I conclude with a discussion of capacity-building in research and institutions that is required in order for benefits from algorithmically managed marketplaces to be realized for stakeholders and broader society. 

Deciding What's Fair: Challenges of Applying Reinforcement Learning in Online Marketplaces

Machine learning (ML) methods already permeate environmental decision-making, from processing high-dimensional data on earth systems to monitoring compliance with environmental regulations. Of the ML techniques available to address pressing environmental problems (e.g., climate change, biodiversity loss), Reinforcement Learning (RL) may both hold the greatest promise and present the most pressing perils. This paper explores how RL-driven policy refracts existing power relations in the environmental domain while also creating unique challenges to ensuring equitable and accountable environmental decision processes. We focus on how RL technologies shift the distribution of decision-making, agenda-setting, and ideological power between resource users, governing bodies, and private industry. 

Power and Accountability in RL-driven Environmental Policy

Rapid advancements in artificial intelligence (AI) along with the deployments of more data-driven pervasive technologies are currently offering incredible possibilities and solving problems in innovative ways. The pace of AI's development is so fast that it is hard to follow in details all the ethical and social implications. Therefore, it is legitimate to assess the effect of AI-based technologies on day to day quality of life. This paper analyzes the effect of the user's Quality of Experience (QoE) when using these new AI-based services on the Quality of Life (QoL). In particular, this paper focuses on two aspects: security and democratization. The security threats of any new technology defines the QoE especially nowadays after revealing warning signs regarding privacy threats and lack of public trust. Furthermore, democratization perception in terms of technology used is also a crucial feature that governs steering AI development towards ethically serving the interests of the society; and therefore improving the QoE and QoL.  Hence, we propose a framework to provide comprehensive analysis of the current status in the existing literature while focusing on the anticipated tools for democratization. In addition, we highlight some research gaps and suggest future opportunities to be addressed.

Aligning AI with Security and Democracy Perceptions to Improve QoE and QoL

This paper develops an approach to assess reinforcement learners with collusive pricing policies in a testing environment. We find that algorithms are unable to extrapolate collusive policies from their training environment to testing environments. Collusion consistently breaks down, and algorithms instead tend to converge to Nash prices. Policy updating with or without exploration re-establishes collusion, but only in the current environment. This is robust to repeated learning across environments. Our results indicate that frequent market interaction, coordination of algorithm design, and stable environments are essential for algorithmic collusion.

Robust Algorithmic Collusion

With AI systems becoming more powerful and pervasive, there is increasing debate about keeping their actions aligned with the broader goals and needs of humanity.  This multi-disciplinary and multi-stakeholder debate must resolve many issues, here we examine two of them.  The first is to clarify what demands stakeholders might usefully make on the designers of AI systems, useful because the technology exists to implement them.  We introduce the framing of cognitive architectures to make this technical topic more accessible.  The second issue is how stakeholders should calibrate their interactions with modern machine learning researchers.  We consider how current fashions in machine learning create a narrative pull that participants in technical and policy discussions should be aware of, so that they can compensate for it.  We identify several technically tractable but currently unfashionable options for improving AI alignment.

Demanding and Designing Aligned Cognitive Architectures

Public policies that supply public goods, especially those involve collaboration by limiting individual liberty, always give rise to controversies over governance legitimacy. Multi-Agent Reinforcement Learning (MARL) methods are appropriate for supporting the legitimacy of the public policies that supply public goods at the cost of individual interests. Among these policies, the inter-regional collaborative pandemic control is a prominent example, which has become much more important for an increasingly inter-connected world facing a global pandemic like COVID-19. Different patterns of collaborative strategies have been observed among different systems of regions, yet it lacks an analytical process to reason for the legitimacy of those strategies. In this paper, we use the inter-regional collaboration for pandemic control as an example to demonstrate the necessity of MARL in reasoning, and thereby legitimizing policies enforcing such inter-regional collaboration. Experimental results in an exemplary environment show that our MARL approach is able to demonstrate the effectiveness and necessity of restrictions on individual liberty for collaborative supply of public goods. Different optimal policies are learned by our MARL agents under different collaboration levels, which change in an interpretable pattern of collaboration that helps to balance the losses suffered by regions of different types, and consequently promotes the overall welfare. Meanwhile, policies learned with higher collaboration levels yield higher global rewards, which illustrates the benefit of, and thus provides a novel justification for the legitimacy of, promoting inter-regional collaboration. Therefore, our method shows the capability of MARL in computationally modeling and supporting the theory of calculus of consent, developed by Nobel Prize winner J. M. Buchanan.

NeurIPS 2021 Workshop PERLS Submissions

Deciding What's Fair: Challenges of Applying Reinforcement Learning in Online Marketplaces

Power and Accountability in RL-driven Environmental Policy

Aligning AI with Security and Democracy Perceptions to Improve QoE and QoL

Robust Algorithmic Collusion

Demanding and Designing Aligned Cognitive Architectures

Calculus of Consent via MARL: Legitimating the Collaborative Governance Supplying Public Goods