Blog 2: Procurement for Employing and Using AI

In a 2021 primer on AI and Procurement, Mona Sloane, Rumman Chowdhury, and colleagues write: “Procurement is the gateway for technology infrastructure implementation, and therefore has long term effects on cities, communities, and on agencies themselves.” As such AI procurement is an important governance tool of AI Localism. The terms for purchasing AI technologies and services can determine the way they are designed and/or delivered, and can also shape markets outside of government. Writing in The Regulatory Review, Lavi M. Ben Dor and Cary Coglianese reaffirm the need for local AI governance to protect residents and their data from use in private sector AI for public purposes without oversight. They point to a case in Texas, where the school board acquired an algorithm created by a private company to evaluate teachers, but the method by which decisions were made by the technology was concealed under trade secret protection. Such instances demonstrate the need for standards to assess, deploy, and monitor how governments procure AI technology and consider its risks.

In the below, we provide a few examples of how cities have leveraged procurement and ordinances to tackle and prevent some of the harmful impacts the use of AI may have on residents, in particular, vulnerable communities, such as AI bias resulting in discrimination and marginalization. Specifically, we discuss efforts to address the threat of racial profiling by surveillance technology and the growing body of governance practices mandating disclosure and routine reporting on the acquisition and use of AI by local governments.

Public AI Procurement 

• In Baltimore, Maryland, the city council passed a restrictive ordinance banning public agencies and private citizens from purchasing facial recognition technology, criminalizing the technology. Balancing explainability and transparency of AI tools created or bought by governments is needed to allow for accountability to constituents of how their data is being used, and by which mechanisms. 
• Berkeley, California, passed an ordinance that requires surveillance technology procured by city agencies to demonstrate the risk and benefits associated with specific technologies before getting permission to procure these tools. The Detroit City Council similarly approved an ordinance requiring transparency of all public surveillance and procurement processes for greater local awareness and transparency. 
• California’s state assembly introduced a bill that would require prospective contractors for high-risk applications, including automated decision systems, to submit an impact assessment in order to be considered for the contract. Similarly, the city of Amsterdam developed ‘contractual terms,’ including transparency and explainability of how the algorithms function, for algorithms purchased by local governments from third-party suppliers.
• In 2022, the state of Vermont proposed a bill that requires the Secretary of Digital Services to audit all automated decision systems created or procured by state agencies, as well as outline standards for procurement development. The New York City Council also requires public agencies to annually disclose information about AI systems and the data collected and analyzed by the tool.
• In 2016, amid growing concerns about the use of surveillance technology in Oakland, California, the city formed a permanent Privacy Advisory Commission (PAC), the first of its kind in the United States. The PAC consists of nine experts who advise the city council on the “purchase and use” of surveillance technology and help craft policies and legislation related to data use and privacy with input from locals to create a smart city model that is privacy- and resident-focused.
• Washington state created an automated decision working group, which convenes representatives across sectors to develop recommendations for enhanced procurement and auditing of AI systems used by the local government.

Conclusion

Local governments are imposing checks and balances to oversee how AI tools are procured in order to ensure constituents’ safety and avoid corporate monopolization of AI service delivery to the public sector. Recent governance practices have set in motion new rules of play that prioritize human impact over technological novelty. Indeed, AI can help streamline government practices and improve digital transformation and smart city initiatives. 

Procurement of local AI tools is incredibly important for local policymakers to drive further data-driven actions and improve existing services. The above has provided a peek into the ways such governance is occurring, demonstrating a key lesson—that local AI use requires risk assessment, public awareness, and transparency to strengthen trust and justify decisions made with AI to residents.

***

We are deeply grateful to Hubert Beroche, President of Urban AI, Christophe Mondin, Researcher at CIRANO, Mona Sloane, Sociologist at New York University and University of Tübingen AI Center, and Ben Snaith, Researcher at the Open Data Institute, for reviewing this blog.

In our next blog post, we will discuss how other local institutions engage, learn, and develop AI for a more informed public.