Utah audit of Banjo deal highlights concerns with AI, government contracts

SALT LAKE CITY — Government entities should be careful with individuals' sensitive data, investigate a technology company's claims about artificial intelligence or machine learning, and assess possible risks before entering into an agreement.

Those are a few of the recommendations and guidelines from an audit report released Monday, which was sparked by Utah's multimillion-dollar agreement with Utah-based tech firm Banjo to use surveillance and develop a crime-protection program. That agreement was scrutinized and halted last year after the alleged racist past of the company's CEO was brought to light.

In all, the Commission on Protecting Privacy and Preventing Discrimination led by the Office of the Auditor outlined 12 points for government entities to consider before entering an agreement with an artificial intelligence or machine learning-centered tech company.

Those are:

• Limit sharing of sensitive data
• Minimize sensitive data collection and accumulation
• Validate technology claims, including capability review
• Rely on objective, repeatable metrics
• Assess threat models
• Perform in-depth review of artificial intelligence or machine learning algorithms
• Demonstrate privacy compliance: privacy-specific items and protection
• Review steps taken to mitigate discrimination
• Determine ongoing validation procedures
• Require vendor to obtain consent of individuals contained within training datasets
• Vet key vendor personnel
• Evaluate vendor corporate management and vendor solvency

"As we began our work, it became clear that the availability of some clear, thoughtful principles aimed at these emerging technologies would be a powerful resource for Utah's government agencies," said Utah Auditor John Dougall, in a statement after the report was released.

"As the depth of expertise on the commission would not be easy to re-create for every potential contract for every agency or entity, these new documents help capture that expertise for use statewide to help protect Utahns' privacy and prevent discrimination against them," he added.

Banjo's deal with Utah

The commission was formed last summer during the fallout from the Banjo deal. The Utah-based company rose in the 2010s as a company that promised the "ethical use of artificial intelligence and data."

In 2019, then-CEO and co-founder Damien Patton told KSL NewsRadio that his company scanned millions of social media posts to identify when and where criminal activity was happening. It eventually landed a $20.7 million deal with the Utah Attorney General's Office. Attorney General Sean Reyes praised the deal in 2019 as "incredible technology" that could help law enforcement and first responders.

But the office froze its contract with Banjo in April 2020 after a report about Patton's previous ties with the Ku Klux Klan in the late 1980s and earlier 1990s surfaced. He stepped down from the company on May 8.

The Office of the Auditor announced in June 2020 that it had formed the Commission on Protecting Privacy and Preventing Discrimination designed to review the state contract and its use for public safety. The commission included police, tech experts, business experts and community activists.

Audit findings

Over the past few months, auditors used what they learned from the Banjo contract to piece together advice for state officials when it comes to major tech contracts.

That starts with a government entity fully understanding the data they have. All data that includes personal information "should be filtered and restricted within the government's systems before being transferred into the vendor's application," the report states. The software shouldn't be able to collect sensitive data that isn't needed.

Third-party vendors should also be able to "clearly demonstrate" the success of their marketing claims.

"Do not rely on anecdotes as validation of these claims," the commission wrote. "Government entities should invest in software applications where the value can be measured on an ongoing basis. A reputable vendor should include success criteria in any Request For Proposal (RFP) response, and these should include metrics that are easy to measure and compare across time and vendors."

The vendor should also be able to provide a government entity with risks or threats to their software. They should also be able to demonstrate their ability to comply with privacy regulations.

The report states that vendors must secure consent when capturing biometric characterizations such as through facial recognition or gait analysis technology.

The commission was also formed following the death of George Floyd while in Minnesota police custody in May 2020. His death led to worldwide protests and reopened discussions about social injustice and police reform. The report also tackled concerns about discrimination.

The commission pointed out that sources of data may include implicit or historic biases; it's also possible that a model could introduce new biases.

"Ensure that the vendor has considered the question of bias and discrimination within their software application and that the vendor has mechanisms, such as audit results, to demonstrate that their software application does not disproportionately affect various categories of individuals, particularly any federally protected class," the report recommended.

Given that key issues with Banjo arose from a report about its CEO's past, the report also states that entities should vet "key vendor personnel," especially if they have access to sensitive information. At the same time, an entity should also evaluate the financial history of a third-party vendor "to ensure they can carry out the contract."

The full report guidelines can be found here. A companion document was also released Monday, which can be read here.

"The findings and recommendations made by the commission will ensure that law enforcement agencies are protecting the privacy of individuals, preventing discrimination and at the same time improving the process of vetting vendors," Salt Lake County Sheriff Rosie Rivera said in a statement. Rivera is also a member of the Commission on Protecting Privacy and Preventing Discrimination.

Jeanetta Williams, president of NAACP's Salt Lake Branch and another commission member, added that she was appreciative that the commission was able to include concerns about discrimination.

"Technology is growing increasingly powerful, and it is very easy for discrimination to be perpetuated unless our public entities are careful to prevent it," Williams said.

The state attorney general's office also issued a statement Monday in response to the audit. It supported the recommendations, standards and benchmarks detailed in the commission's report.

"This will be valuable to our office, law enforcement statewide, and all levels of government," the statement reads, in part.

"The AG's Office remains committed to protecting the privacy and civil rights of Utahns, and appreciates the work and expertise of those who contributed to this report," the statement continues. "Within those guidelines, our office also remains committed to innovation and keeping Utah safe with the most effective crime-fighting tools available."

Related topics