image: Héctor Muñoz-Avila, professor of computer science and engineering at Lehigh University and director of the Institute for Data, AI, Society, and Computation (I-DISC), is leading NSF-funded research to improve the transparency and accountability of AI-generated recommendations.
Credit: Lehigh University
People are increasingly turning to large language models (LLMs) for advice on everyday matters, such as where to stay on vacation. But these systems are also being used to inform consequential decisions involving disease diagnosis, law enforcement, and managing electricity demand. How they arrive at their conclusions is often a black box, and users can’t always be sure that the guidance they’ve been given is accurate and fair.
“It’s called the AI alignment problem,” says Héctor Muñoz-Avila, professor of computer science and engineering in Lehigh University’s P.C. Rossin College of Engineering and Applied Science and director of the Institute for Data, AI, Society, and Computation (I-DISC). “It’s the idea that we want AI recommendations to be consistent with human and societal values and based on facts.”
Muñoz-Avila and his team recently received an Early-Concept Grant for Exploratory Research (EAGER) from the National Science Foundation. The program supports exploratory research involving untested but potentially transformative ideas.
The researchers will develop a model that uses causal auditing and symbolic reasoning to improve the transparency and accountability of LLM-based recommender systems. Rather than inspecting an LLM’s internal reasoning directly, the model will infer the assumptions and factors influencing a recommendation by varying the information, constraints, and preferences presented to the system and observing how its recommendations change.
Causal auditing examines the underlying factors behind AI recommendations, while symbolic reasoning represents and analyzes the relationships among those factors and the resulting recommendation. Together, these approaches could help users better understand why a particular recommendation was made and what information influenced it.
Testing the assumptions behind AI advice
In some situations, says Muñoz-Avila, users may know that a particular recommendation is appropriate. But in other scenarios, such as the aftermath of a hurricane, the guidance may be less reliable. That’s because neither the user nor the AI has a complete picture of conditions on the ground, including how much the storm may have affected infrastructure and access.
“After a disaster, the control center for a rescue operation can be chaotic and operating with limited information,” he says. “So when rescuers are describing a scenario and asking the AI if they should send a rescue team to Area A or Area B, the AI may recommend the team go to Area A. But that recommendation may implicitly depend on the road into that location being open and accessible. And that assumption may not be supported by the available information."
The proposed model will essentially audit these recommendations. It will review both the user’s input and the AI’s output, looking for inconsistencies between the intended goal of the recommendation and the facts, constraints, and preferences on which the recommendation appears to depend. In the rescue scenario, that means evaluating the AI’s recommendation about which region to target.
To identify potential inconsistencies, the model will test counterfactuals, or hypothetical alternatives, such as “Would Area A still be recommended if the road was inaccessible?” If the AI changes course and recommends sending rescuers to Area B, the system could flag that the original recommendation was sensitive to road access, prompting the command center to verify whether the road is accessible.
Toward real-time AI auditing
Muñoz-Avila envisions the model as a plug-in that users could one day use to audit AI recommendations in real time.
“Our system could be looking at what the user is typing and what the system is giving them,” he says. “And when there’s an inconsistency, a window could open up and explain what the inconsistency is.”
With the user’s permission, the tool could then query the AI directly with follow-up questions to test or refine its recommendations based on the available information.
“These systems guide decisions that affect our health, our economy, and our natural resources,” he says. “We want to make them as reliable as possible.”