But simply knowing the right words to say in response to a moral conundrum isn't the same as having an innate understanding of what makes something moral. The researchers also reference a previous study showing that criminal psychopaths can distinguish between different types of social and moral transgressions, even as they don't respect those differences in their lives. The researchers extend the psychopath analogy by noting that the AI was judged as more rational and intelligent than humans but not more emotional or compassionate.
This brings about worries that an AI might just be "convincingly bullshitting" about morality in the same way it can about many other topics without any signs of real understanding or moral judgment. That could lead to situations where humans trust an LLM's moral evaluations even if and when that AI hallucinates "inaccurate or unhelpful moral explanations and advice."
Despite the results, or maybe because of them, the researchers urge more study and caution in how LLMs might be used for judging moral situations. "If people regard these AIs as more virtuous and more trustworthy, as they did in our study, they might uncritically accept and act upon questionable advice," they write.
Great, so the headline of the article directly feeds into the issue the scientists are warning about when it comes to public perception of AI morality