Humans see what they believe

Users make sense of AI behaviour to figure out how to make it do what they want, but they don't weigh all of their observations equally. They approach AI with expectations informed by their prior experiences and selectively attend to feedback that conforms to their beliefs. Users misinterpret AI behaviour in ways that preserve the tenability of their existing mental models and they miss potentially corrective feedback because they're looking in the wrong places or don't know what to look for.

Wishful thinking yields spurious causation

This expectation-driven learning process produces classifiable mental model errors. Users come to believe they caused actions the AI performed or doubt they can cause actions they genuinely direct. They come up with overly elaborate explanations for AI behaviours that they want to believe they can influence and these explanations do not account for AI behaviours that users do not notice. The forms of these errors follow from what users want AI to help them do and the role that they want to play in their cooperation. The resulting failures are systematic and predictable from the conditions under which they arise.

Confusion causes ritualistic patterns of misbehaviour

When users believe they can make AI do what they want, they act on that belief. They perform actions with no effects in synchrony with AI behaviour, because they think that's what makes it do what they want. They perform actions with no specific intentions, believing that any input is better than none. They modify how they provide input to the AI to signal their desires, even when the AI is insensitive to the techniques that users believe make a difference. Users who find causation in coincidence repeat superstitious behaviors that reveal what they misbelieve.

Users can hurt themselves in their confusion

Confused users develop emotional and analytical attitudes that change how they attend to the AI's output. Although frustration initially spurs critical observation and hypothesis testing, users who try to influence AI behaviour find ways to do so, regardless of whether their methods truly work. Recognizing their mistakes and realizing that they cannot influence the AI how they want to can make users feel uninvolved in the interaction. Disengaged and uncritical users stop attending to the feedback that could correct their mental models, stop noticing when their actions fail, and settle into behavioral patterns that only feel effective. The result is a self-reinforcing cycle: confusion produces attitudes that produce behaviors that produce further confusion.