Description
AI's reasoning is hidden. With code, you can open it up and see what's there - no mystery. With AI, you only see inputs and outputs, not what's inside.
AI has something resembling a human mental model, but you can't inspect it.
Impact
- Misalignment between your mental model and AI's goes undetected
- Hard to even suspect something is wrong
