Comment by no_wizard
Because whats inside our minds is more than mathematics, or we would be able to explain human behavior with the purity of mathematics, and so far, we can't.
We can prove the behavior of LLMs with mathematics, because its foundations are constructed. That also means it has the same limits of anything else we use applied mathematics for. Is the broad market analysis that HFT firms use software for to make automated trades also intelligent?
Your first sentence is a non-sequitur. The fact that we can't explain human behavior does not mean that our minds are more than mathematics.
While absence of proof is not proof of absence, as far as I know, we have not found a physics process in the brain that is not computable in principle.