My theory is that you already have something providing ssh agent service
in the past some xserver environments started an ssh-agent for you just in case of, and for some reason i don't remember that was annoying and i disabled it to start my agent in my shell environment as i wanted it.
also a possibility is tharlt there are other agents like the gpg-agent that afaik also handles ssh keys.
but i would also look into $HOME/.ssh/config if there was something configured that matches the hostname, ip, or with wildcards* parts of it, that could interfere with key selection as the .ssh/id_rsa key should IMHO always be tried if key auth is possible and no (matching) key is known to the ssh process, that is unless there already is something configured...
not sure if a system-wide /etc/ssh/ssh_config would interfere there too, maybe have a look there too. as this behaviour seems a bit unexpected if not configured specially to do so.
replace passphrase with private key and you're very correct.
passphrases used to login to servers using PasswordAuthentication are not stored in the agent. i might be wrong with technical details on how the private key is actually stored in RAM by the agent, but in the context of ssh passphrases that could be directly used for login to servers, saying the agent stores passphrases is at least a bit misleading.
what you want is:
also an idea:
all depends on the level of security you want to achieve. additional TOTP could improve security too (but beware that some authenticator providers might have "sharing" features which could compromise the TOTP token even before its first use.