Something else worth mentioning; SSH has built-in support for single-sign-on, via SSH agent forwarding. As long as my public key is available anywhere (and that's exactly what it's designed for), then I can be authenticated by any system, anywhere. Thus, a problem which is so vexing in so many other scenarios, is very cleanly addressed by SSH.
Fundamentally, a password is a shared secret. So you send your password to a server, you are trusting that server not to lose or misplace it. In contrast, an SSH public key doesn't require nearly such careful management.
SSH agent forwarding is extremely risky. Anyone with appropriate permissions (legitimate or illegitimately gained) on a machine you have connected to can use your credentials to open a new connection to any machine you have access to.
This is a very valid issue. I have been using ssh-ident* for this reason. You can add keys to ~/.ssh/identities and it'll start up an agent for each identity when you use it. This means if the machine you connect to is indeed stealing your keys, at least they only have one key, not all the keys you're currently using.
There are some caveats; such as having to specify what domains/IPs should use which identity, and also having to alias ssh to get it to work nicely. So it's not perfect, but it does solve the problem you talked about.
Don't you have to be actively connected to the machine for this to work? i.e. the server I haven't connected to for a few months has no way of opening connections on my behalf at this point, right?
But once you connect, a year later - in those 3 minutes before you disconnect, the attacker might have authenticated as yourself to 100 other machines -- and appended their own key to .ssh/authorized_keys on these machines, so that the compromise no longer needs you to be connected.
Agreed! On nomenclature - you're saying ssh agent forwarding, but the functionality you describe is actually simply public keys on remote machines. The only time you need to forward is when you're logging into another machine through a third machine -- and as other commenters have pointed out, you're extending your trust model to that third machine.
Fundamentally, a password is a shared secret. So you send your password to a server, you are trusting that server not to lose or misplace it. In contrast, an SSH public key doesn't require nearly such careful management.