If your .ssh directory and files have the wrong permissions, ssh-add will fail.

chmod 700 ~/.ssh
chmod 644 ~/.ssh/authorized_keys
chmod 644 ~/.ssh/known_hosts
chmod 600 ~/.ssh/*_rsa
chmod 644 ~/.ssh/*_rsa.pub