If your .ssh directory and files have the wrong permissions, ssh-add will fail.
chmod 700 ~/.ssh chmod 644 ~/.ssh/authorized_keys chmod 644 ~/.ssh/known_hosts chmod 600 ~/.ssh/*_rsa chmod 644 ~/.ssh/*_rsa.pub