Seting up sudo access with PAM and ssh-agent

For Fedora, first install the following package:

$ sudo yum install pam_ssh_agent_auth

Then edit /ets/sudoers, and add the following line:

Defaults    env_keep += "SSH_AUTH_SOCK"
Defaults    timestamp_timeout  = 0  # Not necessary, but turns off caching

Finally, edit /etc/pam.d/sudo, and add something like this (this should be adjusted to your preferences):

auth       sufficient file=~/.ssh/authorized_keys

Update: On Debian 9, I installed

$ sudo apt-get install libpam-ssh-agent-auth