I use ssh, scp, sshfs and x2go with key-based authentication, ie. not entering remote passwords when I connect. There are pros and cons of this, but I think it is more secure.

I use a non-empty passphrase but do not want to enter it on every connection. Once per X session is enough for me.

Here is how I set it up on Debian 11 (“bullseye”):

# do this once and set a good passphrase:
ssh-keygen
# then for each of your accounts on remote hosts:
ssh-copy-id username@otherhost

Configure ssh agent and ssh-add to run when your X session starts. I use the gnome-keyring service as agent:

sudo apt install gnome-keyring

In XFCE – Settings – Session and Startup – Application Autostart, I have two entries with trigger “on login” :

• “SSH Key Agent (GNOME Keyring: SSH Agent)”
• “ssh-add” – created by me, command: ssh-add

This setup will bring up a visual prompt for your ssh passphrase right after XFCE login. The default ssh-askpass looks quite ugly, so I installed a more modern one:

sudo apt install ssh-askpass-gnome

On Debian, that package sets itself as default ssh-askpass “alternative”. If in doubt , try this:

sudo update-alternatives --config ssh-askpass

The resulting prompt looks like this for me (“Adwaita Dark” theme):

After all this your ssh, scp, sshfs, x2go and other ssh based tools should be able to connect to your remote accounts without password prompts.

Update: I took convenience one step further and enabled “Launch GNOME services on startup” in the Advanced tab of Session and Startup in the XFCE setting, as described in the XFCE wiki.

This activates GNOME Keyring which “is integrated with the user’s login, so that their secret storage can be unlocked when the user logins into their session”. This means it will store your once entered ssh passphrase on disk, using your Linux login as the only secret that you still have to enter (as you log in as usual).