SSH fails to connect on private interface over IPSec tunnel

Posted on

QUESTION :

I have Client and Server in different locations connected via an IPSec tunnel, if I try to ssh from Client to Server over the internet everything works fine. I have tried from several Linux and Cygwin installations and the same thing happens from each client. I When I try to ssh from Client to Server’s internal address, the connection hangs with the message:

debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]

Client’s last message is:

debug1: SSH2_MSG_KEXINIT sent

Initially I thought this would be due to an outdated version of ssh on Client not being able to auth properly, but it works without issue over the public interface. Server accepts connections without issue from clients on the same LAN.

Client is 172.24.20.228/23
Server is 192.168.2.24/24 and ssh.domain.tld publicly.

debug info:
Client over public:

$ ssh user@ssh.domain.tld -v -p 2222
OpenSSH_6.7p1, OpenSSL 1.0.1k 8 Jan 2015
debug1: Connecting to ssh.domain.tld [1.2.3.4] port 2222.
debug1: Connection established.
debug1: identity file /home/robbiecrash/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_rsa-cert type -1
debug1: identity file /home/robbiecrash/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA KEY:KEY:KEY
debug1: Host '[1.2.3.4]:2222' is known and matches the ECDSA host key.
debug1: Found key in /home/robbiecrash/.ssh/known_hosts:9
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/robbiecrash/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Offering DSA public key: /home/robbiecrash/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 434
debug1: Authentication succeeded (publickey).
Authenticated to ssh.domain.tld ([1.2.3.4]:2222).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.

Server shows nothing unexpected.

Client over internal:

$ ssh user@192.168.2.24 -vvv -p 2222
OpenSSH_6.7p1, OpenSSL 1.0.1k 8 Jan 2015
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.2.24 [192.168.2.24] port 2222.
debug1: Connection established.
debug1: identity file /home/robbiecrash/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_rsa-cert type -1
debug1: identity file /home/robbiecrash/.ssh/id_dsa type 2
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/robbiecrash/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [192.168.2.24]:2222
debug3: load_hostkeys: loading entries for host "[192.168.2.24]:2222" from file "/home/robbiecrash/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug1: SSH2_MSG_KEXINIT sent

It then hangs there forever. The server shows this:

root@thoth:/var/log# /usr/sbin/sshd -d -p 2222
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: private host key: #3 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2222'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 172.24.20.228 port 5448 on 192.168.2.24 port 2222
debug1: Client protocol version 2.0; client software version OpenSSH_6.7
debug1: match: OpenSSH_6.7 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3
debug1: permanently_set_uid: 104/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: client->server aes128-ctr umac-64-etm@openssh.com none [preauth]
debug1: kex: server->client aes128-ctr umac-64-etm@openssh.com none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]

and exits when I CTRL+C the client:

Connection closed by 172.24.20.228 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 29528

ANSWER :

This looks to me like an MTU issue. There are a number of ways this can be solved/mitigated depending on your setup.

If your router supports it, the most convenient way is probably to do MTU clamping. Alternatively, you can try setting a smaller MTU on the server and client – although that is less then ideal.

(While I new all of this from hard experience already, I just discovered that SSH sets a “Don’t Fragment” bit in the packet header, which compounds MTU problems)

Leave a Reply

Your email address will not be published. Required fields are marked *