How do I properly SSH into a server behind a NAT while maintaining the tightest possible security?

I have a VPC in Amazon AWS. There’s a NAT Server running in a Public Subnet that connects to an Internet Gateway. I have a bunch of servers running in various private subnets within the VPC. I’d like to SSH into the servers that are in the private subnet. All my servers are running AWS Linux (CentOS).

Currently, I can SSH into the NAT Server using my private key. The NAT server allows SSH connections only from my current development IP. Then I can SSH into the servers in the private subnets only if I setup SSH Login on those servers, or if I put a key file on the NAT server and then use them to SSH. For security, it seems like I shouldn’t do either of these things.

Is there a preferred best-practice way to make this connection? It seems like there should be a way to connect with a single SSH call from my home development machine running Apple OSX.

You should not put your secret key on the gateway, and you don’t have to 🙂

setup your local SSH config so you can use the NAT gateway for port forwarding when you need it:

create an entry in your ~/.ssh/config that sets up local forwards to the hosts you want to connect to:

Host natgw-fwd
        User ec2-user
        HostKeyAlias natgw-fwd.my.domain
        HostName 54.182.32.11
        LocalForward 1025 10.0.2.1:22

then add one entry per host forwarded with an HostKeyAlias:

Host internal-one
        User ec2-user
        HostKeyAlias internal-one.ec2.internal
        HostName localhost
        Port 1025

bring the tunnel up in one shell:

ssh -C -v natgw-fwd

and connect to the internal hosts in another shell:

ssh internal-one

When using lots of “quick + short” connections in addition to a shell or two, as with a tool like dsh, latency for setup and teardown of single connections gets more noticeable, and I use ControlMaster and ControlPath to enable connection sharing. The limitations don’t bother me because I rarely use agent or X11 in such a scenario.

I made some research and found an article that appears to be related to what you are asking about.

SSH and bastion servers

By default, Linux instances in EC2 use SSH key files for authentication instead of SSH usernames and passwords. Using key files can reduce the chance of somebody trying to guess the password to gain access to the instance. But using key pairs with a bastion host can present a challenge—connecting to instances in the private subnets requires a private key, but you should never store private keys on the bastion.

One solution is to use SSH agent forwarding (ssh-agent) on the client. This allows an administrator to connect from the bastion to another instance without storing the private key on the bastion. That’s the approach I’ll discuss in this post.

http://blogs.aws.amazon.com/security/post/Tx3N8GFK85UN1G6/Securely-connect-to-Linux-instances-running-in-a-private-Amazon-VPC

Hopefully you will find the solution in creating a bastion server and using SSH agent forwarding in your client as the article recommends.

@Don King’s answer was very helpful and here is the tutorial he suggested. It has instructions for OSX and Windows.

Here is the summary of what I did when logging in from a local OSX machine. This assumes that the basic infrastructure is already correct in your VPC. See this tutorial for details.

1. Open an outbound SSH connection on the NAT security group that allows connections to the security group of the server in the private subnet.

2. Open an inbound SSH connection on the security group of the server in the private subnet that allows a connection from the NAT security group.

3. On OSX, add your public_key.pem to your local keychain: ssh-add -K .aws/public_key.pem

4. SSH from your local machine to the NAT using -A (ssh-agent): ssh -A ec2-user@ip_of_nat

5. SSH from the NAT to the server in the public subnet: ssh ec2-user@ip_of_private_server

Potential Issues:

• make sure ForwardAgent is set to yes in your OSX /etc/ssh_config file. It should be yes by default.