- Change the default SSH port
- IP restrict SSH access to your network(s), VPN(s), or jump box(es)
- Use a jump box
- Rate limit to 3-10 attempts per minute (with IPTables)
- Don’t use root unless you need to
SSH is a wonderful thing. It lets you connect to servers, configure things, download files and more. It also makes you feel like a wizard, and it’s a great party trick around non-techies, unless you live in Seattle where everyone is a software engineer/sysadmin/analyst (in which case the de facto talking points are “AWS vs Azure”, “why emacs is the best ever and everything else sucks because I grew up with this and my reality is the only one that matters” and “yeah, I’m a big deal, I lead team X at company Y and we’re changing the world” anyway, so SSH will hardly impress). But SSH can also be a security headache if implemented incorrectly, which it is every day in companies and projects both small and enterprise.
Since I’m a security person, I’ll address how to secure SSH from the perspective of a potential attacker rather than a DevOps engineer who lives and breathes infrastructure-as-code, CD/CI, Chef/Puppet/Terraform/Ansible, and some-other-impressive-buzzword-on-job-postings.
Changing default ports is cool
Unless you have a compelling reason to keep SSH on port 22 (some CTO who hasn’t touched a computer in 25 years said so, it’s in a deployment policy nobody knows who wrote that nobody will ever change so it’s gospel, there’s a well-articulated and discussed technical requirement/standardization, etc), change it. To anything. Pick a number. If I’m ever attacking, I’m looking for low-hanging fruit. While it depends on the context, if it’s not on port 22 it’s not on my radar. And the last thing I need in my life is to get flagged for running a port scan on 60,000+ ports. Most boxes keep it on 22 though, so it makes life easy. Really, just change it.
IP restricting SSH access is even cooler
Let me begin by sharing an nmap scan on a “highly secure, HIPAA compliant Telehealth service”, which I’m currently doing a write-up on and using it as an example why Telehealth scares me (stay tuned for that)
This is for a box that hosts confidential patient health information.
There are a few things wrong with this picture. First, the FTP and SSH servers are exposed to the public internet. Anybody, from any machine, can connect to them. This box isn’t sitting behind any load balancer, and I know there’s no WAF or IDS/IPS. I also know that in this deployment, FTP is used instead of SFTP or FTPS. There are problem with all of this, but most important is open access to SSH and leaves it vulnerable to brute force attack. You might think that there’s no way there are too many environments out there with such obvious, mind-blowing problems. You’re wrong. Banks do it. Hospitals do it. Governments do it. Fortune 50s do it. It’s everywhere.
Unless you have a compelling reason, you should be IP restricting SSH access to your trusted network(s) or IP(s). Doing this solves many problems from attacker detection to brute force to shielding in case credentials are ever accidentally leaked. There’s always a way around anything, with examples for this case being IP spoofing and a compromised host within a trusted/allowed network. However, dissuading attackers and making it difficult is half the protection. IP restriction not being a perfect, end-to-end protection against all SSH attacks doesn’t make it worthless. Unless it’s an IP or IP range you explicitly trust, don’t allow it. If I’m an attacker and your SSH is IP restricted and on a non-default port, my internet-wide scans probably won’t catch it and I won’t be trying to brute force your credentials.
Jump boxes are like stairs
Jump boxes are the choice between using the elevator or stairs to go up a story or two. When you take the stairs, you’re thankful — you feel better about yourself, stretched the legs and probably saved some time. Elevators are usually pretty safe, but sometimes the cables snap (not so much in the US, but it happens in Asia and Europe enough that I know people that it’s happened to), the engine dies, or for whatever reason you get stuck. Stairs don’t have that problem because they’re a lot more secure, and there’s a lot less room for things to do wrong. Same thing with jump boxes. Not using one (taking the elevator) is fine most of the time, but things can and will eventually go wrong for somebody. And if somebody wants to physically sabotage the elevator (there’s an attacker targeting your SSH server(s)), then it’s pretty likely you’ll have a bad time… eventually.
For any super important, mission-critical, or classified servers, always use a jump box, if not multiple jump boxes. Ideally, use a jump box to connect to any device or server you would be upset if it were compromised should your credentials be stolen from a host being compromised. Say Bob wants to SSH to server B. Server B only allows connections from Server A. Bob connects to Server A, which IP restricts to Bob’s secured network. Once on Server A, Bob connects to Server B.
Imagine the tremendous pain an attacker would have to go through to exploit a proper deployment of that type. Unless I was getting paid enough to retire, I’d move on to low-hanging fruit. I’d also be worried about the forensic fingerprints I’d be leaving to identify myself if my exploitation was successful, because a person or organization willing to set up jump boxes will probably try to figure out who did it. Thanks, but no thanks. So use a jump box. Or two. Or more than that. But at least one.
Rate limiting is fun and easy to do
Regardless of your infrastructure setup and proactive security measures, if an attacker has the opportunity to brute force, they will. Why? It’s fast and easy to do. You know how winter is always coming in Game of Thrones? Well, Python is ALWAYS coming in real life, and it will get to any insecure SSH server. Have you ever used pxssh? I have. It works just fine. I can make scripts around any environment or situation I want, because Python is easy to write in, and even if you’re not a software engineer (like I’m not) you can customize scripts like Nike shoes. Synchronization primitives? Booleans? No problem! Don’t know it? Google it!
SSH authentication should always be rate limited. A good baseline is anywhere from 3-10 attempts per minute, depending on your personal risk tolerance. I err on the side of caution and advise others to do the same, so I rate limit at 3 attempts per minute. There are instances where usernames and passwords are mistyped, but attention to detail and care should be championed across any project or organization. With that, I’ve never seen or have personally botched SSH authentication more than 3 times in any given minute. And if that limit is reached, just wait another minute. The inconvenience to legitimate users either doesn’t exist or is far outweighed by the benefit in preventing brute force. There’s no reason not to do it.
Rate limiting SSH authentication attempts protects you from practically everything other than declining stock performance and Walking Dead fillers that overwhelm you with disappointment. With enough time and attempts, any weak combination can be guessed. Since everything is automated nowadays and hardware performance is hardly at a premium, anything without some form of rate limiting or IPS is in danger from anybody who’s focused on that target. Protect yourself from it. All you need to do is add an IPTables entry.
Root is the root cause of your problems
Unless absolutely necessary, use non-root users with limited permissions. Lock away /etc/passwd, config files and anything else of value. Don’t use root, ever, for anything that doesn’t 100% need it, and if you do use a root user, take every precaution.