Photo by Jonny Birkelund
This week Linux Online interviews Bob Toxen, Linux/Unix security expert with 26 years of experience behind him. Bob founded and runs Fly-By-Day Consulting, Inc., a security consulting business. He's also the author of "Real World Linux Security" a real jewel among the Linux books out there. One thing that the Linux world can be grateful to Bob for is that not only does he write about keeping your Linux system secure, he's a Linux advocate and he believes in Linux as the best bet for secure networking.

Linux Online: First, thanks granting us the interview. It's a topic that everybody is interested in and we're thrilled to get answers from a person of your expertise.

Bob Toxen: It is my pleasure and honor to be invited to do this interview.

Linux Online: Could you tell us how you got started in the field of computer security?

Bob Toxen: I was an undergraduate Computer Science major at University of California, Berkeley. The main Computer Science Unix system, the "Cory Hall" system, was used both for graduate and faculty research and undergraduate teaching. My freshman year, Ken Thompson, the Bell Labs researcher who, with Dennis Ritchie, invented Unix and C, came to Berkeley on sabbatical to teach and work on Unix. Ken designed Berkeley's Operating Systems course, which I had the pleasure to take the following year.

The System Administrator and the graduate students refused to allow undergraduate students to participate in research, which involved improving the Unix kernel and utilities. The SysAdmin and some of the graduate students were, in my opinion and the opinions of some my friends, arrogant, elitist, and restrictive of equipment intended for all students and faculty.

We thought this wrong at a state university. Little did we know that this was the way of the world and that we should get used to it. Instead, we broke Unix system security, added what now are called Trojan horses to the set-UID programs and programs commonly invoked by root, and in the Unix kernel itself. We then proceeded to enhance the system kernel and utilities, clean up old core files, and generally do what would have been "a good thing" if we had had permission to do it. (We published an article on this work in UNIX Review magazine; the article is available at http://www.cavu.com/85jan.html.)

During this time, having no contact with any other gray hats, we learned the basics of both good system security (by seeing the mistakes the SysAdmins made) and cracker techniques. We kept this up for years, breaking into the other Unix systems on campus until we finished school and joined the grind in Silicon Valley 50 miles south. I want to emphasize that, while our actions were wrong, we never damaged data or committed malicious acts. Rather, our quest was for knowledge. Many of these experiences and the lessons learned are discussed in the chapter on case studies. It makes for interesting and humorous reading.

Linux Online: Basically, if I understand it correctly, the Internet as we know it was designed to link computers and people from universities. These people pretty much trusted each other, so tight security was not built in to the system. The problem is that we have to add new security measures all the time. It is said that the the level of security has not kept up with the size and scale of the Internet as it is today, is that correct?

Bob Toxen: This is quite correct. The Internet, which was Arpanet at the time, was for researchers and U.S. Department of Defense contractors. It was small so any troublemakers would be caught and chastised. Threats of job dismissal and arrest kept people honest. What Arpanet and its ICMP and TCP/IP protocols were good at was surviving a nuclear blast. If New York City was vaporized then traffic going through it from, say, Washington, DC to Los Angeles would, within a few minutes, be rerouted through an alternative route with no loss of data. This capability has been tested during U.S. nuclear weapons tests.

There has been no major redesign of the protocols or network hardware configuration. Instead, we add patches such as making the TCP/IP packet sequence number random and add SSH (Secure SHell) and SSL (Secure Socket Layer) public key encryption on top of it.

I think that the real problem today is that many companies, agencies, and individuals do not demand secure solutions from their vendors nor allow their SysAdmins enough time and budget to implement what security is available to them. Security takes time and money that seems to compete with "more urgent" problems like finishing projects on time, going public, or ease-of-use. Recovering from a break-in is ten or 100 times more expensive than making an organization's network and systems secure to begin with. Still, most of my clients contact me only after a break-in or if one of their clients demand a secure network.

The Internet is wonderful at quick and efficient dissemination of information. Crackers take full advantage of this while System Administrators usually fail to. The crackers scan the Internet for systems vulnerable to the latest security bug, and then attack these systems.

At this point, the security issues are well understood by security experts. Getting the word out to SysAdmins and users, then getting them to take the necessary steps to secure their systems is the hard part. Most people do not realize that almost any un-hardened system on the Internet will be broken into in the next few weeks, months, or, if they are really lucky, not for a year or so. One of my client's systems was broken into after being on the web only a few weeks. Another one found his break-in discussed on the front page of the Wall Street Journal. A third suffered weeks of downtime.

Unfortunately, not only do people often neglect to harden their systems, but they make no preparation for fast recovery if a break-in does occur. A little advance preparation allows for much faster recovery. In my book, I devote roughly 25,000 words to preparing for and recovering from an intrusion.

Linux Online: I don't mean to scare anybody with my previous question. It's obvious that large corporations, banks and governmental agencies go to great lengths to make their systems virtually tamper proof or "hacker" proof to use popular jargon. (Cracker being the appropriate word here) The problem is then that small and medium-sized companies and individual users machines and networks are exposed to great risks, aren't they?

Bob Toxen: Actually, many large corporations, banks, and government agencies do not make a serious effort to protect their systems and either have or will be broken into. Even Microsoft and Egghead did not keep crackers out of their critical data. It would be trivial to hijack web connections to The New York Times' web site. The techniques to protect their networks are in my book.

Linux Online: We've seen that even companies like Microsoft are open to attacks. We saw, in particular, in one week that Microsoft was effectively "hacked" and suffered two successful DDoS attacks. This is maybe a easy question, but at the same time it's controversial. Don't you think that this might rule out Microsoft products as a viable way of running a secure network connected to the Internet?

Bob Toxen: I do not see the grouping of these incidents as statistically significant. Presently there is not a good defense against a good DDoS (Distributed Denial of Service) attack because organizations, ISPs, and the large backbone networks do not reject incoming traffic with obviously spoofed (fake) source addresses nor can one quickly get them to stop such attacks routed through their networks.

Unfortunately, some firewall companies (even some of the biggest) have made misleading claims that that their firewalls can block DDoS attacks. What they mean is that they can block a few specialized attacks (such as the SYN Flood attack) that no modern Linux system is vulnerable to, though many Windows systems may be.

Microsoft's new Windows XP (from what I have been told) does not even bother to prevent ordinary users from generating such packets with spoofed source addresses. Further, there is no coordinated way for such DDoS attacks to be traced easily back to their source and the offender shut down.

DDoS attacks can affect any device with an IP address: servers, routers, desktops, palm-pilots, printers, etc. The underlying operating system has nothing to do with it. A flood is a flood. As with buildings in a flood, those that are well-built and which have a secure foundation will be the first open when the flood has subsided. The same is true of computer systems.

Interestingly enough, however, the technology to block packets with spoofed source addresses exists. In fact, it exists at most companies and ISPs at the firewall and router level. One company I know of is building it into the new telecommunications equipment it is designing. ISPs and others just do not bother to use it. It is called "Egress filtering", and simply means rejecting packets with fake source addresses (or which otherwise appear to be evil), that are trying to escape from one's network. I make a point of including Egress filtering in every firewall I build for clients (and my own network).

Hopefully, in the next couple of years, the national governments will encourage everyone to do this filtering, and many of these problems will become much smaller and less common. The Government also needs to require organizations and ISPs to respond quickly to attacks traced to their sites, or be faced with immediate disconnection from the Internet.

It was asked whether this rules out Microsoft products from consideration when building a secure and reliable network. My opinion and that of most security experts I have talked with is YES. I have read with amusement and sadness that some of Cisco's firewall/router products are vulnerable to IIS (Microsoft's web server) bugs, including the very recent Code Red worm, because IIS is used in these products and some have had backdoors too. This amusement is because Cisco boasts better security than Unix-based solutions because they wrote their own "highly secure" operating systems.

It is important to note that some insurance companies are charging higher rates when providing anti-cracker insurance to companies using Microsoft products. The Yahoo, IBM, Compaq, Egghead, CDNow, Barnes& Noble, The Washington Post, and the U.S. government's Whitehouse, FBI, CIA, NSA, and IRS web servers all appear to use Unix or Linux.

When Egghead's web server was broken into and four million customer credit card numbers were exposed to crackers, it was running Windows. Egghead learned the hard way, in my opinion. Star Trek fans may find it interesting that NSA's web server is named gary7. In Star Trek TOS, Gary Seven was the covert operative who returned from the future to fix sabotage to the United States' first manned rocket to the moon moments before lift off.

Linux Online: Do you think that Open Source alternatives and in particular Linux are providing much more secure solutions for corporate and individual networking needs?

Bob Toxen: Definitely. Linux and BSD Unix are the most secure general purpose operating systems, and continue to become even more secure. I have had great success in using Linux firewalls to protect Linux, Unix, and Windows networks from harm. Similarly, Linux and BSD Unix make excellent servers for web hosting, email, file sharing, and other purposes that have better security records than the alternative operating systems.

I am finding that SysAdmins in shops that are "all Windows" or "all Unix" are very receptive to the use of Linux firewalls, and even SMB servers using Samba. This is very refreshing.

Linux Online: The average corporate executive who may very well be a "layperson" in terms of information technology might be tempted to ask the question "How can a product whose source is open and free for all to see give me a higher level of security?" Would you like to answer that question for him or her?

Bob Toxen: Computer security may be considered similar to an ordinary lock on one's home or car. Most people know that inside the lock is a row of spring loaded pins. If just the right combination is pushed up with the matching key the lock will open. This is open source technology. What DOES need to be kept secret is the individual account password or the shape of the key.

Companies with proprietary systems commonly falsely claim that this makes their systems more secure. Security experts call this "Security by Obscurity" and it has been known for decades to not work. Assume, for the sake of argument, that Windows relies on Security by Obscurity. Last year crackers got their hands on the source code for windows. That would mean that every one of the millions of Windows systems in the world suddenly would lose all security because the "Security by Obscurity" suddenly would be lost.

One should assume that crackers have all of the source so the design must not depend on confidentiality of the design. If a cracker gets a password, only those systems or accounts depending on that password are compromised. This is why, in the book, I recommend using different passwords for different purposes when security is important. Different passwords should be used for different online merchants, home and office systems, personal and root accounts, etc.

Linux Online: There is an added issue here as well. It has been discovered that proprietary companies can and do add "back doors" into their products. (not mentioning any names, of course) This should be, according to the terms of the GPL and other "open" licenses, impossible, right?

Bob Toxen: Some well-known proprietary systems have had a backdoor with a "hardwired" password or no password at all. You do not know if they exist or not in your version. At the same time their Marketing departments claim better security than Linux or Unix.

Open Source does not need to rely on the GPL to prevent back doors. Any interested person can look at the source themselves and then compile it. Then they can be SURE that there are no back doors.

Linux Online: In your book you talk about the "seven deadly sins" of security. When I was a boy growing up one teacher told me that he thought "sloth" or laziness to be the most deadly sin of them all. Do you find laziness to be the number one cause of intrusion, that is people who don't bother to keep up with the latest exploits and fix the offending apps? Or maybe the problem is ignorance, which is not one of the classic sins, but obviously, if you don't know how to set up a secure system in the first place, that can be pretty bad.

Bob Toxen: Almost every SysAdmin I know works hard long hours so I would not call it sloth. I would use the term procrastination. Most intend to harden their systems "real soon now", but still want to have some sort of personal life. One of my reviewers had questioned whether my listing deadly sin #7 as procrastination was reasonable.

When I am consulted to help recover from a compromise, about half the time the SysAdmin had known about vulnerabilities that he was going to fix "real soon now". By listing it as one of the seven deadly sins, I put pressure on people to stop procrastinating, and, if more time or budget is needed, to point it out to management. This helps everyone.

Linux Online: It was interesting to find that you give ample treatment to the issue of Spam or unsolicited e-mail. Most people find this annoying, but are we looking at a potential security risk here?

Bob Toxen: Nobody complained that I put it in, and a number of people actually have thanked me. ;^)

Seriously, security is keeping out undesired data and people and spam most certainly is undesired. It costs people wasted time reading it and wasted bandwidth, computrons, and disk space. I consider it to be a security problem. This does not even include the possibility of viruses in the spam email. I understand that some of the most widespread distribution of the current SirCam virus is by spammers using Windows when sending to their thousands or millions of victims.

Linux Online: When you wrote your book did you have to take into account that people might not only use the book to secure their systems but there might also be readers who were looking for tips on how to break into systems? In other words, did you have to take care not to reveal "too much" information?

Bob Toxen: Absolutely! I know of two books on Linux "security" that are really about breaking into systems rather than securing them. This harms the forces of good and helps evil. Before discussing any exploit I weighted the value of disclosing it against the possible harm. For almost every exploit I provide a defense.

For those few I discuss without providing good solutions, the exploits already are so well-known among crackers that I am not helping crackers by revealing them. Rather, my intent is to show SysAdmins just how at-risk their systems are.

Some exploits that I know were not mentioned in the book because I considered that revealing them would do more harm than good.

Linux Online: In your own business as a security consultant, do you see reports of intrusion increasing at a high rate?

Bob Toxen: It rises and falls over time. The first quarter of this year saw a dramatic increase in break-ins of Linux systems. Much of this was due to well-known vulnerabilities in popular software and distributions that people had not bothered to patch. The subsequent publicity has helped to reduce the problems in the second quarter, with systems being hardened. I have noticed a trend in the third quarter: clients wanting Linux firewalls and monitoring of their servers for attack and defacing of web pages.

Most of the recent problems have been with buffer overflow attacks. These really are pretty obscure problems; it takes a very talented cracker to create an exploit for buffer overflow bugs. Unfortunately, if he posts it to the web, any script kiddie can use it. This is causing much of the Linux source code to be audited by experts, to find and fix these problems. When a client hires me to audit its source code, often it is surprised at the number of buffer overflow problems I find and fix. Additionally, recent releases of popular distributions have better security, principally due to less services enabled by default, than previous ones.

When a client has me harden a system with a recent Linux distribution, fewer fixes need to be applied than were needed for older distributions. It appears that some of the vendors have read my book, and that thrills me. Still, one must not be lazy. The recent trend to "clicking" on "high", "medium", or "low" security during an install, rather than taking the time to understand exactly what services are being enabled is reducing, not improving, security.

Linux Online: Do you think that in the end computer security is really something like the income tax system, something which basically works but contains loop-holes that will never disappear or are we really heading toward the day when our networks are as secure as Fort Knox?

Bob Toxen: I do not think that the U.S. income tax system works very well.

I do believe that Linux security is improving substantially each year due to improvements in it, increased awareness by SysAdmins and management, and by increased availability of information. Those who care to can lock down their networks to be very hard to penetrate. I hope that application of the techniques in my book is a good starting point. One must balance the cost of security against the cost of being broken into multiplied by the likelihood of being broken into.

It is critical to subscribe to mailing lists reporting newly discovered security bugs and patches and then applying them; this is discussed in the book. Time, management support, and a willingness to be slightly inconvenienced (because security is not convenient) are necessary too.

Linux Online: Thanks for taking the time out of your busy schedule to answer our questions.

Bob Toxen: It has been my pleasure. I am always happy to help increase Linux system security.

As we said before, Bob is the author of Real World Linux Security, a best selling book on securing your Linux system. He can be reached via his security company, Fly-By-Day Consulting