This week, Linux Online interviews Thomas Besemer, founder of The Besemer Group, a leader in providing embedded Linux solutions. He's also the author of many articles and an upcoming book entitled "The Embedded Linux Reference Guide". Embedded Linux is a hot topic nowadays and Mr. Besemer was nice enough to take the time to answer our questions.

Linux Online: How did you get involved with development on the embedded scale?

Tom Besemer: I grew up in machine shops, and always wanted to control the machines I was building. That's all about embedded computer control. For years, as a developer, myself and many others have been faced with using commercial products which cost a lot; I got involved with Embedded Linux after Jim Ready, founder of MontaVista software started looking at it.

I've known Jim for years, and he's a great spokesman for the industry - he convinced me that this was good new technology to evaluate. Embedded Linux is fun, it's new and it's refreshing. It's also a challenge; Wind River has had the market for years, and it would be nice to see an alternative to VxWorks.

Linux Online: You have a very varied and extensive career in the computer world, and now you have concentrated largely on embedded development. Do you find this area more interesting than others?

Tom Besemer: What's most interesting about it is that it is new, and there are a great number of people not only interested in Embedded Linux, but working on it. Linux has grown from a small group of intellectual people to a wide spread group of individuals who contribute back to the organization. And investment in Embedded Linux has been great by the financial community; take a look at the level of funding Monte Vista (http://www.mvista.com), LynuxWorks (http://www.lynuxworks) and Lineo (http://www.lineo.com) attracted last year - this is hot stuff, and hot stuff can be a blast to work on! The synergy can give established vendors a run for their money, and change the way we do embedded.

So is it fun? Very. Yet it's more than fun; it is challenging, it is new, and it requires me to re-think how I have been doing things for years. I think most engineers will agree with me when I say that a big part of what gets us up in the morning is doing something new, fun and challenging. This is what Embedded Linux is about.

Linux Online: Most people have the impression that a computer is a just PC. I mean, from the time your alarm clock wakes you up, you've had your coffee, you watched the morning news and then gone to the office in your car, you've used about 10 computers already before you even switched on your office work station. That's what embedded is all about, isn't it?

Tom Besemer: That is exactly what embedded is about; all the computers in your life which you don't see. When you make a phone call, an embedded computer handles the call routing. When you start your car, a set of embedded computers aid in it's operation and help keep emissions down. A VCR is a computer, as is a cell phone and a digital watch.

A buzz word today is "Internet Appliance." Just what is that? I think the answer is very open; it could be a toaster which you read email on. It could be a radio plugged into a DSL connection. Regardless of what an "Internet Appliance" is, Linux is a natural for all of this - network support, standard API through System V IPC or Posix interfaces, and growing acceptance by both the commercial and private sector.

Linux Online: Now they're talking about embedding OSes in clothing, conventional eye glasses (read your e-mail while you're walking through the park) your dining room table- a big list there - the sky's pretty much the limit here, isn't it?

Tom Besemer: The sky is the absolute limit, yet the market is driven by issues of economy. Embedded will continue to grow at a tremendous rate as the cost of processors drops and the capability of processors increases.

What's great about Embedded Linux, with it's GPL license model, is that it's royalty free on deployment, as opposed to commercial solutions; developers of consumer products such as Set Top Boxes, where margins on profit are low, can deploy with Linux without a need to pay an OS vendor re-occurring revenue on a per unit shipped basis.

The drawback of Linux being GPL is that it is hard on the Embedded Linux vendors; they can't license the kernel, yet have to spend hundreds of thousands of dollars modifying it to work in an embedded capacity. There is a concept in the industry, especially among VC funded developers, called "stabilized development environments." Vendors such as Monte Vista, LynuxWorks and Lineo will invest a lot of time and money into stabilizing releases.

It's my personal feeling that the Linux community, as a whole, must help support these vendors by including their engineering in standard Linux distributions (as the vendors see fit). The efforts by these firms makes sense, and changes the Linux kernel/distribution. Yes, I am asking for support for these commercial vendors - I don't think it's fair to ask them to re-engineer the kernel each time it changes.

Embedded Developers need stabilized releases to work with; they aren't experimenters, but deployers of product. Time to market is always an issue, and through that, these type developers cannot be in the Operating System business; their business is their product.

Embedded Linux vendors can offer a great deal to the community through their large capital investment, which results in stabilized releases of Linux suitable for Embedded Development. A balance needs to be achieved to ensure that the vendors have financial interest in supporting the Linux community while keeping the distribution available to the general user for free. I can't, as a consultant, ask a client to work with "top of the tree" as an experiment, yet I can't ask the vendors to give away their engineering efforts.

So this is a question back to all the readers; how do we ensure that the vendors are able to generate income while keeping the spirit of GPL alive? Do we need vendors?

Linux Online: Due to the open source nature of Linux, do you think it's safe to say that the development of these types of devices will be easier and faster?

Tom Besemer: It would be hard to say "Easier and Faster" today; Wind River, with VxWorks, has a very stable development environment which runs on a variety of hosts, and targets a variety of processors. Their tools are very mature. In many ways, doing an embedded software project with Linux today is "slower and harder."

I just, in the middle of editing my responses to you, got off the phone with a client up in San Jose who I will visit tomorrow. They are MIPs/VxWorks based. I said "hey, you have got to get on the Linux bandwagon.." The answer was "I'd love to, but the tool chain is not mature, and I'm not in the OS business..."

However, with Linux, all source code is available. It can be a big advantage. When doing a new driver, the developer has many examples on how to proceed. The potential drawback on this is GPL; a large organization who invests a lot of money into proprietary hardware isn't going to be that interested in putting their driver work back out as GPL (at all times).

The major advantage of using Linux for Embedded is that a great deal of resources are there. The tools aren't quite ready for prime time, but then, see the above comment on vendors - these organizations can help stabilize baselines and provide the kind of tools Embedded Developers require.

Linux Online: You're writing a book called "The Embedded Linux Reference Guide". The books scheduled for release in the summer. How is that coming along?

Tom Besemer: It's a brutal project. Part of the issues are in researching everything, and being able to present a reasonable manuscript that many will gain from. Writing a book is hard work; I've written lot's of technical documentation in the past, but most of that was geared towards corporations. When documents of this nature were 80% done, we'd distribute and hold meetings, make decisions and go on. With a book, you have to get to that 100% level, it's got to have lot's of examples, and every single example has to be tested.

However, all these examples will end up GPL, and back in public domain, so the effort is good for the Linux community, while the exposure is good for me.

And I'm not the only one writing a book: I have heard that both Sams and New Riders have books on this coming out.

How's it coming along? Ask me in two months :-))... I think it's going well. Also, check the "HOWTOs" - great stuff, and many times, all the answers are there - they are all of what a good book is about - clear answers to daily questions. There are times when I feel I should just be generating 12 chapters of "HOWTO," but in the same breath, I love the thought of a bound and published work.

Linux Online: Apart from the fact that Linux is open source, and you've got the code available to look at and modify, what makes Linux so well suited for embedded development

Tom Besemer: Your question is also the answer. Lot's of examples, as most Embedded Products are about custom hardware, which require custom Drivers - having a head start through a set of examples is a big win.

License issues are also very important; like my example earlier in this interview about Set Top Boxes, developers of consumer products face a tough and competitive market. Not having license fees on a per unit shipped basis can be a big cost advantage.

I think the other advantage is the fact that the API is a standard "Unix" like API. I've hired a lot of people over the years for clients, and one thing that is always certain is that college graduates have had some experience in a Unix or Linux type environment, but not in environments which you find with commercial embedded operating systems. A huge advantage of deploying an embedded system with Linux is that hiring managers can hire people out of school and put them to work right away, without need for extensive training.

Embedded is big, yet the number of engineers who are capable of doing it is a limited and specialized. Embedded Linux gives a potential advantage to any organization simply out of the ability to hire people who understand the API right away. This allows them to have one or two "hot shot" specialized engineers to handle the ROM/Flash/Driver aspects of their embedded project, while letting less experienced people work on the applications. This is a big win.

Linux Online: Let's be fair. Are there any drawbacks to using Linux in embedded development?

Tom Besemer: There are several drawbacks, many of which will work themselves out with time (and contributions back to the distribution).

A "Reduced Footprint" is a good example; how to build a very small binary that fits into an economical ROM/Flash device is a very important issue, and it's not that easy to do today. Commercial vendors are working on this problem.

Another big one, along the same lines, is "Execute in Place" - the ability to embed Linux into ROM or Flash. The Linux loader, with shared libraries, does dynamic linking of object files when applications are loaded into memory for execution. A dynamic loader cannot do address fixups on an image in ROM.

Without changes to the build and load environment, the physical layout of an application is that data and bss segments follow the text segment in actual memory layout; when running out of RAM, this is a non-issue. When running out of ROM, the data and bss segments need to be repositioned into RAM before the application can run. The current load and run scheme does not support this.

There are options; "copy on write" support schemes can help, where the first write to data or bss causes the page to be copied into memory, and the MMU/TLB is updated to show this new physical address in the same virtual address space that the application was linked with. The fork() system call does model this, in a certain sense.

But mostly, Execute in Place is not native today in the kernel environment, and will need support to make Embedded Linux viable in small, low margin products such as MP3 players.

Diagnostic tools are another issue; typical applications such as Netscape operate as a single instance, without a lot of coordination with other processes. An Embedded application will either be comprised of multiple Linux processes, or a set of Posix threads within one Linux process. They all must operate in harmony, and GDB just does not cut it when it comes to debug of a system where several tasks communicate with each other, and are event driven by interrupts.

Along these lines, visibility is also a problem; the 'ps' command is pretty limited. As an integrator, I need to know what each task is doing and when, what is moving through queues, what is up with semaphores, and I need to be able to query the system at any time.

Vendors will help solve this problem through offerings of tools. Contributors, such as the folks who did the Linux Trace Tool, can help a great deal. And of course, I hope my book can help.

Embedded Systems are "Event Driven" by nature; interrupts happen, and these interrupts require service by application code. Response time is interesting, and the fact that often multiple processes or threads will run as the result of the "event" is interesting. GDB is about fixing problems with code threads - you debug the single instance of the code, and solve problems like array boundaries or bad coding; once those problems are solved, then we integrate. In integration, we find numerous problems - timing, race conditions, etc... Diagnoses of these problems can prove difficult at times, as the entire system must be operating as a whole, without "stalls" what would occur through a breakpoint.

Visibility into the kernel and the applications is very important, but most important is non-intrusive visibility; the ability to "monitor and diagnose" a running system with multiple tasks and/or processes.

I currently have a shell based tool, coupled with a driver, which allows me to call subroutines for execution in the kernel from the shell, do io re-maps from the shell, and display and set memory - I'll put this out by summer, as this is the kind of tool which helps you in the lab, during integration - the ability to invoke kernel level functions during runtime, and display and set kernel memory from the shell when debugging such facilities as PCI based interfaces.

This tool isn't complicated, in fact, it's beauty is in how simple it is - from the shell, without the need to load a new module, I can invoke diagnostic kernel routines and query the state of the hardware (which is either memory mapped, or mapped into PCI space). A big win when faced with "strange system behavior."

Linux Online: There have got to be a lot of drawbacks to using Microsoft products in embedded development, I'd imagine :-)

Tom Besemer: As you know, I was delayed in answering this interview; I was on the road, using Windows to check email. While trying to answer your interview questions, Windows crashed (again). Now that I am back in my office, using Linux, I am able to complete this electronic interview without fear that my system will crash in the middle of my efforts.

A notable characteristic of Linux is that it lacks the "Blue Screen of Death.." This omission and operating characteristic is very refreshing!!

On Microsoft in the embedded world? I hear that they have this thing called CE. I don't know what it is, but can tell you that half the time you get called as a consultant is when things are not going as well as desired, and a client needs help. In this, I conclude that either CE is so good that nobody needs to call me, as there aren't any problems, or that nobody is using it.

I personally have never met a group who is using Microsoft products in an embedded capacity. Whether Microsoft software is viable in an embedded capacity, I don't know, and will never know.

Linux Online: Despite the promise of embedded development with Linux, do you think we might see projects having difficulty getting funding due to the slow-down in the high tech sector

Tom Besemer: Clearly, there is a slowdown in certain sectors of the industry. I'm not a financial person, so it's not appropriate that I answer this question. I can't really see, on the long haul, how the current market can effect Embedded Linux. If anything, it may help it, as the cost of getting into Embedded Linux can be much less than alternative commercial solutions.

However, it may affect the commercial Embedded Linux vendors, and that does hurt the community as a whole, as I feel that their survival is important to making Linux mainstream in the embedded sector. Just as I, a consultant, need clients to keep me alive while I do my book, the vendors need business to keep their efforts going.

Linux Online: We've just heard that some units of PlayStation2 are going to be shipped in Japan carrying the Linux OS on them. That speaks well for Linux in the embedded world doesn't it?

Tom Besemer: Any commercial organization that deploys with an Embedded Linux based project is legitimizing the viability of Linux as an embedded OS. It's good for the Linux community.

A year ago people said I was out of my mind when I spoke of using Linux in an embedded capacity. Six months ago I had a potential client ask me about the availability of SCSI drivers under a commercial Embedded OS; I could not find any, but was able to point to numerous available drivers for Linux. The depth of what is available is wonderful.

My question for all commercial organizations deploying an embedded solution with Linux is simple: are you going to take what is out there to your advantage and not return your efforts, or are you going to embrace this technology, do your engineering, and then support the Linux community through submission of your work to the distribution?

It's sort of like the pennies in the check out counter of your local store; take one when you need one, but be sure to leave one when you can. If the commercial developers take more than they give, then they need a reminder of what GPL is all about.

Wind River (VxWorks) has recently purchased BSDi. I think this is their response to Embedded Linux. In a conference call/presentation on this, which is available to listen to on their web site (http://www.windriver.com), they stated that the BSD license model is more "business friendly."

I would guess that they will pose a serious challenge to Embedded Linux, with their financial might and the potential business advantages of the BSD license model. It could be that that in a year, we don't talk about Embedded Linux, but about Embedded BSD. And we also talk about how Wind figured out how to license the concept of a "Unix like" operating system in the embedded capacity...

Linux Online: On the same idea, do you think we'll all be wearing IBM's wristwatches with Linux embedded in them, before long?

Tom Besemer: I think that GPL will help prevent large organizations from monopolizing the market. But then, see above; is GPL with Linux going to be the solution, or is it going to be a BSD license model with Wind River dictating how a "Unix like OS works in the embedded sector?"

I don't think we can live without the large vendors and organizations. They have resources. However, it's not clear whether their financial might can help or hurt the Embedded Linux and/or Linux community at this time.

I think GPL helps keep it all in check, but let's see what Wind River does with BSD.

In the end, I think it is going to come down to the integrity of the people who use the freely available Linux technology, and actions always speak louder than words. Let's see what these big players do, and if it's not acceptable, well, what are the options?

Linux Online: What's your favorite embedded Linux project out there and why?

Tom Besemer: My favorite project? All the ones you and I don't see; it's all the people who are trying, with either success or struggle, to embed Linux. Without these people, there would not be an Embedded Linux movement. Let's hope we hear from them in the form of submissions back for general distribution.

Myself, I've got a 5 year old who loves trains. In any form; real or model. We are building an H.O. scale train set which is controlled by Linux. I'll be putting all the code from this in my book, and back out in public domain, while my son just enjoys the efforts. So, I guess that is my favorite project.

Linux Online: Thanks a lot Tom for taking the time to answer my questions.

Tom Besemer: I appreciate your expression of gratitude, but really, thank Linus; without his vision, we would not be having this interview, nor would there be commercial companies who have attracted large amounts of VC.

I think this should all serve as a reminder that one person can change the world, and that all thanks really are due to Linus, and his vision on what Linux should be about; he is clearly one man who's efforts continue to change the way all companies do business.

You can check out Tom's articles on embedded Linux and find out more about his work in this field at his website http://www.tbcorp.com.