Tobi Waves

by Mark Burgess

The increasing availability of mobile communication devices changes our society. Appointments become fluid and can be changed up to the last minute as an SMS will inform the other party of the new location in space and time. People who are physically in the same room can easily avoid any communication with each other as they keep connected to their "friends" over their mobile devices.

On a sociological level the availability of mobile communication devices has not yet been integrated into the framework of our social standards. Where is it appropriate to use a mobile phone? Does the availability of a connection to your peer grant you permission to change appointment times at the last minute? Is it acceptable to have a mobile and turn it off?

Challenges for system management in this context are: Diversity as many different technologies will be around until (if) a unified standard emerges. Stability in the face of environmental noise.

We must find new ways of keeping the systems within our realm of responsibility in some organization. Firewalls make little sense in an environment with a wild mix of interconnected private and company devices. VPNs are giving a hint at things to come.

How can we find system management methodologies for diverse, mobile and changing device populations. Looking at natural behavior of birds (swarms) or ants (hives) give clues on how organization can work in such environments. Even today kids can be seen to swarm around town governed by SMS messages they exchange.

For system management this means that strict central control is a thing of the past. Maybe stable existing structures which allow for other devices to integrate them self will work. There is no point in trying to stop the advance of these new technologies. We rather have to integrate them into our environment and adapt the environments to them.

A secure system is one where the risks are known and have been deemed acceptable.

by Richard Moore of IBM

Richards group is occupied with getting RAS (Reliability, Availability and Serviceability) into Linux.

In order to get Linux established with previous "Big Iron" customers, a whole new set of requirements becomes important.

As Reliability is not really achievable, the aim is to reach pseudo reliability which means to hide failing elements of the system from other system components, probably taking a performance hit while doing so.

In a 2 CPU machine the failure of a single CPU can be recover gracefully by shifting the workload to the one working CPU.

The Serviceability component means that the system must have the means to detect failing components, best before they fail completely and then replace them. Serviceability is not limited to error detection but encompasses all elements which make a system serviceable. So this includes manuals, problem correlation, debugging tools, logging. Compared to what is available for IBM 390 the Linux offering is still in an embryonic state. Many tools are available, but they are often not yet integrated into the mainline kernel nor is there a consensus on which tools to use. The consensus is still Syslog which is not easily machine parseable and thus does not lend itself to automation.

The big advantage of Linux is that there is virtually no old code in the system compared to old systems like Windows or big iron machines which have a rich heritage of old code. Linux developers have a tendency to not shy away from ruthlessly eliminating bad code. They rather break an interface then keeping a bad one around. The effect is, that the systems are cleaner and easier to maintain.

The documentation problem gets solved in part by the much better code quality in Linux (due to peer review) and the extreme size of the kernel developer community. Also because source is available a lot of documentation is in the source itself.

Due to the versatile workloads of Linux systems all functionality in the serviceability must be tailored to the specification of the local setup. There is not much use in a mobile phone dumping core into its flash ram.

Murry Stokley of FreeBSD Mall is one of the FreeBSD release engineers. In this talk he told about FreeBSD development, organization and what is new and cool in FreeBSD 5.0.

Development and release management

Everything is on CVS

There is a current branch and a stable branch. Material from the current branch gets merged back into the stable branch when they have gotten enough testing.

Over the last 12 months 160 people have committed code directly via cvs to the FreeBSD kernel. Non commiters are welcome to submit patches via the gnats bug tracking system.

FreeBSD is highly organized with elected leadership, developer documentation, release engineering, core team.

Tinderbox environments constantly test the current release.

Release 4.x remains supported in the foreseeable future as most FreeBSD sites have very high stability requirements.

New Features In 5.0

Support for kernel scheduled entities which leads to better threads

Device file system

Bluetooth and Firewire support

Mandatory Access Control

UFS2 with bigger inodes to store extended attributes

GEOM - modular disk I/O transformation framework

Device monitoring daemon devd to manage pcmcia and other plugable devices

Soft Updates (fs enhancement) with snapshots and background fsck

No more perl dependency in the base system

New platforms: ia64 and sparc64

More information is on (www.freebsd.org ...)

by H. Peter Alvin of Transmeta

As the Linux kernel developed, the root file system became more important and had to be able to live in more and more different places, like configurable locations on the disk or even on the network. Eventually even in RAM as initrd entered the scene. This caused all sorts of special cases needing handling to support all these variants. And all this is happening inside the kernel which is tough for development as testing is really painful. So the ideal case would be to be able to organize the booting process in user space.

The solution is to have a virtual root for the system, called rootfs using ramfs code. As the kernel starts, / is rootfs and the actual root filesystem becomes a simple overlay mount. This means that it is possible to use an initial ram disk and get rid of it later. As the kernel threads live in their own world (rootfs).

To simplify matters further, initrd gets replaced with initialramfs which is populated by loading one or several cpio archives. The cpio archives can come from the disk, from the net or even be compiled into the kernel itself. They provide the files required in early user space. To allow initialramfs to be small and still provide a useful environment, a special stripped down C library called klibc was developed to provide library code for this case.

Programming in this environment is almost as simple as normal userspace programming. Malloc works as expected, file and socket handling is there. Still there are restrictions, as all the rest of the system is not up yet. STDIO is available but it is very slow especially for reading, as klibc does not implement buffering to save space.

Candidates for early Userspace

With this infrastructure the possibilities become endless. The following candidates for early user space come to mind:

Partition detection

Determining the root filesystem type

Network booting

Caveat

Note that this is very much an ongoing development and only available in 2.5.x. (www.kernel.org ...) has more information.

For the last few weeks Regula and I have been watching the 1994 drama series My So-Called Live. I totally love this show, too sad it got canceled after only 19 episodes. It's the story of 15 year old Angela Chase, her family and high school friends. I won't attempt to tell the story of the show as it is not really the story which makes it live. It's more the depiction of Angela's "so-called" life, as well as the lives of the people around her.

I seem to have a knack for getting hocked on TV Series that get canceled (Farscape is another example). Here I knew what was going to come, as I had bought the DVD box-set long after the fact. All the more amazing to see that fan community is still alive and kicking (www.mscl.com ...). Because the series ended so abruptly it offerd fertile ground for fan fiction, also called episode 20 fan fiction.

After watching the penultimate episode tonight I went on the net to research if the people behind MSCL had done other things I might want to see. The producers of the Series are Ed Zwick and Marshall Herskovitz of the Bedford Falls Production Company named after the setting of "It's A Wonderful Life". They started working together in 1983 and have since produced several award winning shows, unfortunately not all with great ratings. Some died an early death like MSCL. First there was the all around successful thirtysomething, then after MSCL (www.amazon.com ...) there was Relativity which also got rave reviews but low ratings. Now they are back with Once and Again (www.amazon.com ...) where ratings and reviews seem to be more in sync again.

Unfortunately there seem to be no DVDs of thirtysomething available, but the first season of Once and Again was just released on DVD, so there is at least something to console me when the last episode of MSCE will be watched tomorrow night.

I also found a lengthy article about Zwick and Herskovitz and Bedord Falls at (www.angelfire.com ...)

Our department is taking part in the ETH Laptop Project. This means, we are helping our students to make better use of their laptops. Currently this means we are developing a Linux and a Windows setup tailored to the requirements of our students. These setups will make it simple for them to integrate their laptops with our Unix Environment. We also have struck a deal with IBM which offers the students IBM Laptops at competitive prices and we will put our own Windows and Linux on these boxes.

Today I have been trying to get the IBM Windows XP installation which is already on the laptop when the students buy it, into a form so that it contains all the latest security stuff and fixes from MS and updates from IBM as well as our locally developed packages. When all the stuff was in, I used the sysprep tool to 'reseal' the machine, so that when the students boot it, it will come up with the usual short setup where the user can define the admin password and has to enter the serial number. Well, that was the plan at least. When I tried to reboot after the sysprep step, Windows came only halfway up and then complained, that setup could not continue because two processes were accessing the registry. BOOM. Reboot.

Over the course of the day I tried the whole spiel in many variations, searched the web, hunted through newsgroups. As every try took about 40 minutes, this problem was really painful to debug. Eventually and counter to all I expected, it finally worked. Unfortunately I had twisted and turned so many knobs that I am not sure which one was actually responsible for the sudden success. So tomorrow I will be at it again, trying to verify my recipe for success.

I so am glad that I can mostly work with Unix systems and only have to use windows occasionally. Whenever there is a problem with a windows box I feel like I was forced to wear a thick winter gloves while trying to repair a watch, blindfolded and someone occasionally moving the watch around.

But hey, I am stronger than windows! Eventually it sits up and begs for food, but the process always is extremely annoying.

Today around 9am our main Solaris server started acting up. Its performance got patchy. We eventually found that it was suffering from excessive TCP retransmits of up to 1000%. This means that for each packet it sends out on the net it has to try 10 times until it is successful. This is an extremely hight value, or so Virtual Adrian tells us.

We started searching franticly for the reason of the problem, as performance on the server and even more on its clients was suffering badly. After about one hour of web hunting with and traffic dumping, we gave in to the pressure from the street and rebooted the beast, hoping that probably some internals of the kernel had been thrown out of whack and after a reboot all would be well. And indeed it was, at least for a few minutes. Then the server started misbehaving again, driving its TCP stack through the roof. As rebooting did not help, we went back to tcpdumping and etherealing. I did learn a lot about pcap filter syntax ...

'tcp[tcpflags] amp (tcp-rst) !0 ampamp tcp[tcpflags] amp (tcp-ack) =0'

but nothing about the reason for the retransmits. Fortunately, at this stage, the retransmit rate was not always at 1000% so work was possible for our users.

Then, in the early afternoon, Manuel found that the root disk of the server causing SCSI timeouts. As if we didn't have enough on our hands already. SCSI timeouts make the machine stop and wait for several seconds at a time. Together with the server, most people using its resources, were experience the same freezing problem on their workstation.

What a day. I have been writing emails about what was happening to our users all day long, but things were really stating to look bad. Our wonderful reputation for high quality service and superb uptime was going down the drain. It seemed though that most users were not blaming us at this stage, probably due to the fact that I kept them up to date with what was happening.

Around that time David found, that in the latest Solaris kernel patch there was a fix for some TCP stack issue which might be related to the retransmits we were still suffering from. He started to put in this patch so that we could activate it when we rebooted. This was going to be necessary anyway as I was preparing to replace the root disk with a fresh device.

Then, suddenly just minutes before the reboot, the server went back to normal, the retransmits were gone and performance was good again, no traces left.

So here I am, another day older and not much smarter about what was causing todays network problems. I can imagine things like that there is a bug in the Solaris TCP stack which can be triggered by a rouge packet and this would cause the symptoms we experienced today but I suspect, once the real reason is known, it will be way less spectacular.

Once upon a time, there was this firm, the little Toy Factory, they were building these neat and simple woodblock toys. Kids could use them in various setups. Clever kids could even create their own additional woodblocks and hoock them on. Woodblocks seemed quite simple, but the trick with these toys was their clever overall design which made it easy integrate them with other Woodblock toys and even create your own additions. The factory was really successful with the kids who knew about their toys. Partly also because each Woodblock toy also came with a complete manual explaining not only how to use it but also giving detailed account about how this particular WBT had been constructed. Woodblock toys guaranteed for hours of satisfying and creative playtime together with your friends.

Not far to the north west there was this other company, the Lolly Makers, they produced shapely and tasty lollies in many colorful designs. Kids who tried them were really taken by the great taste of the sweet lollies. The lollies sold very well and soon most kids who lived in the vicinity of the lolly factory could be seen wandering the streets with a lolly in their mouth. Interestingly enough these kids seemed to loose all interest in playing with the woodblock toys or other kids apart from talking about the latest 'inventions' of the Lolly Makers. The lollies seemed be all they needed. Rumors had it that the lolly makers were using addictive and psychoactive substances in their creations. But whoever uttered any suspicions in this direction soon got letters from a big firm in the city, who advised them to refrain from telling any further lies about the Lolly Makers.

Over the last few days I have redesigned the Website of my Department, and implemented it purely with CSS2. I had to discover the hard way that even 4 years after the standard has been published, we are not there yet. While Mozilla is shining bright with its good implementation, many other entries like Opera, Konqueror and IE are working hard to do a good job but fail in odd places. What is amazing is where they do not work. Opera, for example, can not grok, that a box which is defined by its distance from all edges of the browser window is fully defined an can be displayed propperly. It just ignores part of the settings to be able to draw in the wong place and size.

Why does this not get fixed? First I thought there must be many bugs, and they just don't get round to fix this particular one. But then I got another theory: Everybody who does serious CSS2 Webdesign is working hard to make sure that his pages work with all the players in the CSS arena (Opera, Mozilla, Konqueror and Internet Explorer). Therefore these stupid bugs don't disappear as nobody will notice them but Webdesigners who then successfully hide the bugs from the end users by going the extra mile to make their pages work with all browsers.

How about a website which collects pages that pass all CSS/HTML conformance tests at (validator.w3.org ...) with flying colors, but got axed because display problems in some browser prevented them from working fully 'cross platform'

With todays low disk prices, it becomes intriguing to use several large harddisks as backup media. On the simples level they can be used like tapse to store dump files. But having all this quickly accessible storage space, it would be way cool to have some snapshot like quality where the backups are readily accessible by the users. Today I read about a cool idea for backup which is exactly what I have been wishing for the program implementing this is called FauBackup (faubackup.sourceforge.net ...). I could smack me for not thinking off this myself. Anyhow, here is how it works.

First Day Make a full copy (file by file) of the work partition to the backup partition.

Second and every following Day Walk the work disk and compare the file i-node change time of the files with the time the last backup was taken. Anything that staid the same gets hard linked from the previous backup to the current backup. Everything that has changed gets copied over.

Additionally compressing the files in the backup could lead to significantly lower disk usage. The advantages of this approach are: a) Users can restore their own files by just going into the backup partition. What I am not clear yet, is how to prevent them from modifying the backup. b) Backup space requirements are minimal as only changed files go into the new backup and old ones are hard linked. c) if we run out of space old backups can simply be removed as all backups are complete and independent of each other.

Now I just have to decide which computer is going to be my first victim. And I have to figure out if I am going to be stubborn and write my own implemntation implemntation of this idea or if I can convince myself to use faubackup.

In 14 days we will be moving out of our good old flat, to a new house. Or rather we will move out, and after Easter we will move into the new house.

Yesterday we finally chose the new table and chairs we were going to buy for the house. We spent a great deal of time figuring out what table we were going to get and where we would get it without spending all the money we have left after buying that house.

What we sort of ignored all the time, was the chairs. We did not ignore them completely, no, but we did never discuss their color. So when we finally were ready to order everything, yesterday at the furniture place, the guy wanted to know which color the chairs should be (the part which is not wood).

We have a pitch black floor, a fair beech wood table, white walls, transparent light gray white curtains. The chairs should contrast to their surroundings without taking all the attention. Aargh ... what a decision 20 minutes before they are going to close the shop for the night.

We went back and forth amongst all the colors on offer and finally settled on a fair grayish color with a hint of blue. With the order sent and the price agreed, we are left to worry if this was really the color which is right for us. Will not be to delicate? Will it fit with the rest of our furniture? I see six to eight weeks of worrying to worry ahead of us. Because of all our special wishes, it will take them that long to produce and ship the things.

Don't say it ... just don't! I'm going to bed now.

I read in the paper today that the George W is going to ask Congress for 75 billion to finance his crusade against Saddam. Lets assume this will amount to about to maybe 100 billion in the end. This is about 500$ for each American citizen or 5,000$ for each person living in Iraq. I don't know how much the average income in Iraq is, but I guess with 5,000$ per person you should do well. Think even how many people in some other 3rd world country, one which has not even oil, could be sustained with this kind of money.

I saw Bowling for Columbine (www.amazon.com ...) last autumn, today heard Michael Moore's (www.michaelmoore.com ...) acceptance speech for the Oscar. He seems to be one of the few outspoken, witty Americans around these days, who are actively working for a better future for America in this world and not apart from it. A country currently governed by fear and through fear of an artificially enhanced enemy whom to fight will serve as a method to ensure W's reelection by people who might otherwise just realize what disservice they are doing themselves by choosing leaders who don't seem to have the understanding nor the intention to play by the rules for the greater good of the world.

In the radio news today, the presenter remarked, that the US was detaining POWs in Guantanamo under the label of 'Unlawful Combatants', refusing them any of the rights POWs should have, according to international law. He continued, that Iraq might call the US Soldiers exactly the same and would not even be all the out of line as the US is fighting this war with out UN mandate and is thus in blatant violation of the very fabric the chances for stability and prosperity of the world are based upon.

All Friday Regula have been packing our stuff into cardboard boxes. On Saturday morning at 8:30 10 of our friends arrived in Aarburg, ready to help us move all our belongings from the flat at the Feldstrasse to our new house at the Aarweg in Olten.

We had packed 90 boxes which had to be carried down 3 floors at the old place and up 4 floors in the new place. And this was only the beginning. The boxes alone filled the Van we had rented. A further three rides were necessary to transport all our furniture. I had hoped we would manage with two rides. I was so wrong. I promise, I will never buy anything new unless I give something old away ...

At 4pm all was done and everybody had had a gracious share of lifting and stair climbing. Thanks to the delicious lunch my mother had prepared for the whole crew we were tired, but at least well fed. (Thanks Mom!)

A big thanks to Alexa, Claudia, Doris, Gabi, Regula, Christoph, David, Fritz, Manfred, Manuel for their help. If you are ever going to move house, count me in.

I was searching google groups today when I lumbered onto this post from February 1998: (groups.google.com ...)

Amazing, these days many people blame W but six years ago, the Clinton administration has proposed a war on Iraq as well, only that they did listen: (www.cnn.com ...) It seems CNN has not edited their archives, even thought these days, US media don't report anti war rallies unless many hundred thousends attend. They rather bring an extendes feature on the few people marching for the war. It seems that independent and critical journalism is "out". Remember Watergate?

Today Akamai kicked Al Jazeera off their network. The Register has some thoughts on this (www.theregister.co.uk ...)

I got up at 4:15 am today. In order to get to Eindhoven on time, I had to take the 7 am flight from Zurich and thus the 5:20 train from Olten. Guess I am lucky to have no problem with getting up early.

In Schiphol I met Rudi and Alexios. Together we rode to Eindhoven and arrived at the TU/e just in time for the meeting. The other members of the program committee were there as well: Walter, Edwin, Brenda, Peter, Bastiaan, Xander, Jos and Fred. And there was also Marielle and Sabina from Iconic who will be doing all the actual organization work for the conference.

This is my first time on a PC. What I found the most difficult part, is to figure out which talks would be relevant 1.5 years from now, as the conference will be held at the end of September 2004.

At the meeting we did not really decide anything fixed, apart from the meaning of SANE: System Administation and Network Engineering Conference.

We tossed around a ton of names of potential invited speakers, had a short discussion about the deadlines, debated whether we should demand 400 word abstracts or extended abstracts like LISA. We also collected places where we wanted to distribute the Call for Papers.

I guess most things will get sorted on our PC mailing list, as we move on.

A week ago, a journalist of the swiss NZZ Newspaper had been directed to me by the universities press office. The guy was looking at writing a story on Open Source software and the press office people knew I had some projects in this area. It was the week of the Blaster worm. So after talking for about 40 minutes to the journalist on Wednesday he calls on Thursday to let me know that the story on Open Source had been shelfed, and he had to write a piece on Blaster. Oh, and by the way, would I know anything about the worm. I had been battling it for the last two days, so I said yes. With the effect that I was quoted in the article on Blaster.

The following week a guy from Swiss national television (SF DRS) called, they were doing a piece on Blaster too, with a 'confront MS angle'. They had read my name in the newspaper, I agreed to the interview. I organized coaching from our press office, as the TV people were from that hard hitting investigative show, and I didn't want to be caught in the middle. I was really careful not to make any too harsh comments. They have not yet broadcast it, so I don't know if I am going to look good or bad.

In the meantime Sobig has come back to haunt us, now in its F incarnation. The press office sent the local TV people straight to me when they inquired about a person to talk to regarding Sobig.

As you can see in the screenshot, I am the IT expert now. I picked that profession after the TV people told me that "Systemmanager" was way too complex and the press office people insisted, that "Security Specialist" was going to cause trouble with the folks from CS or central IT Security. Just in case you ever wondered how you become an expert.

I guess I just had my 15 minutes of fame :-).

I have known it for a about a year. SciFi has not renewed its commitment to Farscape which essentially meant the end of the show. I am sure none of the folks at SciFi who made these decisions had watched the show. How else could it be that they killed the single most captivating, intelligent, humorous, sexy, thrilling, emotional and realistic sci-fi show ever conceived. Almost every episode I watched left me tingling all over, ideally wondering how they were able to come up with these great episodes over and over again.

It's almost a year since the cancellation, the Farscape fan-base still seems to be going strong at (www.savefarscape.co ...) plotting away on a strategy to get the show back on the air. Never loose hope.

Today I also went to the website of SciFi wondering what they were doing regarding Farscape. And indeed, they have this page "Farscape Memories" with articles from different actors remembering their time shooting the series. What hypocrisy first they pull the plug on the show and then they act all bleary eyed.

If I was rich, I mean real rich, I would just order a few more years of Farscape from the Creature Shop ... I guess I should not have gone into OpenSource, but rather have turned commercial at an early age, sucking in big bucks during the Internet Bubble then I could do more than keeping my fingers crossed for another network picking up the show again.

The other night I went to see "Wilbur wants to kill himself" (xrl.us ...) a movie about a guy, his brother, a women kid living in some Scottish town. The guy has a book shop, falls in love with the women and the brother tried to kill himself. The movie is by Lone Scherfig a Danish writer/director who was quite successful with the movie Italian for Beginners. I loved the story and the acting and above all the Scottish accent.

The strangest thing happened to me a few minutes into the film. When Alice (the women) entered the scene for the first time. She looks exactly like a friend of mine. I am quite sure it was not here, because she does not speak with a Scottish accent, and she is not an actress. Never the less I kept comparing her and Alice all through out the film. How would she act in a similar situation? And just to make it more complex, Alice is just a role, played by Shirley Henderson (xrl.us ...) but written by Sherfig, so how can it have any bearing on my friends behavior, or do looks influence how people behave, did Sherfig write the part of Alice for Shirley? Are Alice and my friend twins separated a birth?

Well it was a wired experience, but definitely a great movie. Go watch it!

I admit, I am a soap opera junkie. I just love watching them. My current favorite is The West Wing written by Aaron Sorkin (en2.wikipedia.org ...). Tonight I made it through the final episode of the first season. This time the outlook is great, as the show is till running in the US, now in its fifth season.

The West Wing is about people working in the White House for the President. It's also about the President himself. President Bartlet, played by Martin Sheen is a liberal Democrats dream. What a contrast to the current reality. An American friend told me recently that she finds it rather disturbing to watch the show, seeing what the reality could also be. Michael Moor even nominates Bartlet for President, along with Oprah in his latest book.

I am watching the West Wing on DVD, there was also the 15 minutes Making Of feature on one of the disk. There the actors talk about their parts and some bits of the sets are shown. Hearing the actors talk was odd. They were so different from the roles they play in the show. Not only did they talk differently, but also their body language was altered. I have never noticed something like this before in such a show. The characters in the West Wing make a very authentic impression on me, much more than the actors themselves actually. At least as far as the story line and the dialogs are concerned this has to be credited to Aaron Sorkin. Maybe Michael Moor should be nominating Aron Sorkin along with Oprah and Bartlet, someone has to write their lines after all.