Tobi Waves

by Theodore Ts'o

History

VA Linux wanted to be the IBM of the Linux world, but forgot to ask whether maybe IBM wanted to become the IBM of Linux. This was a huge mistake.

What makes Linux significant

One of the first projects done primarily over the Internet without a central group contributing all the code.

Difference to other projects. Many of the people involved are on a mission to distribute the knowledge about the system. Even people who can not bootstrap their own system are welcome.

Open Source and Free Software

The Open Source movement brought the whole "making source available is cool" concept into the industry without the political issues of the GPL.

Intellectual property laws are going to become more and more of a problem, so we have to become involved.

The huge number of Open Source licenses leads to incompatibilities when working with libraries. The important lesson here is to pick the right license from the start, as relicensing is difficult as soon as you have received a lot of code contributions.

Rapid Development and Open Source

Infrastructure work or documentation writing are necessary tasks but they are not very sexy. We need to cater for them. Incremental changes are not the only thing.

Even though there has not been a lot of innovation coming from Redmond. This does not mean they are sitting on their hands. They are working on vertical integration technologies like their new user interface description language which will be used on all their products.

User Interface Testing

Have programmers watch a users working with their software. Do not let them talk to the users and help them. They will see all the problems which are to be fixed.

The Limits

Software with a short shelf life like tax software or games needs a huge investment in development and have to recover the cost quickly, So this will probably never work with opensource concepts.

Binary drivers are a fact of life due to paptent and even legal issues. There is not clear solution for this yet in connection with Open Source.

Closing

It's been a great 10 years, but there is still a lot todo.

by Robert Griessmer

For the technical Information, read my earlier entry about Urs Hoelzels talk.

About cheep hardware: If one writes his software under the assumption that hardware is going to fail anyway. This makes lots of problems much simpler. If the number of machines grows really high, you can employ people who spend their days swapping broken ones.

These days Google designs their own bare-bone boards with only the necessary components for the stuff, the machines are required to-do.

Google Philosophy

Make the worlds information search able.

Do things that matter, and do them algorithmically so that they are scalable.

Every engineer has a free day per week to work on his own fun projects.

Google Tricks

Spell checking happens statistically. Google does not know what is right, but what is most popular.

Google provides an API for programmers.

Experimental projects (labs.google.com ...)

Google Zurich Office

Right in the city center. Entire range of software development. Meant to attract European engineering talent. Tight integration with the Mountain View offices.

Jobs: (www.google.ch ...)

User Questions

Q: What is your crawling bandwidth

A: Well about this (shows with hands) Size.

...

A: oh and ... no comment ... I can't really talk about this.

by David Rosenthal

Open Source in Government

Could government require that only Open Source software be submitted for a new text processing system to be procured? Yes, but only if the fact of it being Open Source was directly tied to the core functionality of the software and requirements in this area. This point may be difficult to make, depending on the nature of the software.

If GPL software is modified by one branche of the governemnt and passed on to a different branch. This is a sort of publication. Accordning to the GPL license the second branche gets a license with the code which allows it to redristribute the modified version without restrictions, takeing it effectively out of the control of the first party.

Copyright

Every program of an "individual nature" is protected by copyright law. "Individual" means, that the code is potentially recognizable as being written by that particular author. This right does not have to be registered or patented or anything. Every author has it.

In Swiss law this means that I as an Author have the exclusive right to decide whether and how my product is used. I can give all or some of my rights up to a third party .. aka grant a license. GPL or any of the other Open Source licenses operate on this principle.

GPL can be enforced. 19.5.2004 the Munich District upheld the GPL in a case of netfilter team vs firewall vendor.

Contract Law

If I modify an Open Source product for a third party (contract) I can become liable for any problems with the whole product if I do not exclude this in the contract. This is not special for Open Source products but it may easily go under the radar. Becareful when drafting your contracts.

Warranty and liability can be limited, but only to the extent allowed in the law. Swiss Law: Any limitation of liability for gross negligence or willful intent is void.

Do not trust in disclaimers. Be open about the limitations of your product. Reduce the expectations in your product.

Software Patents

European patent law is fundamentally different from US patent law. In the US "everything under the sun, made by man" can be patented. In Europe patent are strictly limited to technoligical inventions. And even there. Patents have to be new, not obvious to someone skilled in the 'art' and there must be no prior art. This is not going to change in the next 10 to 20 years at least.

Patents are granted easily, but they may not be valid. In the first 9 Months after publication they are very simple to kill. Greenpeace does this all the time in the biotech area.

The likely hood of a big player (aka MS) going against a small one are slim as the PR damage would be huge, not to speak of all the anti-trust issues.

Conclusion

Open Source doesn't have particular legal problems. But make sure you read the fine print of the licenses.

Legal issues with OSS are very similar to commercial software.

Worrying about patents is of no use.

by Poul-Henning Kamp

Compters are now 50 years old. Unix is 30 years old.

We write code on the screen and not on paper. but thats about it.

Unix has blown more chances at being a big success than any other operating system. It's about making the same mistakes again.

The problem

Programming happens in the brain and not in the computer. Throw your thinking at problems not more hardware. Programmers should have slow machines.

Instead of baking a bigger cake. Unix companies fight about the same piece.

The state today

Uncountable Linux distros, a handfull of BSDs, IBM AIX - IBM the Unix way, Sun Solaris - Unix the Sun Way, HP-UX - Unix cul de sac, Mac OS X - yea, its Unix, but don't worry about it.

Unix Standards

Very weak, Incomplete, Ambigous,

POSIX. Everybody made sure that their product was covered by it. Its not a standard but rather a panorama of things to-do. MVS is POSIX compliant, and so is Windows.

The one good standard is the "POSIX 1E" security extension. Which was never formally adopted, but everybody sticks to it religiously.

The Linux Standards Base will fail because it defines what we have today. It repeats the mistakes of POSIX. Its not about how Unix should be.

Should we save Unix

_No_

Architectural mess.

No significant invention in the last 20 years

Everybody thinks in his box.

_Yes_

You get the source

The only alternative is LongHorn

Can Unix be saved - No

no market model

no cooperation to generate a market

too much politics

Can Unix be saved - Yes

Start thinking outside your box

Stop bickering about irrelevant details (BSD/Linux, Gnome/KDE)

Work on the real problems. Fight for open data. Fight the software patent mafia.

Invent things! Plan9 (namespaces), Sun (Java), Apple (User Interfaces), Your Name could be here,

Find the next Web.

Quotes

The only thing Unix has invented is Unix.

The KDE people sit in the KDE box. They have a little hole in their box to see the GNOME people.

I don't care about your license as long I get your source. I even wrote a license for it.

Tutorial by Theodore Ts'o

What to do when data got lost

Don't panic !

Stop and think, figure out what happened. create a backup with

|dd if/dev/hda1 of/dev/hdb1 bs1k convsync,noerrors|

If you have no spare disk, buy one. The disks are way cheaper than the data.

Disks have a life span of 2-3 years, if they are in heavy use ... you might want to swap them preventively just to be sure.

Physical issues

Harddrives can not only experience head crashes but also "high rides". This is the name for incidents when a head flies higher than normal. This condition will only get noticed when data is read back. The new solaris zfs tries to catch this problem by reading back recently written data whenever it has spare time.

Hard drives survive only about 50'000 power downs due to the controlled head crash happening in the landing zone ... This can be a real issues for laptop configurations where frequent disk spin-downs are used to save batteries.

Some harddrives are not designed for continuous use. This will be noted in the spec sheet ... Make sure you check the spec of cheap disks you plan on using in your web-server.

A small head crash will not necessarily cause an immediate disk failure. It could just chip off small amounts of material from the disk surface which will then fly around in the hard-drive case. This condition will cause an increasing number of additional head-crashes which again will chip of material ... This means that it is a good practice to take a full linear image backup of a 'damaged' disk as soon as possible. Errors may well increase as you work on fixing it.

Get a new disk if you find any bad blocks on a disk.

Modern disks

Disks used to have a simple physical geometry. This is still visible in the head, cylinders, sector geometry specifications. Modern disks use constant bit rate and multiple long spiraling tracks to fit more data. This is all hidden by the controller and exposed through a simple linear block number to the OS.

The only thing one can assume about physical disk layout is, that two blocks which are numerically close together will normally have a short seek time.

EFI/GUID partitioning schemes

_Universally Unique IDs (aka GUID)_

A GUID is a 16 Byte number. Either a random number. Collision probability 1/2^64 (birthday paradox). Another method is to take the mac address of the computer plus a hires time stamp. A 3 bit code in the UUID/GUID shows the method used to create the GUID.

The EFI/GUID partitioning scheme uses a GUID to identify each disk as well as each partition. Partition types are "well-known" GUIDs, but still GUIDs (16 Byte) this allows to have unique identifiers for each filesystem type without a central registry.

An EFI/GUID partitioned disk contains an old style MBR patition table in the first sector which claims that the whole disk is covered by a special partition type. This prevents old OSes from messing with an EFI disk. Linux can do EFI partitions on any machine you run linux on. The only special problem is to have a boot loader which is able to deal with it.

About the FAT FS

Because all files are stored as single linked lists, random access is very hard. This also makes file fragmentation very bad. On top of it FAT uses a first free block allocation scheme which again furthers fragmentation.

Inode based Filesystems (FFS)

Stores only the filename and a link to the inode in the directory. The inode then stores all the meta information on the file. This allows to create hard-links.

For short files all blocks are linked directly from the inode. Longer files are created with indirect blocks. Even longer ones are stored with double or even triple indirect blocks.

Inode based filesystems are very fragmentation resistant. This is the reason why there are no defragmenters for Linux.

Old FFS filesystems like UFS allow to specify the physical geometry of the disks to optimize the physical allocation of the filesystem elements. Newer FFS implementations do not bother with this anymore, as there is nothing to be known about disk geometry anyways.

How to recover from accidents

_Overview_

Ask yourself what has happened?

What is the lowest level where you have problems. Always fix the lowest level first.

How important is the data?

When was the last backup performed.

Create a plan of attack before you do anything else.

_Hardware Level_

First indication are often console messages from ide/scsi driver. If you catch a correctable error, you may be able to replace the drive before it actually breaks.

If you see BadCRC errors on a new system it may indicate a simple cabling problem.

The "dev xx:yz" elements in disk errors identify the device file minor/major number affected by the error and thus the partition.

Use |e2fsck -c| to mark bad blocks and see what files are affected.

Check S.M.A.R.T. logs.

In any case make a full image (dd) backup of the disk.

For the image backup you may use |dd_resque| from (www.garloff.de ...) it will alter its block size when it hits a problem to recover as much data as possible without loosing speed while reading is easy, and it has a progress bar.

Partition Table Corruption

If the filesystem can not be found, it may be "only" a problem with the partition table.

|fdisk -l| will show what is there.

Make a backup copy of the MBR. (dd is your friend)

|gpart -W /part.table /dev/hda| can scan the disk for filesystems and reconstruct the partition table. Old filesystems from old partitions still sitting on the disk may confuse gpart.

Filesystem Corruption Problem

Errors may be reported by |e2fsck| during quick boot check or during a full check.

EXT2/3 can also detect errors as it runs ... the actions it should take in this case can be configured at mount time or through tune2fs. For laptops 'remount-ro' is advisable. Servers should better 'panic' as this allows the system to get back into a sensible stat and not limp along. Often such minor corruptions are fixed in the |e2fsck| phase.

In general running |e2fsck| with -y (yes to everything) is fine as you can normally not do anything else than say yes anyway, but |e2fsck| may move orphaned inodes and disconnected directories into 'lostfound' and this should be cleaned up before booting the system fully. The 'file' command can help to identify files. The locate database can help identify the original location of the directory.

e2fsck will not notice blocks with wrong data which are part of a file as it does not maintain any CRCs.

Undeleting Files

In EXT3 unlink will zero out inodes and can thus not be recovered. (this may be 'fixed' at some point')

Undelete on a system level is not possible with EXT3.

Use userspace delete/undelete tools.

Oh and make backups.

|grep -ab regexp /dev/hda1 | awk -F: 'printf(%x\n", ($11023)/1024);}'| (use 4095,4096 for 4k blocks)

Gives the disk blocks where the regexp was found. Then use |lde| to examine the blocks visually.

e2image

The |e2image| tool lets you create a backup of the inode table.

The latest (not released yet) debugfs can use the inode table from an e2image backup, this allows to recover lost files. Even an accidental mkfs can be reverted to a large extent (contents of the root directory will be in lost+found).

It is good practice to run e2image every night.

S.M.A.R.T.

This is the internal health monitoring system of modern hard disks. It will give early warning about disk problems in the waiting.

|smartctl| and |smartd| are your friends here.

Conclusion

Make backups. Save your sanity.

by Greg Koah-Hartmann

Most Unix systems have a device filesystem. So does Linux with devfs. There are three main problems with it.

The code is ugly and beyond repair

The namespace is not LFS compliant

The author of the code has out of the loop for about two years.

A new solution has to be found, as the state of the /dev tree without some automatic management is not tenable. In Debian for example there are 18'000 static entries in there. And on the other hand there are USB plug and play devices which tend to get a different device name every time they are plugged in.

The only thing udev can not do, is to detect a process trying to access a device node that does not exist and then load the relevant driver. This feature of devfs does not seem crutial though.

In the kernel 2.6 there are two main components which make a new and simple solution possible:

The kernel can call a program called |/sbin/hotplug| whenever new devices are connected to the system.

The sysfs filesystem (mounted under /sys) contains all information about devices known to the kernel.

Udev provies a small userspace daemon which manages the /dev tree. It can populate it with a small set of default devices like ttys at boot time and then go on to add all other devices known to the system. It is configurable via simple text file with rules about the naming of the devices. These rules can be pretty sophisticated. Usb devices can be identified according to their vendor or product string as well as through any other property they provide. It is even possible to make udev run an external program which examines the device and then decides how the /dev entry should be called.

All distributions have adopted udev for their linux 2,6 editions. There are some teething problems with distros not using the official udev helper scripts. The author himself maintains the gentoo package.

Udev has to be started VERY early in the boot process, so that other programs can access the devices. Depending on the setup it may be necessary to add udev to initrd. Volume managers and RAID setups are mentioned.

*

by Chris Welti

Motivation

Historically applications running over wide area networks was very bad and people got used to this. Today there is a lot of bandwidth available and the quality is quite good. Still there are performance is not as good as one might expect. Due to the big number of different entities involved in todays complex networks it is very difficult to debug end to end performance issues.

What is PERT

In December 2002 a group called PERT (analog to CERT) was created to help debug such end to end performance issues. PERT is a virtual team of experts. There are cross-discipline experts as well as subject experts. This gets the group both, a good overall as well as deep insight in specific areas.

What do they do

PERT accepts reports about performance problems and analyzes the end to end situation. Information is gathered from all parties involved. Once problems are identified, PERT organizes the implementation the solution as there is often a lot of know-how and know-who involved in getting this fixed.

Performance Hints

TCP buffer size on the sender and the recipient side should be about 2MB on machines which are trying to do fast transfers over wide area links. Default is often 64k. In a real world application this caused file transfers to go from 10 MB/s to 90 MB/s. Rule of thumb: |buffer = bandwidth * round-trip time|.

Duplex miss-match. This is mostly because of hand configured duplex settings on network cards. If one side is configured by hand this normally causes the other side to automatically assume it is on a half duplex link, regardless of the actual manual settings. Rule: always have both sides either manual or auto-negotiated.

Internal buffers of SSH are too small for fast transfer over vast high latency links (see first rule). Needs to be adjusted at compile time! See also (www.psc.edu ...) for more on this.

Small txqueuelen for Giga Bit Ethernet adapters on Linux. Make it larger: |ifconfig eth0 txqueuelen 1000|.

Large round trip times for

Methodology

After identifying the network path, entities and technologies involved PERT applies a divide and conqueror approach to home in on the bottle-neck.

Join the Effort

|pert-discuss@switch.ch|

by Rik van Riel

CKRM (Class-based Kernel Resource Management) is a framework for assigning resources to processes and network connections. Resources can be CPU time, memory, disk IO bandwidth and inbound socket connections. Additional virtual resources like "number of tasks" make the prevention of fork-bombs possible.

In contrast to other Resource Management systems, CKRM comes with a virtual filesystem |/rcfs| which can be used to get information but also t configure the resource classes. The rcfs supports normal permissions to govern access rights. Users can use the system for fine grain control of their own resources on the system. If I have the right to use 50% of the cpu with my processes, I can further define how these 50% are distributed among my processes.

|echo 1234 gt/rcfs/task_class/tc1/targt| moves process 1234 into the task class tc1.

For the automatic classification of processes and network connections there are classification engines available. For example the RBCE (Rule Based Classification Engine).

Inbound connection control can be used to protect server applications from remote denial of service attacks by assigning local addresses to a higher priority resource class.

CKRM is not yet in the mainline kernel. Several people from IBM are working on this full time. The current state of this extension is Beta level. Some interface changes can be expected when mainline kernel integration starts.

More info on (ckrm.sf.net ...)

by Amon Ott

Why

Classic Access Control is not sufficient to define secure access scenarios:

The granularity is way too small (only user group read write execute)

Every user can decide the access permissins for his files.

root can do everything

Solution

Other models (not one, but several appropriate ones) are required for describing security policies according to the requirements of the project.

RSBAC is a framework for implementing access control systems.

It can control individual users and programs as well as incoming and outgoing network connections.

The first stable version has been released in March 2000.

RSBAC can be extended with loadable modules.

Auditing and logging is supported at every level.

Architecture

Subjects - are processes acting on behalf of a user.

Object Types - like FILE, DIR, PROCESS, USERS, NETDEV, ...

Requests - abstraction of what the a subject wants to-do with an object. (eg. R_LINK_HARD, ...)

RSBAC acts on system calls. When a system call is received, the call gets intercepted and passed on to the decision-making facility where a decision is taken if the system call should be performed or not. Once a system call is performed a notification is generated so that the access control system knows what is happening and can take this into account for further decisions.

Models

RSBAC supports number of different access control models.

AUTH - Can be used to restrict which UIDS a process can change to.

Role Compatibiliti (RC) - Subjects and objects are sorted into roles and object types. The rules are then described based on the roles and object types. This makes simple to keep rules stable even though users and objects change.

ACLs - Who may access which object with which rights. RC Roles can be used in this.

File Flags (FF) - Secure Delete, Append Only.

Linux Capabilities (CAP) - lets you control normal Linux capabilities from outside the process.

Process Jails (JAIL) - Like BSD Jails (the better chroot)

Resource Control (RES) - File size, Memory, CPU time, ...

Pageexec (PAX) - anti stack smashing ...

How to get it

Kernel patch from www.rsbac.org

Test ist with the iso images from (www.adamantix.org ...)

Quote

The basic idea of RSBAC is to introduce a second level of security to make sure that errors and mistakes one makes in the first level do not lead to disaster.

by Fredy Künzler

Things I learned

There is no money in ADSL. Swisscom expects the market to be saturated next year.

In summer 2005 Swisscom will offer SDSL (probably 2mb symertical) everywhere. It will be over-booked and not guaranteed availability.

SDSL is like ADSL but without Voice Line on the same wire.

Bluewin 40 million loss per year.

Sunrise can only survive because of their GSM license.

There is no Money in ADSL

End user pays CHF 45.55 (+VAT)

Provider pays CHF 31.20 for the ADSL link to Swisscom. In addition to this the provider has to pay for the network bandwidth between his network and the ADSL backbone (backhaul) this cost CHF 391 for 1 Megabit/s per Month. With moderate overbooking he can fit 40 ADSL customers into one Megabit. This adds another CHF 10 for each of his cutomers.

This means at the end of the day the ISP gets about 4 CHF per ADSL link and month.

The normal wholesale price for 1 Megabit/s connectivity is CHF 100 per Month.

Whats worse, in spring 2003 WEKO got Swisscom to lower the prices they charge for ADSL connectivity by 20%. Swisscom is fighting this decision in court. If they winn, all ISP will have to back the 'missing' 20% back to Swisscom. This will cause a great many of them to go out of buisness.