Tobi Waves

Tutorial by Theodore Ts'o

What to do when data got lost

Don't panic !

Stop and think, figure out what happened. create a backup with

|dd if/dev/hda1 of/dev/hdb1 bs1k convsync,noerrors|

If you have no spare disk, buy one. The disks are way cheaper than the data.

Disks have a life span of 2-3 years, if they are in heavy use ... you might want to swap them preventively just to be sure.

Physical issues

Harddrives can not only experience head crashes but also "high rides". This is the name for incidents when a head flies higher than normal. This condition will only get noticed when data is read back. The new solaris zfs tries to catch this problem by reading back recently written data whenever it has spare time.

Hard drives survive only about 50'000 power downs due to the controlled head crash happening in the landing zone ... This can be a real issues for laptop configurations where frequent disk spin-downs are used to save batteries.

Some harddrives are not designed for continuous use. This will be noted in the spec sheet ... Make sure you check the spec of cheap disks you plan on using in your web-server.

A small head crash will not necessarily cause an immediate disk failure. It could just chip off small amounts of material from the disk surface which will then fly around in the hard-drive case. This condition will cause an increasing number of additional head-crashes which again will chip of material ... This means that it is a good practice to take a full linear image backup of a 'damaged' disk as soon as possible. Errors may well increase as you work on fixing it.

Get a new disk if you find any bad blocks on a disk.

Modern disks

Disks used to have a simple physical geometry. This is still visible in the head, cylinders, sector geometry specifications. Modern disks use constant bit rate and multiple long spiraling tracks to fit more data. This is all hidden by the controller and exposed through a simple linear block number to the OS.

The only thing one can assume about physical disk layout is, that two blocks which are numerically close together will normally have a short seek time.

EFI/GUID partitioning schemes

_Universally Unique IDs (aka GUID)_

A GUID is a 16 Byte number. Either a random number. Collision probability 1/2^64 (birthday paradox). Another method is to take the mac address of the computer plus a hires time stamp. A 3 bit code in the UUID/GUID shows the method used to create the GUID.

The EFI/GUID partitioning scheme uses a GUID to identify each disk as well as each partition. Partition types are "well-known" GUIDs, but still GUIDs (16 Byte) this allows to have unique identifiers for each filesystem type without a central registry.

An EFI/GUID partitioned disk contains an old style MBR patition table in the first sector which claims that the whole disk is covered by a special partition type. This prevents old OSes from messing with an EFI disk. Linux can do EFI partitions on any machine you run linux on. The only special problem is to have a boot loader which is able to deal with it.

About the FAT FS

Because all files are stored as single linked lists, random access is very hard. This also makes file fragmentation very bad. On top of it FAT uses a first free block allocation scheme which again furthers fragmentation.

Inode based Filesystems (FFS)

Stores only the filename and a link to the inode in the directory. The inode then stores all the meta information on the file. This allows to create hard-links.

For short files all blocks are linked directly from the inode. Longer files are created with indirect blocks. Even longer ones are stored with double or even triple indirect blocks.

Inode based filesystems are very fragmentation resistant. This is the reason why there are no defragmenters for Linux.

Old FFS filesystems like UFS allow to specify the physical geometry of the disks to optimize the physical allocation of the filesystem elements. Newer FFS implementations do not bother with this anymore, as there is nothing to be known about disk geometry anyways.

How to recover from accidents

_Overview_

Ask yourself what has happened?

What is the lowest level where you have problems. Always fix the lowest level first.

How important is the data?

When was the last backup performed.

Create a plan of attack before you do anything else.

_Hardware Level_

First indication are often console messages from ide/scsi driver. If you catch a correctable error, you may be able to replace the drive before it actually breaks.

If you see BadCRC errors on a new system it may indicate a simple cabling problem.

The "dev xx:yz" elements in disk errors identify the device file minor/major number affected by the error and thus the partition.

Use |e2fsck -c| to mark bad blocks and see what files are affected.

Check S.M.A.R.T. logs.

In any case make a full image (dd) backup of the disk.

For the image backup you may use |dd_resque| from (www.garloff.de ...) it will alter its block size when it hits a problem to recover as much data as possible without loosing speed while reading is easy, and it has a progress bar.

Partition Table Corruption

If the filesystem can not be found, it may be "only" a problem with the partition table.

|fdisk -l| will show what is there.

Make a backup copy of the MBR. (dd is your friend)

|gpart -W /part.table /dev/hda| can scan the disk for filesystems and reconstruct the partition table. Old filesystems from old partitions still sitting on the disk may confuse gpart.

Filesystem Corruption Problem

Errors may be reported by |e2fsck| during quick boot check or during a full check.

EXT2/3 can also detect errors as it runs ... the actions it should take in this case can be configured at mount time or through tune2fs. For laptops 'remount-ro' is advisable. Servers should better 'panic' as this allows the system to get back into a sensible stat and not limp along. Often such minor corruptions are fixed in the |e2fsck| phase.

In general running |e2fsck| with -y (yes to everything) is fine as you can normally not do anything else than say yes anyway, but |e2fsck| may move orphaned inodes and disconnected directories into 'lostfound' and this should be cleaned up before booting the system fully. The 'file' command can help to identify files. The locate database can help identify the original location of the directory.

e2fsck will not notice blocks with wrong data which are part of a file as it does not maintain any CRCs.

Undeleting Files

In EXT3 unlink will zero out inodes and can thus not be recovered. (this may be 'fixed' at some point')

Undelete on a system level is not possible with EXT3.

Use userspace delete/undelete tools.

Oh and make backups.

|grep -ab regexp /dev/hda1 | awk -F: 'printf(%x\n", ($11023)/1024);}'| (use 4095,4096 for 4k blocks)

Gives the disk blocks where the regexp was found. Then use |lde| to examine the blocks visually.

e2image

The |e2image| tool lets you create a backup of the inode table.

The latest (not released yet) debugfs can use the inode table from an e2image backup, this allows to recover lost files. Even an accidental mkfs can be reverted to a large extent (contents of the root directory will be in lost+found).

It is good practice to run e2image every night.

S.M.A.R.T.

This is the internal health monitoring system of modern hard disks. It will give early warning about disk problems in the waiting.

|smartctl| and |smartd| are your friends here.

Conclusion

Make backups. Save your sanity.

by Greg Koah-Hartmann

Most Unix systems have a device filesystem. So does Linux with devfs. There are three main problems with it.

The code is ugly and beyond repair

The namespace is not LFS compliant

The author of the code has out of the loop for about two years.

A new solution has to be found, as the state of the /dev tree without some automatic management is not tenable. In Debian for example there are 18'000 static entries in there. And on the other hand there are USB plug and play devices which tend to get a different device name every time they are plugged in.

The only thing udev can not do, is to detect a process trying to access a device node that does not exist and then load the relevant driver. This feature of devfs does not seem crutial though.

In the kernel 2.6 there are two main components which make a new and simple solution possible:

The kernel can call a program called |/sbin/hotplug| whenever new devices are connected to the system.

The sysfs filesystem (mounted under /sys) contains all information about devices known to the kernel.

Udev provies a small userspace daemon which manages the /dev tree. It can populate it with a small set of default devices like ttys at boot time and then go on to add all other devices known to the system. It is configurable via simple text file with rules about the naming of the devices. These rules can be pretty sophisticated. Usb devices can be identified according to their vendor or product string as well as through any other property they provide. It is even possible to make udev run an external program which examines the device and then decides how the /dev entry should be called.

All distributions have adopted udev for their linux 2,6 editions. There are some teething problems with distros not using the official udev helper scripts. The author himself maintains the gentoo package.

Udev has to be started VERY early in the boot process, so that other programs can access the devices. Depending on the setup it may be necessary to add udev to initrd. Volume managers and RAID setups are mentioned.

*

by Chris Welti

Motivation

Historically applications running over wide area networks was very bad and people got used to this. Today there is a lot of bandwidth available and the quality is quite good. Still there are performance is not as good as one might expect. Due to the big number of different entities involved in todays complex networks it is very difficult to debug end to end performance issues.

What is PERT

In December 2002 a group called PERT (analog to CERT) was created to help debug such end to end performance issues. PERT is a virtual team of experts. There are cross-discipline experts as well as subject experts. This gets the group both, a good overall as well as deep insight in specific areas.

What do they do

PERT accepts reports about performance problems and analyzes the end to end situation. Information is gathered from all parties involved. Once problems are identified, PERT organizes the implementation the solution as there is often a lot of know-how and know-who involved in getting this fixed.

Performance Hints

TCP buffer size on the sender and the recipient side should be about 2MB on machines which are trying to do fast transfers over wide area links. Default is often 64k. In a real world application this caused file transfers to go from 10 MB/s to 90 MB/s. Rule of thumb: |buffer = bandwidth * round-trip time|.

Duplex miss-match. This is mostly because of hand configured duplex settings on network cards. If one side is configured by hand this normally causes the other side to automatically assume it is on a half duplex link, regardless of the actual manual settings. Rule: always have both sides either manual or auto-negotiated.

Internal buffers of SSH are too small for fast transfer over vast high latency links (see first rule). Needs to be adjusted at compile time! See also (www.psc.edu ...) for more on this.

Small txqueuelen for Giga Bit Ethernet adapters on Linux. Make it larger: |ifconfig eth0 txqueuelen 1000|.

Large round trip times for

Methodology

After identifying the network path, entities and technologies involved PERT applies a divide and conqueror approach to home in on the bottle-neck.

Join the Effort

|pert-discuss@switch.ch|

by Rik van Riel

CKRM (Class-based Kernel Resource Management) is a framework for assigning resources to processes and network connections. Resources can be CPU time, memory, disk IO bandwidth and inbound socket connections. Additional virtual resources like "number of tasks" make the prevention of fork-bombs possible.

In contrast to other Resource Management systems, CKRM comes with a virtual filesystem |/rcfs| which can be used to get information but also t configure the resource classes. The rcfs supports normal permissions to govern access rights. Users can use the system for fine grain control of their own resources on the system. If I have the right to use 50% of the cpu with my processes, I can further define how these 50% are distributed among my processes.

|echo 1234 gt/rcfs/task_class/tc1/targt| moves process 1234 into the task class tc1.

For the automatic classification of processes and network connections there are classification engines available. For example the RBCE (Rule Based Classification Engine).

Inbound connection control can be used to protect server applications from remote denial of service attacks by assigning local addresses to a higher priority resource class.

CKRM is not yet in the mainline kernel. Several people from IBM are working on this full time. The current state of this extension is Beta level. Some interface changes can be expected when mainline kernel integration starts.

More info on (ckrm.sf.net ...)

by Amon Ott

Why

Classic Access Control is not sufficient to define secure access scenarios:

The granularity is way too small (only user group read write execute)

Every user can decide the access permissins for his files.

root can do everything

Solution

Other models (not one, but several appropriate ones) are required for describing security policies according to the requirements of the project.

RSBAC is a framework for implementing access control systems.

It can control individual users and programs as well as incoming and outgoing network connections.

The first stable version has been released in March 2000.

RSBAC can be extended with loadable modules.

Auditing and logging is supported at every level.

Architecture

Subjects - are processes acting on behalf of a user.

Object Types - like FILE, DIR, PROCESS, USERS, NETDEV, ...

Requests - abstraction of what the a subject wants to-do with an object. (eg. R_LINK_HARD, ...)

RSBAC acts on system calls. When a system call is received, the call gets intercepted and passed on to the decision-making facility where a decision is taken if the system call should be performed or not. Once a system call is performed a notification is generated so that the access control system knows what is happening and can take this into account for further decisions.

Models

RSBAC supports number of different access control models.

AUTH - Can be used to restrict which UIDS a process can change to.

Role Compatibiliti (RC) - Subjects and objects are sorted into roles and object types. The rules are then described based on the roles and object types. This makes simple to keep rules stable even though users and objects change.

ACLs - Who may access which object with which rights. RC Roles can be used in this.

File Flags (FF) - Secure Delete, Append Only.

Linux Capabilities (CAP) - lets you control normal Linux capabilities from outside the process.

Process Jails (JAIL) - Like BSD Jails (the better chroot)

Resource Control (RES) - File size, Memory, CPU time, ...

Pageexec (PAX) - anti stack smashing ...

How to get it

Kernel patch from www.rsbac.org

Test ist with the iso images from (www.adamantix.org ...)

Quote

The basic idea of RSBAC is to introduce a second level of security to make sure that errors and mistakes one makes in the first level do not lead to disaster.

by Fredy Künzler

Things I learned

There is no money in ADSL. Swisscom expects the market to be saturated next year.

In summer 2005 Swisscom will offer SDSL (probably 2mb symertical) everywhere. It will be over-booked and not guaranteed availability.

SDSL is like ADSL but without Voice Line on the same wire.

Bluewin 40 million loss per year.

Sunrise can only survive because of their GSM license.

There is no Money in ADSL

End user pays CHF 45.55 (+VAT)

Provider pays CHF 31.20 for the ADSL link to Swisscom. In addition to this the provider has to pay for the network bandwidth between his network and the ADSL backbone (backhaul) this cost CHF 391 for 1 Megabit/s per Month. With moderate overbooking he can fit 40 ADSL customers into one Megabit. This adds another CHF 10 for each of his cutomers.

This means at the end of the day the ISP gets about 4 CHF per ADSL link and month.

The normal wholesale price for 1 Megabit/s connectivity is CHF 100 per Month.

Whats worse, in spring 2003 WEKO got Swisscom to lower the prices they charge for ADSL connectivity by 20%. Swisscom is fighting this decision in court. If they winn, all ISP will have to back the 'missing' 20% back to Swisscom. This will cause a great many of them to go out of buisness.

A tutorial by Joost van Dijk.

About the IPv6 Header

It is much simpler than the IPv4 header. It has a fixed length of 40 Bytes.

Rule 1: Make the frequent case fast

Complex things like fragmentation are handled with extension headers. Because IPv6 routers do not do fragmentation anyway they can do with just looking at the base header. Fragmentation happens in the sending host. The sender uses Path MTU discovery.

With the flow label in the header, The router can see which packets belong together without looking into the packets themselves.

Extension headers can be inserted between the IPv6 header and the payload data. These extension headers come in a predefined order. This order again ensures that the routers only have to look at the first few extension headers because only they can contain information relevant to a router.

Each header has a field called next header which defines the type of the next header.

Implementations

Most current OSes and routers support IPv6. Since Windows XP SP2 there is a production quality IPv6 implementation for Windows as well. The first prototypes were available as a separate downloads from MS.

Enabling it on Windows XP SP2

netsh interface ipv6 install

IPv6 addressing

The first few bits of an IPv6 address define the type of the address. Every IPv6 interface has a link local address (private local address space).

Addresses are written as 8 colon separated tuples of 4 hex digits.

One sequence of zeros can be abbreviated to a double ::

2001:0000:0000:0000:0000:3233:da33:3ad3 becomes 2001::3233:da33:3ad3

The loop back address is ::1

Global Unicast Addresses are built like this

001 - 3 bit
Top Level Aggregator (eg. RIPE) - 13 Bit 
Reserved (these bits could be added the TLA or NLA field in the future) - 8 Bit Next Level Aggregator (ISP) - 24 Bit
Site Level Aggregator (Subnet) - 16 Bit
Interface Address - 64 Bit

RFC 3587 obsoleted this format recently. In the future, the toplevel registrars will decide where the borders are.

Because of the hierarchical nature of addressing the routing tables will become much shorter for IPv6 routing.

Currently only 3 TLAs are defined. Today most new addresses are from the 2001:: Sub-TLA Assignment range. There you get 13 bit sub TLA and 19 bit NLA.

The 48 bit Ethernet addresses can be mapped to the 64 bit IPv6 interface address: (first 24 bit, FFFE, last 24 bit) this is not required though. You can use a random number. Just make sure you get no duplicates.

Addresses starting with 0 ending with an IPv4 address can be used for automatic tunneling.

Current versions of the host dns lookup tool will find IPv6 addresses and it will do reverse lookups automatically when given a numeric address.

Multicast addresses

They start with "FF" there are some well known addresses like

ff02::1 - all nodes on the link

ff02::2 - all routers on the link

ff05::1:3 - All DHCP server at this site

There is a special entry in the routing table for multicast FF00::/8.

ping -c 2 -I eth0 ff02::1 will find all hosts on the local link.

In IPv6 there is a new version of the ICMP protocol (known from ping) it is now also used for ARP and multi cast group membership management.

Get the IPv6 routing table on Linux

route -A inet6

or use the shortcut notation

route -6

Getting an IPv6 address in IPv4 land through tunneling

On way, is to use 6to4. (tldp.org ...) Note that the gateway 192.88.99.1 is a global any-cast address which will automatically go to the closest IPv6 gateway. The 6to4 approach requires you to have a public IPv4 address on your machine or a NAT gateway which can do protocol 41 NAT (protocol 41 is used for 6to4 tunneling)

The new Teredo protocol allows even boxes behind a NAT gateway to get connected to IPv6. Windows XP SP2 has this feature built in. On Linux there is an implementation called miredo which can do the same.

A tutorial by Gerald Carter.

General Things

The big accomplishment of the Samba team is, that they document stuff which MS does not document.

In October 2004, support for Samba 2.x will be dropped.

The configuration parameters parsing and the autoconf files in samba 3 are larger than the whole samba distribution of 13 years ago.

Samba 4 is a complete rewrite from ground up. Don't wait for it!

Samba 3.2 will get backports of some 4.0 features like their RPC code. Better ACL support. Make sure Samba servers look even more like Windows Servers.

There are about 3 people working on Samba 3.x and 3 people working on Samba 4.

A further goal for Samba 4 is to make CIFS protocol as a viable alternative NFS. Unix extensions are being worked on to workout the wrinkles with non unified UIDs.

Samba 3 tibits

The big underestimated tool in samba 3 is net. It is similar to its windows namesake. Unfortunately there is not much documentation on it. But if you start it without parameters it will tell you what it does.

If you run samba without netbios support and you want to use several different configurations on the same server you can add virtual interfaces and then use the %I option for loading different configurations depending on the interface the client connected to.

Samba always tells its version number. This is not a security issue, because if knowing the samba version allows someone to hack into a samba server, this means that there is a bug in samba which needs to be fixed.

Per-service parameters, set in the [global] section will become the default for all services which do not set the parameter explicitly

To reduce the load on your samba server, use the deadtime option in the [global] section. It is set to 0 by default. If you set it to 15 samba will kill seemingly dead connections (happens a lot with print clients) after 15 minutes without negative effects on the client side in general.

In the samba config file you can access environment variables using the %$(ENVVAR) syntax.

SWAT the samba administration GUI will probably be integrated into samba by letting smbd execute swat for connections on port 901.

Windows will not show any shares ending in $. This is only cosmetic though, it does not prevent connections to the share. Using the 'browseable' setting may make more sense as this will prevent listing of the share from the server side (still no security, but you are free to choose the name).

Configuring samba for guest access

[global]
map to guest = bad user
guest ok     = yes
username map = /file
...

And make /file contain

# map everyone to an invalid share 
foo = *

Samba Authentication

Windows uses a challenge response system when authenticating users. This requires both ends to share a common secret. Windows does not store plain text passwords, it does encrypt them, but there is no salt in lanman hashes (windows encrypted passwords). Even worse due to the challenge response system, anyone who is able to get a copy of a encrypted password can use this with a properly hacked smbclient to access the corresponding windows account. Lanman v2 hashes added some measures to prevent 'man in the middle' attacks, but the base problem remains. This means you have to be much more careful to prevent 3rd parties from accessing encrypted windows passwords as they do not even have to be cracked before they can be used.

There is also a positive side to this, because due to the challenge response approach, a hostile (hacked) server will not be able collect passwords from users trying to log on.

Samba can use multiple passdb backends. If several passdb backends are defined in smb.conf, samba will search all backends. If a password gets changed, samba will change it in the passdb backend where the password came from. If a new user is added it gets added to the first passdb backend defined in passdb backend

For storing additional information per user, use at least the tdbsam backend. The text based smbpasswd can only store the most basic information.

Quote: LDAP is not that difficult, but the problem is that people try to walk before they crawl.

Samba needs a Unix account for every user.

Note that smbpasswd does not allow entering the password on the commandline anymore, but it can take input from stdin now:

(echo pass;echo pass)|smbpasswd -a user -s

Access

If users have problems with the fact that they can connect to other users home directories, put the following in your [homes] share.

[homes]
valid users = %S

Instead of using complex mask settings for files and directories, you can set the inherit permissions parameter and manage the permissions on the Unix directory level. This allows to have only one group share with different access permissions down the tree.

Share-level ACLs are done internally in samba, so they do not require any filesystem acl support.

MS-DFS

With MS-DFS, a server can send a transparent referral to a client so that it queries a different server. To make it work the client password must work on both servers.

In smb.conf:

[global]
host msdfs = yes

[dfs]
msdfs root = yes
path = /export/dfs

In /export/dfs do:

ln -s 'msdfs:server1\share1,server2\share2,...' directory

This will cause requests for \\server\dfs to be transparently redirected to \\server1\share1 or \\server2\share2 is the first one is missing.

Smaba can even do DFS proxies. In smb.conf on sever1 do:

[proxyshare]
msdfs proxy = \\server2\anothershare

Printing

On a printer share you can define how much space must be left (in kb) before a new job is accepted:

min print space = 5000

In RPC based printing the %c value contains the number of pages to print.

If using samba as a printer server, you may want to be able to define the default configuration data which is installed together with a printer driver. For this install one printer (lets call it seedprinter) with the driver you are interested in, and change the printer defaults from windows and then call rpcclient with the magic setprinterdata value _p_f_a_n_t_0_m_ this will copy the printer configuration data of 'seedprinter' as the default for all printers who are using the same printer driver as well as for any new printer which is associated with this driver.

rpcclient -U printadmin -c "setprinterdata seedprinter _p_f_a_n_t_0_m_ xxx" server

The xxx argument is ignored, so use just any string ...

The caveat about this is, that when we tried it during the talk it did not work.

NetBIOS

Samba 3 works fine with netbios disabled. Just don't start nmbd, make sure all your servers are in DNS and use the following in your smb.conf file:

[global]
...
name resolve order = host
disable netbios = yes

Several Samba Servers using the same authentication source

To have several Samba servers authenticate against the same user database you can setup one samba server as a PDC and make the other Samba instances into clients of the samba server. Make sure you do not provide winbind with a user id or group id mapping range config so that it falls back to using the user and group ids provided by the Unix host.

Windows Integration

When storing user profiles on samba you may want to use the patch %H/.winprofile/%a as logon path this will store the users profile on a 'per windows release' basis. Note the logon path is not your home!

A PDC requires a machine trust account for each host who is using it. These accounts get created when a machine joins the domain. This means that samba must have appropriate scripts defined to be able to run these scripts, machines must join using the 'root' account of the samba server. This means you need a samba password for root, and the whole setup may make you feel rather edgy :-). The samba folks are working on this.

If you ever want to migrate a Windows NT4 PDC to a Samba domain controller the command net rpc vampire is your friend as it will suck all the account information out of an existing PDC. This relies on the availability of the scripts mentioned in the previous paragraph.

A tutuorial by Tom Limoncelli

This is the second time I attended this tutorial. This time I got the first half, Check the (people.ee.ethz.ch ...) entry.

New Hire Process

Draw up a check list for new hires.

Let new users choose their username.

Visit the new hire on their first day for a short chat about the system.

Give them a 2 page handout with the most important things new users need to know.

Show them how to print.

Show them how to access/install software.

Show them how to get help.

Do a follow-up visit in their the new users second week.

Things people expect to be fast

There are some support requests, people expect to be handled quickly. If it takes us a long time todo these things, our image will suffer badly. Identify the tasks that are supposed to be quick and make sure that they are.

Users for example expect reseting passwords or getting new IP addresses to be quick. Take this into account when deciding on what to work first. Resetting a password really does take very little time, so if you do it immediately, customers will be happy and you do not have to work any harder.

If something is put on hold, tell the user when he can expect the problem to be solved.

You should also look at the damage created by a problem persisting when deciding on the priority for dealing with it. Involve the customer in this decision as he may know more about the side effects of the problem.

Make sure you understand what problem the user has before starting to work. Some people do not report their problems but rather give instructions what support has todo for them. This can lead to interesting situations where there would be a perfectly simple solution to a problem but because the user never told you the problem and you did not ask, you start working on the complex solution to an unknown problem.

The visibility paradox

The best System Managers do not get recognized because everything works and people almost forget about them. So we have to become active to make sure people know that we are working for them.

Have a monthly meeting with the leader of each group you are working for. These meetings can be very short ("30 minutes are enough"). Let them talk and mention your things in passing.

Be physically visible. Have stickers on the computers about how to contact support.

Make sure the office layout lets customers see the people first they are supposed to talk to (front line support) when they walk into your space.

Have a yearly town hall meeting with all your users. Have a lecture on a current topic and then have questions and answers. Don't be afraid of unhappy users who might complain in public, this is much better than people complaining about your behind your back.

When spam^H^H^H^Hmailing all your users ... Make sure that grammar and spelling are correct, keep extra short, have the important information first. Create a useful subject. People will NOT even start to read the mail when they do not see a reason to do so in the subject.

Customer satisfaction

Users will resist answering complex questionnaires. But if you send a short evaluation mail whenever a request ticket gets closed, you may get better response: One question, three possible answers Happy, Indifferent, Unhappy, with links to paste into the browser.

Helpdesk Scope

How about this? We know which things we are responsible for, and for all the other things people bring to us, we know where to refer them to.

Infinite scope but clearly defined responsibility.

If you walk up to a computer with unsupported hardware. Tell the user that this HW was not supported, but that you were allowed to work for 30 minutes on the problem. Then try to fix it for 60(!) minutes and then if you are not successful, tell the user that they would have to buy a supported card.

Continued in (people.ee.ethz.ch ...)

A Keynote by Paul Kilmartin

eBay is the worlds 68th largest economy right behind Kenia.

WalMart took 41 years to grow to somewhat over 100 billion dollar. MS took 19 years to grow over 30 billion, eBay took 8 years to reach 3 billion. Managing growth is a challenge.

eBay started on the first Monday in September 1995 (labour day) out of eBays founders bedroom on a 30$ account with a local ISP. In January 1997 eBay had 4 pcs, at a co-location facility with 10 Mb/s. At that time it was supporting 200'000 auctions. Growth continued in 1999 they had over 200 servers and over 200 Mb/s dedicated bandwidth.

In 1999 things stared getting really serious as eBay had a 22h outage due to its hap-hazard IT architecture not keeping up with the rapid growth. The first attempt to get to grips with this was to throw more hardware and clustered computers at it. This did get them some headroom, but things only started to really get better when they started to split the application out into more individual clusters.

The base idea was to restructure the whole architecture so that one server failing should not take down the whole site. This was implemented by having the same code base on all servers, but running only part of it on any particular machine.

Special precautions had to be put in to prevent clever users from manually changing URLs into executing parts of the ebay functionality on the 'wrong' machine.

By January 2001 eBay another 11 hour outage happened as the whole dual attached SCSI setup for storage went south. At this point a new IT strategy was established at eBay: It is not possible to find 'the ultimate' setup. Everything we setup has a maximum life of 3 years. We do not only need good solutions to our current problems but we also have to plan for its replacement with the next better thing.

Today: 100 Database Instances on Sun 440's and 6800's all in VCS auto failover. 7800 Mb/s bandwidth. Search and Listings on 1100 CPUs on 127 Servers, Mail on 271 Server, ...

Since the 2001 outage the average availability has been around 99.9% ...

Quotes:*

Most support organizations are more interested in closing tickets than fixing problems.

Using market leaders means other people find problems before we do and fixes are introduced quickly.

We are a live entity with millions of active customers, not a cadaver to be dissected. Vendors have to bring fixes for problems not experimental solutions. (Running Sun Explorer is forbidden on production machines.)

We can't set up cameras on the grassy knoll to record the next Kennedy assassination.

A presentation by Luca "ntop" Deri

With todays fast networks packet capture has become a problem.

The problem is two fold, first the packets have to be captured, and second there must be spare cpu cycles for analyzing the packets.

The tool of the trade is the pcap library which provides a unified interface to packet capture. It presents the same API on every OS while it is highly customized on the machine side.

The problem is that pcap performance is not very performant. Especially in problem cases when denial of service attacks happen pcap is not able to capture the traffic because it does not deliver the necessary performance.

There are specialized cards available for packet capture but they do not have pulic APIs and are very expensive.

Lucas tests have shown that normal Linux packet capture is really the worst among Linux (0.2%) , FreeBSD (34%) and Windows (68%).

There have been various attempts to improve this, but even the best solutions were only able to capture (11.7%) of the traffic in the worst case.

In essence this mean using Linux for packet capture is a very bad idea. FreeBSD is better by almost a magnitude.

Lucas idea was to create a special package capture architecture for Linux by providing a new socket type Socket Packet Ring (PF_RING). PF_RING provides a ring buffer in memory for each socket. The application can then read from the ring buffer with mmap. The socket has facilities to record the fact that it had to overwrite a packet in the ring before it was read. This decouples the application from the kernel and improves performance substantially. This implementation is network driver neutral and quite fast (47%) still a bit slower than FreeBSD and still over 50% of the traffic lost in the worst case.

The interesting thing at that point was that the CPU was running at 30% in that situation and loosing 50% of the traffic. Luca found that disabling and enabling the interrupts in the kernel were preventing it from going into packet capturing mode fast enough. The rtirq patch was able to solve this last problem. Luca ended up with a System that was as fast as FreeBSD in capturing packets but with much more CPU to spare.

This solution is about twice as fast as commercial netflow capturing probe selling for much higher prices than the hardware cost for running Lucas solution.

Luca has been investigating ways to further improve performance and found that Gigabit Ethernet drivers on Linux could be programmed much more efficiently by exploiting the cards local packet buffers. A second issue is that Linux does memory allocation and de-allocation whenever it reads from the network card which takes a long time.

Luca has published his work in a project called nCap which provides a accelerated variant of libpcap that lets you recompile your old libpcap applications to reach much better speed.

In an attempt to further improve performance Luca has created a custom gigabit Ethernet card driver that programs the Ethernet card to make its traffic data available directly in the computers memory, freeing the CPU totally from this task, letting it work on traffic analysis exclusively. Luca calls this new method 'straight capture'. This method gives you traffic capture at device speed. With the limitation that only one application per card.

(www.ntop.org < ...)

(www.nmon.net < ...)

A talk by Wietse Venema

Wietse tells how he got into security in the early nineties at Eindhoven university, when trying to figuring out who was scanning their network, writing tcpwrapper in the process and learning lots about network programming.

In 1995 Wietse and Dan Farmer announced their automatic network scanner called SATAN in a white paper. For some people this announcement was introducing the death of the Internet. A lot of discussions happened about a sensible way to release such a dangerous software. Finally it was released on a limited basis first and then to every body at the same time. This was even featured on CNN. The whole excitement about the tool was totally overdrawn though, neither an increase nor a decrease of break-in activity was noted in the wake of the release.

This was just another episode in the ongoing debate over full disclosure or as a US Military put it: "If my systems have a problem, I would rather hear it from a friend."

Finally Wietse talked about his experience when writing PostFix. Work on PostFix started in 1996 out of frustration about the continued security problems of sendmail.

Quote: Some problems in software are found because and others are found because so many users use it.

Today PostFix is the standard in many Systems and building the cornerstone of many large organizations infrastructure.

The release of the PostFix mailer was touted by the IBM PR department in an article in the New York Times. This caused Lou Gersner (the CEO at that time) to start asking Questions about IBMs strategy in OpenSource. A year later IBM had fully embraced Linux from its smallest to its largest systems.

Quotes:

You can run from Windows but you can't escape from it. Suddenly you UNIX-based mail server because a major vehicle for email worms and other malware.

SPF is evil

Spammers don't destroy the infrastructure, it's the well-meaning people with poorly designed coutnermeasures.

On security of OpenSource vs. ClosedSource: You don't need source to find bugs.

The number of people who contribute source to PostFix I haven't got to spend much time fixing before integration is about two.

Security initiative are great, but only for new systems.

You can all layers of security around old systems, but these layers will also have bugs.

More on security (seclabs.cs.ucdavis.edu ...)

a presentation by Rüdiger Weis

My interpretation: TCG is basically a system for vendors who do not trust users. Even though the licenses which go with TCG enabled products with licenses that would not be valid in Europe to enforce it with hardware support. TCG 1.2 fixes some of the problems, but only some.

In principle a security infrastructure in every computer would be nice if it did not come with all the (evil) 3rd party interests and did not have all the problems still to be found in TCG 1.2.

The systems that prevent access to copyrighted songs can also be used to prevent any other documents. It would even be possible to prevent access to some documents at a later date by changing the appropriate access keys.

Imagine the copyright holder looses his keys, all copies of his work may become inaccessible.

The cryptographic foundations of TCG 1.2 stand on pretty week (new) legs cryptographically as many new methods are being employed in this system.

While TCG supports long keys of 2048 bits, it is still possible to use 512 bit keys and weak 160 bit sha-1 keys this raises questions regarding the cryptographic viability of this system.

Neat: https://www.trustedcomputing.org uses an invalid SSL certificate.

Bundesamt für Sicherheit in der Informationstechnik (www.bsi.bund.de ...)

A presentation by Clifford Wolf

SubMaster is a system for distributed software development where every developer can develop on his own repository and then forward patches to a central repository. The patches received at the central repository go into a system from where their integration into the main repository can be controlled by the project maintainer using specialized tools.

With SubMaster you get a rich tool set for working in such an environment.

SubMaster uses SubVersion as its data repository but wraps it so that distributed development becomes possible.

SubMaster comes with two main scripts sm for maintaining local repositories and uploading patches and smap for applying the patches to the main repository.

Further information is available on (www.rocklinux.org ...)