Tobi Waves

Still early today, and I have learned my first lesson: Always negotiate first and then make a deal, don't try to change it later. I was asked to give a talk at the NordU 2003 conference. My talk is scheduled on Thursday. I assumed now that I am here for the week, that I would get free admission to the tutorials, if there was space. Similar to what I am used to from the SANE conference. Unfortunately it seems that the rules here in Sweden are different. No free admission for speakers ... As I said, I guess I should have asked first, before picking the cheaper flight over the weekend.

Then again, the Internet access is quite decent here, the chairs in the terminal room are comfortable and I got a nice big Apple Mac LCD screen to work with. At the actual conference I can go to all the talks, so I guess I am not missing all that much. If things around here stay as quiet as they are presently I might even get some work done.

Here in the terminal room they have these beautiful new iMacs with the 1440x900 flat panel screen. After I had played around for a bit with all the nice gadgets and got impressed by the sleek performance of their new Safari Web Browser I decided to get to work and installed the new rootless X Server Apple is now offering for download. All went well and I was soon looking at my first xterm. Work could begin, so I thought, but then it hit me. Where the hell are all the keys on this Swedish Mac Keyboard? No {, no ], no * not even a $*. Basically everything I need for programming was missing. Eventually I discovered that while the keys were not labeled all the necessary keys were reachable using combinations of Alt and Shift. I went on and even found how to start the Xquartz server with the swedish keymap applied, only to discover the in this configuration the Alt key was acting as Meta and thus lost all its power to create any of my most wanted characters.

If I ever buy a mac, I would have to get a 3 button mouse and a proper keyboard with it. I wonder how programmers at Apple work? Maybe there is a special high clearance programmers keyboard available with all the secret keys labeled properly and working. This model which would wreck havoc in the Mac user population, if leaked to the unwashed, easily confused, masses. It would leave Apples hot-lines go into meltdown as desperate users try to get help in picking the right keys. What is the world coming to?

Yesterday I have been telling various people that I found it rather annoying to not be allowed to go to tutorials. And guess what, just now, Kristen Nielsen of the program comity was here and told me that it was all a misunderstanding, and that I was welcome to join any of the tutorials I want. So I guess I will have to start writing conference reports sooner than I expected.

Tools for Creating Happy Users by Tom Limoncelli and Christine Hogan

I only joined the class in the afternoon, so I can not report on the material covered in the morning. Below I have taken some notes on the topics I found most interesting.

Handling Support Calls

The first topic was "handling support calls". Tom made a case, that handling support calls can be taught and is not just something one knows or doesn't. On the first level, each call can be broken down in to: Greeting, Whats wrong? Fix It, Verify It.

After the greeting, the second step is to get a complete problem statement. Where, When, Who, What? No special tech knowledge is required here, just social skills, active listening and probably some checklists. A complete problem statement will make the actual problem solving much simpler.

Getting a problem statement is already quite sensitive. Eg. one should never ask things which encourage the user to lie: "Is it plugged in?" is bad. Better: "Lets check both ends of the power cord". When a problem statement has been created the problem has to be verified/reproduced/scripted, because otherwise there is no way of verifying the effectively of an eventual fix.

The third step is where the tech/fixing skills are really important. In trying to find a solution, it can be helpful to involve the user into selecting which approach is taken, to fix the problem, and thereby taking their situation (eg time pressure) into account. Keep in mind that you want to know what caused the problem to go away, so only do One thing at a time.

The forth and probably most important step is, that both the person who fixed the problem and the user who reported it verify and agree that the problem has been fixed.

Unless you monitor something, you can not call it a service!

Customers rely on the services we provide, so if there is a problem with one of the services, we should know before the customers do. "Yes, we are already working on it, we expect to have this fixed by 10am, we will send out mail when it works again." is much better than. "Oh, for how long has the company website not been working anymore?"

Monitoring allows to find patterns in breakage, it assists in capacity planning and if combined with a knowledge base, solutions to known problems can be shipped out with the alarm.

My points: Optimize the monitoring for the case when all is well, because this is the state it will run most of the time. Don't have the monitoring system take 'counter action' because this leads into to the 'nanny trap' where successive layers of software are fixing the problems of the underlying system instead of fixing the root cause of the problem. Check out (isg.ee.ethz.ch ...) and also (people.ee.ethz.ch ...) for two tools in this area.

Want to learn more?

Incidentally Tom and Christine have written a book: The Practice of System and Network Administration (www.amazon.com ...) which expands on the soft topics of system administration.

Solaris Internals: Architectural Tips and Tidbits by Richard McDougall richard.mcdougal@eng.sun.com

In this tutorial, Richard highlighted various components of Solaris. Below I have listed the things I found interesting with an emphasis on Solaris 9 features and Solaris 8 tricks.

The Solaris kernel is preemptive. There are only very few non-preemption points in critical code paths. Combined with the threaded kernel this allows for very scalable handling of IO interrupts.

In the 64bit transition of Solaris only long and pointer were changed from 32 to 64bit the other types stayed the same size. (32bit -gt ILP 32, 64bit -gt LP 64).

Oracle profits from 64bit by being able to cache all diskblocks and thus acces data without system calls. Memory mapping a whole database is not unreasonable with todays memory prices.

64bit code has bigger pointers and longs it has to move more data. This which results in a  5% performance loss. If other 64bit features are used this loss of performance is not significant.

Solaris 8 has a new modular, live kernel debugger called mdb which replaces adb and crash. It allows to look at things like the list of open files and other kernel data structures. In Solaris 9 adb and crash are removed. In Solaris 9 mdb even understands c structs defined in header files. It uses this information for "nice" memory dumping.

The kstat command is used to show kernel statistics: eg. kstat -n~system_misc. Kstat is written in perl and uses a perl module to access data. This module can be easily used in other programs.

About the Solaris Virtual Memory System

VM is all done on demand, pages get loaded as they are used, but the memory reservation (in swap) happens on exec, fork and break. The effect of this is, that a lot of swap gets reserved but it does not actually get used as long as it does not get touched by the code. Forking apache reserves the same amount of memory for the forked copy as for the original but it only copies the bits of the original that get modified (copy on write). The advantage is that Solaris will be able to actually provide all the memory a process got allocated in the first place and will not die halfway through the operation because it promised more memory than what was actually available. (Linux and AIX are less conservative as they hand out reservations without looking at the available memory.)

The loading of programs uses a similar process where the file on disk becomes part of the OS memory (it gets memory mapped) and then get successively loaded (demand paged) into RAM as when parts of the program get executed.

When multiple copies of a program get started (or forked) they will all share the same memory as long as they do not modify it (copy on write).

How to determine how much memory a process really uses? The text segment of each process is read only so all instances of the same program or shared library are shared. The data segment is initially shared but will get split partly as the programs modify their statically allocated memory and copy on write happens. With

pmap -x  PID

I get a detailed report of what memory the process really uses. The interesting part is the total of private memory (or anon memory in Solaris 9) which shows the amount of memory which is exclusive to the process. Solaris 9 gives us pmap -S which will show how much memory is reserved in swap.

Memory Access Speed

Solaris 9 U1 has some support for NUMA (non uniform memory access time architectures) architectures. This means that Solaris can handle the fact that different sections of the memory have different access times. While Suns multi CPU boxes are essentially symmetrical, so all memory has the same access time, the larger variants like Ex800 or F15k are slightly NUMA. Solaris 9 U1 deals with this fact by building memory latency groups and taking the access time relative to the cpu into account when allocating memory to a process. This happens automatically but there are also APIs for influencing this when writing programs.

How to determine if there is a memory shortage

Solaris 8 has an all new paging system which has much better performance than previous versions. The vmstat -p~3 command will show a detailed report on the paging activity. By looking out for high number in the Anonymous column this indicates that there is a shortage of physical memory. So this is the way to check if any bad swapping is happening. Make sure all VM tunables are removed from /etc/system when migrating to Solaris 8.

In Solaris 9 the mdb -k command ::memstat gets a detailed list of memory usage.

CPU Usage Accounting

In Solaris 8 interrupts (eg from the Network Card) are accounted as idle time while Solaris 9 does account them correctly.

With trapstat in Solaris 9 it is possible to see how many interrupts that are occurring.

How to determine where the kernel is spending it stime

The command lockstat -kIi997 sleep 10 will monitor what the kernel is doing for 10 seconds and sample the kernel threads 997 time a second and then show you what it found. lockstat sleep 10 will tell how many locks of what type occurred where within the last 10 seconds. The mpstat 1 will show some per processor statistics. Especially the column Intr will show if the box is spending a lot of time being interrupted. If Smtx is high, this indicates that the CPU is spinning on mutexes while it unsuccessfully tries to acquire a lock.

Kernel level Threading

Up to Solaris 8 the default thread model was based on the idea of a user-level scheduler mapping user-land threads onto kernel threads (LWP). Thread switching happens on blocked threads only. So this is almost like cooperative multitasking with the obvious problems. The nice thing is that this can be very fast as the handing of between threads is very light weight and could create very efficient programs. Unfortunately the whole system is complex and quite unreliable the massive number of thread patches is a witness to this fact.

An alternative threading library is sitting is /usr/lib/lwp it turns all threads into kernel threads and scales really well because there is now only one scheduler. All the fairness problems go away. What is really cool about it, a simple

LD_LIBRARY_PATH=/usr/lib/lwp program

will make a threaded application use the kernel threads. Note that this is only working really well from S8U7 (kernel jumbo from feb 2002) in Solaris 9 the new threading lib is the default.

A better top: prstat

Solaris 8 includes something like top called prstat it has various options to show all sorts of statistics: -m will show per process microstat where it is possible to see much better where a process is spending its time. With out options it acts just like top. With -t you get stats on a per user level.

Tricks with truss

See what is happening in the program besides system calls

truss -d -u a.out,libc program

Find out in which system calls take how much time

truss -c program

Filesystem Tuning

Machines which access many files concurrently might profit from setting ufsninode and ncache to higher values. Searching for "system tuning manual" on (docs.sun.com ...) will give more information. This could also be of interest for Machines running diskless which access many different files concurrently over NFS as inode cache misses are even more expensive.

In Solaris 9 there is an option to get checksumming on the device driver level to make sure files do not change on disk.

The kernel parameters ufs_LW and ufs_HW make sure that UFS does not consume too much memory (write throttling) this has a rather negative impact on performance as their default value is too low. In Solaris 9 this is therefore set to higher values. Richard suggested the following to be put in /etc/system of Solaris 8 systems especially if they have a lot of memory:

set ufs_LW=4194304
set ufs_HW=67188864

Check (kr.sun.com ...) and (www.princeton.edu ...) for some information.

Another important tunable parameter is maxphys. It sets the maximal junk of data the filesystem will write to disk in one go. For SCSI this is ste to 128k by default which is way too low. Richard suggests to set this to 2 MByte.

set maxphys=2097152

Check (206.231.101.22 ...)

To learn more ...

For more information, Richard recommends his Solaris Internals book: (www.amazon.com ...) or (www.solarisinternals.com ...)

Yesterday Tom Limoncelli said that he thought that one day tutorials were probably the best method for learning new things. Today I did the 'self experiment' and spent the whole day in Richard McDougall's Solaris tutorial. Only minutes after the tutorial has finishes I can confirm that I really do feel exhilarated and would like to try out all the thing Richard touched upon. But looking more closely at what exactly it is, that I learned from today's slew of slides and explanations, all that remains are my notes, a headache and quite a number of areas I would like to investigate. I have not yet learned anything in the sense that I have tested and applied what Richard has been explaining in the real world. So even though I would have never "learned" as much as I heard today, I guess I would have profited more from less information and more hands-on training on real world problems. Or if hands-on was not possible, then probably paper exercises where I had to think up solutions which then would have been discussed later on.

The main problem with this more thorough approach is, that it would be way less sexy than the information blast method and people who pay for these tutorials want something out of them. At worst people might complain, that while they had learned how to make their Sun run faster they effectively had figured it out on their own and wondered why they had to pay so much for an instructor to only ask them questions.