Tobi Waves

a presentation for Kris Buytaert

OpenMOSIX is a platform for HPC clustering.

An OpenMOSIX cluster is self organizing. It has no central node. New nodes are discovered as they join the cluster.

Each process is split in two parts, the frontend part which does all the IO and a backend part which does 'the math'. The frontend node stays where it was started. The backend will migrate to another node where it will be executed. Backend parts can continue to migrate at a later stage if the situation changes.

OpenMOSIX uses an economics model to figure out where a process should migrate to.

OpenMOSIX does not require any special libraries to run application. All the core functionality is in the kernel. There is a set of special tools for administration of the cluster. A lot of configuration information is in the /proc interface.

Applications do not have to be recompiled to run on an OpenMOSIX cluster.

For performance reasons, there is no security between the nodes of an OpenMOSIX cluster. So the setup has to be done on a private network.

OpenMOSIX does not do Batch queuing. It can be combined with condor for this to get the best from both worlds (excellent scheduling from condor and simple process migration and check-pointing from OpenMOSIX.

New Features: Migration of shared memory, port to 2.6, no more OpenMOSIX file system (there are enough cluster file-systems already like luster or gfs), migration of most features into user-space.

HOWTO: (www.faqs.org ...)

Homepage: (openmosix.sourceforge.net ...)

OpenMOSIX Knoppix (bofh.be ...)

a presentation by Marco Pfatschbacher

Normally a load-balancer is a physical box sitting in front of the nodes doing the actual work. In this setup the load-balancer becomes a single point of failure. To have high reliability, we need a second load-balancer with fail-over.

Marco presented a method to setup a group of hosts with load sharing/balacing functionality. Instead of using a dedicated load-balancer, the worker nodes are sitting on the same Ethernet segment and each node receives all traffic and just consumes the traffic it is supposed to use.

In order to receive all traffic, all nodes setup a virtual interface with the same Ethernet address and ip number. Simple repeaters have no problems with this, but switches are normally not happy when they do see the same mac address on several ports. The trick to solve this problem, is to configure the physical interfaces to respond with proxy ARP responses telling the switch the Ethernet address of the virtual interface. This will make the switch to always flood the network with traffic destined for the IP address of the virtual interface.

The nodes now use a distributed filtering approach (nms.lcs.mit.edu ...) to decide for each incoming TCP connection which node is going to handle it.

High-Availability is implemented through a small daemon ifstated and CARP (www.newsforge.com ...) to redistribute incoming connections appropriately if one node becomes unavailable.

Known Limitations and further work: Load-Sharing is static and stateful packet filtering (PF) can not be used.

A talk by Sjoera Nas

Bits of Freedom is an Dutch NGO funded by private parties. Their topics are privacy, freedom of speech, spam, e-voting and copyright. In September 2004 they did a test how simple it was to get Dutch ISPs to take down a web page which contains an obviously public domain text.

7 out of 10 providers acted swiftly by taking down the alleged violating document.

The full paper: (www.bof.nl ...)

taught by Dave Thomas

Programmers are like artists, they can only be successful if you have fun doing it. The programmer sitting in front of an empty editor buffer is like an artist in front if a blank canvas.

A good programmer picks the language appropriate for the problem.

Ruby as a language was created in 1994 by Yukihiro Matsumoto (Matz). He combined concepts from various other languages into a new language. Despite what one might expect, the new language is actually beautiful and coherent.

Ruby is similar to Perl in the sense that it does not force a programming paradigm on the user. It is rather a multi-paradigm language allowing procedural, object oriented, functional as well as meta programming.

About the language

All Ruby objects AND classes inherit from the default Class Object which has a default new method on the class that calls the default initialize method on the object. Each method can be overridden.

class Song
   def initialize(a_title)
     @title = a_title
   end   
   attr_reader :title
   attr_accessor :artist
end

a_song = Song.new("Hello")
a_song.title
a_sont.artist = "Sam"

The attr_* functions are meta programming elements, they create an attribute reader and an accessor method respectively.

def title
  @title
end

def artist=(val)
 @artist = val
end

In Ruby you always have to use accessor methods to get to object variables. The advantage is that your code will always stay the same regardless if you actually do something when a variable is set, or just set it directly. The = in the method definition is part of the method name. So even though it looks like an assignments it is actually a method call.

Strings can contain embedded ruby code

"string#{arbitrary ruby code}string"

Methods can be called with blocks of code

def example
  yield arg1,arg2
end
example {|var1,var2| puts var1 puts var2}

or

example do |var1,var2| puts var1 puts var2 end

The yield function will execute the block passed to the example call. In the block, arg1 and arg2 are accessible as var1 and var2.

Ruby does exception handling

def my_file_open(name)
  f= File.Open(name)
  yield f ensure f.close
end

my_file_open("file") do
 |file| line = file.gets
 puts line
end

this will make sure f.close gets called even when the block runs into problems and throws an exception.

Blocks can be nested ...

DBI.connect("DBI:Pg:my_db") do |db|
  db.transaction do
    db.execute("SELECT ..." do |stmt|
      stmt.each do |row|
        # process row
      end
    end
  end
end

In Ruby, variables are untyped while objects are typed by the thing they do (method names). The means if an object has the right methods, you can use it as a replacement for another object. In Ruby this is called Duck Typing (If it walks and talks like a duck it might as well be duck). This helps for things like unit testing. The type of an object is what it can do.

Meta programming

Ruby classes and objects are "open". This means you can add new methods or overwrite existing ones.

class String
  def encrypt
    tr "a-z","b-za"
  end
end

a = "cat"
puts a.encrypt

Adds a new encrypt method to rubies standard String class.

Languages should allow you todo cool things even when it would allow stupid people to horribly break everything.

Classes and Modules get executed at 'definition time'.

class Logger
 if ENV['DEBUG']
   def log(msg)
     STDERR.puts "LOG: " + msg
   end
 else
   def log(msg)
   end
 end
end

A module is a class that can not be instantiated (static method). Class methods in normal Classes have to be prefixed with the Class name to separate them from object methods.

module Dictionary
 WORDS = {}
 File.read("/usr/share/dict/words").split.each do |word|
   WORDS[word]= true
 end 

 def Dictionary.known_words?(word)
   WORDS[word]
 end
end

class Dave
  def Dave.hello
    puts "Hello"
  end
  Dave.hello
end

Method names can end in =, ! and ? in addition to the normal characters and numbers. By convention, = is for 'set' methods, ? is for test methods and ! is for dangerous methods.

Methods always get executed on an object (aka receiver) if no object is mentioned, then the default receiver 'self' will be used. Inside a class definition the default receiver is the current class.

class Dave
  def Dave.hello
    puts "Hello"
  end
  hello
end

Dave.hello

Subclasses inherit class and instance methods of their parents.

module ActiveRecords
  class Base
    def Base.set_table_name(new_name)
      @table_name = new_name
    end
  end
end

class Book lt ActiveRecord::Base
  set_table_name "volumes"
end

Note that @table_name is a variable of the class Base. This works because class Base is actually an object of the class Class.

Additional Ruby Stuff

A good IDE for ruby is Freeride

Rails THE is Ruby framework for web applications. (www.rubyonrails.org < ...)

Ruby has a database interface called DBI (equal to the one in perl) and it also has a much more powerful one called ActiveRecord that maps databases to classes, objects and methods.

class Book ltActiveRecord::Base
end

will access the database "books" and create all sorts of useful methods for accessing the information inside the database. For dumb database like mysql which do not allow to define a internal consistency rules, you can use ActiveRecord to define restrictions on what the database should accept.

Why Ruby

Lightweight

Transparent, Ruby is obvious and easy to read

Portable

OpenSource (MIT, Artistic)

Easy to learn, things work the way you or rather Mats expects.

Ruby is fun.

Stable Language. The language has not changed much over the recent versions, most action happens in the extension libraries.

Resources

Websites (www.ruby-lang.org ...) and (www.rubygarden.org ...) and the newsgroup comp.lang.ruby

Ruby Programmer guide ... (www.pragmaticprogrammer.com ...) (1st edition is available for free online)

taught by Bill Dudney

About Eclipse

Eclipse is a rich-client-platform (rcp) that gives you a lot of functionality for building rich clients, this means that a lot of functionality is contained in the client. Eclips itself is such a "client".

Currently there are about 100 companies who donate resources to eclipse development.

The main focus of Eclipse is still Java, but there is a growing number of plugins for using Eclipse to write code in other languages.

Eclipse consists of 1000s of tools. A perspective is Eclipses way of only showing the tools required to do the task at hand. Eg the Java perspective only shows the tools relevant for Java development.

The Java editor is aware of the Java language and has lots of java specific functions like context help and command completion. It can even help to fix bugs by suggesting fixes to common errors.

Eclipse is fully integrated with JUnit. It can automatically generate JUnit test skeletons for any class you select. The Eclipse Java editor has lots of functions fo code generation. At the touch of a button it can generate all that fluffy code that does not do anything and still has to be repeated many times. (Ruby solves this problem with its META programming functionality in the language itself.)

Eclipse can deal with Ant (the java make alternative) files, but internally Eclipse has its own build system, so it does not keep Ant files in sync. NetBeans on the other hand, uses Ant internally as its build system so if you are into Ants, you may want to look an NetBeans.

Debugging Java with Eclipse

Eclipse can remote debug java applications. With this it is possible to fully separate the application from the eclipse environment. This comes especially handy with big java apps like Tomcat. Use options -Xdebug and -Xrunjdwp: to start the application with remote debugging hooks enabled (this is a special function of the Sun JDK, so it will only work when running with the Sun JVM.

The Eclipse debugger can be enhanced with custom java code (toString methods) to teach it how to represent custom types when printing variables.

J2EE development with Eclipse

If you are doing web application development you should get WTP (Web Tools Project) which provides full J2EE integration for Eclipse.

Installing WTP: First get GEF (Graph Editor Framework), EMF (Eclipse Modeling Framework) and Visual Editor Project and only then you can install WTP successfully.

taught by Alex Russell

What is AJAX?

An acronym for Asynchronous JavaScripts AND XML

Ever wondered how Google does their cool new apps like (maps.google.co ...) or how something like (shared.snapgrid.com ...) works? There is no real standard yet but a keyword: AJAX. Wikipedia has an evolving page about it: (en.wikipedia.org ...)

The key to AJAX applications is that everything happens on the same page. No page reloading is required.

The browser turns from a dumb page renderer into a protocol client.

Where to use AJAX

AJAX is for interactive applications, for static documents, the current xhtml pages are perfect: Accessible, fast, ... only use AJAX when you can make the users lives better and not worse.

One of the big challenges is cross-browser compatibility. While MSDN and Mozilla provide good references for their browsers, they obviously do not talk all that much about compatibility problems. There is a good resource for such information on (www.quirksmode.org ...)

Getting XMLHttpRequest working

In order to be able to make http requests from within a JavaScript you need a XMLHttpRequest object. While it is readily available in modern browsers, it is not so trivial in IE. The most popular way of doing this today is to use conditional compilation in IE.

/@cc_on @/
/@if (@_jscript_version gt= 5)
... code ...
@end @/

Here is a tutorial (www.webpasties.com ...)

More Code Snippets on (www.fiftyfoureleven.com ...)

innerHTML vs DOM

The fastest and most simple way for altering webpages on the fly is to use the innerHTML propperty of a node. The problem is, that setting innerHTML will replace the node completely and with it all it's properties. So a new node will have to have its properties re-attached, even if it has the same ID as the old node.

If you use DOM for manipulating content, your code on the browser side will have to do more, since that data from the server arrives in XML and not pre-generated HTML as you would with innerHTML. The advantage is that you can do much more fine grain manipulations of the content.

DOM vs innerHTML benchmark (www.quirksmode.org ...)

Send JavaScript from the server?

The fastest way of communicating with the server is to send data encoded as JavaScript from the server and using eval() on the client. The advantage of this is that we do not have to parse the data actively on the client side, but we can use the browsers JavaScript parser to interpret the data. The client side can then access the data structures in JavaScript directly to generate the relevant HTML/XML code. This is especially efficient if you are dealing with large tabular data structures.

There is a standardized subset of JavaScript called JSON (JavaScript Object Notation) for this lightweight data exchange method. More about this on (www.crockford.com ...)

REST API

Communication between an Ajax-style UI and your Server should happen via the REST (Representational State Transfer) API of your Web application. (en.wikipedia.org ...) and (www.xfront.com ...)

Debugging JavaScript

The most simple way is to open the JavaScript console of your browser to see any errors the JavaScript engine generates. For automated testing you may want to use a JavaScript implementation that can be started from the command line like Mozillas Rhino (www.mozilla.org ...) project. There are also several Mozilla extensions that help:

LiveHTTPHeaders (livehttpheaders.mozdev.org ...) shows the HTTP headers exchanges between Mozilla and the server.

Venkman a graphical JS debugger (www.hacksrus.com ...)

Ethereal is also helpful as you can see what really happens on the wire. If you do not have the necessary permissions to sniff data off the wire, you may want to redirect your browser through a proxy where you can dump the data that traverses it.

Links

Examples (dojotoolkit.org ...)

Simple AJAX Toolkit (www.modernmethod.com ...)

About using Ajax on Rails the Ruby web toolkit (www.onlamp.com ...)

Ruby on Rails (www.rubyonrails.org ...)

An evolving browser UI toolkit with pluggable widgets that separate HTML from JavaScript Code (dojotoolkit.org ...)

CPAN for Java Script (www.openjsan.org ...)

Wiki with AJAX Framework overview (www.ajaxpatterns.org ...)

Nice widget and screen effects library for ajax applications (openrico.org ...)

taught by Evan Lenz

XSLT is a language for processing XML documents. XSLT itself is written in XML. The output generated by an XSLT 'program' can be anything, but normally it is used for generating (X)HTML documents which can then be displayed by a browser.

XSLT uses the XPATH language for addressing 'nodes' in an XML document.

XPATH Expressions

An XPath expression is made up from several steps separated by /.

step/step/step

A step consists of three elements: The axis to identify a set of nodes relative to the current-context node. The node test to filter out relevant nodes from the set of nodes selected by the axis and finally any number of optional predicates to further filter which nodes get selected.

axis::node-test[predicate][predicate]

XPath expressions can return 4 types of data:

node-set which is a number of zero or more nodes without duplicates

number a floating point number

string a unicode string

boolean true or false

XPath knows seven different types of nodes:

Root - the toplevel node of a document is called "/"

Element - 'tags'

PI - Processing instructions lt?xml ...?gt

Comment - Comment tags

Text - Character data, including white space!

Attribute - lttag attrib="xxx"gt

Namespace - lt?xml-stylesheet ...?gt

When selecting XPath nodes, you can use different 14(!) axis. By default, you use the child:: axis. Default means, that you don't even have to mention it. So an expression like section is actually child::section. The other common axis is attribute:: it also has an abbreviation, called @ so instead of writing attribute::section you can write @section

The other 12 axis are: descendant-or-self:: which can be abbreviated as // and parent:: aka .. and self:: aka . the remaining axis do not have abbreviations: ancestor:: following-sibling:: preceding-sibling:: following:: preceding:: namespace:: descendant:: ancestor-or-self::

XPath example

XPATH looks at an XML document as if it was a filesystem. Like when navigating a filesystem, there is a context node where from where XPATH looks at the document.

ltarticlegt
ltheadinggtHellolt/headindgt
ltparagtParagraph ltemphgt1lt/emphgtlt/paragt
ltparagtParagraph 2lt/paragt
lt/articlegt

If the first para node is the current context node the expressions would return the following:

* - emph
emph - emph
.. - para
../* - heading,para,para
../para[2] - the second paragraph
/article/* - heading,para,para

The XSLT Processing Model

The most important command in xslt is xsl:apply-templates. A lot of people do not use this function properly since they write only one big template, instead of writing many small ones for different purposes.

If you use multiple templates, xslt will invoke a conflict resolution protocol when several templates match for a particular element.

Selection happens by priority:

-0.5  match=""
-0.25 match="xyz:" 
 0    match="name"
  .5  match="nameA/nameB" 

You can override the priority of a template by setting the priority attribute explicitly.

Whenever XSLT processing is stared, the processor executes the template-rule for /. If you do not supply a template-rule the processor will use its built-in template rules (there is one for each of the seven node types).