SIDUS : Single Instance Distributing Universal System

For proper English, please refer to this Linux Journal article published in November 2013. It was written by Marianne Corvellec based on Emmanuel Quemener's articles.

All items in this documentation, code snippets, etc. are part of the license CeCILL. It is therefore necessary to respect the four freedoms of free software to operate SIDUS in its own infrastructure. If you intend to use in your SIDUS infrastructure, let her know author, Centre Blaise Pascal or quote it!
First presentation with video of SIDUS on Scipy 2013

SIDUS is the contraction of Single Instance Universal Distributing System and purposes to oversimplify the administration of large amount of machines.

Its latin origin of set of stellar bodies is an allegory: we let you find one that suits you best =).

SIDUS has two fondamental properties :

  • unicity of configuration : two machines starting SIDUS run exactly the same Operating System.
  • use of local resources : processors and RAM used are local host one.

SIDUS is not :

  • neither LTSP for Linux Terminal Server Project: LTSP offers simplified management of thin clients by providing a X11 or RDP access to a server: and it supports the entire processing load. Conversely, SIDUS operates entirely (or at the discretion of the user) any machine that hangs there. Only the storage operating system is offset to third machines.
  • neither FAI for Fully Automatic Installation: FAI and Kickstart offer a complete easy installation to minimize or eliminate any action by the administrator. Conversely, SIDUS offers a unique system in a tree incorporating both the base system and all installed applications manually.
  • neither a network LiveCD: LiveCD starts a minimalist system, necessarily fixed. It is always possible to create your own LiveCD but it is a tricky operation. With SIDUS, it is possible to install on the fly for all users new software.

Five Ws & Two Hs

Why ?

Time of administration system of computer equipment (compute node, post or personal workstation, virtual machine testing) increases with the number and diversity. Thus, all these materials essentially share the same files, but each on its own disk. How to limit the installation time and administrative machinery while retaining the flexibility related to their destination?

This is the challenge faced by SIDUS (for Single Instance Universal Distributing System), developed at the Centre Blaise Pascal originally mainly to simplify the task of the system administrator only deal with the management of hundreds of machines of all kinds for all destinations (technical platforms multi-node, multi-core, GPU, etc.).

What ?

One approach to network booting and provides a perfectly generic system for all machines.

When ?

The first version of SIDUS of February 2010, it was originally on Debian Etch. It followed all the developments of the Debian distribution, essential base (or exclusive) of Blaise Pascal Centre.

Early 2013, SIDUS serves nearly 90 permanent cluster nodes to CBP (up to 120 simultaneous based loans), GPGPU servers, workstations and multi-core positions COMOD (for Compute On My Own Device).

For Who ?

SIDUS is for all those who want to simplify the task and that:

  • Have groups of machines having all the same destination:
    • Nodes computing cluster
    • Machines free service room
    • Virtual machines in training
  • Want to analyze equipment without ever touching the system

Where ?

The Centre Blaise Pascal, hotel projects ENS-Lyon in the field of computing and scientific computing, SIDUS uses for all its amenities including uniformity must be preserved as much as possible: a simple reboot should be sufficient to replace the system in its original condition.

The Scientific Centre of Numerical Modeling (PSMN) computing center ENS-Lyon, now uses SIDUS over a hundred knots and prepares its generalization for the establishment of Meso@Equip (nearly 200 additional nodes). On september 2013, the number of nodes working under SIDUS exceeds 320.

The chemistry laboratory uses “COMOD” for some jobs “on demand.”

Members of the laboratory at the institute, and the LBMC and LJC are interested in “COMOD” it remains to define the conditions of access which are not in the hands of CBP.

University Joseph Fourier, through its thematic schools on scientific computing, used since 2011 for infrastructure SIDUS work of auditors.

How much ?

8 Neoware thin clients inflated CPU and memory in early 2010 and diverted from their original purpose, we are approaching the 120 machines to CBP using this system.

Some researchers use chemical laboratory via SIDUS COMOD: availability by providing the ability to deploy a full working machine on the job in seconds.

When thematic Les Houches School on scientific computing, SIDUS was served nearly fifty machines simultaneously.

As for the price of the software (question asked last Scipy 2013), all the components that it uses is Open Source, SIDUS is also, as well as all related documentation! SIDUS is under CeCILL.

If you use SIDUS, notify the author and make the promotion!

If you want to deploy SIDUS on your facilities and get help, you can contact the author.

How ?

SIDUS is based on a simple majority and proven components available on most GNU / Linux distributions.

We will now describe how to install our own SIDUS!

Installation

Preparing the System

We need to prepare a bit to accommodate our SIDUS. For this, we hand over several services to deploy our clients: DHCP, TFTP, NFS servers. You maintain a good relationship with your IT department or you are free enough to unconstrained access to LDAP and DNS servers defined:

  • DHCP service provides our client an IP address but diffuse and additional information: the address of the TFTP server using the next-server variable and the name of the binary PXE, often called pxelinux.0.
  • TFTP service when the scene. It offers everything needed by TFTP for booting the system: the binary pxelinux.0, the kernel and start the client system. If we need to provide parameters to a particular customer, Our method a specific document whose name is builds from its MAC address (prefixed with 01 and which : are replaced by -).
  • NFS server then calls in the loop: it will provide the root system by its protocol (so nfsroot). It is in this root, for example /srv/nfsroot/sidus we will install our client system.

Our configurations we use respectively isc-dhcp-server, tftpd-hpa and nfs-kernel-server for DHCP, TFTP and NFS servers.

Look in more detail the configuration of the different services we have include:

In Linux Journal article, next-server is defined as 172.16.20.13, it's better to put it inside LAN: 10.13.20.13 is a better choice.

For the DHCP server in the configuration file /etc/dhcp/dhcpd.conf :

next-server 10.13.20.13;
filename "pxelinux.0";
allow booting;

For the TFTP server in the /srv/tftp folder, we have 3 files and the pxelinux.cfg folder . pxelinux.0 file is included inside the package syslinux-common:

./pxelinux.0 
./vmlinuz-Sidus
./initrd.img-Sidus
./pxelinux.cfg

In the pxelinux.cfg folder, we have the default file. An example of a startup file to the client specifying everything you need to start. It has two inputs, tmpfs and iscsi. We will return later to the second input iscsi:

  • the kernel vmlinuz-Sidus
  • the bootstrap system initrd.img-Sidus
  • the NFSroot server 10.13.20.13 with mountpoint /srv/nfsroot/sidus
DEFAULT tmpfs

LABEL tmpfs
KERNEL vmlinuz-Sidus
APPEND console=tty1 root=/dev/nfs initrd=initrd.img-Sidus nfsroot=10.13.20.13:/srv/nfsroot/sidus,rsize=8192,wsize=8192,tcp ip=dhcp aufs=tmpfs

LABEL iscsi
KERNEL vmlinuz-Sidus
APPEND console=tty1 root=/dev/nfs initrd=initrd.img-Sidus nfsroot=10.13.20.13:/srv/nfsroot/sidus,rsize=8192,wsize=8192,tcp ip=dhcp aufs=iscsi ISCSI_TARGET_IP=10.13.20.13 ISCSI_INITIATOR=iqn.2013-04.zone.sidus.target:default root=LABEL=ISCSI 

For the NFS server, the configuration takes a line in the file / etc / exports. Here we open a read-only access to the machines is between the IP 10.13.20.128 and 10.13.20.254:

/srv/nfsroot/sidus 10.13.20.128/255.255.255.0(ro,no_subtree_check,async,no_root_squash)

Once these three DHCP, TFTP and NFS services configured we can install a complete SIDUS.

Be careful though! It will also require a root containing the accounts of users (NFS version 4) and a process for their identification/authentication (LDAP or Kerberos). We have deployed SIDUS in environments where these services were provided by third party servers as completely autonomous environments. The installation of an OpenLDAP server with SSL or Kerberos server out of this, we will simply specify the client files that we need to be adapted to our site infrastructure configurations here for the LDAP identification/authentication and NFS version 4 users folders.

Deboostrap

Debootstrap For installation of a system as a root. It requires several parameters such as the installation root, the hardware architecture, distribution and archive Debian FTP or HTTP request to download.

Warning: there begins the “specialization” of our facility, originally built around a Debian distribution. This tool is familiar with all Debian-like distributions: it will be available in the system derived from the spiral (starting with Ubuntu). It is however quite easy to do this on-like Redhat, Fedora incorporating such a clone, Febootstrap, but we do not tested.

Debootstrap also accepts as input a list of archives (you know Debian is very fussy about licensing separating archives main, contrib and non-free), a list of packages to be included and a list of packages to exclude. We would have been delighted to here, specify the complete list of packages to include or exclude, but unfortunately, this approach is a dead end, so we will install as soon as this command debootstrap, a set of tools it seems necessary to include early (eg kernel firmware for extended support of all hardware and a set of audit tools).

We have for convenience defined environment variables corresponding to the root of our system $SIDUS and a command for executing a control chroot with a special package installation option. The variable $MyInclude includes the list of desired packages (separated by commas) and $MyExclude list of rejected packets including:

export SIDUS=/srv/nfsroot/sidus
alias sidus="DEBIAN_FRONTEND=noninteractive chroot ${SIDUS} $@"

Here are examples of MyInclude and MyExclude we use

export MyInclude="adduser,apt,apt-utils,aptitude,aptitude-common,aspell,aspell-en,aufs-tools,bsdmainutils,btrfs-tools,busybox,ca-certificates,clusterssh,console-setup,cpio,cron,cups-pdf,debian-archive-keyring,dmidecode,dselect,emacs,environment-modules,ethtool,firmware-bnx2,firmware-linux,firmware-linux-nonfree,gnupg,gpgv,groff-base,htop,hwinfo,hwloc,iftop,ifupdown,info,initramfs-tools,install-info,iotop,iperf,ipmitool,iproute,iptables,iputils-ping,isc-dhcp-client,isc-dhcp-common,kmod,ldap-utils,less,libapt-inst1.5,libapt-pkg4.12,libboost-iostreams1.49.0,libcwidget3,libept1.4.12,libgcrypt11,libgdbm3,libgnutls26,libgpg-error0,libidn11,libkmod2,libncursesw5,libnet-ldap-perl,libnewt0.52,libnfnetlink0,libnss-ldap,libp11-kit0,libpam-ldap,libpipeline1,libpopt0,libprocps0,libreadline6,libsigc++-2.0-0c2a,libsqlite3-0,libssl1.0.0,libstdc++6,libtasn1-3,libudev0,libusb-0.1-4,libxapian22,linux-headers-3.2.0-4-amd64,linux-image-3.2.0-4-amd64,locales,logrotate,lsof,man-db,manpages,mbw,mtr,mutt,nano,net-tools,netbase,netcat-traditional,nfs-common,nscd,ntpdate,open-iscsi,openssh-server,pciutils,procps,python-ldap,readline-common,rsyslog,screen,scsitools,sdparm,ssh,ssmtp,sudo,tasksel,tasksel-data,tmux,traceroute,tshark,udev,usbutils,vim,wget,whiptail,xinit,python-html2text"

export MyExclude="nano,exim,mysql-server,mysql-server-5.5,mysql-server-core-5.5,network-manager,apache2,apache2-mpm-worker,apache2-utils,apache2.2-bin,apache2.2-common,libapache2-mod-dnssd,libapache2-mod-php5,r-cran-fecofin,libmpich1.0gf,gerris,gspiceui,qucs,ktimetrace,kseg,ghdl,earth3d,libopenigtlink1,qtdmm,scilab-overload,gmsh,klogic,g++-doc,openturns-wrapper,xorsa,r-cran-rpvm,labplot,zygrib,libteem1,magnus,libcomplearn-dev,libtorque2,torque-common,torque-server,gridengine-client,gridengine-exec,gridengine-master,gridengine-qmon,gnuplot,gnuplot-nox,rtai,rtai-doc,libhdf5-dev,libhdf5-1.8,libgd2-xpm"

By default, the next command install un Debian Wheezy distribution in /srv/nfsroot/sidus root for an AMD64 architecture from Debian mirror http://ftp.debian.org/debian :

time debootstrap --arch amd64 --components='main,contrib,non-free' --include=$MyInclude --exclude=$MyExclude wheezy $SIDUS http://ftp.debian.org/debian

Following this command, we need to take some precautions:

  • normally, if the Debian package is a service, it starts after installation. We must therefore inhibit the launch of this service by defining a hook:
    printf '#!/bin/sh\nexit 101\n' > ${SIDUS}/usr/sbin/policy-rc.d
    chmod +x ${SIDUS}/usr/sbin/policy-rc.d
  • packages require access to the list of processes, systems, peripherals, virtual memory pointers devices. We must therefore “binder” mount these files from the host system SIDUS system:
    sidus mount -t proc none /proc
    sidus mount -t sysfs sys /sys
    mount --bind /run/shm ${SIDUS}/run/shm
    mount --bind /dev/pts ${SIDUS}/dev/pts

Specific packages installation

In order to simplify the installation of packets belonging to the same family, has created many Debian meta packages, prefixed with “science”: science-chemistry denotes, for example all chemistry software packages. The installation order of all scientific packages is done by a single command. As we are fond of completeness, we will “also” add packages “suggested” (note the option –install-suggests is present only from the Wheezy distribution):

time sidus apt-get install --install-suggests -f -m -y --force-yes science-*

During installation, the longest phases are downloading packages representing several GB and initial configuration of some packages (such as Perl and LaTeX) (depends of Internet connectivity and official mirrors).

In the best case scenario, it takes 45 minutes to a full tree 32GB.

Purge

Unfortunately, this installation bulimia is not without effect. Packages still install some “evil” and a purge of a few, including an installer Matlab!

time sidus apt-get purge -y -f --force-yes matlab-*

Environnement local

The local environment is significant and, by default, nothing is set. The default is the U.S., we parameterize:

  • local environment: ${SIDUS}/etc/locale.gen
  • timezone : ${SIDUS}/etc/timezone
  • keyboard : ${SIDUS}/etc/default/keyboard

LDAP & NFS third party servers

LDAP and NFS services require a small configuration. Import prefined files is better.

In our case, we chose to put them all on one Web server http://MyServeur.MySite/sidus. A wget can directly retrieve and place them in the right place with the -O option. For example, the file nsswitch.conf:

wget -O  ${SIDUS}/etc/nsswitch.conf http://MyServeur.MySIte/sidus/nsswitch.conf

So, here are the files to download:

  • The LDAP authentication:
    • ${SIDUS}/etc/nsswitch.conf,
    • ${SIDUS}/etc/libpam_ldap.conf,
    • ${SIDUS}/etc/libnss-ldap.conf,
    • ${SIDUS}/etc/ldap/ldap.conf
  • The NFS folders for users :
    • for NFSv4 :
      • ${SIDUS}/etc/default/nfs-common,
      • ${SIDUS}/etc/default/idmapd.conf,
      • ${SIDUS}/etc/fstab
    • for NFSv3 :
      • ${SIDUS}/etc/fstab

Startup sequence

How SIDUS sharing without duplicating? We will be inspired by the mechanism used in some LiveCD installation of the root system is the superposition of two layers, one read-only (nfsroot system) and the other read / write (a TMPFS in the case easiest). The two layers are bonded by glue AUFS, the successor project UnionFS.

Everything is in a single “hook” at startup: rootaufs placed early in the initrd boot. It is based on five steps:

  1. creating temporary files /ro, /rw and /aufs
  2. moving NFSroot from initial mount point to /ro,
  3. mounting a remote share, a local or remote partition, or TMPFS volume to another mount point /rw
  4. superposition of the two folders /ro et /rw in /aufs folders
  5. moving /aufs to original mount point

This script, rootaufs, in located in ${SIDUS}/etc/initramfs-tools/scripts/init-bottom

In Linux Journal article, rootaufs as destination file is missing.

The original script was inspired by the project rootaufs from Nicholas A. Schembri (http://code.google.com/p/rootaufs/). He was deeply modified to adapt our infrastructure: a version is available on http://www.cbp.ens-lyon.fr/sidus/rootaufs :

wget -O ${SIDUS}/etc/initramfs-tools/scripts/init-bottom/rootaufs http://www.cbp.ens-lyon.fr/sidus/rootaufs

We have not finished yet to have a functional system we create a special initrd for our NFS boot:

  • aufs in ${SIDUS}/etc/initramfs-tools/modules
  • eth0 as DEVICE in ${SIDUS}/etc/initramfs-tools/initramfs.conf
sidus update-initramfs -k all -u
In Linux Journal article, vmlinux-Sidus is written, not vmlinuz-Sidus as TFTP kernel file.

Then just copy the kernel and boot loader in the definition :

cp ${SIDUS}/boot/vmlinuz-3.2.0-4-amd64 /srv/tftp/vmlinuz-Sidus
cp ${SIDUS}/boot/initrd.img-3.2.0-4-amd64 /srv/tftp/initrd.img-Sidus

Initially, we explored the possibility of offering a second NFS share as read/write persistence associated with a client reboots changes. This version, though functional, demanded the opening of an atomic NFS from each customer with what we imagine as a burden on their server!

We have preferred another approach persistence in the form of a network of iSCSI disk, which we saw in our definition file /srv/ftp/ pxelinux.cfg/default, with the definition of LABEL=iscsi. With this, to persist, we create an iSCSI shared by client. Mount parameters of iSCSI disk is defined in the command line.

For reasons of simplicity, the volume has offered the client IP and we provide only the server iSCSI volume. The login and password to access the default password is in the file rootaufs.

Some tricks

  • for configuring the DHCP name (hostname), remove /etc/hostname
  • set the /etc/resolv.conf with a static definition
  • in /etc/network/interfaces : set a loopback
  • modify GDM3 to start necessary after NSCD and DBUS daemons
  • set /etc/security/limits.conf (essential in a HPC environment)
  • set /etc/fstab with NFS user accounts server

For virtual systems based on VirtualBox

  • Install VBoxLinuxAdditions.run in SIDUS system

For systems with InfiniBand card

  • Force loading modules in /etc/modules and regenerate the initrd
  • Insert in /etc/rc.local a script to retrieve the Ethernet IP address and build an IP address for Infiniband card.

For systems with Nvidia graphics boards

For the majority of Nvidia cards, packages available in Debian Wheezy allow complete installation owners drivers, libraries OpenGL, OpenCL and CUDA. Be careful though if you want to use the ICD (Installable Client Loader) OpenCL AMD processors simultaneously operate your AND your graphics card: we had to install yourselves all the infrastructure Cuda and OpenCL “from scratch”.

For systems with AMD/ATI graphics boards

For the majority of ATI cards, packages available in Debian Wheezy allow a full install proprietary drivers, libraries OpenGL, OpenCL and CUDA.

Removing binding "system"

In order to properly install SIDUS, we were forced to bind host and chroot closely.

Here are the commands to set the system as original one:

umount ${SIDUS}/run/shm
umount ${SIDUS}/dev/pts
sidus umount /proc/sys/fs/binfmt_misc
sidus umount /proc
sidus umount /sys

Deleting temporary folders

rm -r ${SIDUS}/run/* ${SIDUS}/tmp/* 
lsof | grep ${SIDUS}  | awk '{ print $2 }' | sort -u | xargs -I '{}' kill '{}'

Administration

The administration of a SIDUS instance is administered like any system chroot: it however are precautions to take, all the administrative functions that do not require the same constraints from each other.

Update Debian

This is certainly the most common function. The problem of up-to-date is twofold:

  • Packages can replace services starting with /etc/init.d: a sequence is going to try to stop and restart the service
  • Packages require the presence of a /proc mounted

To achieve up-to-date a SIDUS instance installed in ${SIDUS} safely, simply chain the following commands:

  • Before update commands:
    mount --bind /dev/shm ${SIDUS}/dev/shm 
    mount --bind /dev/pts ${SIDUS}/dev/pts 
    chroot ${SIDUS} mount -t proc none /proc
  • Update commands :
    chroot ${SIDUS} aptitude update
    chroot ${SIDUS} aptitude -y full-upgrade
    chroot ${SIDUS} aptitude clean
  • After update commands :
    umount -l /proc
    umount -l ${SIDUS}/dev/shm
    umount -l ${SIDUS}/dev/pts
    lsof | grep ${SIDUS}  | awk '{ print $2 }' | sort -u | xargs -I '{}' kill '{}'  

The last line of the downstream control is a precaution to prevent the process started in the instance running on the host.

Demonstration for Scipy 2013

Next imagesSidus4Scipy2013 and Sidus4Client can judge the effectiveness of Sidus to provide a complete environment on a single workstation.

These two images require both the open source virtualization tool VirtualBox available on Linux platforms, MacOS X, Windows and Solaris.

The SIDUS instance is integrated into the virtual machine sidus4scipy after import of machine Sidus4Scipy. It offers a full Gnome desktop with Python tools presented at Scipy 2013.

The virtual machine sidus4client integrates the minimum virtual machine to start a client using SIDUS instance. After a few seconds, a Gnome desktop opens.

Some elements:

  • Space required to install the complete image is 8GB
  • To import:
    • Ask resetting the Mac address (otherwise, all persons on the same network have the same IP address: communication difficulties are to be expected :)
  • Before starting the virtual machine:
    • Open settings
    • Go into the network settings and check the network interface is eth0 Linux en0 MacOS
    • Go into the settings and create a shared folder myhost
  • It is preferable to have at least 3 GB on the machine creating two virtual machines
  • During startup, the U.S. QWERTY keyboards
  • Usernames and passwords for the root and generic accounts created 4
    • root & Sidus2013
    • alpha & Alpha13
    • beta & Beta13
    • gamma & Gamma13
    • delta & Delta13
    • (see all other Greek letters in a dictionary!)
    • omega & Omega13
Caution! In an effort to internationalize (Scipy 2013)
  • The default keyboard is an American Qwerty
  • The default language is English
  • The time zone of the Austin

It is very easy to iron in a “French” mode for the keyboard and time via commands

  • For SIDUS server
    • dpkg-reconfigure keyboard-configuration
    • dpkg-reconfigure tzdata
  • For customer SIDUS
    • sidus dpkg-reconfigure keyboard-configuration
    • sidus dpkg-reconfigure tzdata

Starting on third party machine

It is also possible to start the instance on another machine by selecting the configuration of the virtual machine network that you intend to use (usually eth0) interface,

  • Either by directly connecting the machine to another via the Ethernet port, a necessary condition is
    • RJ45 port supports autoconfiguration in cross harbor
  • Either by connecting the host computer and the client device on an external switch

In both cases, forwarding must be activated : /proc/sys/net/ipv4/ip_forward set to 1.

Emmanuel Quemener 2013/09/28 23:06

en/developpement/productions/sidus.txt · Dernière modification: 2015/01/07 10:04 (modification externe)