Ceci est une ancienne révision du document !
SIDUS is the contraction of Single Instance Universal Distributing System and purposes to oversimplify the administration of large amount of machines.
Its latin origin of set of stellar bodies is an allegory: we let you find one that suits you best .
SIDUS has two fondamental properties :
SIDUS is not :
Time of administration system of computer equipment (compute node, post or personal workstation, virtual machine testing) increases with the number and diversity. Thus, all these materials essentially share the same files, but each on its own disk. How to limit the installation time and administrative machinery while retaining the flexibility related to their destination?
This is the challenge faced by SIDUS (for Single Instance Universal Distributing System), developed at the Centre Blaise Pascal originally mainly to simplify the task of the system administrator only deal with the management of hundreds of machines of all kinds for all destinations (technical platforms multi-node, multi-core, GPU, etc.).
One approach to network booting and provides a perfectly generic system for all machines.
The first version of SIDUS of February 2010, it was originally on Debian Etch. It followed all the developments of the Debian distribution, essential base (or exclusive) of Blaise Pascal Centre.
Early 2013, SIDUS serves nearly 90 permanent cluster nodes to CBP (up to 120 simultaneous based loans), GPGPU servers, workstations and multi-core positions COMOD (for Compute On My Own Device).
SIDUS is for all those who want to simplify the task and that:
The Centre Blaise Pascal, hotel projects ENS-Lyon in the field of computing and scientific computing, SIDUS uses for all its amenities including uniformity must be preserved as much as possible: a simple reboot should be sufficient to replace the system in its original condition.
The Scientific Centre of Numerical Modeling (PSMN) computing center ENS-Lyon, now uses SIDUS over a hundred knots and prepares its generalization for the establishment of Meso@Equip (nearly 200 additional nodes). On september 2013, the number of nodes working under SIDUS exceeds 320.
The chemistry laboratory uses “COMOD” for some jobs “on demand.”
Members of the laboratory at the institute, and the LBMC and LJC are interested in “COMOD” it remains to define the conditions of access which are not in the hands of CBP.
University Joseph Fourier, through its thematic schools on scientific computing, used since 2011 for infrastructure SIDUS work of auditors.
8 Neoware thin clients inflated CPU and memory in early 2010 and diverted from their original purpose, we are approaching the 120 machines to CBP using this system.
Some researchers use chemical laboratory via SIDUS COMOD: availability by providing the ability to deploy a full working machine on the job in seconds.
When thematic Les Houches School on scientific computing, SIDUS was served nearly fifty machines simultaneously.
As for the price of the software (question asked last Scipy 2013), all the components that it uses is Open Source, SIDUS is also, as well as all related documentation! SIDUS is under CeCILL.
If you use SIDUS, notify the author and make the promotion!
If you want to deploy SIDUS on your facilities and get help, you can contact the author.
SIDUS is based on a simple majority and proven components available on most GNU / Linux distributions.
We will now describe how to install our own SIDUS!
We need to prepare a bit to accommodate our SIDUS. For this, we hand over several services to deploy our clients: DHCP, TFTP, NFS servers. You maintain a good relationship with your IT department or you are free enough to unconstrained access to LDAP and DNS servers defined:
next-server
variable and the name of the binary PXE, often called pxelinux.0
.pxelinux.0
, the kernel and start the client system. If we need to provide parameters to a particular customer, Our method a specific document whose name is builds from its MAC address (prefixed with 01 and which :
are replaced by -
)./srv/nfsroot/sidus
we will install our client system.Our configurations we use respectively isc-dhcp-server, tftpd-hpa and nfs-kernel-server for DHCP, TFTP and NFS servers.
Look in more detail the configuration of the different services we have include:
For the DHCP server in the configuration file /etc/dhcp/dhcpd.conf
:
next-server 10.13.20.13; filename "pxelinux.0"; allow booting;
For the TFTP server in the /srv/tftp
folder, we have 3 files and the pxelinux.cfg
folder . pxelinux.0
file is included inside the package syslinux-common:
./pxelinux.0 ./vmlinuz-Sidus ./initrd.img-Sidus ./pxelinux.cfg
In the pxelinux.cfg
folder, we have the default
file. An example of a startup file to the client specifying everything you need to start. It has two inputs, tmpfs and iscsi. We will return later to the second input iscsi:
vmlinuz-Sidus
initrd.img-Sidus
10.13.20.13
with mountpoint /srv/nfsroot/sidus
DEFAULT tmpfs LABEL tmpfs KERNEL vmlinuz-Sidus APPEND console=tty1 root=/dev/nfs initrd=initrd.img-Sidus nfsroot=10.13.20.13:/srv/nfsroot/sidus,rsize=8192,wsize=8192,tcp ip=dhcp aufs=tmpfs LABEL iscsi KERNEL vmlinuz-Sidus APPEND console=tty1 root=/dev/nfs initrd=initrd.img-Sidus nfsroot=10.13.20.13:/srv/nfsroot/sidus,rsize=8192,wsize=8192,tcp ip=dhcp aufs=iscsi ISCSI_TARGET_IP=10.13.20.13 ISCSI_INITIATOR=iqn.2013-04.zone.sidus.target:default root=LABEL=ISCSI
For the NFS server, the configuration takes a line in the file / etc / exports. Here we open a read-only access to the machines is between the IP 10.13.20.128 and 10.13.20.254:
/srv/nfsroot/sidus 10.13.20.128/255.255.255.0(ro,no_subtree_check,async,no_root_squash)
Once these three DHCP, TFTP and NFS services configured we can install a complete SIDUS.
Be careful though! It will also require a root containing the accounts of users (NFS version 4) and a process for their identification/authentication (LDAP or Kerberos). We have deployed SIDUS in environments where these services were provided by third party servers as completely autonomous environments. The installation of an OpenLDAP server with SSL or Kerberos server out of this, we will simply specify the client files that we need to be adapted to our site infrastructure configurations here for the LDAP identification/authentication and NFS version 4 users folders.
Debootstrap For installation of a system as a root. It requires several parameters such as the installation root, the hardware architecture, distribution and archive Debian FTP or HTTP request to download.
Warning: there begins the “specialization” of our facility, originally built around a Debian distribution. This tool is familiar with all Debian-like distributions: it will be available in the system derived from the spiral (starting with Ubuntu). It is however quite easy to do this on-like Redhat, Fedora incorporating such a clone, Febootstrap, but we do not tested.
Debootstrap also accepts as input a list of archives (you know Debian is very fussy about licensing separating archives main, contrib and non-free), a list of packages to be included and a list of packages to exclude. We would have been delighted to here, specify the complete list of packages to include or exclude, but unfortunately, this approach is a dead end, so we will install as soon as this command debootstrap
, a set of tools it seems necessary to include early (eg kernel firmware for extended support of all hardware and a set of audit tools).
We have for convenience defined environment variables corresponding to the root of our system $SIDUS
and a command for executing a control chroot
with a special package installation option. The variable $MyInclude
includes the list of desired packages (separated by commas) and $MyExclude
list of rejected packets including:
export SIDUS=/srv/nfsroot/sidus alias sidus="DEBIAN_FRONTEND=noninteractive chroot ${SIDUS} $@"
By default, the next command install un Debian Wheezy distribution in /srv/nfsroot/sidus root for an AMD64 architecture from Debian mirror http://ftp.debian.org/debian :
time debootstrap --arch amd64 --components='main,contrib,non-free' --include=$MyInclude --exclude=$MyExclude wheezy $SIDUS http://ftp.debian.org/debian
Following this command, we need to take some precautions:
printf '#!/bin/sh\nexit 101\n' > ${SIDUS}/usr/sbin/policy-rc.d chmod +x ${SIDUS}/usr/sbin/policy-rc.d
sidus mount -t proc none /proc sidus mount -t sysfs sys /sys mount --bind /run/shm ${SIDUS}/run/shm mount --bind /dev/pts ${SIDUS}/dev/pts
In order to simplify the installation of packets belonging to the same family, has created many Debian meta packages, prefixed with “science”: science-chemistry denotes, for example all chemistry software packages. The installation order of all scientific packages is done by a single command. As we are fond of completeness, we will “also” add packages “suggested” (note the option –install-suggests
is present only from the Wheezy distribution):
time sidus apt-get install --install-suggests -f -m -y --force-yes science-*
During installation, the longest phases are downloading packages representing several GB and initial configuration of some packages (such as Perl and LaTeX) (depends of Internet connectivity and official mirrors).
In the best case scenario, it takes 45 minutes to a full tree 32GB.
Unfortunately, this installation bulimia is not without effect. Packages still install some “evil” and a purge of a few, including an installer Matlab, we ruffles hair!
time sidus apt-get purge -y -f --force-yes matlab-*
The local environment is significant and, by default, nothing is set. The default is the U.S., we parameterize:
${SIDUS}/etc/locale.gen
${SIDUS}/etc/timezone
${SIDUS}/etc/default/keyboard
LDAP and NFS services require a small configuration. Import prefined files is better.
In our case, we chose to put them all on one Web server http://MyServeur.MySite/sidus. A wget
can directly retrieve and place them in the right place with the -O
option. For example, the file nsswitch.conf
:
wget -O ${SIDUS}/etc/nsswitch.conf http://MyServeur.MySIte/sidus/nsswitch.conf
So, here are the files to download:
${SIDUS}/etc/nsswitch.conf
, ${SIDUS}/etc/libpam_ldap.conf
, ${SIDUS}/etc/libnss-ldap.conf
, ${SIDUS}/etc/ldap/ldap.conf
${SIDUS}/etc/default/nfs-common
, ${SIDUS}/etc/default/idmapd.conf
, ${SIDUS}/etc/fstab
${SIDUS}/etc/fstab
How SIDUS sharing without duplicating? We will be inspired by the mechanism used in some LiveCD installation of the root system is the superposition of two layers, one read-only (nfsroot system) and the other read / write (a TMPFS in the case easiest). The two layers are bonded by glue AUFS, the successor project UnionFS.
Everything is in a single “hook” at startup: rootaufs placed early in the initrd boot. It is based on five steps:
/ro
, /rw
and /aufs
/ro
,/rw
/ro
et /rw
in /aufs
folders/aufs
to original mount point
This script, rootaufs
, in located in ${SIDUS}/etc/initramfs-tools/scripts/init-bottom
The original script was inspired by the project rootaufs from Nicholas A. Schembri (http://code.google.com/p/rootaufs/). He was deeply modified to adapt our infrastructure: a version is available on http://www.cbp.ens-lyon.fr/sidus/rootaufs :
wget -O ${SIDUS}/etc/initramfs-tools/scripts/init-bottom http://www.cbp.ens-lyon.fr/sidus/rootaufs
We have not finished yet to have a functional system we create a special initrd
for our NFS boot:
aufs
in ${SIDUS}/etc/initramfs-tools/modules
eth0
as DEVICE in ${SIDUS}/etc/initramfs-tools/initramfs.conf
sidus update-initramfs -k all -u
Then just copy the kernel and boot loader in the definition :
cp ${SIDUS}/vmlinuz /srv/tftp/vmlinux-Sidus cp ${SIDUS}/srv/nfsroot/boot/initrd /srv/tftp/initrd-Sidus
Initially, we explored the possibility of offering a second NFS share as read/write persistence associated with a client reboots changes. This version, though functional, demanded the opening of an atomic NFS from each customer with what we imagine as a burden on their server!
We have preferred another approach persistence in the form of a network of iSCSI disk, which we saw in our definition file /srv/ftp/ pxelinux.cfg/default
, with the definition of LABEL=iscsi
. With this, to persist, we create an iSCSI shared by client. Mount parameters of iSCSI disk is defined in the command line.
For reasons of simplicity, the volume has offered the client IP and we provide only the server iSCSI volume. The login and password to access the default password is in the file rootaufs
.
/etc/hostname
/etc/resolv.conf
with a static definition/etc/network/interfaces
: set a loopback/etc/security/limits.conf
(essential in a HPC environment)/etc/fstab
with NFS user accounts server/etc/modules
and regenerate the initrd
/etc/rc.local
a script to retrieve the Ethernet IP address and build an IP address for Infiniband card.For the majority of Nvidia cards, packages available in Debian Wheezy allow complete installation owners drivers, libraries OpenGL, OpenCL and CUDA. Be careful though if you want to use the ICD (Installable Client Loader) OpenCL AMD processors simultaneously operate your AND your graphics card: we had to install yourselves all the infrastructure Cuda and OpenCL “from scratch”.
For the majority of ATI cards, packages available in Debian Wheezy allow a full install proprietary drivers, libraries OpenGL, OpenCL and CUDA.
In order to properly install SIDUS, we were forced to bind host and chroot closely.
Here are the commands to set the system as original one:
umount ${SIDUS}/run/shm umount ${SIDUS}/dev/pts sidus umount /proc/sys/fs/binfmt_misc sidus umount /proc sidus umount /sys
rm -r ${SIDUS}/run/* ${SIDUS}/tmp/*
lsof | grep ${SIDUS} | awk '{ print $2 }' | sort -u | xargs -I '{}' kill '{}'
The administration of a SIDUS instance is administered like any system chroot: it however are precautions to take, all the administrative functions that do not require the same constraints from each other.
This is certainly the most common function. The problem of up-to-date is twofold:
/etc/init.d
: a sequence is going to try to stop and restart the service/proc
mounted
To achieve up-to-date a SIDUS instance installed in ${SIDUS}
safely, simply chain the following commands:
mount --bind /dev/shm ${SIDUS}/dev/shm mount --bind /dev/pts ${SIDUS}/dev/pts chroot ${SIDUS} mount -t proc none /proc
chroot ${SIDUS} aptitude update chroot ${SIDUS} aptitude -y full-upgrade chroot ${SIDUS} aptitude clean
umount -l /proc umount -l ${SIDUS}/dev/shm umount -l ${SIDUS}/dev/pts lsof | grep ${SIDUS} | awk '{ print $2 }' | sort -u | xargs -I '{}' kill '{}'
The last line of the downstream control is a precaution to prevent the process started in the instance running on the host.
Next imagesSidus4Scipy2013 and Sidus4Client can judge the effectiveness of Sidus to provide a complete environment on a single workstation.
These two images require both the open source virtualization tool VirtualBox available on Linux platforms, MacOS X, Windows and Solaris.
The SIDUS instance is integrated into the virtual machine sidus4scipy after import of machine Sidus4Scipy. It offers a full Gnome desktop with Python tools presented at Scipy 2013.
The virtual machine sidus4client integrates the minimum virtual machine to start a client using SIDUS instance. After a few seconds, a Gnome desktop opens.
Some elements:
It is very easy to iron in a “French” mode for the keyboard and time via commands
dpkg-reconfigure keyboard-configuration
dpkg-reconfigure tzdata
sidus dpkg-reconfigure keyboard-configuration
sidus dpkg-reconfigure tzdata
It is also possible to start the instance on another machine by selecting the configuration of the virtual machine network that you intend to use (usually eth0
) interface,
In both cases, forwarding must be activated : /proc/sys/net/ipv4/ip_forward
set to 1
.
— Emmanuel Quemener 2013/09/28 23:06