Ci-dessous, les différences entre deux révisions de la page.
Les deux révisions précédentes Révision précédente Prochaine révision | Révision précédente Prochaine révision Les deux révisions suivantes | ||
en:developpement:productions:sidus [2013/09/28 22:44] equemene |
en:developpement:productions:sidus [2014/11/04 17:22] equemene [Startup sequence] |
||
---|---|---|---|
Ligne 1: | Ligne 1: | ||
====== SIDUS : Single Instance Distributing Universal System ====== | ====== SIDUS : Single Instance Distributing Universal System ====== | ||
- | {{ :developpement:productions:sidus.png?200 |}} | + | <note important>For proper English, please refer to this Linux Journal [[http://www.linuxjournal.com/content/november-2013-issue-linux-journal-system-administration|article]] published in November 2013. It was written by Marianne Corvellec based on Emmanuel Quemener's articles.</note> |
- | <note important>Work in progress</note> | + | {{ :developpement:productions:sidus.png?200 |}} |
- | <note warning>All items in this documentation, code snippets, etc. are part of the license [[http://www.cecill.info/index.fr.html|CeCILL]]. It is therefore necessary to respect the four freedoms of free software to operate RESIDUE in its own infrastructure. If you intend to use in your RESIDUE infrastructure, let her know [[emmanuel.quemener @ ens-lyon.fr | author]], Centre Blaise Pascal or quote it!</note> | + | <note warning>All items in this documentation, code snippets, etc. are part of the license [[http://www.cecill.info/index.fr.html|CeCILL]]. It is therefore necessary to respect the four freedoms of free software to operate SIDUS in its own infrastructure. If you intend to use in your SIDUS infrastructure, let her know [[emmanuel.quemener @ ens-lyon.fr | author]], Centre Blaise Pascal or quote it!</note> |
<note tip>First [[https://conference.scipy.org/scipy2013/presentation_detail.php?id=199|presentation]] with [[http://www.youtube.com/watch?v=J5myH0y_bks|video]] of **SIDUS** on [[http://conference.scipy.org/scipy2013|Scipy 2013]]</note> | <note tip>First [[https://conference.scipy.org/scipy2013/presentation_detail.php?id=199|presentation]] with [[http://www.youtube.com/watch?v=J5myH0y_bks|video]] of **SIDUS** on [[http://conference.scipy.org/scipy2013|Scipy 2013]]</note> | ||
Ligne 17: | Ligne 17: | ||
* **use of local resources** : processors and RAM used are local host one. | * **use of local resources** : processors and RAM used are local host one. | ||
- | **SIDUS** n'est donc : | + | **SIDUS** is not : |
- | * **neither LTSP** for //Linux Terminal Server Project//: LTSP provides simplified management of thin clients by providing a X11 or RDP access to a server: and it supports the entire processing load. Conversely, SIDUS operates entirely (or at the discretion of the user) any machine that hangs there. Only the storage operating system is offset to third machines. | + | * **neither LTSP** for //Linux Terminal Server Project//: LTSP offers simplified management of thin clients by providing a X11 or RDP access to a server: and it supports the entire processing load. Conversely, SIDUS operates entirely (or at the discretion of the user) any machine that hangs there. Only the storage operating system is offset to third machines. |
- | * **neither FAI** for //Fully Automatic Installation//: FAI or Kickstart offer a complete easy installation to minimize or eliminate any action by the administrator. Conversely, SIDUS offers a unique system in a tree incorporating both the base system and all installed applications manually. | + | * **neither FAI** for //Fully Automatic Installation//: FAI and Kickstart offer a complete easy installation to minimize or eliminate any action by the administrator. Conversely, SIDUS offers a unique system in a tree incorporating both the base system and all installed applications manually. |
- | * **neither a network LiveCD**: LiveCD starts a minimalist system, necessarily fixed. It is always possible to create your own LiveCD but it is a fairly expensive operation. With SIDUS, it is possible to install the fly on all its customers instantly software to reconfigure. | + | * **neither a network LiveCD**: LiveCD starts a minimalist system, necessarily fixed. It is always possible to create your own LiveCD but it is a tricky operation. With SIDUS, it is possible to install on the fly for all users new software. |
===== Five Ws & Two Hs ===== | ===== Five Ws & Two Hs ===== | ||
Ligne 28: | Ligne 28: | ||
Time of administration system of computer equipment (compute node, post or personal workstation, virtual machine testing) increases with the number and diversity. Thus, all these materials essentially share the same files, but each on its own disk. How to limit the installation time and administrative machinery while retaining the flexibility related to their destination? | Time of administration system of computer equipment (compute node, post or personal workstation, virtual machine testing) increases with the number and diversity. Thus, all these materials essentially share the same files, but each on its own disk. How to limit the installation time and administrative machinery while retaining the flexibility related to their destination? | ||
- | This is the challenge faced by RESIDUE (for Single Instance Universal Distributing System), developed at the Centre Blaise Pascal originally mainly to simplify the task of the system administrator only deal with the management of hundreds of machines of all kinds for all destinations (technical platforms multi-node, multi-core, GPU, etc.). | + | This is the challenge faced by SIDUS (for Single Instance Universal Distributing System), developed at the Centre Blaise Pascal originally mainly to simplify the task of the system administrator only deal with the management of hundreds of machines of all kinds for all destinations (technical platforms multi-node, multi-core, GPU, etc.). |
===== What ? ===== | ===== What ? ===== | ||
Ligne 77: | Ligne 77: | ||
===== How ? ===== | ===== How ? ===== | ||
- | RESIDUE is based on a simple majority and proven components available on most GNU / Linux distributions. | + | SIDUS is based on a simple majority and proven components available on most GNU / Linux distributions. |
We will now describe how to install our own SIDUS! | We will now describe how to install our own SIDUS! | ||
Ligne 95: | Ligne 95: | ||
Look in more detail the configuration of the different services we have include: | Look in more detail the configuration of the different services we have include: | ||
+ | <note important>In [[http://www.linuxjournal.com/content/november-2013-issue-linux-journal-system-administration|Linux Journal]] article, next-server is defined as ''172.16.20.13'', it's better to put it inside LAN: 10.13.20.13 is a better choice.</note> | ||
For the DHCP server in the configuration file ''/etc/dhcp/dhcpd.conf'' :<code> | For the DHCP server in the configuration file ''/etc/dhcp/dhcpd.conf'' :<code> | ||
next-server 10.13.20.13; | next-server 10.13.20.13; | ||
Ligne 145: | Ligne 146: | ||
export SIDUS=/srv/nfsroot/sidus | export SIDUS=/srv/nfsroot/sidus | ||
alias sidus="DEBIAN_FRONTEND=noninteractive chroot ${SIDUS} $@" | alias sidus="DEBIAN_FRONTEND=noninteractive chroot ${SIDUS} $@" | ||
+ | </code> | ||
+ | |||
+ | Here are examples of ''MyInclude'' and ''MyExclude'' we use | ||
+ | <code> | ||
+ | export MyInclude="adduser,apt,apt-utils,aptitude,aptitude-common,aspell,aspell-en,aufs-tools,bsdmainutils,btrfs-tools,busybox,ca-certificates,clusterssh,console-setup,cpio,cron,cups-pdf,debian-archive-keyring,dmidecode,dselect,emacs,environment-modules,ethtool,firmware-bnx2,firmware-linux,firmware-linux-nonfree,gnupg,gpgv,groff-base,htop,hwinfo,hwloc,iftop,ifupdown,info,initramfs-tools,install-info,iotop,iperf,ipmitool,iproute,iptables,iputils-ping,isc-dhcp-client,isc-dhcp-common,kmod,ldap-utils,less,libapt-inst1.5,libapt-pkg4.12,libboost-iostreams1.49.0,libcwidget3,libept1.4.12,libgcrypt11,libgdbm3,libgnutls26,libgpg-error0,libidn11,libkmod2,libncursesw5,libnet-ldap-perl,libnewt0.52,libnfnetlink0,libnss-ldap,libp11-kit0,libpam-ldap,libpipeline1,libpopt0,libprocps0,libreadline6,libsigc++-2.0-0c2a,libsqlite3-0,libssl1.0.0,libstdc++6,libtasn1-3,libudev0,libusb-0.1-4,libxapian22,linux-headers-3.2.0-4-amd64,linux-image-3.2.0-4-amd64,locales,logrotate,lsof,man-db,manpages,mbw,mtr,mutt,nano,net-tools,netbase,netcat-traditional,nfs-common,nscd,ntpdate,open-iscsi,openssh-server,pciutils,procps,python-ldap,readline-common,rsyslog,screen,scsitools,sdparm,ssh,ssmtp,sudo,tasksel,tasksel-data,tmux,traceroute,tshark,udev,usbutils,vim,wget,whiptail,xinit,python-html2text" | ||
+ | |||
+ | export MyExclude="nano,exim,mysql-server,mysql-server-5.5,mysql-server-core-5.5,network-manager,apache2,apache2-mpm-worker,apache2-utils,apache2.2-bin,apache2.2-common,libapache2-mod-dnssd,libapache2-mod-php5,r-cran-fecofin,libmpich1.0gf,gerris,gspiceui,qucs,ktimetrace,kseg,ghdl,earth3d,libopenigtlink1,qtdmm,scilab-overload,gmsh,klogic,g++-doc,openturns-wrapper,xorsa,r-cran-rpvm,labplot,zygrib,libteem1,magnus,libcomplearn-dev,libtorque2,torque-common,torque-server,gridengine-client,gridengine-exec,gridengine-master,gridengine-qmon,gnuplot,gnuplot-nox,rtai,rtai-doc,libhdf5-dev,libhdf5-1.8,libgd2-xpm" | ||
</code> | </code> | ||
Ligne 175: | Ligne 183: | ||
===== Purge ===== | ===== Purge ===== | ||
- | Unfortunately, this installation bulimia is not without effect. Packages still install some "evil" and a purge of a few, including an installer Matlab, we ruffles hair!<code> | + | Unfortunately, this installation bulimia is not without effect. Packages still install some "evil" and a purge of a few, including an installer Matlab!<code> |
time sidus apt-get purge -y -f --force-yes matlab-*</code> | time sidus apt-get purge -y -f --force-yes matlab-*</code> | ||
Ligne 189: | Ligne 197: | ||
LDAP and NFS services require a small configuration. Import prefined files is better. | LDAP and NFS services require a small configuration. Import prefined files is better. | ||
- | In our case, we chose to put them all on one Web server **http://MyServeur.MySite/sidus. A ''wget'' can directly retrieve and place them in the right place with the ''-O'' option. For example, the file ''nsswitch.conf'':<code> | + | In our case, we chose to put them all on one Web server **http://MyServeur.MySite/sidus**. A ''wget'' can directly retrieve and place them in the right place with the ''-O'' option. For example, the file ''nsswitch.conf'':<code> |
wget -O ${SIDUS}/etc/nsswitch.conf http://MyServeur.MySIte/sidus/nsswitch.conf | wget -O ${SIDUS}/etc/nsswitch.conf http://MyServeur.MySIte/sidus/nsswitch.conf | ||
</code> | </code> | ||
Ligne 221: | Ligne 229: | ||
This script, ''rootaufs'', in located in ''${SIDUS}/etc/initramfs-tools/scripts/init-bottom'' | This script, ''rootaufs'', in located in ''${SIDUS}/etc/initramfs-tools/scripts/init-bottom'' | ||
- | The original script was inspired by the project **rootaufs** from Nicholas A. Schembri (http://code.google.com/p/rootaufs/). He was deeply modified to adapt our infrastructure: a version is available on http://www.cbp.ens-lyon.fr/sidus/rootaufs :<code> | + | <note important>In [[http://www.linuxjournal.com/content/november-2013-issue-linux-journal-system-administration|Linux Journal]] article, ''rootaufs'' as destination file is missing.</note> |
- | wget -O ${SIDUS}/etc/initramfs-tools/scripts/init-bottom http://www.cbp.ens-lyon.fr/sidus/rootaufs | + | The original script was inspired by the project **rootaufs** from Nicholas A. Schembri (http://code.google.com/p/rootaufs/). He was deeply modified to adapt our infrastructure: a version is available on http://www.cbp.ens-lyon.fr/sidus/rootaufs : |
+ | <code> | ||
+ | wget -O ${SIDUS}/etc/initramfs-tools/scripts/init-bottom/rootaufs http://www.cbp.ens-lyon.fr/sidus/rootaufs | ||
</code> | </code> | ||
Ligne 231: | Ligne 241: | ||
<code>sidus update-initramfs -k all -u</code> | <code>sidus update-initramfs -k all -u</code> | ||
- | Then just copy the kernel and boot loader in the definition :<code> | + | <note important>In [[http://www.linuxjournal.com/content/november-2013-issue-linux-journal-system-administration|Linux Journal]] article, ''vmlinux-Sidus'' is written, not ''vmlinuz-Sidus'' as TFTP kernel file.</note> |
- | cp ${SIDUS}/vmlinuz /srv/tftp/vmlinux-Sidus | + | Then just copy the kernel and boot loader in the definition : |
- | cp ${SIDUS}/srv/nfsroot/boot/initrd /srv/tftp/initrd-Sidus | + | <code> |
+ | cp ${SIDUS}/boot/vmlinuz-3.2.0-4-amd64 /srv/tftp/vmlinuz-Sidus | ||
+ | cp ${SIDUS}/boot/initrd.img-3.2.0-4-amd64 /srv/tftp/initrd.img-Sidus | ||
</code> | </code> | ||
Ligne 262: | Ligne 274: | ||
==== For systems with Nvidia graphics boards ==== | ==== For systems with Nvidia graphics boards ==== | ||
- | For the majority of Nvidia cards, packages available in Debian Wheezy allow complete installation owners drivers, libraries OpenGL, OpenCL and CUDA. Be careful though if you want to use the ICD (Installable Client Loader) OpenCL AMD processors simultaneously operate your AND your graphics card: we had to install all the infrastructure pilot Cuda and OpenCL "from scratch" afford. | + | For the majority of Nvidia cards, packages available in Debian Wheezy allow complete installation owners drivers, libraries OpenGL, OpenCL and CUDA. Be careful though if you want to use the ICD (Installable Client Loader) OpenCL AMD processors simultaneously operate your AND your graphics card: we had to install yourselves all the infrastructure Cuda and OpenCL "from scratch". |
==== For systems with AMD/ATI graphics boards ==== | ==== For systems with AMD/ATI graphics boards ==== | ||
Ligne 270: | Ligne 282: | ||
===== Removing binding "system" ===== | ===== Removing binding "system" ===== | ||
- | In order to properly install RESIDUE, we were forced to bind host and chroot closely. | + | In order to properly install SIDUS, we were forced to bind host and chroot closely. |
Here are the commands to set the system as original one: | Here are the commands to set the system as original one: | ||
Ligne 326: | Ligne 338: | ||
====== Demonstration for Scipy 2013 ====== | ====== Demonstration for Scipy 2013 ====== | ||
- | Les images suivantes [[http://www.cbp.ens-lyon.fr/sidus/sidus4scipy2013.ova|Sidus4Scipy2013]] et [[http://www.cbp.ens-lyon.fr/sidus/sidus4client.ova|Sidus4Client]] permettent de juger de l'efficacité de Sidus à offrir un environnement complet sur un poste de travail unique. | + | Next images[[http://www.cbp.ens-lyon.fr/sidus/sidus4scipy2013.ova|Sidus4Scipy2013]] and [[http://www.cbp.ens-lyon.fr/sidus/sidus4client.ova|Sidus4Client]] can judge the effectiveness of Sidus to provide a complete environment on a single workstation. |
- | Ces deux images nécessitent toutes les deux l'outil de virtualisation Open Source [[http://www.virtualbox.org|VirtualBox]] disponibles sur les plates-formes Linux, MacOS X, Windows et Solaris. | + | These two images require both the open source virtualization tool [[http://www.virtualbox.org|VirtualBox]] available on Linux platforms, MacOS X, Windows and Solaris. |
- | L'instance SIDUS est intégrée dans la machine virtuelle **sidus4scipy** issue de l'importation de la machine [[http://www.cbp.ens-lyon.fr/sidus/sidus4scipy2013.ova|Sidus4Scipy]]. Elle offre un bureau Gnome complet avec des outils Python présentés lors de Scipy 2013 | + | The SIDUS instance is integrated into the virtual machine **sidus4scipy** after import of machine [[http://www.cbp.ens-lyon.fr/sidus/sidus4scipy2013.ova|Sidus4Scipy]]. It offers a full Gnome desktop with Python tools presented at Scipy 2013. |
- | La machine virtuelle **sidus4client** intègre la machine virtuelle minimale permettant de démarrer un client utilisant l'instance SIDUS. Après quelques secondes, un bureau Gnome s'ouvre. | + | The virtual machine **sidus4client** integrates the minimum virtual machine to start a client using SIDUS instance. After a few seconds, a Gnome desktop opens. |
- | Quelques éléments : | + | Some elements: |
- | * l'espace nécessaire pour installer l'image complète est de 8 Go | + | * Space required to install the complete image is 8GB |
- | * à l'importation : | + | * To import: |
- | * demander la réinitialisation de l'adresse Mac (dans le cas contraire, toutes les personnes sur le même réseau local auront la même adresse IP : des difficultés de communication sont à prévoir :) | + | * Ask resetting the Mac address (otherwise, all persons on the same network have the same IP address: communication difficulties are to be expected :) |
- | * avant le démarrage de la machine virtuelle : | + | * Before starting the virtual machine: |
- | * ouvrir les paramétrages | + | * Open settings |
- | * aller dans le paramétrage réseau et vérifier l'interface réseau : c'est **eth0** sous Linux, **en0** sous MacOS | + | * Go into the network settings and check the network interface is eth0 ** ** Linux ** ** en0 MacOS |
- | * aller dans le paramétrage de partage et créer un dossier **MyHost** | + | * Go into the settings and create a shared folder myhost ** ** |
- | * il est préférable de disposer d'au moins 3 Go sur la machine créant ces deux machines virtuelles | + | * It is preferable to have at least 3 GB on the machine creating two virtual machines |
- | * lors du démarrage, les claviers sont en QWERTY américains | + | * During startup, the U.S. QWERTY keyboards |
- | * les identifiants et mots de passe du root et de 4 comptes génériques créés | + | * Usernames and passwords for the root and generic accounts created 4 |
- | * **root** et **Sidus2013** | + | * **root** & **Sidus2013** |
- | * **alpha** et **Alpha13** | + | * **alpha** & **Alpha13** |
- | * **beta** et **Beta13** | + | * **beta** & **Beta13** |
- | * **gamma** et **Gamma13** | + | * **gamma** & **Gamma13** |
- | * **delta** et **Delta13** | + | * **delta** & **Delta13** |
- | * (voir pour toutes les autres lettres grecques dans un dico !) | + | * (see all other Greek letters in a dictionary!) |
- | * **omega** et **Omega13** | + | * **omega** & **Omega13** |
- | <note warning>Attention ! Dans un souci d'internationalisation (Scipy 2013) | + | <note warning>Caution! In an effort to internationalize (Scipy 2013) |
- | * le clavier par défaut est un américain Qwerty, | + | * The default keyboard is an American Qwerty |
- | * la langue par défaut est l'anglais | + | * The default language is English |
- | * le fuseau horaire celui de Austin | + | * The time zone of the Austin |
- | Il est très facile de repasser dans un mode "français" pour le clavier et le fuseau via les commandes | + | It is very easy to iron in a "French" mode for the keyboard and time via commands |
- | * Pour le serveur SIDUS | + | * For SIDUS server |
- | * ''dpkg-reconfigure keyboard-configuration'' | + | *''dpkg-reconfigure keyboard-configuration'' |
- | * ''dpkg-reconfigure tzdata'' | + | *''dpkg-reconfigure tzdata'' |
- | * Pour le client SIDUS | + | * For customer SIDUS |
- | * ''sidus dpkg-reconfigure keyboard-configuration'' | + | *''sidus dpkg-reconfigure keyboard-configuration'' |
- | * ''sidus dpkg-reconfigure tzdata'' | + | *''sidus dpkg-reconfigure tzdata'' |
</note> | </note> | ||
- | ===== Démarrage sur une machine tierce ===== | + | ===== Starting on third party machine ===== |
- | Il est aussi possible de démarrer l'instance sur une autre machine en sélectionnant dans la configuration de la machine virtuelle l'interface réseau que vous comptez utiliser (généralement ''eth0''), | + | It is also possible to start the instance on another machine by selecting the configuration of the virtual machine network that you intend to use (usually ''eth0'') interface, |
- | * soit en connectant directement la machine à une autre via le port Ethernet, une condition est nécessaire | + | * Either by directly connecting the machine to another via the Ethernet port, a necessary condition is |
- | * un port RJ45 supportant l'autoconfiguration en port croisé | + | * RJ45 port supports autoconfiguration in cross harbor |
- | * soit en connectant la machine hôte et la machine cliente sur un commutateur externe | + | * Either by connecting the host computer and the client device on an external switch |
- | Dans les deux cas, le //forwarding// doit être activé : ''/proc/sys/net/ipv4/ip_forward'' mis à ''1''. | + | In both cases, //forwarding// must be activated : ''/proc/sys/net/ipv4/ip_forward'' set to ''1''. |
- | --- //[[emmanuel.quemener@ens-lyon.fr|Emmanuel Quemener]] 2013/07/02 18:08// | + | --- //[[emmanuel.quemener@ens-lyon.fr|Emmanuel Quemener]] 2013/09/28 23:06// |