We had a great time here! Even though the weather was cold, our Belgian friends out here were very welcoming. The organization was perfect, the WiFi too, and the conferences very enlightening. We had the chance to meet people from very different projects, like Jorge Salamero Sanz who was speaking about Samba 4 and SSO integration, Jan-Piet Mens speaking about the modern DNS challenges, Garrett Honeycutt from PuppetLabs (which leaded a great practical initiation to Puppet) and the surprising Evgeny and his “Database is enough” presentation.

Jonathan CLARKE, CTO at Normation, was present to introduce Rudder, a Configuration Management software aimed towards simple usage and extensibility.

We also met our local friends here to share a small sandwich and a not so small beer: François BAYART / A-Kaser and Fabrice FLORE-THEBAULT / themr0c.

This 2012 edition of the LOADays was a great success, so we are eagerly waiting for the 2013 one !

Infrastructures IT : Concilier best practices et contraintes du quotidien

Disponibilité, prévention des risques, sécurité, industrialisation, gestion des changements, partage des connaissances … autant de sujets clés de la gestion de services informatiques qu’il est souvent difficile de concilier avec les contraintes technologiques, budgétaires et décisionnelles.

Nous avons eu l’honneur de compter parmi nos intervenants Mark Burgess, précurseur dans la gestion de configuration logicielle, et CTO fondateur de la société CFEngine AS, leader mondial du domaine, et Enrico Bigaignon, Satellites Control Division, Eutelsat qui a pu nous faire partager son retour d’expérience en matière de mise en place de solutions de gestion de configuration.

Vous retrouverez ci-après le contenu des présentations (hormis celle de M. Bigaignon que nous ne pouvons pas diffuser pour raisons de confidentialité).

Mark Burgess a d’abord présenté l’état de l’art du domaine (téléchargement de la présentation) :

Puis, la société CFEngine et son offre ont été introduits (téléchargement de la présentation) :

Ensuite, Jonathan Clarke a introduit le positionnement de Normation et les objectifs de la solution Rudder (téléchargement de la présentation) :

Notre prochaine matinale sera organisée le jeudi 7 juin, sur le thème La gestion de configuration dans les infrastructures IT, un sujet souvent esquivé : les vrais bénéfices d’une best practice entâchée de scepticisme. Pour ne pas la rater, suivez ce blog ou abonnez-vous à notre newsletter (un envoi tous les deux mois). Rendez-vous très prochainement !

Chez Normation, nous utilisons Scala depuis plus de 3 ans pour notre plus grand plaisir – et quelques irritations.

Quelques irritations donc, le plus souvent liées à l’immaturité de l’ecosystème, avec en premier lieu un outillage au mieux fonctionnel… Oh qu’il aura fallu attendre longtemps pour avoir un plugin Eclipse ou IntelliJ correct ! Et je ne parle pas de quelque chose d’aussi poussé que ce que permettent les IDE Java, simplement un éditeur qui ne plante pas, souligne les erreurs (celles qui en sont réellement), et qui permet de faire la complétion correctement… L’autre point important de frustration est la lenteur du compilateur scalac. Venant de Java ou de langages typés dynamiquement, on comprend mieux ce que signifie ce xkcd…

Mais ces irritations ont été très largement contrebalancées par tout un tas d’avantages. Si je ne devais en retenir seulement trois, mes préférences personnelles iraient à:

• En premier lieu (et c’est tout de même primordial lorsque l’on passe 10h par jour à coder), Scala m’a permis de retrouver la joie de coder. Alors qu’avant, nous ragions sans cesse sur Java, ce langage qui n’arrête pas de se mettre entre notre pensée et le code produit, nous avons découvert un langage qui nous permettait de simplement écrire les algorithmes tels qu’on les imagine. L’API collection ultra-complète est déterminante sur ce point, et elle est tout simplement fantastique.
• Scala nous a permis de pouvoir nous lancer dans des refactorings massifs sans appréhension, en sachant qu’une fois qu’il sera terminé, tout marchera (presque . C’est un peu comme passer de SVN à Git, et de découvrir la joie de faire des branches sans craindre le merge à venir. Ici, le facteur clé est le système de type de Scala, qui permet de définir simplement des propriétés fortes sur les données (“non, l’ID d’une personne n’est pas une chaîne de caractères, c’est un ID”) que le compilateur va vérifier pour nous, de manière beaucoup plus exhaustive et systématique que ne le feraient des tests.
• Enfin, faire du Scala, c’est avoir la chance de travailler dans un écosystème naissant, bouillonnant de bonnes idées et de personnes brillantes, capable de trouver des solutions nouvelles et souvent intéressantes à nos problèmes. Cet esprit attire clairement des profils d’un excellent niveau et a aujourd’hui un impact direct au niveau RH : on peut facilement rencontrer des candidats motivés et très bons, avec finalement peu de cas où le candidat ne mérite pas le poste…

… est-il applicable dans toutes les entreprises ?

Évidemment, lorsque nous avons choisi cette technologie, nous étions une exception: une start-up sans bagage technique, sans dizaines de niveaux hiérarchiques à convaincre, sans équipes à former…

Aussi, à la veille des ScalaDays 2012, avec un buzz de plus en plus pressant autour de ce langage, on peut se demander si choisir Scala aujourd’hui en entreprise, c’est vraiment raisonnable ? Ou est-ce toujours une lubie de start-up ?

J’ai eu l’occasion d’intervenir en novembre dernier dans le cadre d’une réflexion pour le choix d’une pile technologique pour le nouveau produit d’une entreprise. Voici les slides présentés à cette occasion[1]:

Et pour ceux qui se posent la question… Non, Scala n’a pas été retenu dans l’entreprise en question, car il ne convenait pas à la volonté stratégique d’externaliser une partie importante du développement sans avoir de leader Scala en interne.

[1] légèrement revus (graphique indeed.com mis à jour, nouvelles références) et anonymisés

We took the opportunity to introduce Normation, presenting our mission, our strategic partnership with CFEngine AS, and the aims of Rudder. For those who weren’t there, we thought we’d share an extract of this presentation:

It was a great event, with many interesting questions and discussions, and we’d like to thank everyone who attended, and of course Mark for coming all the way to Paris for this, and his warm comments about Normation.

If you’re curious about Normation or Rudder, we’d love to hear from you – contact us by email, Twitter (@Normation and @RudderProject) or IRC (#rudder on FreeNode)!

Today, nearly every modern computing-related company uses directly or indirectly a virtualization software. As seen with the recent Cloud computing trend, it is a sector that recently exploded in popularity and it is now accessible to everyone.

There are now a LOT of alternatives to virtualize an OS :

• Isolation : OpenVZ, VServer, LXC, UML, BSD jails, Solaris zones…
• Paravirtualization : Xen
• Full virtualization : VMWare Player/Workstation/ESX/ESXi, KVM/Qemu, Virtualbox, Bochs…

When having to deal with a large pack of machines for building packages, testing software on multiple OSes or optimally use a powerful server with lots of RAM, you rapidly should, if not have to, realize that Virtual Machines can make your life significantly easier. It was the case for us.

Here at Normation, we love free software (as in free speech, but a free beer is fine too, thanks), so we decided to use something that was FOSS, multi OS compatible (ruling out all isolation softwares), and convenient to use. We chose to use the libvirt/KVM and LVM mix. (ESX/VirtualBox users, do not leave yet. You will be much pleased to learn that libvirt is compatible with your software too ! You will loose the graphical virt-manager, but virsh might come in handy for you if you have scripting needs by example)

Overview

First, let’s deepen a bit the details on these two softs :

• KVM : KVM is a port of the Qemu fullvirt software to the standardized Linux virtualization layer. It consists of two parts : the qemu/kvm binary and the KVM kernel module that handles every CPU/Memory/device operation depending of the CPU support.
• libvirt : libvirt is a RedHat sponsored project that tries to be a “meta” hypervisor, offering a standardized set of instructions to manage VMs independently of the underlying virtualization software. It currently supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox and VMware ESX.
• LVM : The Linux Logical Volume Manager, which enables a much more flexible storage volume management compared to regular MBR partitions, and permits to create “pools” of volumes if needed.

Solution overview

Now let’s see how to create a powerful hypervisor in no time !

Installation of the solution

Shopping list

To install this solution, you will need the following prerequisites :

• A debian distribution, preferably squeeze (or wheezy/testing if you want the latest features) has been used in this post. However, any Linux distribution that bundles libvirt and KVM will do !
• A free partition on the disk, with lots of space (As much as what your VMs will eat, 40GB is a bare minimum)
• A PC that supports the hardware virtualization CPU instructions (AMD SVM or Intel VT-X, this isn’t mandatory but you WILL have performances issues if you do not have it), a minimum of 2GB of RAM, and a fast disk drive. (You might want to have a hardware RAID on a production server, or anything having decent I/O performances)
• A basic comprehension of how GNU/Linux and a PC works

Installation of the packages

You only have to fire up this one-liner, and you will have every tool at hand:

• Debian users:

aptitude update
aptitude install libvirt-bin kvm qemu lvm

• RedHat users:

yum install libvirt qemu-kvm lvm2

• SuSE/OpenSuSE users:

zypper refresh
zypper install libvirt kvm qemu lvm2

I assume that your hypervisor is also your client (the one that will manage your VM), and that you have a X server/a Desktop environment already installed. If not, or if you just want to use the CLI tools, do not try to install this on the server. Note that you can also install this on a remote machine and access the hypervisor using an SSH tunnel (The applications creates it automatically if needed. There are other ways but I will let you the pleasure to read the manual if you are interested)

aptitude install virt-manager
# or
yum install virt-manager
# or
zypper install virt-manager

Virt-Manager

LVM pool creation

This part is optional, you can also use plain files to store the VMs data, but please bear in mind that I personally had some real performance issues with some hardware/filesystem combinations, which were resolved using this. Also, we seek to create a production-ready environment so you really should do it.

Remember when we talked about a big free partition on the drive ? Now let’s use it for real. Here I assume it is /dev/sda4, but your mileage may vary.

fdisk /dev/sda
# Type t, 4, and 8e to switch the partition type to Linux LVM. I will help the kernel and LVM tools to locate the partition and use it.
pvcreate /dev/sda4
vgcreate (vgname) /dev/sda4 # Change to a meaningful name for you

Now we have a (vgname) LVM Volume Group ready to store all our VMs.

Virt manager usage

At this point, the hypervisor machine is ready. Now we just need to set up in libvirt the storage pools for the VMs and the network parameters. You can use virt manager or virsh (the libvirt shell) to do this.

First, you need to add whichever user who will have access to the system to the libvirt system group, to grant them access to the libvirt control socket.

for i in user1 user2 user3
do addgroup $i libvirt
done

Virsh uses XML templates to do that, like this one for the network:

<network>
<name>default</name>
<uuid>65f40d0d-219a-2dd9-a6ac-3a612c26e36b</uuid>
<forward mode='nat'/>
<bridge name='virbr0' stp='on' delay='0' />
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
</dhcp>
</ip>
</network>

Please note that there is no clean and user friendly wizard to do this kind of operation using virsh, you either must have a template at hand and edit it or use the virt manager GUI which is way more user friendly.

Virt Manager is a Graphical User Interface to the libvirt software. It enables the user to get back his clean interface to see which VMs are launched, edit them or create them at will, much like Virtualbox or VMWare Player.

Start it from your desktop menu, and it will at first try to connect to a local hypervisor. If you have one, bingo, no more steps to connect are required, you are in, however, if your hypervisor is a remote machine, you have to click File -> Add a connection and use the QEMU/KVM hypervisor with a SSH remote connection method.

Now that you are connected, right click the name of your connection (like “localhost (QEMU)”) and choose “Details”: This is where you can tune some of libvirt’s access to the machine.

Network parameters

Here you can tune how libvirt will set up connections to the outside world or create internal networks. If a simple NAT is all you need, nothing more to do here than enable the “default” network to start on boot. You might want to create an internal network or a second outside one, if so click the “plus” sign below the network names and follow the guide.

Storage parameters

Here you will have to create storage pools to store your VMs. A default pool has already be created for those who want to use plain files, and stores them in /var/lib/libvirt/images. You will probably need two more pools : A LVM one and somewhere to store your OS installation ISOs if you use a remote hypervisor.

Here the same routine applies: Click the “plus” sign and follow the guide, and do not forget to select an appropriate pool type.

As of my installation, I like having my ISOs in /home/ISO, so if you want to do this, here is the procedure:

mkdir -p /home/ISO
chown -R libvirt-qemu:libvirt /home/ISO

and add this directory as a pool. Please note that nothing forbids you to use the VMs Volume Group to create a logical volume to store your ISOs and mount it on /home/ISO !

lvcreate --name ISO --size 16G (VGName)
mke2fs -j /dev/(VGName)/ISO
# And append to your fstab the following entry :
/dev/(VGName)/ISO /home/ISO ext3 defaults 0 2

Now we have everything set up and ready for use !

Everyday usage

Useful procedures

You will likely create VMs using virt manager’s interface, and there is a quick and easy wizard to do it: you just have to click the first icon in the main interface.

To backup a VM, you have two options :

• Use a dedicated software inside the VM, like backup-manager or Bacula/Amanda and store the backups using FTP/NFS/…
• Backup the VM raw data directly

The first solution is much more economic in terms of used space, but if you have lots of backup-dedicated space you can use the second solution, which offers a easier and quicker recovery alternative. Remember that you can not backup a VM raw data while it is running of you WILL have serious data corruption in your backup. You can use dd for that, or virt-clone.

Note that if you chose to store your VMs using plain QCOW2 or RAW images, it is easier: a simple copy and it is all done.

Which brings me to the LVM method first advantage, after the performance : snapshots !

LVM magic

You can use snapshots to reduce your disk memory usage, using a single base image for several VMs, or have a minimal downtime during backups : Stop the VM, snapshot, immediately backup and restart it.

Simply put, here is the magic command to create a snapshot :

lvcreate --snapshot /dev/(VGName)/(VMName) --name (SnapshotName) --size (VM Logival Volume Size)
# After you have backuped the snapshot (if you do not consider the snapshot to be a backup itself), delete it.
lvremove /dev/(VGName)/(SnapshotName)

Further than that, LVM enabled us to make a simple failover configuration and get a quick and efficient fault tolerance, using DRBD, every VM is replicated over the network in a raid1-like fashion between two hypervisors and in case the primary machine went to fail, we only need to switch DRBD from secondary to primary on the secondary host and reboot the VM’s and here, incident bypassed !

Windows machine emulated in KVM

Usage tips

Here is a list of things that may prove useful to you then :

• IMPORTANT: If you boot a Linux VM, you might want to add “elevator=noop” to your Linux boot command line to force the disk scheduler to let the host machine handle the disk writes reorganisations (like tunnelling tcp over tcp, it is bad to have two schedulers trying to do each other’s job). For example, on GRUB2 on Debian, you have to append “elevator=noop” after GRUB_CMDLINE_LINUX in /etc/default/grub.
• IMPORTANT: Remember that the remote display will always be slower than a real dedicated connection directly to the VM. In other terms, you will be much more comfortable using an SSH / NX / RDP client instead of the console, if you have this choice.
• Do not forget to set the networks to “autostart”, or you will get an error after a reboot if you try to launch a VM
• The connection to the VMs screen is done via VNC, and sometimes if you close one VM window and try to reopen it, the VNC client will get stuck trying to connect. If it happens, restart virt manager
• If you use a recent version of libvirt and virt manager (like the ones in Debian testing, I did not test squeeze ones), you might prefer to switch your VM display to the SPICE protocol, much more efficient than VNC but younger. I had great performance and nearly no bug with it, so feel free to test
• Do not expect hype features like 3D acceleration yet. As of now, libvirt is primarily aimed towards data serving machines accessed using SSH/NX/RDP. However, a lot of advance has been made recently with SPICE and the QXL video emulation. Please see here for more details.
• You have the choice to install your OS using either your host CD-Rom drive, or ISO images. I advise you to prefer ISO images, they are much more faster to install.
• If you do not need sound devices, delete them from the VM configuration. No real reason for that, it is just a bit cleaner. LibVirt will not try to add one on a remote hypervisor though.

Mais qu’est-ce qu’un barcamp ? D’après Wikipedia :

Un BarCamp est une rencontre, une non-conférence ouverte qui prend la forme d’ateliers-événements participatifs où le contenu est fourni par les participants qui doivent tous, à un titre ou à un autre, apporter quelque chose au Barcamp. C’est le principe pas de spectateur, tous participants.

(Source et plus d’infos : http://fr.wikipedia.org/wiki/BarCamp)

L’objectif de cette semaine c’est d’améliorer Rudder, de faire qu’il nous plaît à tous, avant la sortie de la version 2.4.0 début mars, et travailler sur les sujets qu’on néglige parfois : communication, communauté, documentation, tests, qualité, ergonomie…

On profite de l’occasion pour faire une opération portes ouvertes : vous êtes les bienvenus chez nous tous les jours de 9h jusqu’à tard… Passez-nous faire un petit coucou, tester la dernière version de Rudder, nous donner votre avis, grignoter un morceau… Promis, on ne vous demandera pas vraiment de travailler

L’équipe de Normation au grand complet (9 personnes) vous donne donc rendez-vous tous les jours au 87 rue de Turbigo, 75003 Paris ! C’est au métro Temple sur la ligne 3 ou République sur les lignes 5, 8, 9 ou 11.

En particulier, un évènement communautaire autour de Rudder / CFEngine aura lieu jeudi soir pour se rencontrer et discuter (ou troller) autour d’un verre (ou deux).

Ca nous ferait très plaisir de vous voir ici alors n’hésitez pas à passer ! Si vous ne pouvez pas venir, rejoignez-nous sur IRC (#rudder sur FreeNode) ou suivez-nous sur Twitter (@normation et @RudderProject).

As always, FOSDEM was a great event, with thousands of open source users, contributors and developers all mixing together and sharing their findings, software and thoughts. A real spirit of collaboration and possibility always reigns at FOSDEM, and this year was no exception.

For our part, we introduced Rudder, our open source configuration management tool in the Configuration Management devroom on the Sunday (see the official talk page). Our talk was well received, with many interesting questions. Thanks to those who were there, and those who made it possible!

Our slides are available on SlideShare, or you can just view them here:

We also did our share of collaborating, and plan-making. Developers from three of the main components of Rudder, CFEngine, FusionInventory and OpenLDAP, were at FOSDEM, and we got together to discuss each project, their integration into Rudder and ways to improve both our use of them and the community’s experience of them.

Some promising ideas were thrown around, hopefully you’ll hear more about them here soon!

This year, four of us from Normation will be attending, all the way through, from the world-famous, record-breaking beer event to the last talks on the Sunday.

We’ll be introducing Rudder, our open source configuration management tool in the Configuration Management devroom on the Sunday. Our talk will show how Rudder’s approach enables everyone in the IT department to benefit from the advantages of configuration management, without necessarily needing to learn a complex tool, or even get their hands dirty. We’ll describe and demonstrate how this is possible, and dive into the technical architecture that makes it work. Read more on the official talk page.

We’re always interested in meeting new people and discussing configuration management, tools like CFEngine, open source… or even LDAP and Scala, for the more adventurous

Come and have a chat, we’ll be easily recognisable in our flashy black t-shirts with the Rudder logo!

Just before FOSDEM, we’re giving a CFEngine 3 training session. Some seats are still available, so why not come and kill two birds with one stone – some professional training at the end of the week, and a good open source event at the weekend!

Looking forward to seeing you in Brussels!

Going to Brussels for FOSDEM already and could spare a few days before hand to learn a new skill? Or just need a good reason to go to Brussels and then stay on for FOSDEM?

Whatever your interest, you may want to attend our CFEngine training session in Brussels on February 1,2 and 3 (that’s Wednesday-Friday before FOSDEM)!

Flexible course programme

These 3 days can be split up as follows, so that you can build your own training content, depending on your expectations and requirements:

• Day 1 morning: What is configuration management? CFEngine architecture and the “desired state” approach
• Day 1 afternoon and Day 2: CFEngine Fundamentals: hands-on training covering basic CFEngine installation, syntax and features through examples. At the end of the course, attendees will have an understanding and recommended best practices for designing a CFEngine-based configuration management system.
• Day 3: CFEngine Industrialization: Moving on from elementary examples to real-world requirements, including complex parameters, reporting, interaction with other tools and planning changes.

Except the first morning, all of this course is very hands-on, and attendees will be expected to bring a laptop so that they can get CFEngine up and running in their own environment. All courses are given by a trainer who is a recognized CFEngine expert, community contributor and has received training certification from CFEngine AS. The course language is English.

Interesting in participating?

To sign up visit http://cfengine-training-brussels-february-2012-eorg.eventbrite.com/ (payment by credit card or PayPal accepted), or email commercial AT normation.com to get an official quote or submit a purchase order.

Any questions?

Just ask us by email (contact AT normation.com), Twitter (@normation) or IRC (#normation on FreeNode).

Looking forward to seeing you in Brussels!

CFEngine 3 is a very secure tool, that relies on keys to identify hosts and authorize connections. To set up a secure CFEngine infrastructure, you ought to exchange keys between hosts (note that if you don’t have confidential data on your promises, you can skip the security offered by the public key system).

Key exchange

The client-server communication is based on keys (very much like ssl). Each node must have a copy of the other public keys, obtained either via a remote copy (with scp), by trusting keys automatically, or using the bootstrap system introduced in Cfengine 3.2.0 (that would still require the server to accept automatically keys).

If you don’t want to accept keys based on automatic trust, you need to manually exchange them to set up the system. This article will show you how to do this using the cf-runagent tool, by connecting to each host from the main CFEngine server.

Assume a server with the following ip : 192.168.56.150
Assume a client with the following ip : 192.168.56.152

Client side

The initial promises on the client will need to configure the cf-serverd to run on it, and try to fetch its real promises from the policy server :

Note that the client trusts keys from the server

Server side

On the server, you’d have the following promise for the server component, authorizing hosts to connect, but not trusting their keys

Note: only the server itself is accepted on trust, hosts from the subnetwork 192.168.56.* can connect but their keys are not automatically trusted, and their DNS name is not deemed as reliable.

First execution (without prior key exchange)

If the agent on the client tries to connect to the server, without previous key exchange, you’ll have the following output on the server :

To relieve the burden of copying keys with scp, or the risk of trusking keys of every hosts, you can do a manual key exchange, from the server, using the cf-runagent interactive mode

Interactive key exchange

The interactive key exchange will happen from the server to the client. It will ask for the user to trust the key, and execute the remote promises of the client (in this case, the initial promises fetch the real promises from the server)

Here we accepted the key of 192.168.56.152 on the server, and the client could connect to download its new promises, and then apply them

Note: To prevent any risks, the new promises should not have 192.168.56.150 in the trustkeyfrom

Note 2: Since the version 3.2.0, if you are willing to automatically accept keys from the clients on the servers, you don’t need to copy any promises on the client, the bootstrap procedure from the Nova edition has been backported in the community edition; and it uses its own embedded promises