Mais qu’est-ce qu’un barcamp ? D’après Wikipedia :

Un BarCamp est une rencontre, une non-conférence ouverte qui prend la forme d’ateliers-événements participatifs où le contenu est fourni par les participants qui doivent tous, à un titre ou à un autre, apporter quelque chose au Barcamp. C’est le principe pas de spectateur, tous participants.

(Source et plus d’infos : http://fr.wikipedia.org/wiki/BarCamp)

L’objectif de cette semaine c’est d’améliorer Rudder, de faire qu’il nous plaît à tous, avant la sortie de la version 2.4.0 début mars, et travailler sur les sujets qu’on néglige parfois : communication, communauté, documentation, tests, qualité, ergonomie…

On profite de l’occasion pour faire une opération portes ouvertes : vous êtes les bienvenus chez nous tous les jours de 9h jusqu’à tard… Passez-nous faire un petit coucou, tester la dernière version de Rudder, nous donner votre avis, grignoter un morceau… Promis, on ne vous demandera pas vraiment de travailler

L’équipe de Normation au grand complet (9 personnes) vous donne donc rendez-vous tous les jours au 87 rue de Turbigo, 75003 Paris ! C’est au métro Temple sur la ligne 3 ou République sur les lignes 5, 8, 9 ou 11.

En particulier, un évènement communautaire autour de Rudder / CFEngine aura lieu jeudi soir pour se rencontrer et discuter (ou troller) autour d’un verre (ou deux).

Ca nous ferait très plaisir de vous voir ici alors n’hésitez pas à passer ! Si vous ne pouvez pas venir, rejoignez-nous sur IRC (#rudder sur FreeNode) ou suivez-nous sur Twitter (@normation et @RudderProject).

As always, FOSDEM was a great event, with thousands of open source users, contributors and developers all mixing together and sharing their findings, software and thoughts. A real spirit of collaboration and possibility always reigns at FOSDEM, and this year was no exception.

For our part, we introduced Rudder, our open source configuration management tool in the Configuration Management devroom on the Sunday (see the official talk page). Our talk was well received, with many interesting questions. Thanks to those who were there, and those who made it possible!

Our slides are available on SlideShare, or you can just view them here:

We also did our share of collaborating, and plan-making. Developers from three of the main components of Rudder, CFEngine, FusionInventory and OpenLDAP, were at FOSDEM, and we got together to discuss each project, their integration into Rudder and ways to improve both our use of them and the community’s experience of them.

Some promising ideas were thrown around, hopefully you’ll hear more about them here soon!

This year, four of us from Normation will be attending, all the way through, from the world-famous, record-breaking beer event to the last talks on the Sunday.

We’ll be introducing Rudder, our open source configuration management tool in the Configuration Management devroom on the Sunday. Our talk will show how Rudder’s approach enables everyone in the IT department to benefit from the advantages of configuration management, without necessarily needing to learn a complex tool, or even get their hands dirty. We’ll describe and demonstrate how this is possible, and dive into the technical architecture that makes it work. Read more on the official talk page.

We’re always interested in meeting new people and discussing configuration management, tools like CFEngine, open source… or even LDAP and Scala, for the more adventurous

Come and have a chat, we’ll be easily recognisable in our flashy black t-shirts with the Rudder logo!

Just before FOSDEM, we’re giving a CFEngine 3 training session. Some seats are still available, so why not come and kill two birds with one stone – some professional training at the end of the week, and a good open source event at the weekend!

Looking forward to seeing you in Brussels!

Going to Brussels for FOSDEM already and could spare a few days before hand to learn a new skill? Or just need a good reason to go to Brussels and then stay on for FOSDEM?

Whatever your interest, you may want to attend our CFEngine training session in Brussels on February 1,2 and 3 (that’s Wednesday-Friday before FOSDEM)!

Flexible course programme

These 3 days can be split up as follows, so that you can build your own training content, depending on your expectations and requirements:

• Day 1 morning: What is configuration management? CFEngine architecture and the “desired state” approach
• Day 1 afternoon and Day 2: CFEngine Fundamentals: hands-on training covering basic CFEngine installation, syntax and features through examples. At the end of the course, attendees will have an understanding and recommended best practices for designing a CFEngine-based configuration management system.
• Day 3: CFEngine Industrialization: Moving on from elementary examples to real-world requirements, including complex parameters, reporting, interaction with other tools and planning changes.

Except the first morning, all of this course is very hands-on, and attendees will be expected to bring a laptop so that they can get CFEngine up and running in their own environment. All courses are given by a trainer who is a recognized CFEngine expert, community contributor and has received training certification from CFEngine AS. The course language is English.

Interesting in participating?

To sign up visit http://cfengine-training-brussels-february-2012-eorg.eventbrite.com/ (payment by credit card or PayPal accepted), or email commercial AT normation.com to get an official quote or submit a purchase order.

Any questions?

Just ask us by email (contact AT normation.com), Twitter (@normation) or IRC (#normation on FreeNode).

Looking forward to seeing you in Brussels!

CFEngine 3 is a very secure tool, that relies on keys to identify hosts and authorize connections. To set up a secure CFEngine infrastructure, you ought to exchange keys between hosts (note that if you don’t have confidential data on your promises, you can skip the security offered by the public key system).

Key exchange

The client-server communication is based on keys (very much like ssl). Each node must have a copy of the other public keys, obtained either via a remote copy (with scp), by trusting keys automatically, or using the bootstrap system introduced in Cfengine 3.2.0 (that would still require the server to accept automatically keys).

If you don’t want to accept keys based on automatic trust, you need to manually exchange them to set up the system. This article will show you how to do this using the cf-runagent tool, by connecting to each host from the main CFEngine server.

Assume a server with the following ip : 192.168.56.150
Assume a client with the following ip : 192.168.56.152

Client side

The initial promises on the client will need to configure the cf-serverd to run on it, and try to fetch its real promises from the policy server :

Note that the client trusts keys from the server

Server side

On the server, you’d have the following promise for the server component, authorizing hosts to connect, but not trusting their keys

Note: only the server itself is accepted on trust, hosts from the subnetwork 192.168.56.* can connect but their keys are not automatically trusted, and their DNS name is not deemed as reliable.

First execution (without prior key exchange)

If the agent on the client tries to connect to the server, without previous key exchange, you’ll have the following output on the server :

To relieve the burden of copying keys with scp, or the risk of trusking keys of every hosts, you can do a manual key exchange, from the server, using the cf-runagent interactive mode

Interactive key exchange

The interactive key exchange will happen from the server to the client. It will ask for the user to trust the key, and execute the remote promises of the client (in this case, the initial promises fetch the real promises from the server)

Here we accepted the key of 192.168.56.152 on the server, and the client could connect to download its new promises, and then apply them

Note: To prevent any risks, the new promises should not have 192.168.56.150 in the trustkeyfrom

Note 2: Since the version 3.2.0, if you are willing to automatically accept keys from the clients on the servers, you don’t need to copy any promises on the client, the bootstrap procedure from the Nova edition has been backported in the community edition; and it uses its own embedded promises

You are kindly invited to take a glance at the complete presentation hosted here if you want to learn more about this exciting part of the Debian systems.

Your first Debian package

First, you will need to pull all the required dependencies required to build any package – please ensure that you have some deb-src entries in your sources.list (using Synaptic or your good old fashioned CLI editor) or else you will get a “E: You must put some ‘source’ URIs in your sources.list” error.

apt-get update
apt-get install build-essential devscripts debhelper

Then we will try to build a simple package, with minimal dependencies :

apt-get build-dep dash
apt-get source dash
cd dash-(version)
debuild -us -uc (No package signing)

After a while, you hopefully get your fresh debian package(s) in the upper directory :

# ls -l ../*.deb
ash_0.5.5.1-7ubuntu1_all.deb
dash_0.5.5.1-7ubuntu1_i386.deb

Now, here are some few tips about the debian packaging system:

Every package to be built has the same structure:

• A plain sources directory, sometimes patched with debian specific code bits,
• a “debian” directory.

Inside this “debian” directory you get all the files that will describe how to build the package and every meta informations about it.

For example, you have regular version attached to the source directory, plus a Debian specific one, in dash’s case :

1. 0.5.5.1 is the source version, called “upstream” version
2. -1 is the Debian specific patch level (relative to the patches in the Debian bugtracker)

Let’s see some important components of the “debian” directory :

The “compat” file

This file specifies the compatibility level of the package definition, the current level is 7.

The “control” file

This file describes the place that this package will take in the APT ecosystem, with key metadata that help the system to identify the target architecture, the dependencies … the format is based on the rfc-822, which you can also see in HTTP and Mail headers.

Essential: yes

For example, the Architecture definition, which defines on which architecture this package will work:

• any : Will build on any architecture
• all : This package is architecture independant
• amd64, hurd-i386 : Will only build and run on these platforms

You can also encounter entries like $(shlibs:dep), that automatically analyse on which libraries depends the build binaries and automatically fill the dependencies list with the associated packages, or the Essential entry (found in the dash package) which specifies that this package is critical for correct system operation. (trying to delete it will trigger the following warning)

WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  bash dash (due to bash)
0 upgraded, 0 newly installed, 5 to remove and 60 not upgraded.
After this operation, 6756 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]

The “rules” file

This file is the bridge between the APT system and the upstream/patched sources and it is basically a GNU Makefile, it specifies how the binaries/bytecodes are to be build before packaging them, basically specifying options for the configure script or triggering a make task.

There are five mandatory targets :

• build : How to configure and build the package
• binary, binary-arch, binary-indep : How to compile the binaries
• clean : How to clean up the rubbish

Hopefully, you will try someday to build something much more complex than dash. And you will be very limited by the syntax of a basic makefile to do your work.

This is where the helpers save the day, the most used one being debhelper : Its objective is to factor the most common tasks used for binary compilation and source preparation, like configure scripts runs, documentation preparation …

Debhelper is used in the rules files with keywords like package.dirs and package.docs, the helpers being called are redacted in various languages, and there are some more modern implementations of it: CDBS (critisized for his complexity and slowness) and dh (created in 2008 to “kill” CDBS, identifiable by all his commands prefix : dh_*)

Here is a sample of a trivial dh usage :

#!/usr/bin/make -f
%:
dh $@
override_dh_auto_configure :
dh_auto_configure --with-kitchen-sink
override_dh_auto_build :
make world

and a quick reference of some useful dh_* statements :

• binary – arch : build install
• dh_testdir : Test directories according to parameters
• dh_testroot : Test the sources according to parameters
• dh_installchangelogs : Install changelogs
• dh_installexamples : Install examples (in /usr/share/doc/package/examples)
• dh_link : Create links
• dh_strip : Strip binaries
• dh_compress : Compress files
• dh_fixperms : Adjust permissions
• dh_installdeb : Install deb files
• dh_shlibdeps : Build shared libs dependencies
• dh_gencontrol : Generate a control file
• dh_md5sums : Generate md5 sums
• dh_builddeb : Build the deb file

According to statistics, dh grows rapidly comparing to plain debhelper, while CDBS stays stable. The current consensus is: use dh if you can, you might also want to get some documentation about schroot and sbuild if you have complex needs (which enables you to build packages using LVM snapshots to always have a clean build environment), or about package maintenance using VCS systems (See the Vcs-Git: http://smarden.org/git/dash.git/ entry in dash’s control file)

You are now encouraged to play around with simple packages like dash, like bumping versions using the changelog file or creating your own simple package !

Get involved !

When you will be a little more experienced, you might consider to get involved in the Debian project, you will then have multiples possibilities to do so:

• Adopt an unmaintained package (using wnpp-alert or http://www.debian.org/devel/wnpp)
• Get involved in a packaging team (http://wiki.debian.org/Teams), which is recommended for new contributors, teams are often happy to share experience
• Get some new softwares in Debian (Please, only useful and non-redundant things, we don’t want another hot-babe scandal)
• Try to ease Debian-Ubuntu collaboration (Launchpad teams)
• Find a mentor : http://mentors.debian.net

This gave me the opportunity to outline the importance of optimizing the “setup” phase of machines when using cloud computing.

Today’s tools and APIs make it very easy to create and destroy instances, but every time you do start a new instance, you’ve got to actually set it up to the state where it’s ready to do what you need (ie, be a web server, load balancer, database, whatever…).

This is an important phase to optimize: it’s the one where you’re paying for, but not yet able to use, the service

Three approaches seem to co-exist here:

1. Doing it manually: Obviously, this is one to avoid. Not only is it excessively time-consuming, but it’s also error-prone. (Ever tried doing the exact same complicated thing a dozen times over without making the odd mistake or going crazy?)
2. Pre-configuring in static images: This is a very common approach. You know how you want things, so you set them up once, and make an image of that state (ISO, AMI, whatever…). You can then use your image to create your new machine just the way you want it, no time wasted! That’s great, but wears out over time -it’s a very brittle approach. If you ever need to change that configuration, not only will it take a while (get the image, change it, upload it again, etc…), but you also have no way of propagating the change to existing instances based on the old image.
3. Using a configuration tool: That’s a deliberately vague title – I’m referring to any software approach to configuring your machine, be it a bunch of home grown scripts or a fully blown configuration management tool, such as CFEngine, Puppet or Chef.

If you’ve read any of the previous posts on this blog, you’ll know that we strongly favor the last approach

Configuration management tools have many benefits over the previous approaches: using a dedicated, purpose-built tool to set up your servers means that you can reach new levels of control and flexibility, that you cannot achieve with static, brittle pre-built images. Need to make a change? Just update your reference configuration rules, and the change will be applied to all your instances, old and new, over the next few minutes.

Enough teasing, the slides below tell the rest of the story. I introduced the main tools I mentioned before, and provided a quick overview of how CFEngine works. Those at the session also got a sneak preview of our own open source project, Rudder, to be publicly announced very soon…

I’d welcome any comments on how you setup your cloud instances… and be happy to discuss all these topics at another open source conference in the future!

Nous y présenterons une session mercredi à 14:40 (au Patio, Amphi 3), en français, pour raconter notre vécu de la gestion de désastre et comment la gestion de configuration nous a aidé à nous en remettre. On détaillera également les raisons de notre choix d’outil, qui a porté sur CFEngine 3.

En espérant vous y voir nombreux ! N’hésitez pas à venir discuter avec nous, avant ou après la session, et nous nous ferons un plaisir de partager avec vous les dernières avancées que Normation a créé autour de la gestion de configuration…

[1] Toute l’équipe de Normation sera aux RMLL !

Nous organisons des sessions de formation à Paris, les 15, 16 et 17 juin, et à Bruxelles les 4, 5 et 6 juillet.

Trois formules sont disponibles :

• Matinée d’introduction à la gestion de configuration (demie-journée non-technique)
• Fondamentaux Cfengine (2 jours)
• Industrialisation Cfengine (Fondamentaux + 1 jour)

Le détail du programme et les tarifs sont disponibles sur notre site web.

L’inscription est ouverte, contactez-nous pour vous inscrire. Plus que quelques jours pour profiter du tarif early bird !

Normation est partenaire officiel de Cfengine AS, et réalise des formations Cfengine certifiées à travers le monde.

We’ve been testing this beta version, and we are quite thrilled about it. This post won’t go through every detail of the changelog (Eystein is much better at it that I would be) but will run over the changes that improve our ways of using Cfengine.

Bug fixes

Some nagging bugs have been corrected, most notably:

• Bad class evaluation: sometimes, the evaluation of complex class conditions was invalid
• Segfault when calling a function in trustkeyfrom or allowconnects: when using the escape function in these lists, the agent or the server crashed
• Sanitization of the environment variables: on certain servers, the processes could not be correctly detected. It was because the number of columns of the output of program calls was limited to 80 columns by the system and not overriden in the agent.
• Packages promises didn’t define resulting classes if an action was taken: if a packages promise ran a command (ie, “aptitude install foo”), no classes were set on completion. Reasonable defaults are now used (a 0 return code means success, anything else is a failure), but can still be overridden.
• Memory leak in the Cfengine server (cf-serverd) when opening many secure connections: the memory consumption of the server daemon would jump occasionally if serving many clients.

New features

An improved release system

For more robust and reliable stable versions, features have been added to industrialize building and releasing Cfengine itself:

• New scripts to automate release, so that human error is mostly ruled out;
• Units test, to prevent regressions. Users are even encouraged to create unit tests;
• Easier compilation from sources. The following lines are enough to build Cfengine from sources:

  # ./autogen.sh [CONFIGURE OPTIONS]
  # make clean
  # make
  

Improved reporting in file editing

Classes can be set on every atomic file edit, meaning that each line insertion, replacement or removal can be tracked and reported.

Let’s say we want to enforce the content of the file /tmp/foo to be two lines containing foo and bar (yes, this is just an example), and be able to know if these lines are added or were already there (for the sake of simplicity, we won’t bother about knowing if other lines were present). The following promise defines a class for each added line, for each repaired line and for each line that couldn’t be added:

Note the reports in the bundle edit_line which does the reporting based on the classes defined in the insert_lines promise.

The first execution would result in:

# /var/cfengine/bin/cf-agent -Kf ./test.cf
R: Line foo added
R: Line bar added

So each line has been added, the agent did its job. A second execution would result in :

# /var/cfengine/bin/cf-agent -Kf ./test.cf
R: Line foo already present
R: Line bar already present

Which proves that no one tampered with the file while we were away.

Of course, this level of detailed reporting would probably be excessive in a real-life example, but this illustrates how classes can be used to build your own custom reporting systems.

Improved string manipulation

Cfengine 3 has pretty neat functions to read files, parse them and extract data into arrays (for instance, reading data from /etc/passwd). The 3.1.5 version introduces the same neat functions for parsing variables, rather than files. And that’s really cool to be able to manipulate complex strings rather than files if you, like me, generate your promises with external tools.

Here is an example that illustrates this new feature:

• We generate a promise file, with a variable containing a list of users, and parameters for them
• The variable is parsed and data are extracted in an array
• The array is used to edit a passwd file, and a report is made if the line is added. This is a bit different from the ways described in a previous article on file editing.

Please note that the data are inserted in a random order. Indeed, the keys of an array are not ordered, being indexed by a kind of hash.

These new features are really great, and we’d like to thank the team behind Cfengine 3 and the Community for their great job at making Cfengine 3 a really powerful and reliable product.