How about a new Rudder (2.9 release) for 2014?

Rudder_2.9_liberty_ShipHappy New Year!

At the foot of the Christmas tree was a new version of Rudder, 2.9 “Liberty ship”! (for the curious: http://en.wikipedia.org/wiki/Liberty_ship)

To begin 2014, our latest version, 2.9, further extends usability and adaptability of the previous version, 2.8. There are relatively few new features in this version though it will improve the way you use Rudder.

In “Liberty ship”, the main additions and new features are:

  • ncf: Rudder now comes with ncf, the CFEngine framework created by Normation (https://github.com/Normation/ncf),
  • Rules by category: organise your Rules by category,
  • Application of Directives: verify and select the Rules applicable to a Directive directly from the Directive’s configuration page,
  • Directives form: the Directives form is more intuitive and functional,
  • Node-Server communication parameters: CFEngine Security parameters can be configured directly via the web interface,
  • CFEngine Binaries: no more need to enter the full path on the command line in order to use CFEngine binaries in Rudder,
  • Other enhancements to the web interface: save the number of entries displayed in tables, hide unnecessary information, non-applied Directives are more visible…

Lire la suite

CFEngine vs. Puppet vs. Chef vs. Ansible vs. Salt

Last month Infoworld published an article titled “Review: Puppet vs. Chef vs. Ansible vs. Salt” written by Paul Venezia, that prompted many people in the configuration management community to get in touch with us to ask why CFEngine was not included. Our answer was simple: “No, we have no clear reason as to why this has occurred and it is bizarre not to have included CFEngine in the article.”

The article seeks to present the reader with a choice of configuration management technologies according to the following needs:

“In some cases, we may be talking about large installations that exist only to support very large applications or large installations that support myriad smaller services. In either case, the ability to wave a wand and cause them all to bend to the will of the admin cannot be discounted. It’s the only way to manage these large and growing infrastructures.”

Certainly, the scope indicated suggests CFEngine should be on the shortlist!

While the article is titled and written in the style of a review we would seek to remind readers that any review of such technology that dedicates 5 to 7 paragraphs per technology is hardly exhaustive. Indeed, we need a decent testbed to deploy all these tools to in order to back up assertions and claims with hard data for this to be considered a factual review. Instead, at best, it can be considered a primer to the options available.

We thought we’d try and help out here and share some content that could easily be inserted into the existing article at Infoworld should the editors feel inclined to provide a more complete picture to their readership…

CFEngine in a nutshell

CFEngine also started out as an open source project, in 1993. Both the technology and the author, Mark Burgess, receive recognition for this on-going contribution and have been called the Grand/Godfather of Configuration Management. The latest version of CFEngine continues to build on scientific research conducted by Mark Burgess. The concept of promise theory, first proposed by Mark in 2004, and the facility to exploit this has been available since version 3 of CFEngine, released in 2009.

Isn’t CFEngine just a by-product of scientifc research? Yes and no. Yes, CFEngine continues to receive plenty of interest from academia. However, in 2008, CFEngine AS was founded to support the ever-growing commercial interest as enterprise had not missed the opportunity to benefit from such learning. Today, JP Morgan, LinkedIn, IBM and many other organisations publicly acknowledge the value they get from using CFEngine for configuration management. CFEngine is by far the most widely used technology in production and is a core sysadmin skillset, even for today’s cloud architects!

You can find CFEngine deployed on many different platforms: AIX support came in 2005, we now use agents across Android-based deployments and of course Microsoft’s platform is supported as well, albeit only the enterprise version of CFEngine and that requires purchase of a license. Either way, if you are running a mixed environment then CFEngine is a very interesting proposition for you.

Not one single tool uses the same approach, all tools require the understanding and use of their specific vocabulary and syntax. CFEngine is no exception and with that there is the inevitable learning curve. Where some of the other tools (Puppet and Chef) have opted to leverage the Ruby ecosystem and require the dependencies and resources that go along with such a decision, CFEngine’s unequaled strength in this regard is its lightweight approach. Written in C, with a very small footprint and far fewer dependencies, CFEngine continues to execute significantly faster and with greater reliability than any other tool thus far.

CFEngine in a Web UI

ssh-config-in-Rudder-GUICFEngine AS offers an enterprise version with a Web UI, for a fee. There is also Rudder, from Normation that is free (100% open source, free software, with enterprise-grade support available). In terms of what one can get done using a Web UI and as we provide an enterprise Web UI in the form of Rudder we’ll talk about what you can do with CFEngine using Rudder.

With Rudder you have a complete web interface with which to administer and interact with your configuration management technology (CFEngine).

You can configure and manage nodes and their configuration. You are able to automate common system administration tasks (installation, configuration), enforce configuration over time (configuring once is good, ensuring that configuration is valid and automatically fixing it is better), establish and maintain an inventory of all managed nodes, and report on the compliance of configurations (desired vs actual state) by node or group(s).

ReportsRudder’s workflow combines templates known as ‘Techniques’, from which new instances are created as ‘Directives’. Together, with Nodes that are grouped together according to user-specified patterns, Configuration Rules are generated that are responsible for applying the Directives to the associated Group(s). All of this is done via the web interface. Separation of user roles is also possible allowing for the implementation of change control throughout each and every stage of interaction with Rudder.

 

“Is my infrastructure compliant?” Rudder reports on the state of all configuration items under management providing a clear signal as to whether you have converged on your desired configuration state. Rudder will show you when configurations have failed to apply and display pertinent information to help with further investigations to resolve these configuration incidents.

In terms of what comes out-of-the-box, Rudder is loaded with predefined ‘techniques’ that cover most common configuration management needs and are easily customized and extended via the GUI for specific and complex use cases. In the latest version, Rudder also offers the ability to manage isolated groups of nodes via Rudder relay servers.

So, we hope Infoworld will appreciate the support and update their article in light of these arguments. Feel free to tweet to Paul and Infoworld to request they correct the omission and include CFEngine – you know it makes sense!

Rudder goes on patrol (2.8.0)

Rudder_2.8_Patrol_BoatWe have just released the latest version of Rudder, v2.8 ‘Patrol Boat’. In this post we will share with you what has been fixed, improved and the new features we’ve added in this release.

This release cycle was shorter than usual, just under a month in fact, though it did not stop us working just as hard to make it another exciting release for our community. We continued to make improvements to Rudder’s user experience (Rudder application configuration is now possible via the web interface) and introduce more new features (relay servers)!

In keeping with our nautical theme for the 2.x series of Rudder, we named this version ‘Patrol Boat’ based on the improvements in workflow and security. The main changes and new features are:

  • Configuration of settings directly from the web interface: no more need to manually edit Rudder’s config file!
  • Relay servers: you can now manage multiple networks from Rudder with relay servers, enabling node isolation across your target infrastructure,
  • Validation workflow: this functionality is now accessible from the REST API,
  • HTTPS is active by default: better application security without any need to modify configuration,
  • Better display of information pertaining to Directives: several improvements in the Directives interface, key information is easier to find and digest, superfluous information has been reduced,
  • Techniques: support for Android added, a new Technique to manage partitions as well as many other enhancements throughout the Techniques library
  • Rudder now uses the very latest version of CFEngine which is version 3.5, released in May, 2013,
  • Improvement in the performance of reporting features: agent execution event logs are now stored in a separate table reducing the load on indexing and greatly improving the responsiveness of reporting,
  • Numerous improvements relating to packaging (status option for the rudder-server-root init script, better log management, failed inventories management, …),
  • Various minor enhancements to the web application: more extensive application event logging, warnings are now displayed when a rule has not been applied to any group and/or when a rule does not apply to any Directive.

Configuration via the web interface:

In previous versions of Rudder, in order to modify an application setting you had to do two things: locate and modify the parameter in the configuration file /opt/rudder/etc/rudder-web.properties and then restart the web application server from the command line: /etc/init.d/jetty restart

In version 2.8, we have begun to make available certain settings of the configuration file to enable management of these settings directly from the web interface, without having to restart the web application. The configurable settings are:

  • Change messages: for each and every configuration change made via the web interface, it is possible to stipulate that the user submits a reason for the change. This feature’s parameters are:
    • Enable the feature,
    • Make the request for a reason mandatory,
    • The message to be displayed that informs the user of this functionality and how to proceed.

Configure_CM

  • Validation Workflow: to validate changes effected via the web interface before deployment to your environment, the following parameters are now accessible from the web interface:
    • Activate change requests,
    • Enable self-validation / deployment by the user.

Configure_CR

Relay servers:

Rudder now enables the deployment of relay servers, so what are relay servers?

A server relay is a form of proxy between a Rudder server and a group of nodes (these nodes could be on another subnet that is inacessible to the Rudder server). The Rudder relay server is a server-managed node, that relays the promises, generated by the Rudder server, to a group of sub-nodes and relays the reports generated from these nodes to the Rudder server.

The short and simple procedure for using Rudder with relay servers is documented here: http://www.rudder-project.org/rudder-doc-2.8/rudder-doc.html#relay-servers. To get started using relay servers just follow the instructions and you’ll be up and running in minutes!

Workflow validation using REST API:

In 2.7, we added an API to manage the majority of the elements in Rudder (Rules, Directives, Groups and Nodes). Yet even with workflow valiation activated there was no way in 2.7 to validate the changes via the API.

This functionality has now been added in version 2.8. Each Change Request can now be managed via the API. It is now possible to request a list of all change requests according to various filters, read the details of individual change requests and thereafter accept or refuse these requests including the ability to modify their information, such as in the case of automating bulk-reassignment of change requests for example.

With this latest addition of Rudder it is now possible to use Rudder headless, i.e. independently of the web interface. Deployment of changes can also be automated: for example deployment of changes can be launched every day at 23:00…

For more information and several examples please see our REST API documentation.

HTTP/S by default:

Rudder is now configured to use HTTP/S by default. Thus, all communication with the Rudder server is encrypted (including the REST API) ensuring security communications are enabled and active for all interactions with infrastructure configuration services under Rudder’s management.

Managing Directives in the web interface:

We have made several changes in the way Directives are displayed in the web interface:

  • The version of the Technique used by each Directive is now displayed in the Directives tree view list. No more need to look in each and every Directive to check they are all up to date.

Directive_tree_version2

  • The Directive‘s internal information (Identity, Technique used, …) takes up much less space as it is now hidden by default, allowing the user to better focus on configuration settings.

2.7Avant (Version 2.7)

2.82.8

  • for certain Directives, the optional sections are no longer displayed by default. For more information please see Techniques below.

New additions to Techniques:

We are especially grateful to our community for the following two contributions:

  • Android is now completely supported in all Techniaues, allowing the distribution and application of CFEngine promises compatible with Android. We would like to thank Cédric Cabessa (GENYMOBILE) who developed this with the support of Matthieu Cerda (Kegeruneku aka Matthieu Cerda at Normation)
  • A new Technique that enables partition management on the server. This is our first technique created entirely by the community, a huge thank you to Olivier Mauras (coredumb) for their contribution!
  • Various Techniques were updated to not display, by default, their optional sections (cf improvements to Directives above ). The Techniques concerned are:
    • aptPackageManagerSettings Version 1.0
    • zmdPackageManagerSettings Version 1.0
    • zypperPackageManagerSettings Version 1.0
    • fileManagement Version 2.0
    • checkGenericFileContent Version 3.2
    • copyGitFile Version 1.5
    • dnsConfiguration Version 1.1
    • sshConfiguration Version 2.0

Rudder now uses CFEngine 3.5:

All Rudder agents are now using CFEngine 3.5.2 (previously 3.4.4), allowing us to benefit from all the new functionality offered in the latest version of CFEngine (https://cfengine.com/docs/3.5/whats-new.html) as well as all the latest bug fixes!

With the addition of the latest version of CFEngine it was necessary to update our techniques library as CFEngine 3.5 is more strict about parsing of promises.

Before upgrading to our latest version, users must ensure they have already upgraded their server to one of the following versions of Rudder: 2.4.11, 2.6.8 and 2.7.5.  The full upgrade procedure is documented here: http://www.rudder-project.org/rudder-doc-2.8/rudder-doc.html#_upgrade_rudder

General improvements to overall performance:

In order to perform certain tasks (reporting, date of the last report, …) we run queries on the entire content of all reports in the database even though we are only interested in a subset of that data.

We now store the execution event logs of Rudder agents in a separate table, this allows us to stop losing time by performing queries on tables containing massive amounts of data and instead run queries in a more optimized fashion, bringing significant performance increases.

The change in performance for displaying a page that lists nodes and the various reporting pages are quite impressive, we’ve seen at least a 30% improvement already.

When migrating to version 2.8, an update script is launched. This script takes charge of the database modifications in order to migrate each node’s execution data to a separate table and build new indexes. The script is expected to take some time to run so don’t worry if that is the case when running the upgrade on your deployment of Rudder.

Other significant changes:

We have introduced several small changes which will make your use of Rudder a more pleasant experience.

  • Inventories in error no longer get to stay in /var/rudder/inventories/incoming, blocking the arrival of new inventories. Instead, they are now delivered to /var/rudder/inventories/failed,
  • Added a new function “status” to /etc/init.d/rudder-server-root to display the health of Rudder’s components
  • Added a human-friendly error log type (error in rudder-users.xml) and new log events (when a Change Request is no longer valid),
  • The function “delaycompress” is now applied  by logrotate to all log files and we’ve improved apache2 log entries,
  • Orphaned rules: a warning is displayed when a Rule is not applied to any group and/or Directive.

Summary:

That’s it for Rudder 2.8 Patrol boat! In order to upgrade to this version it is necessary to use Techniques compatible with CFEngine 3.5. To this end, you must have already upgraded your version of Rudder server to one of the latest versions in the following branches: 2.4, 2.6 and 2.7 (starting from 2.4.11, 2.6.8 et 2.7.5). The complete procedure is available and documented here.

As stated earlier, this release is not yet declared “stable“. In all cases, when we release a new version of Rudder it has been thoroughly tested, and we consider the release enterprise-ready for deployment. To be declared “stable” we prefer to wait until a version has been available and running in production for several months. As such, we expect version 2.7 of Rudder to be declared stable very soon.

We are deeply grateful for the continued support and enthusiasm of our Rudder community. We welcome your feedback, ideas and bug reports. We are listening to you! Please do not hesitate to contact us via IRC (#rudder on Freenode) or the Rudder community mailing lists (https://www.rudder-project.org/site/community/mailing-lists/), we will be delighted to welcome you and respond to any questions or feedback you might wish to share.

The changelog for Rudder v2.8 ‘Patrol Boat’ is here.

The documentation for Rudder v2.8 ‘Patrol Boat’ is available online (or directly via your Rudder server!)

Vagrant: Virtual machine provisioning made easy

During Devops Days Paris 2013 earlier this year (18/19 April 2013) I was surprised to discover that very few people had heard of Vagrant. Luckily; there was an open space dedicated to Vagrant (and its usage) where 15-20 people attended but only 5 among them knew about Vagrant (including my colleague, François Armand and myself). This tools deserves to be better known, so the aim of this post is to introduce you to Vagrant and share one of several ways we regularly use Vagrant at Normation.

We began using this tool extensively during July 2012. At that time Rudder 2.4.0~beta3 was about to be released. We needed to test installation as much as possible, thus we needed to start from a fresh, clean environment for each test. This required the creation of several VMs (a server and some nodes) followed by installing and configuring Rudder (server or agent) on each of them.. Doing this manually took a lot of time, was laborious and could be prone to error. All the time spent on installation was wasted every time we destroyed those VMs after a test. Now all of this is automated with Vagrant and we no longer lose time configuring virtual machines.

What is Vagrant?

Vagrant is an open source virtual environment building tool created in January 2010 by Mitchell Hashimoto, and developed on his spare time. As Vagrant became more and lore popular worldwide, in November 2012 Mitchell founded Hashicorp, a company dedicated to Vagrant. With 3 years of development, the huge community and the support of Hashicorp, Vagrant has become a proven and reliable software.

Vagrant’s approach is very simple:

  • Create the VM, based on a base box a VM provider.
  • Customize  it, still using the VM provider
  • Configure it, on startup, using a provisioning tool..
  • Control it directly with vagrant commands, or ssh in (vagrant ssh)

Once you have configured your template (Vagrantfile) you can then re-use it and thus automate the entire installation process!

Vagrant uses a single configuration file: the Vagrantfile. In this file you describe how your virtual machines will be created and configured. Everything you need to know will be centralized in that file. How to configure a Vagrantfile is clearly documented on the website http://docs.vagrantup.com/v2/.

Vagrant will use three things to create the VM:

  • A base box: These are VMs prepared for vagrant usage (vagrant user, access defined). You can create your own boxes however many are already available at vagrantbox.es ,
  • A VM provider: It will create the VM using the base box and customize it. Numerous providers are supported (Virtualbox, AWS, …),
  • A provisioning tool: It will be run at the VM boot, it can run a shell script or use a configuration management tool (CFEngine, Chef, Puppet, …).Vagrant

So there are a huge variety of possibilities and it can be adapted to all of your needs and abilities.

Vagrant is very simple to install and use, you just need to choose your provider (virtualbox is the easiest one, other providers are also available).

Creating your first VM is very easy,  just run the following in a dedicated directory:

$ vagrant box add base http://files.vagrantup.com/lucid32.box
$ vagrant init
$ vagrant up

Vagrant offers you several commands to manage your environment :

  • init: initiate a vagrant environment with a bare Vagrant file
  • up: start  VM(s)
  • ssh: ssh into the VM passed as parameter
  • status: show status of all VMs
  • destroy: destroy VM(s)
  • halt: stop VM(s)
  • box: manage base boxes

… and a lot more!

So, to use vagrant the only things you need to know are how to declare a VM (very easy and straight forward with the Vagrantfile), and how to configure it – depending on your needs!

Vagrant gives you automation and I do think that automation is one of the best things ever invented. Automation helps you get back the time spent doing things manually so that you can make even better use of your time on other tasks. With Vagrant you do the preparation only once, then (almost, if we take into account Murphy’s Law) everything works as expected. At Normation, we use it in three areas: our development environment, testing environment and embedded demo. I’ll explain in details how we manage to do this with vagrant, and all advantages we gain by using it in a follow-up blog post.

Vagrant is not the only tool for virtual environment automation and provisioning, there is Veewee (creation of base boxes, developped with love by Patrick Debois) and Cobbler (my colleague, Mathieu Cerda, gave a talk about Cobbler at LOADays 2013, here is a link to his slides). They are very good and I encourage you to take a look at them! Easy automation is now a reality, it would be a shame not to take advantage of this!

Normation special offer of Rudder support for free!

We understand that starting a new project is often time-consuming.

Part of this challenge is making sure you follow best practices, ensuring your design choices during planning will not impact production deployment and service continuity thereafter, that your team is up to speed on your technology choices and fundamentally that you are confident that your are in control.

We want to help!

Whether you just want to evaluate Rudder or you have already begun to deploy our solution in production, we want you to have the best possible experience.

To this end we have developed a special offer of three Rudder support tickets, provided by our configuration management experts at Normation, for free!

  • Professional, Enterprise-grade support backed by our team of Rudder experts,
  • Support ticket tracking via a dedicated interface hosted on our client portal,
  • Dedicated web interface, email or phone support according to your preference,
  • 1 hour Guaranteed Intervention Time (GIT),
  • 1 day Guaranteed Workaround Time (GWT).

To benefit from this offer and request your free support please complete and submit this form.

Normation

87 rue Turbigo
75003 Paris

Tel : 01 83 62 26 96
Fax : 01 83 62 29 38
Contact us

Sign up for our newsletter

English
Français

Follow us