Open source software is the rule in IT infrastructure automation. But what business models do companies like us apply, and how do these affect product decisions and open source users? The so called “open core” model is common, but we believe it introduces schizophrenia, as Chef just announced they do too. This post will explain why, present our own business model and help you understand why this may change a lot of things.
(Français) Le printemps amène le soleil et les bourgeons, et nous une nouvelle version de Rudder (2.10) !
Sorry, this entry is only available in Français.
Happy New Year!
At the foot of the Christmas tree was a new version of Rudder, 2.9 “Liberty ship”! (for the curious: http://en.wikipedia.org/wiki/Liberty_ship)
To begin 2014, our latest version, 2.9, further extends usability and adaptability of the previous version, 2.8. There are relatively few new features in this version though it will improve the way you use Rudder.
In “Liberty ship”, the main additions and new features are:
- ncf: Rudder now comes with ncf, the CFEngine framework created by Normation (https://github.com/Normation/ncf),
- Rules by category: organise your Rules by category,
- Application of Directives: verify and select the Rules applicable to a Directive directly from the Directive’s configuration page,
- Directives form: the Directives form is more intuitive and functional,
- Node-Server communication parameters: CFEngine Security parameters can be configured directly via the web interface,
- CFEngine Binaries: no more need to enter the full path on the command line in order to use CFEngine binaries in Rudder,
- Other enhancements to the web interface: save the number of entries displayed in tables, hide unnecessary information, non-applied Directives are more visible…
Last month Infoworld published an article titled “Review: Puppet vs. Chef vs. Ansible vs. Salt” written by Paul Venezia, that prompted many people in the configuration management community to get in touch with us to ask why CFEngine was not included. Our answer was simple: “No, we have no clear reason as to why this has occurred and it is bizarre not to have included CFEngine in the article.”
The article seeks to present the reader with a choice of configuration management technologies according to the following needs:
“In some cases, we may be talking about large installations that exist only to support very large applications or large installations that support myriad smaller services. In either case, the ability to wave a wand and cause them all to bend to the will of the admin cannot be discounted. It’s the only way to manage these large and growing infrastructures.”
Certainly, the scope indicated suggests CFEngine should be on the shortlist!
While the article is titled and written in the style of a review we would seek to remind readers that any review of such technology that dedicates 5 to 7 paragraphs per technology is hardly exhaustive. Indeed, we need a decent testbed to deploy all these tools to in order to back up assertions and claims with hard data for this to be considered a factual review. Instead, at best, it can be considered a primer to the options available.
We thought we’d try and help out here and share some content that could easily be inserted into the existing article at Infoworld should the editors feel inclined to provide a more complete picture to their readership…
CFEngine in a nutshell
CFEngine also started out as an open source project, in 1993. Both the technology and the author, Mark Burgess, receive recognition for this on-going contribution and have been called the Grand/Godfather of Configuration Management. The latest version of CFEngine continues to build on scientific research conducted by Mark Burgess. The concept of promise theory, first proposed by Mark in 2004, and the facility to exploit this has been available since version 3 of CFEngine, released in 2009.
Isn’t CFEngine just a by-product of scientifc research? Yes and no. Yes, CFEngine continues to receive plenty of interest from academia. However, in 2008, CFEngine AS was founded to support the ever-growing commercial interest as enterprise had not missed the opportunity to benefit from such learning. Today, JP Morgan, LinkedIn, IBM and many other organisations publicly acknowledge the value they get from using CFEngine for configuration management. CFEngine is by far the most widely used technology in production and is a core sysadmin skillset, even for today’s cloud architects!
You can find CFEngine deployed on many different platforms: AIX support came in 2005, we now use agents across Android-based deployments and of course Microsoft’s platform is supported as well, albeit only the enterprise version of CFEngine and that requires purchase of a license. Either way, if you are running a mixed environment then CFEngine is a very interesting proposition for you.
Not one single tool uses the same approach, all tools require the understanding and use of their specific vocabulary and syntax. CFEngine is no exception and with that there is the inevitable learning curve. Where some of the other tools (Puppet and Chef) have opted to leverage the Ruby ecosystem and require the dependencies and resources that go along with such a decision, CFEngine’s unequaled strength in this regard is its lightweight approach. Written in C, with a very small footprint and far fewer dependencies, CFEngine continues to execute significantly faster and with greater reliability than any other tool thus far.
CFEngine in a Web UI
CFEngine AS offers an enterprise version with a Web UI, for a fee. There is also Rudder, from Normation that is free (100% open source, free software, with enterprise-grade support available). In terms of what one can get done using a Web UI and as we provide an enterprise Web UI in the form of Rudder we’ll talk about what you can do with CFEngine using Rudder.
With Rudder you have a complete web interface with which to administer and interact with your configuration management technology (CFEngine).
You can configure and manage nodes and their configuration. You are able to automate common system administration tasks (installation, configuration), enforce configuration over time (configuring once is good, ensuring that configuration is valid and automatically fixing it is better), establish and maintain an inventory of all managed nodes, and report on the compliance of configurations (desired vs actual state) by node or group(s).
Rudder’s workflow combines templates known as ‘Techniques’, from which new instances are created as ‘Directives’. Together, with Nodes that are grouped together according to user-specified patterns, Configuration Rules are generated that are responsible for applying the Directives to the associated Group(s). All of this is done via the web interface. Separation of user roles is also possible allowing for the implementation of change control throughout each and every stage of interaction with Rudder.
“Is my infrastructure compliant?” Rudder reports on the state of all configuration items under management providing a clear signal as to whether you have converged on your desired configuration state. Rudder will show you when configurations have failed to apply and display pertinent information to help with further investigations to resolve these configuration incidents.
In terms of what comes out-of-the-box, Rudder is loaded with predefined ‘techniques’ that cover most common configuration management needs and are easily customized and extended via the GUI for specific and complex use cases. In the latest version, Rudder also offers the ability to manage isolated groups of nodes via Rudder relay servers.
So, we hope Infoworld will appreciate the support and update their article in light of these arguments. Feel free to tweet to Paul and Infoworld to request they correct the omission and include CFEngine – you know it makes sense!
We have just released the latest version of Rudder, v2.8 ‘Patrol Boat’. In this post we will share with you what has been fixed, improved and the new features we’ve added in this release.
This release cycle was shorter than usual, just under a month in fact, though it did not stop us working just as hard to make it another exciting release for our community. We continued to make improvements to Rudder’s user experience (Rudder application configuration is now possible via the web interface) and introduce more new features (relay servers)!
In keeping with our nautical theme for the 2.x series of Rudder, we named this version ‘Patrol Boat’ based on the improvements in workflow and security. The main changes and new features are:
- Configuration of settings directly from the web interface: no more need to manually edit Rudder’s config file!
- Relay servers: you can now manage multiple networks from Rudder with relay servers, enabling node isolation across your target infrastructure,
- Validation workflow: this functionality is now accessible from the REST API,
- HTTPS is active by default: better application security without any need to modify configuration,
- Better display of information pertaining to Directives: several improvements in the Directives interface, key information is easier to find and digest, superfluous information has been reduced,
- Techniques: support for Android added, a new Technique to manage partitions as well as many other enhancements throughout the Techniques library
- Rudder now uses the very latest version of CFEngine which is version 3.5, released in May, 2013,
- Improvement in the performance of reporting features: agent execution event logs are now stored in a separate table reducing the load on indexing and greatly improving the responsiveness of reporting,
- Numerous improvements relating to packaging (status option for the rudder-server-root init script, better log management, failed inventories management, …),
- Various minor enhancements to the web application: more extensive application event logging, warnings are now displayed when a rule has not been applied to any group and/or when a rule does not apply to any Directive.
Configuration via the web interface:
In previous versions of Rudder, in order to modify an application setting you had to do two things: locate and modify the parameter in the configuration file /opt/rudder/etc/rudder-web.properties and then restart the web application server from the command line: /etc/init.d/jetty restart
In version 2.8, we have begun to make available certain settings of the configuration file to enable management of these settings directly from the web interface, without having to restart the web application. The configurable settings are:
- Change messages: for each and every configuration change made via the web interface, it is possible to stipulate that the user submits a reason for the change. This feature’s parameters are:
- Enable the feature,
- Make the request for a reason mandatory,
- The message to be displayed that informs the user of this functionality and how to proceed.
- Validation Workflow: to validate changes effected via the web interface before deployment to your environment, the following parameters are now accessible from the web interface:
- Activate change requests,
- Enable self-validation / deployment by the user.
Rudder now enables the deployment of relay servers, so what are relay servers?
A server relay is a form of proxy between a Rudder server and a group of nodes (these nodes could be on another subnet that is inacessible to the Rudder server). The Rudder relay server is a server-managed node, that relays the promises, generated by the Rudder server, to a group of sub-nodes and relays the reports generated from these nodes to the Rudder server.
The short and simple procedure for using Rudder with relay servers is documented here: http://www.rudder-project.org/rudder-doc-2.8/rudder-doc.html#relay-servers. To get started using relay servers just follow the instructions and you’ll be up and running in minutes!
Workflow validation using REST API:
In 2.7, we added an API to manage the majority of the elements in Rudder (Rules, Directives, Groups and Nodes). Yet even with workflow valiation activated there was no way in 2.7 to validate the changes via the API.
This functionality has now been added in version 2.8. Each Change Request can now be managed via the API. It is now possible to request a list of all change requests according to various filters, read the details of individual change requests and thereafter accept or refuse these requests including the ability to modify their information, such as in the case of automating bulk-reassignment of change requests for example.
With this latest addition of Rudder it is now possible to use Rudder headless, i.e. independently of the web interface. Deployment of changes can also be automated: for example deployment of changes can be launched every day at 23:00…
For more information and several examples please see our REST API documentation.
HTTP/S by default:
Rudder is now configured to use HTTP/S by default. Thus, all communication with the Rudder server is encrypted (including the REST API) ensuring security communications are enabled and active for all interactions with infrastructure configuration services under Rudder’s management.
Managing Directives in the web interface:
We have made several changes in the way Directives are displayed in the web interface:
- The version of the Technique used by each Directive is now displayed in the Directives tree view list. No more need to look in each and every Directive to check they are all up to date.
- The Directive‘s internal information (Identity, Technique used, …) takes up much less space as it is now hidden by default, allowing the user to better focus on configuration settings.
- for certain Directives, the optional sections are no longer displayed by default. For more information please see Techniques below.
New additions to Techniques:
We are especially grateful to our community for the following two contributions:
- Android is now completely supported in all Techniaues, allowing the distribution and application of CFEngine promises compatible with Android. We would like to thank Cédric Cabessa (GENYMOBILE) who developed this with the support of Matthieu Cerda (Kegeruneku aka Matthieu Cerda at Normation)
- A new Technique that enables partition management on the server. This is our first technique created entirely by the community, a huge thank you to Olivier Mauras (coredumb) for their contribution!
- Various Techniques were updated to not display, by default, their optional sections (cf improvements to Directives above ). The Techniques concerned are:
- aptPackageManagerSettings Version 1.0
- zmdPackageManagerSettings Version 1.0
- zypperPackageManagerSettings Version 1.0
- fileManagement Version 2.0
- checkGenericFileContent Version 3.2
- copyGitFile Version 1.5
- dnsConfiguration Version 1.1
- sshConfiguration Version 2.0
Rudder now uses CFEngine 3.5:
All Rudder agents are now using CFEngine 3.5.2 (previously 3.4.4), allowing us to benefit from all the new functionality offered in the latest version of CFEngine (https://cfengine.com/docs/3.5/whats-new.html) as well as all the latest bug fixes!
With the addition of the latest version of CFEngine it was necessary to update our techniques library as CFEngine 3.5 is more strict about parsing of promises.
Before upgrading to our latest version, users must ensure they have already upgraded their server to one of the following versions of Rudder: 2.4.11, 2.6.8 and 2.7.5. The full upgrade procedure is documented here: http://www.rudder-project.org/rudder-doc-2.8/rudder-doc.html#_upgrade_rudder
General improvements to overall performance:
In order to perform certain tasks (reporting, date of the last report, …) we run queries on the entire content of all reports in the database even though we are only interested in a subset of that data.
We now store the execution event logs of Rudder agents in a separate table, this allows us to stop losing time by performing queries on tables containing massive amounts of data and instead run queries in a more optimized fashion, bringing significant performance increases.
The change in performance for displaying a page that lists nodes and the various reporting pages are quite impressive, we’ve seen at least a 30% improvement already.
When migrating to version 2.8, an update script is launched. This script takes charge of the database modifications in order to migrate each node’s execution data to a separate table and build new indexes. The script is expected to take some time to run so don’t worry if that is the case when running the upgrade on your deployment of Rudder.
Other significant changes:
We have introduced several small changes which will make your use of Rudder a more pleasant experience.
- Inventories in error no longer get to stay in /var/rudder/inventories/incoming, blocking the arrival of new inventories. Instead, they are now delivered to /var/rudder/inventories/failed,
- Added a new function “status” to /etc/init.d/rudder-server-root to display the health of Rudder’s components
- Added a human-friendly error log type (error in rudder-users.xml) and new log events (when a Change Request is no longer valid),
- The function “delaycompress” is now applied by logrotate to all log files and we’ve improved apache2 log entries,
- Orphaned rules: a warning is displayed when a Rule is not applied to any group and/or Directive.
That’s it for Rudder 2.8 Patrol boat! In order to upgrade to this version it is necessary to use Techniques compatible with CFEngine 3.5. To this end, you must have already upgraded your version of Rudder server to one of the latest versions in the following branches: 2.4, 2.6 and 2.7 (starting from 2.4.11, 2.6.8 et 2.7.5). The complete procedure is available and documented here.
As stated earlier, this release is not yet declared “stable“. In all cases, when we release a new version of Rudder it has been thoroughly tested, and we consider the release enterprise-ready for deployment. To be declared “stable” we prefer to wait until a version has been available and running in production for several months. As such, we expect version 2.7 of Rudder to be declared stable very soon.
We are deeply grateful for the continued support and enthusiasm of our Rudder community. We welcome your feedback, ideas and bug reports. We are listening to you! Please do not hesitate to contact us via IRC (#rudder on Freenode) or the Rudder community mailing lists (https://www.rudder-project.org/site/community/mailing-lists/), we will be delighted to welcome you and respond to any questions or feedback you might wish to share.
The changelog for Rudder v2.8 ‘Patrol Boat’ is here.
The documentation for Rudder v2.8 ‘Patrol Boat’ is available online (or directly via your Rudder server!)