Blog

The Problem With Configurations

Originally published in Netocratic

Configuration is one of those things we usually take for granted. Yet, as more and more configurations are added the use of configurations becomes hard and complicated. With enough configuration parameters, keeping track of what the current configuration “is” can be quite problematic. Often, people misconfigure systems due to various reasons such as poor documentation or unexpected interactions between different parameters. Sometimes, the configuration itself can be complex and requires a language to express it, raising the bar of required knowledge for defining the configuration. Despite the “boring” nature of configurations they can and do obliterate entire companies in matter of hours.

Read more

Docker, SELinux and the myth of kernel independence

Recently I built docker images for omnibus builds. Omnibus packages must be built on the target distro so I needed images for centos, debian and ubuntu. Usually I build such docker images on my laptop which is running Ubuntu; I try to make the builds as repeatable as possible using the excellent packer tool and when I publish images I build them again (using a build server) on cloud instances. In this case, I was using GCE CentOS 7 instance to perform the build and I was surprised when a build that was smooth on my laptop failed miserably.
Read more

Thoughts about configuration management

Current configuration management tools solve a huge problem, providing a way to place repeatable, eventually coordinated application and server setup on large scale environments. Although we have come a long way since handcrafting our servers, even with Chef or Puppet, I still feel like we’re chasing our tails. Virtual Human Before we had configurable software, we needed to edit a few source files or headers, change a few constants and recompile.
Read more

Using S3 as a cookbook store backend for chef-server 11

chef-server is a modular service - various tasks are delegated to different sub-services, one of which is Bookshelf, Chef’s cookbook store in charge of storing and serving cookbook files. Bookshelf’s API is S3 compatible by design (I can only assume Opscode are using S3 for their SAAS) and can be replaced with S3. Using S3 as the backend store makes sense for clustering, but there’s very little information on the web on how to configure chef-server to use it.
Read more

Autonomation in web systems

Yes, I got the spelling right. Autonomation is a manufacturing methodology originally developed by Toyota (part of the famed Toyota Production System). The idea is sort of like this: you have a machine on a production line; this machine is supervised by a human and is feeding its output to some other machine down the line. Occasionally something goes wrong, perhaps a defective product by the machine. The machine stops itself immediately and the human will fix/change the machine (not the product, the machine!
Read more

Using include_recipe in Chef LWRP

A big part of Chef’s resources internal nuts and bolts has been change over the last year. One of the most welcome changes has been recipe_eval - a mini Chef run that can be used inside Chef LWRP. This mechanism has been put to good use in use_inline_resources dsl directive in LWRP, in chef-stage and in inline_recipe. Now that LWRPs have become so powerful, there aren’t many reasons to use definitions.
Read more

How to prevent Continuous Deployment from turning into a Continuous Disaster

These days, one the of most frequent request we hear from clients is setting up continuous deployment. Every company wants it, every DevOps related conference has sessions about it. However, newcomers tend to miss one of the key points of CI/CD – it really isn’t an automation problem, it’s a cultural problem. From a pure technical perspective implementing a build-deploy pipeline is fairly simple with current tools and this has been the case for several years.
Read more

IT is NOT ops

At the end of DevopsCon I participated in a panel discussing the future of Ops and DevOps. One thread of discussion at that panel followed a statement by Ben Kepes which basically amounted to “in the future, there won’t be any IT Ops guys anymore” (I can’t remember the exact wording, apologies). I was very disturbed by that statement; not because of fear for my profession, but rather because of the lack of dismay in the crowd.
Read more

Cronic Disease

There’s no denying Cron is one of the most useful tools in any *nix O/S but it’s Every so often, there’s a cron job going haywire. Jobs fail to run and nobody notices, the sterile environment (no sane default PATH) eludes users and from time to time, a really bizarre critical issue - like cron filling up the disk with its mail output buffer because a job is looping and spewing an endless stream of error messages.
Read more