Skip to main content Logo (IEC resistor symbol)logo

Quis custodiet ipsos custodes?
Home | About | All pages | RSS Feed | Gopher

Chef: search in recipe based on roles or recipes

Published: 08-10-2013 | Author: Remy van Elst | Text only version of this article

Table of Contents

Chef supports a very powerfull search syntax which allows you for example tosearch all nodes with the graphite-server role and get their IP addresses. Thistutorial shows you how to search based on a role a node has or a recipe a nodehas, plus an example config file with erb syntax. It has an example cookbookwhich sets up collectd as client and graphite as server. It shows you how to usethe search function of Chef to get the IP addresses of the graphite servers andplace those in the collectd config files. This technique is applicable to allkinds of services that use a client-server model, for example, munin, haproxy,zabbix and many more.

If you like this article, consider sponsoring me by trying out a Digital OceanVPS. With this link you'll get $100 credit for 60 days). (referral link)

Lets say you want to build a graphite server which gets data from a lot ofcollectd clients. You can hard code it in the collectd.conf file, but this isnot preferred, what if your graphite server changes? What if you want to add agraphite server and have all your clients automagically also send data to thatserver? Here is where the following comes in handy.

You can use this graphite cookbook. If you add the graphite recipe to anode, it will install everything needed for a graphite server, including the webui. Now, you can also create a role graphite_server and add the recipe tothat, then add the role to a node. This way you have a graphite server running.

collectd 5.1 or higher is required for graphite support. My environmentcurrently runs mostly on Ubuntu 12.04 LTS, which has collectd 4 in therepositories. Therefore I build a package myself, but there are also PPA'savailable. I also run my own repositories, so I can just use the collectd-corepackage, if you don't have a collectd 5.1 or higher package then the followingexample won't work for you.

This very simple cookbook installs the collectd package and sets the configfile. Take a look at it:

package "collectd-core" do    action :installendservice "collectd" do  supports :start =>true, :restart => true, :stop => true  action [:enable, :start]endnode.set[:collectd][:client] = truegraphite_servers = search(:node, 'recipes:"graphite"')template "/etc/collectd/collectd.conf" do    source "collectd.conf.erb"    owner "root"    group "root"    mode 0644    notifies :restart, "service[collectd]"    variables(        :graphite_servers => graphite_servers    )end

The following line does the search magic:

graphite_servers = search(:node, 'recipes:"graphite"')

It searches the Chef server for all nodes with the graphite recipe and makesthat available in this cookbook. Then it passes it on to the template, which wewill discuss below. If you don't want to search on recipes but for example onroles, you can use the following code:

graphite_servers = search(:node, 'role:graphite-server')

or on an attribute set in the node:

graphite_servers = search(:node, 'graphite_server:true')

Now the template (collectd.conf.erb) is a standard collectd template with someerb to enumerate the information in the graphite_servers variable. Skip to thebottom to see it:

# Managed by Chef for node <%= node['fqdn'] -%>.# Do not edit manually, your changes will be overwritten.Hostname <%= node['fqdn'] -%>FQDNLookup falseInterval 30ReadThreads 1LoadPlugin syslogLogLevel infoLoadPlugin cpuLoadPlugin dfLoadPlugin diskLoadPlugin entropyLoadPlugin interfaceLoadPlugin irqLoadPlugin loadLoadPlugin memoryLoadPlugin processesLoadPlugin rrdtoolLoadPlugin swapLoadPlugin usersLoadPlugin networkLoadPlugin iptablesLoadPlugin uptimeLoadPlugin "write_graphite"<Plugin "write_graphite"><% @graphite_servers.each do |graphite_server| -%> <Carbon>   Host "<%= graphite_server['ipaddress'] -%>"   Port "2003"   EscapeCharacter "_"   SeparateInstances true   StoreRates false   AlwaysAppendDS false </Carbon> <% end -%></Plugin>

This part starts a loop, which will loop trough all the values in the array itgot from the cookbook:

<% @graphite_servers.each do |graphite_server| -%>

Then this part does another lookup to get the node's IP address:

Host "<%= graphite_server['ipaddress'] -%>"

You can change that to get any other attribute from a node, in this example weneed the IP address.

This last part ends the loop:

 <% end -%>

This will result in a config file with all the graphite servers you have in yourChef environment. One of the big plus points is that you can add or removegraphite servers whenever you want without the nodes having issues. Need toscale up a few servers? Just deploy some new nodes and all the clients will usethem. Scaling down? No issue, all the clients will stop using them withoutmanual action.

This technique is very applicable to other client-server models, like Munin. Or,any other setup like this.

Tags: chef, collectd, cookbooks, deployment, devops, graphite, roles, ruby, tutorials