Thiago Vinhas

Self-monitoring Chef using Sensu

Fri, 12 May 2017 09:45:00 -0400

Chef is one of the most popular tools for DevOps awesomeness, and you won't believe what's one of the biggest causes of chef-server unavailability: disk full

Yep, you read that right. In 2017, there's still a LOT of people who don't have all their systems being carefully monitored by a trustworthy application. That means that if that logrotate is not really rotating all your logs, or if postgresql is misbehaving and generating a lot of data, sooner or later your system will crash badly. Recovering a database after it crashes for lack of disk space is not fun, and you can't be 100% sure you won't have any data loss.

My grandpa used to say "it's better safe than sorry", and this should be every SysOps/CloudOps/SRE/DevOps Engineer mantra.

I'll show bellow how to easily set up Sensu to monitor the disk space and the overall status of your Chef Server.
You can have sensu installed on the same server as chef-server, but it's highly discouraged. Come on, CPU power and memory are extremely cheap these days and it's not worth the risk to be cheap on this area. A small AWS instance will cost about the same as a couple frappuccinos from Starbucks.

If you don't have Sensu installed, go ahead and read this Five Minute Install document. I won't cover Sensu's installation here, but basically you'll have to add the EPEL repository (if using RHEL/CentOS, sudo yum install epel-release -y), add the sensu repository to yum and install Redis, sensu and uchiwa (optional). There is something though, that the documentation doesn't make clear. You'll have to edit /etc/redis.conf and comment out the line that starts with bind, and change the line protected-mode yes to protected-mode no in order for your clients to be able to communicate with the sensu server. Don't forget to start the services and set them to start on boot time.

Once you set it up, you'll be able to check the dashboard on http://your-ip:3000. You should have the sensu-client started up so you can see it listed on the dashboard.

Set up the checks on the Sensu server

Sensu has a unique way to setup its checks and that's what makes it so versatile. For this example, you'll need two checks and one handler, that will send you an email in case of a failure.

Bellow are the two checks that you'll need on your sensu server

I saved this one as /etc/sensu/conf.d/checks/check-chef-server.rb. Notice that the command has to be run with sudo, because it actually uses chef-server-ctl for the checks.

{
  "checks": {
    "chef-server": {
      "handlers": ["default", "email"],
      "command": "sudo -u root /opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-chef-server.rb",
      "interval": 60,
      "subscribers": ["chef-server"]
    }
  }
}

And this one as /etc/sensu/conf.d/checks/disk.json:

{
  "checks": {
    "check_disk": {
      "handlers": ["default", "email"],
      "command": "/etc/sensu/plugins/check-disk-usage.rb -w 93 -c 96 -W 93 -K 96",
      "interval": 60,
      "subscribers": ["all"]
    }
  }
}

For the handlers, you can save the contents bellow on /etc/sensu/conf.d/handlers/email.json

{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "mail -s 'sensu alert' thiago@vinhas.org"
    }
  }
}

Now let's go to the next step...

Set up sensu-client on your Chef Server

Here, all you have to do is add the sensu repository by running the command bellow and then run yum install sensu -y. If you get a 404 error, you're probably using Amazon Linux, so replace $releasever to 6 and try again.

$ echo '[sensu]
name=sensu
baseurl=https://sensu.global.ssl.fastly.net/yum/$releasever/$basearch/
gpgcheck=0
enabled=1' | sudo tee /etc/yum.repos.d/sensu.repo

You'll now need a /etc/sensu/config.json with the following content:

{
  "redis": {
      "host": "sensu-server.myhost.org",
      "port": 6379
  },
  "transport": {
      "name": "redis",
       "reconnect_on_error": true
  }
}

If you start your sensu client now, you'll see in a few seconds that your chef server will be added to the dashboard. The only check available there though is be keepalive, but we really want to be able to see a problem before the whole server blows up, right?
I'll show you how to check for disk usage, and for the results of chef-server-ctl status, so you get alerted by email in case any of these fail. Sensu can also be integrated with Pagerduty, Opsgenie, ServiceNow, JIRA, Slack, Hipchat and many others.

So what you have to do on your chef-server is add the sensu repo as showed above, run yum install sensu -y and add the two plugins bellow on your /etc/sensu/plugins:

$ cd /etc/sensu/plugins
$ wget https://raw.githubusercontent.com/sensu-plugins/sensu-plugins-disk-checks/master/bin/check-disk-usage.rb
$ wget https://raw.githubusercontent.com/sensu-plugins/sensu-plugins-chef/master/bin/check-chef-server.rb

The check-disk-usage has a ruby gem dependency, and since we don't want to mess with Chef's ruby gems (and even on non-chef systems you want to be a client, it's prudent to use Sensu's embedded ruby), we're going to run:

$ /opt/sensu/embedded/bin/gem install sys-filesystem

You can test the plugin now by running /opt/sensu/embedded/bin/ruby check-disk-usage.rb (include the full path if you're not on the plugins/ directory)

Add the /etc/sensu/config.json file so your sensu client knows how to report to its master, just like you did above.

Now start all the services and set them to start on boot, and you're good to go.

On your Sensu server:

$ service sensu-server start
$ service sensu-api start
$ service sensu-client start
$ service uchiwa start
$ systemctl enable sensu-server sensu-api sensu-client

On your Chef server:

$ service sensu-client start
$ systemctl enable sensu-cliente

You should now install a dashboard like Uchiwa to see your monitored servers/services and its alerts.
You're just a yum -y install uchiwa ; service uchiwa away. Now just access http://your-server:3000 to check out Uchiwa's dashboard.

That's all folks!

Creating a local Chef repository with Pulp

Mon, 01 May 2017 07:00:00 -0400

It's very common these days to leave part, or even your whole farm, on an air gapped environment. It will definitely make it harder for malicious visitors to gain unauthorized access, but you'll have some other things to worry about regarding the management of that infrastructure.

Chef is one of the most popular orchestration tools around, and if you need to bootstrap servers on a airgapped environment, it's important that you keep a local repository at least with the products your company use.

Unfortunately, Chef's rpm packages are made using omnibus, and that means every single dependency of that software will be declared in the package, making the repository's yum metadata HUGE (currently it has about 2.1 GB).

Here is a simple way to create a local copy of the packages, and then creating a local repository using Pulp, the best tool around for creating and managing repositories:

Create a local user, let's say yum with all the defaults and add it to your /etc/sudoers file and login using that user
And then:

$ sudo yum install wget createrepo
$ sudo setenforce 0 # I won't cover SELinux permissions here
$ chmod o+x . # You'll need execute permission in your home directory to serve http

You'll have to create the directories where your repo will be located, and run the wonderful wget tool for mirroring the packages:

$ mkdir -p repos/chef ; cd repos/chef
$ wget -r -N -np -nH --cut-dirs=3  https://packages.chef.io/repos/yum/stable/el/7/x86_64/

Brace yourself because there's a LOT of packages to download and it will take a while. The good thing here is that this command will only download new/changed files, so it's safe to add it to your crontab and let it run every night.
Be aware that Chef imposes a transfer limit per IP, so if you abuse you'll be temporarily blacklisted. Use wget to fetch the packages to a single central location, and redistribute from there internally as needed.

Once done, you'll need to run:

$ createrepo /home/$USER/repos/chef/el/7/x86_64

Now follow Pulp's documentation to successfully install it, and then change the verify_ssl parameter to false on /etc/pulp/server.conf.
Run the following commands so Pulp creates your repository:

$ pulp-admin login -u admin -p admin
$ pulp-admin rpm repo create --repo-id=chef --serve-http=true --serve-https=true \
 --relative-url=chef --description 'Chef Software Repo' \
 --display-name 'Chef Software Repo' --feed file:///home/$USER/repos/chef/el/7/x86_64/
$ pulp-admin rpm repo sync run --repo-id=chef # This will probably take a while
$ pulp-admin rpm repo sync schedules create -s 2017-04-12T12:00:00Z/P1DT  --repo-id chef

Now you can access your own local Chef repository at:

http://ip-address/pulp/repos/

That's it! Hope this helps you.

Automated Cassandra Cluster on AWS

Wed, 19 Apr 2017 04:20:00 -0400

Cassandra has grown a lot the past few years and now is one of the most used NoSQL (hate this term) database for all kinds of projects.
A while ago I worked on a quick project to create a script to automatically launch and setup Cassandra clusters on AWS, with zero human iteration. I decided to share my work in case someone needs to achieve something similar.

All you really need to do is save the script bellow and make sure you have a working python environment and install the following 2 python dependencies:

pip install boto3 argparse

If you want to understand everything that is done, which is always the recommended approach, take a look at my cassandra-cluster Github repository.
Make sure you edit the credentials and any other relevant details you want, and then run the script bellow:

#!/usr/bin/env python
#######################################################
#                                                     #
#  Cassandra cluster launcher                         #
#  Developed by Thiago Vinhas <thiago@vinhas.org>     #
#                                                     #
#######################################################

import boto3
import argparse

##############################################
####           AWS Credentials            ####
##############################################

conn = {
    'aws_access_key_id': 'AKIABLAHBLAHBLAHBLAH',
    'aws_secret_access_key': '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',
    'region_name': 'us-east-1'
}

userdata = "#!/bin/bash \n aws s3 cp s3://my-script-bucket/cassandra-userdata.sh .\n bash cassandra-userdata.sh"
key = 'tvinhas'
securitygroup = 'sg-7d256706'
instanceprofile = 'arn:aws:iam::686787828286:instance-profile/cassandra'

parser = argparse.ArgumentParser(description='You are fine with the defaults...')
parser.add_argument('-a', '--ami', action="store", dest="ami", default="ami-6869aa05", help="Define the AMI to be used")
parser.add_argument('-s', '--subnet', action="store", dest="subnet", default="subnet-23dc877b", help="Define the subnet to be used")
parser.add_argument('-i', '--instancetype', action="store", dest="instancetype", default="t2.large", help="Define the instance type")

args = parser.parse_args()
ami = args.ami
subnet = args.subnet
instancetype = args.instancetype

##############################################
####           Launch Instance 1          ####
##############################################

print "Launching Instance cassandra1..."

ec2 = boto3.resource('ec2', **conn)
create_instance = ec2.create_instances(
    ImageId=ami,
    MinCount=1,
    MaxCount=1,
    KeyName=key,
    InstanceType=instancetype,
    SecurityGroupIds=[securitygroup],
    UserData=userdata,
    IamInstanceProfile={
        'Arn': instanceprofile
    },
    SubnetId=subnet
)

instanceid = create_instance[0].id
instance = create_instance[0]
instance.wait_until_running()

# Tag instance
client = boto3.client('ec2', **conn)
client.create_tags(
    Resources = [instanceid],
    Tags= [{"Key": "Name", "Value": "cassandra1"}]
)

instance.load()

print "Created new instance with id %s" % instanceid
print "IP Address: %s" % instance.public_ip_address
print "Instance Type: %s\n" % instancetype


##############################################
####           Launch Instance 2          ####
##############################################

print "Launching Instance cassandra2..."

ec2 = boto3.resource('ec2', **conn)
create_instance = ec2.create_instances(
    ImageId=ami,
    MinCount=1,
    MaxCount=1,
    KeyName=key,
    InstanceType=instancetype,
    SecurityGroupIds=[securitygroup],
    UserData=userdata,
    IamInstanceProfile={
        'Arn': instanceprofile
    },
    SubnetId=subnet
)

instanceid = create_instance[0].id
instance = create_instance[0]
instance.wait_until_running()

# Tag instance
client = boto3.client('ec2', **conn)
client.create_tags(
    Resources = [instanceid],
    Tags= [{"Key": "Name", "Value": "cassandra2"}]
)

instance.load()

print "Created new instance with id %s" % instanceid
print "IP Address: %s" % instance.public_ip_address
print "Instance Type: %s\n" % instancetype


##############################################
####           Launch Instance 3          ####
##############################################

print "Launching Instance cassandra3..."

ec2 = boto3.resource('ec2', **conn)
create_instance = ec2.create_instances(
    ImageId=ami,
    MinCount=1,
    MaxCount=1,
    KeyName=key,
    InstanceType=instancetype,
    SecurityGroupIds=[securitygroup],
    UserData=userdata,
    IamInstanceProfile={
        'Arn': instanceprofile
    },
    SubnetId=subnet
)

instanceid = create_instance[0].id
instance = create_instance[0]
instance.wait_until_running()

# Tag instance
client = boto3.client('ec2', **conn)
client.create_tags(
    Resources = [instanceid],
    Tags= [{"Key": "Name", "Value": "cassandra3"}]
)

instance.load()

print "Created new instance with id %s" % instanceid
print "IP Address: %s" % instance.public_ip_address
print "Instance Type: %s\n" % instancetype

The ultimate development machine

Mon, 20 Feb 2017 07:20:00 -0500

Back in the day I used to use several VM's on a Linux desktop to be able to do my work efficiently. One VM for each different linux distribution I had to deal with, one VM for each different Windows Server I had to support, and all my coding/scripting/tooling on linux, shared to the VM's using Samba.

As I mentioned here, after over a decade using Linux as my desktop, I started using Windows 10 a couple years ago, after my terrible experience with MacOS after my employer got me a Macbook Pro. Until last year, pretty much all the cool engineers had Macs for their work and mostly because it was easier to use brew to install all sorts of linux/unix tools on the Unix terminal that MacOS runs on. Dealing with all the desktop limitations, poor interface and the despicable keyboard.

I'll try to show on this post how a hardcore linux guy can take the most of Windows 10 in a way you won't even believe that can be happy using Windows.

Install the Windows Subsystem for Linux (aka bash)

When I started using Windows after so many years, I've installed and got pretty addicted to cygwin. It's fast and has pretty much every linux command-line tool you'll ever need. The only problem is that it's still Windows, so you are stuck with all the crazy path names like c:\\Users\\tvinhas\\My\ Documents.
Starting with November 2016's release of Windows (Anniversary Update), hell has frozen over and Microsoft introduced the Windows Subsystem for Linux (aka bash) that is basically a full-featured linux shell on Windows. It's different than cygwin because you're actually running native linux ELF binaries on Windows. It's kinda like the reverse of what Wine does on linux. To install bash on Windows, you'll have to:

Open the app Settings -> Windows & security -> For developers, and click on the Deveoper mode box.
Open the Command Prompt as Administrator and run powershell -command Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

Install ConEmu for the best console experience

ConEmu is the best terminal application you'll find, and it supports any shell available. Spend some time configuring it as you like.
Protip: To have ConEmu starting with bash and directly into your home directory, open its settings and go to Tasks. Create a new task and paste this as the command, adjusting the path to reflect your login: %windir%\system32\bash.exe -cur_console:p -new_console:d:C:\Users\tvinhas\AppData\Local\lxss\home\tvinhas\.
Now make sure you have that task set as default and start it. Edit your .bashrc and add cd ~ to its very end.

Enable eternal bash history

Every good linux guy is lazy and re-typing long commands sucks. With this tip, you can have every single command you ever typed on your terminal available thru ctrl+r. First comment out the following lines:

#HISTSIZE=1000
#HISTFILESIZE=2000

And then add this to your .bashrc:

export HISTFILESIZE=
export HISTSIZE=
export HISTTIMEFORMAT="[%F %T] "
export HISTFILE=~/.bash_eternal_history
PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Install Atom or your favorite IDE/editor

If you're used to Sublime Text or Notepad++, give Atom a shot. You won't regret.

Enable Hyper-V and forget about Virtualbox

I was (still am) a big VMware fan, but there's absolutely no reason for you to install VMware or Virtualbox on Windows 10 since it already has Hyper-V embedded and it's free.
To install it, open Powershell as Administrator and run:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

Install Docker and make your terminal talk directly with it

Docker for Windows is free and 100% compatible with what you use on Linux. It uses Hyper-V with a tiny linux VM it creates to do its magic, so you'll have to enable Hyper-V and that means no VMware and no Virtualbox (Hyper-V can't work together with other hypervisors and vice versa).
Protip: If you're really a console guy, you'll want to run docker from your shiny new bash. To do that, make sure you install docker on Linux by typing apt install docker.io and then add the following somewhere in your .bashrc file:

export DOCKER_HOST=tcp://0.0.0.0:2375

Now run docker run hello-world on your linux terminal to test your installation (make sure you ran source .bashrc or that you opened a new terminal)

That's all for today. I hope you enjoy!

Uninstalling unnecessary apps from Windows 10

Wed, 15 Feb 2017 07:00:00 -0500

MacOS quickly became the standard for development and sysadmin work, and after over 10 years using a handful of Linux desktops as my primary computer, that sounded like the natural upgrade for me. My past employer got me my first Macbook Pro a couple years ago, and after 6 months hating every single bit of it, I decided to give Windows 10 a try and had it installed using bootcamp. I'm pretty sure that hating the Mac had some sort of psychological effect, because I absolutely loved Windows 10 and I have been using it since then.

Living in the 3rd world for as long I did, I got obsessed in shaving every piece of unnecessary software and services from the OS, so I could get the most of the crappy hardware I had available. Even though running Windows 10 on a Surface Book with Intel i7 and 16GB of Memory is already as smooth as you could want, I still keep this habit, so bellow are the first steps I take after a brand new Windows 10 install:

Open Powershell as Administrator and run

get-appxpackage *messaging* | remove-appxpackage
get-appxpackage *sway* | remove-appxpackage
get-appxpackage *commsphone* | remove-appxpackage
get-appxpackage *windowsphone* | remove-appxpackage
get-appxpackage *people* | remove-appxpackage
get-appxpackage *zunemusic* | remove-appxpackage
get-appxpackage *zunevideo* | remove-appxpackage
get-appxpackage *bingfinance* | remove-appxpackage
get-appxpackage *bingnews* | remove-appxpackage
get-appxpackage *bingsports* | remove-appxpackage
get-appxpackage *onenote* | remove-appxpackage
get-appxpackage *alarms* | remove-appxpackage
get-appxpackage *camera* | remove-appxpackage
get-appxpackage *soundrecorder* | remove-appxpackage
get-appxpackage *xbox* | remove-appxpackage
get-appxpackage *officehub* | remove-appxpackage
get-appxpackage *skypeapp* | remove-appxpackage
get-appxpackage *getstarted* | remove-appxpackage
get-appxpackage *3dbuilder* | remove-appxpackage

Now from the Command Prompt (as Administrator):

taskkill /f /im OneDrive.exe
%SystemRoot%\SysWOW64\OneDriveSetup.exe /uninstall
rd "%UserProfile%\OneDrive" /Q /S
rd "%LocalAppData%\Microsoft\OneDrive" /Q /S
rd "%ProgramData%\Microsoft OneDrive" /Q /S
rd "C:\OneDriveTemp" /Q /S
REG Delete "HKEY_CLASSES_ROOT\CLSID\{018D5C66-4533-4307-9B53-224DE2ED1FE6}" /f
REG Delete "HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{018D5C66-4533-4307-9B53-224DE2ED1FE6}" /f

There are a few other things you might wanna check, like running taskmgr going to the Startup tab and disabling anything you don't really need to be running just after Windows starts.