Nagios Plugin for monitoring OpenSolaris Zones CPU und MEM

To get CPU and memory data by zone, there is prstat -Z. Executed from within the global zone it returns a list of all zones, their current memory and CPU utilization, etc.

ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     3       34 2443M 2430M    15%   9:03:05 7.9% app
     4       19 2562M 2014M    12% 288:22:24 1.7% db
     0       51   85M   92M   0.6% 145:57:00 1.6% global
     2      143  483M  192M   1.2%   0:17:52 0.6% web
     1       19 3106M 3111M    19%   0:49:05 0.0% mem

Putting it into a script and passing an command line parameter for the zone name to it, the script can calculate a nagios status. The script returns that nagios status based on warning and critical thresholds, passed to the script together with nagios performance data.

prstat -Z needs to run within the global zone

Usually, our nagios server calls nagios check scripts, which need to run locally on the monitored box, via SSH. All our command definitions use $HOSTADDRESS$ as the target host for the SSH connection. $HOSTADDRESS$ resovles to the host under test during a nagios check run. But prstat -Z needs to run within the global zone. To deal with that I added the IP-address of the global zone as a parameter to the nagios check call:

check_command check_by_ssh_zone_mem!4000!5000!app2!5120!10.0.0.1

In my command definition I changed the SSH target from $HOSTADDRESS$ to $ARG5$ making the check_by_ssh script connect to the global zone instead of the host under test.

Now we get nice CPU and MEM usage graphs for our OpenSolaris zones. As soon as I’ll have added documentation to my nagios check scripts, I’ll publish them on MonitoringExchange for everyone to scrunitize and maybe even use.

Related posts:

• Monitoring tools essentials: Munin vs. Nagios
• Behavior Driven Ops, Kanban vs. Scrum and a new upcoming Monitoring and Trending Tool
• Kent Langley Talks About Joyent, Their New Accelerators And Their Use of Opensolaris

Related posts:

• Monitoring tools essentials: Munin vs. Nagios
• Acceptance Testing with Cucumber
• webrat: Automated Acceptance Testing with RSpec or Cucumber

We don’t want to setup a development environment on our CI server (which requires loading an anonymized production dump - quite a beast). It’s enough for us to have a test environment there, using a checked out development_structure.sql (created on a development box and checked into version control) to create the test database. Unfortunately, the rake tasks provided by rails do not cover this scenario.

So, we simply came up with the following addition to rails’ set of rake tasks:

namespace :db do
  namespace :test do
    desc "Recreate the test database from an existing db/development_structure.sql dump file"
    task :load_structure => [ "db:test:purge" ] do
      ActiveRecord::Base.establish_connection(:test)
      ActiveRecord::Base.connection.execute('SET foreign_key_checks = 0')
      IO.readlines("#{RAILS_ROOT}/db/#{RAILS_ENV}_structure.sql").join.split("nn").each do |table|
        ActiveRecord::Base.connection.execute(table)
      end
    end
  end
end

rake db:test:load_structure reads the checked out development_structure.sql and applies it to the test database (which it recreates first by defining db:test:purge as a pre-requisite). That’s all we need to have a fully up-to-date test database on our CI server! No need to worry about creating a full development environment there just to be able to run migrations and dump the updated schema prior to setting up the test database.

Related posts:

• Seed Data In Ruby On Rails
• GLoc Validation Problems After Rails 2.1 Upgrade
• Setup a Ruby on Rails Project Using the Lighthouse API With ActiveResource

Not writing about and contemplating my experiences, successes and failures from the week’s work meant I wasn’t improving, wasn’t honing my focus. Last week, I had a velocity of 0. For the first time in over six months, I literally delivered no customer value in my weekly sprint. And I chalk it up to not blogging - or at least not holding my strange sort of online weekly retrospective.

Being too busy to hold retrospectives can be dangerous especially if you’re working alone. It’s too easy (and ofttimes down right seductive) to just grab the next story and jump in head first, finishing chores left and right and feeling good about your progress. Luckily, however, I’m religious about tracking and sharing my work with Pivotal Tracker so at least I knew I missed out on doing some for our users last week.

So, what the hell have I been working on? Setting up a new server to host our relaunched portal and putting the finishing touches on our corporate blog. But, wait, why isn’t this customer value? First of all, customers don’t care about how your Nagios install interfaces with Munin to collect server metrics. And while CSS often means the difference between a “ho-hum” site and “nice”, it’s definitely not the reason why users will read your content or yawn and click away.

What’s been keeping you from delivering customer value? Are you honest with yourself, your company and your users about delivering real user value?

Related posts:

• Stop Being Busy. Get Productive!

DRY Cucumber Scenarios

The new cool kid on the block is Cucumber. One of its main features is that you can describe acceptance tests in nearly plain English (or any other language) simplifying discussions with the product managers and other, non-technical stake holders. I’m really impressed by the ease, with which you can describe DRY scenarios like our login feature:

Feature Login
  In order to be able to contribute
  users
  want to login

  Background:
    Given the user "kanban" exists
    And I am logged out
    And I am on the login page

  Scenario: Login a with valid user
    When I fill in "Benutzername" with "auser"
    And I fill in "Passwort" with "secret"
    And I press "Einloggen"
    Then I should see "Meine Seite"
    And I should see "Neuigkeiten auf meiner Seite"

  Scenario Outline: Login with invalid credentials
    When I fill in "Benutzername" with "< user_name >"
    And I fill in "Passwort" with "< password >"
    And I press "Einloggen"
    Then I should see "Einloggen"
    And I should see "Der Benutzername oder das Passwort stimmt nicht"

    Examples:
      |user_name|password|
      |auser|bla|
      |bla|secret|
      |bla|bla|

Background, Scenario Outline and Example are the keywords enabling DRY descriptions.

Hooking Scenrios to your app using webrat

As I described in my previous post about Acceptance Testing with Cucumber, Cucumber comes with a set of pre-defined steps for webrat. So the steps are usually single line calls to webrat’s API:

When /^I open (.+)$/ do |url|
  visit url
end

Acceptance Testing for a pure technical team

So far, so good. Cucumber works great and has a very descriptive language perfectly suited for talking to non-technical folks. In our team, the situation is such that we hardly have to communicate in such great detail with our CEO - he’s mainly interested in the rough workings of a feature, but not in the details (he has other things to worry about ). As the product development team only consists of three tech guys, we wondered whether it’s necessary to have the English language abstraction. Wouldn’t it be nicer just to use RSpec with Cucumber to make it immediately obvious to technical folks, what the test really does (instead of hiding the details in step descriptions?

To be honest, I was not sure whether it was a good idea to go from Cucumber to RSpec. I feared that the tests would get cluttered by syntax and be much longer (and harder to read) than the Cucumber features. But, I gave it a try. And I was surprised. See for yourself - the same feature side by side as RSpec description and Cucumber feature:

Surprisingly enough, re-writing the feature as RSpec description does not bloat it. In fact, for a tech guy, it is easier to read than the feature as you see exactly what is happening right away. Additionally, you’ve got full Editor/IDE support for writing specs.

RSpec vs Cucumber

Cucumber is great. If you talk in detail about features with non-technical folks, it is the way to go. If you do not have to bridge such a gap between tech and non-tech guys, you might be better off using webrat directly from within your specs. It removes one layer of indirection and doesn’t really make things worse to read, if you adapt best practices for structuring your specs from cucumber (like using examples and well named helpers).

Related posts:

• Acceptance Testing with Cucumber
• Behavior Driven Ops, Kanban vs. Scrum and a new upcoming Monitoring and Trending Tool

103 posts and 113 comments. That’s the result of our first year. And, exactly 200 of you have subscribed to our RSS feed - an excellent choice. For all of you who joined recently (and to remind you veterans), check out the five most read posts:

1. Puppet or Capistrano - Use the Right Tool for the Job (Guest post by Andrew Shafer)
2. Puppet vs. Capistrano - a short comparison
3. System Configurations + Code Revisions = Continuous Integration FTW (Guest post by Patrick Debois)
4. Thoughtworks Mingle vs. Pivotal Labs Tracker
5. Agile Tool Vendors: Please don’t try to manage complexity - simplify my life!

And don’t miss out on our three most commented posts:

• Agile on steroids
• Configuration Management remixed: Introducing Carpet
• Visible Ops: Rolling Out Change Management

I hope you enjoy reading Agile Web Operations as much as we do writing it. Keep the comments flowing and don’t forget to tell your friends!

We’re looking forward to the coming years of Agile Web Operations.

Sincerely yours,

Dan & Matthias

Related posts:

• Improving Operations with Agile
• Test First in Operations at The Build Doctor
• Four Agile Posts Everyone Should Read

Common approaches like adding seed data through rails migrations are discouraged, and plugins like seed_fu only work for small amounts of seed data. In seed_fu, you can specify a seed method for your ActiveRecord models like so:

User.seed(:login, :email) do |s|
      s.login = "bob"
      s.email = "bob@bobson.com"
      s.first_name = "Bob"
      s.last_name = "Bobson"
    end

Running the rake db:seed task provided by seed_fu will add all defined models to your test database.

DHH has even standardized a way to load seed data for Rails 3, making the rake db:seed task part of rails and setting up a file called db/seeds.rb for maintaining your seeding code. Using that file, you can load your seed data however you see fit, e.g. seed_fu.

How to Deal With Big Amounts of Seed Data

So far, so good. There are ways to load seed data into your rails test database using Ruby code. But what if, like in our case, you have to seed more than 60,000 Points of Interest and over 16,000 cars? We definitely don’t want to write Ruby code for each of them. The only sane way of handling such amounts of data are database dumps. So I added my own rake db:seed:dump and rake db:seed:load tasks to our Rails 2.3.2 application. As soon as we move to Rails 3, we can call the load task from within db/seeds.rb.

Short and sweet (and completely MySQL specific and dependent on MySQL living in your path ) here are my two rake tasks:

namespace :db do
  namespace :seed do
    require 'db/seed_tables'
 
    desc "dump the tables holding seed data to db/RAILS_ENV_seed.sql. SEED_TABLES need to be defined in config/environment.rb!!!"
    task :dump => :environment do
      config = ActiveRecord::Base.configurations[RAILS_ENV]
      dump_cmd = "mysqldump --user=#{config['username']} --password=#{config['password']} #{config['database']} #{SEED_TABLES.join(" ")} > db/#{RAILS_ENV}_seed.sql"
      system(dump_cmd)
    end
 
    desc "load the dumped seed data from db/development_seed.sql into the test database"
    task :load => :environment do
      config = ActiveRecord::Base.configurations['test']
      system("mysql --user=#{config['username']} --password=#{config['password']} #{config['database']} < db/#{RAILS_ENV}_seed.sql")
     end
  end
end

Note that I use a file called db/seed_tables.rb to define, which tables shall be dumped. It just holds an array of table names like so:

SEED_TABLES = [
  "auxilary_services",
  "background_informations",
  "pois"
]

Using two basic rake tasks and database dumps eases the pain of handling test data for us. How do you manage your test data? Let us know in the comments!

Related posts:

• Setting up a test database on a ruby on rails continuous integration server using SQL instead of schema.rb
• Setup a Ruby on Rails Project Using the Lighthouse API With ActiveResource
• GLoc Validation Problems After Rails 2.1 Upgrade

Ultimately, I had to make a compromise between feasibility and correctness, but I’m pretty satisified with the results. In fact, SiteSucker was able to confirm that, besides images, we did manage a 100% conversion for all the existing urls.

Google Analytics & Full Realization

At first, I spent about a day going through the top 1000 URLs according to the Google Analytics tracker. This was horribly redundant, mind-numbing work but I didn’t see a way to export all 5 MILLION URLs that Google Analytics had on on record for the last month.

I calculated at the rate I needed to go through the top 1000, I’d need almost 7 years to double check the rest. And, of course, I knew we didn’t have that many valid pages in our site.

Site Sucker to the Rescue

I’d used SiteSucker a few times in recent months to double check the health of our site’s link structure. It’s extremely fast and the user interface is very lean (making use of Mac OS’s Console logging application). What I wondered was how I could execute a web crawl not from a site, but from a saved file? Turns out, it’s very easy!

First, ensure the settings of SiteSucker to log the download history (and save that log):

Then, enter the original sitename in the Web URL input and hit enter. You’re off to the races!

For ~20k links, it took almost 20 minutes for SiteSucker to grab them all. Take the finished log output and snip away the unneeded text to the left and right of the url. Go ahead and take the extra time to wrap them up in a nice anchor tag (this will help in the next step).

Copy this file and rename it to reflect the new site where you want to test your redirects. In your favorite editor search/replace the old domain with the test (or new) domain. Wrap it in with basic html & body tags and change the extension to .html. Now, we have a very basic html page containing all the links from the old site.

In SiteSucker, go back into the settings and check the Limits. We want to enforce a maximum level of 1 now. This is because we already have all the relevant links in our file - no point in asking SiteSucker to recrawl the entire site for every original link (this would take days).

Finally, drag the html file you created (containing the test or new domain name and all links) into the Web URL bar and let go. SiteSucker dutifully follows every link and reports its findings. Hopefully, you won’t have too many ERRORs, but if you do, it’s quite easy now to rectify them as you have a log showing exactly which redirects failed!

Don’t let huge numbers of links frighten you into thinking you can’t make your site better. There are plenty of web crawlers available on many platforms - I have also used and recommend Xenu on Wine. Thinking a bit outside of the box can make it easy to turn that mountain into a mole hill again.

How do you maintain your sites’ link “healthiness” ? Let us know in the comments.

Related posts:

• Acceptance Testing with Cucumber

Features

For testing your application, you specify features consisting of scenarios. Each scenario is defined by a set of steps and validations. Here’s a short example:

Feature Contact Form
  In order to tell us something
  users
  want to contact us online without having to leave their browser
 
  Scenario: Submitting with valid infos
    Given I am on the contact page
    When I fill in "Your Name" with "Matthias Marschall"
    And I fill in "Your Message" with "What's up?"
    And I press "Send"
    Then I should see "Thanks for your feedback!"

Usually, one would need to implement fixtures to bind steps to the application code. But cucumber comes with a very useful set of pre-defined steps for webrat, in fact, all of the above steps are already there. You should be able to successfully execute the above example out of the box (given your app already has the contact page up and running).

Full Cycle E-Mail Testing with rails and cucumber

Filling out the contact form should send an email to a feedback account with the information provided. There exists a very handy plugin for dealing with email verification in a rails test environment: rspec-email

After installing it, you can add steps like these to the scenario shown above:

    And "contact@example.com" should receive 1 email
 
    When "contact@example.com" opens the email with subject "Feedback"
    Then I should see "What's up" in the email
 
    When I click the first link in the email
    Then ...

Again, all of the above steps are provided by the plugin (after calling script/generate….).

Without writing a single line of fixture code, I was already able to do a full integration test including form validations and email sending. Not too bad for 2 hrs worth of research. My next steps will be to try out the selenium integration and play around with setting up data in the database for driving my tests. Stay tuned by subscribing to our RSS feed.

Related posts:

• webrat: Automated Acceptance Testing with RSpec or Cucumber

Why not just login to Pingdom?

Good question! My answer - because it’s slow & tedious. It takes me minutes to get the information I’m looking for - namely, the average response time of a few critical pages from yesterday and today. To not have to click through the various cities every day deselecting all the [European|American] sites in order to get a more accurate measurement - this alone is worth its weight in gold!

Don’t get me wrong. The graphs are wonderful for spotting trends, but I don’t need to look at these every day. I want to be able to tell within a second or two if my site is performant.

There’s a reason they made an API

The good folks at Pingdom are number crunchers at heart - they probably also avoid logging into the site when possible because they developed a nice API with which you can access all the info you need.

After requesting an API key, you’re just a few curl commands away from this :

And here’s the corresponding code:

<?php
    define('SERVER_URL', 'https://ws.pingdom.com/soap/PingdomAPI.wsdl');
    define('USERNAME', '<username>');
    define('PASSWORD', '<password>');
    define('PINGDOM_API_KEY', '<api_key>');
    define('PINGDOM_API_STATUS_OK', 0);
    define('RESOLUTION', 'DAILY');
    define('GOOD_TIMING', 500);
    define('POOR_TIMING', 800);

    $client = new SoapClient(SERVER_URL, array( 'trace' => 1,'exceptions' => 0 ));

    $sessionId = loginPingdom($client);
    $responses = fetchPages($client, $sessionId);
    writeResponse($responses);
    logout($client, $sessionId);

function loginPingdom($client) {
    // Login
    $login_data->username = USERNAME;
    $login_data->password = PASSWORD;
    $sessionId = null;

    // Pingdom API function, see details on http://www.pingdom.com/services/api-documentation/
    $login_response = $client->Auth_login(PINGDOM_API_KEY, $login_data);

    // Check if everything is OK
    if (PINGDOM_API_STATUS_OK != $login_response->status) {
        echo'Unable to login to Pingdom';
    } else {
        // Without this value you wont be able to call any other Pingdom API function
        $sessionId = $login_response->sessionId;
    }

    return $sessionId;
}

function fetchPages($client, $sessionId) {
    $locs = array("Amsterdam, Holland","London, UK","Paris, France","Stockholm, Sweden");

    // Today - up through CURRENT_TIME
    $get_times_today->from = mktime("00", "00", "00", date("m"), date("d"), date("Y"));
    $get_times_today->to = CURRENT_TIME;
    $get_times_today->resolution = RESOLUTION;
    $get_times_today->locations = $locs;

    //Yesterday
    $get_times_yesterday->from = mktime("00", "00", "00", date("m") , date("d") - 1, date("Y"));
    $get_times_yesterday->to = mktime("23", "59", "59", date("m") , date("d") - 1, date("Y"));
    $get_times_yesterday->resolution = RESOLUTION;
    $get_times_yesterday->locations = $locs;

    // page names taken from how you entered them in Pingdom
    $pages_to_query = array('Homepage', 'Krankheiten', 'Asthma article');

    $responses = array();

    foreach ($pages_to_query as $checkName) {
        $response = null;

        $get_times_today->checkName = $checkName;
        $get_times_yesterday->checkName = $checkName;

        $times_today = $client->Report_getResponseTimes(PINGDOM_API_KEY, $sessionId, $get_times_today);
        $times_yesterday = $client->Report_getResponseTimes(PINGDOM_API_KEY, $sessionId, $get_times_yesterday);

        if (PINGDOM_API_STATUS_OK != $times_today->status ||
            PINGDOM_API_STATUS_OK != $times_yesterday->status) {
            echo('Error occurred while trying to get list of responsetimes');
        } else {
            $list_times_today = $times_today->responseTimesArray;
            $list_times_yesterday = $times_yesterday->responseTimesArray;

            $response->page = $checkName;
            $response->today = round($list_times_today[0]->responseTime);
            $response->yesterday = round($list_times_yesterday[0]->responseTime);

            $response->y_threshold = $response->yesterday < GOOD_TIMING ? 'green' : ($response->yesterday < POOR_TIMING ? 'orange' : 'red');
            $response->t_threshold = $response->today < GOOD_TIMING ? 'green' : ($response->today < POOR_TIMING ? 'orange' : 'red');

            $responses[] = $response;
        }
    }

    return $responses;
}

function writeResponse($responses) {
    $yesterday = date('d.m.Y', mktime(0, 0, 0, date("m") , date("d") - 1, date("Y")));
	$current = date('d.m.Y', CURRENT_TIME);

    echo "<br/><br/>";
    echo "<table class='responses' title='Response time (ms)' border='1'>";
    echo "    <tr>";
    echo "        <th> Response Times </th>";
    echo "        <th> $yesterday </th>";
    echo "        <th> $current </th>";
    echo "    </tr>";

    foreach ($responses as $response) {
        $response_page = $response->page;
        $response_yest_thresh = $response->y_threshold;
        $response_toda_thresh = $response->t_threshold;
        $response_yesterday = $response->yesterday;
        $response_today = $response->today;

        echo "    <tr>";
        echo "        <td>$response_page</td>";
        echo "        <td><span style='color:$response_yest_thresh;'>$response_yesterday ms</span></td>";
        echo "        <td><span style='color:$response_toda_thresh;'>$response_today ms</span></td>";
        echo "    </tr>";
    }

    echo "</table>";
}

function logout($client, $sessionId) {
    $logout_response = $client->Auth_logout(PINGDOM_API_KEY, $sessionId);
    if (PINGDOM_API_STATUS_OK != $logout_response->status) {
        echo('Error occurred while closing connection');
    }
}
?>

Of course, there are a few customizations you should make - namely the $locs, $pages_to_query, GOOD_TIMING & POOR_TIMIING (both measured in msec) values. But, I hope I’ve gotten you off to a good start.

Questions, comments & criticisms happily taken!

No related posts.