It all began a year ago. We wanted to introduce a new navigation bar, but gradually, gaining feedback (and confidence) that it was the right upgrade for our users. My idea at that time was a random cookie which would show 20% of our users the new navigation while giving them a separate google analytics code for tracking their visit. And it worked well. Within a few weeks we were able to increase the cookie up to 100% and then simply throw it away.
GUI Based A/B testing in a custom CMS
But the idea remained. We now had a successful prototype for A/B testing. The next hurdle was introducing it into the CMS. Implementing a new page in the code for every hairbrained, well-intentioned GUI change is not exactly exciting stuff for web developers to do. But, this is exactly what our half-assed solution required. A product manager added a new A/B test to the backlog for a developer to implement. How about introducing
/page/B/ params to the CMS system?
Development was fairly straightforward. A single new table in the DB to maintain the new A/B mapping, and a few day’s development later, the product management and editorial staff were able to create their own A/B tests!
The next question was how to ensure we served A/B pages as requested, not to mention the myriad of edge cases (B page requests for pages that only existed in A form, etc.). We needed acceptance tests.
Webrat saves the day (again)
As our backend system is behind an SSL authentication mechanism, Mechanize traps the 401 code and we authenticate with our own
do_visit (patch is here). Our build server is a Debian Lenny using Ruby 1.8.7 and Webrat 0.6.0.
Now that we have stable access to the CMS, we can write a webrat test for our A/B testing functionality. Let’s create a B page:
def test_1_cms_create_B_Case do_visit($cms_url) assert_contain "Login" assert_not_contain(PHP_ERROR) fill_in "username", :with => "xyz" fill_in "password", :with => "123" click_button "Login" # create Test Case B in CMS do_visit($cms_url + 'structure/page-edit/2366/page/B/') assert_contain "caseId=B" assert_not_contain(PHP_ERROR) end
Let’s visit the frontend and ensure we see the B page after overwriting the cookie:
def test_2_visit_demo_page_B do_visit($net_url + '/Demo/') assert_not_contain(PHP_ERROR) # overwrite the pageAB cookie to 'B' cookie_jar = webrat.adapter.mechanize.cookie_jar url = URI.parse($net_url) cookie_jar.to_a.each do |cookie| if cookie.name == 'pageAB' cookie.value = 'B' our_cookie = cookie.clone() cookie_jar.add(url, our_cookie); break; end end page = do_visit($net_url + 'ND-intern/Demo/') assert_not_contain(PHP_ERROR) # Verify we get a B page with our B cookie! unless page.root().to_s.include? 'pageB' assert false end end
And that’s it. Of course, you need to do some extra testing for the boundary cases, but I’ll leave this as an excercise for the reader …
Have any webrat tricks you’d like to share with us? Let us know in the comments!