Swab

Swab is a simple WSGI A/B testing tool.

Usage:

 1. Define an experiment:

 from swab import Swab
 s = Swab('/tmp/.swab-test-data')
 s.addexperiment('button-size', ['default', 'larger'], 'order-completed')

 Here we have an experiment called 'button-text', for testing two sizes of
 button on a form.

 The two variants the experment are called 'small' and 'large'. The first
 listed is always taken as the control variant, and results will be
 presented in terms of how much better or worse the other variants have
 performed compared with the control

 The goal associated with the test is called 'order-completed'.

 2. Take a WSGI application:

 from swab import record_goal, show_variant

 def get_button_style(environ):
 if show_variant('button-size', environ) == 'small':
 return 'font-size: 100%'
 else:
 return 'font-size: 150%'

 def myapp(environ, start_response):

 if environ['REQUEST_METHOD'] == 'GET':
 start_response('200 OK', [])
 return ['''
 < html >
 < body >
 < form method="POST" >
 < button style='%s' type="submit" >Complete your order< /button >
 < /form >
 < /body >
 < /html >''' % (get_button_style(environ),)
 ]

 if environ['REQUEST_METHOD'] == 'POST':
 swab.record_goal('order-completed', environ)

 3. Wrap it in the swab middleware:

 app = s.middleware(myapp)

4. Let it run, and after a while browse to http://< server >/swab/results to see which variant is winning.

Testing

To test your competing pages, start Swab in debug mode:

swab = Swab('/tmp/.swab-test-data', debug=True)

Then append '?swab.< experiment-name >=< variant-name >' to URLs to force one variant or another to be shown.

Disable debug mode in production to avoid people messing about with your figures.

Basic design

Each visitor is assigned an identity which is persisted by means of a cookie. The identity is a base64 encoded randomly generated byte sequence. This identity is used as a seed for a RNG, which is used to switch visitors into test groups.

Every time a test is shown (triggered by the show_variant function), a line is entered into a file at < datadir >/< experiment >/< variant >/__all__

Every time a goal is recorded (triggered by the record_goal function), a line is entered into a file at < datadir >/< experiment >/< variant >/< goal >

Each log line has the format :\n.

No file locking is used: it is assumed that this will be run on a system where each line is smaller than the fs blocksize, allowing us to avoid this overhead. The lines may become interleaved, but there should be no risk of corruption even with multiple simultaneous writes. See http://www.perlmonks.org/?node_id=486488 for a discussion of the issue.

Statistical analysis

The further reading section contains links to the various articles that helped me implement the algorithms and statistical formulae used in Swab. However I don't have a background in maths or stats and I would welcome contributions or corrections from anyone with a deeper understanding of these tests.

What is new in this release:

• Added a javascript beacon to record tests (helps exclude bots)
• Better exclusion of bots on server side too
• Record trial app won't raise an error if the experiment name doesn't exist
• Removed debug flag, the ability to force a variant is now always present
• Strip HTTP caching headers if an experiment has been invoked during the request
• Improved accuracy of conversion tracking
• Cookie path can be specified in middleware configuration

What is new in version 0.1.2:

• Minor bugfixes

What is new in version 0.1.1:

• Bugfix for ZeroDivisionErrors when no data has been collected

Requirements:

• Python