Better BDD – a Javascript Protractor experiment – Building a test framework organically

So almost two years in to my test manager role, I seem to have freed up time to get back on the tools somewhat. While there are a huge number of problems to solve, I first decided to have a look at protractor for a few reasons:

  • I didn’t feel we were getting much value out of our UI tests.
  • I wanted to see how my automation principles transferred to javascript.
  • I wanted to have a concrete example to illustrate some points I discuss frequently with my team.
  • Putting this out there will no doubt get a bunch of people telling me how I can do this better (win)!
  • I still have huge misgivings regarding BDD and have vaguely promised to document something. This goes some way toward addressing that (Michele).

With those thoughts in mind, I started with the Protractor tutorial.

Setting up

I had installed node ages ago, so there were a few bumps in the setup due to how admin rights are given. There were some errors out of protractor (lost from my shell session now) and I went home to install in order to avoid proxy annoyances.

Thus:

Running in powershell (Windows 8):

Set-ExecutionPolicy Unrestricted -Scope CurrentUser -Force
npm install -g npm-windows-upgrade
.\webdriver-manager update
.\npm-windows-upgrade

Bits of node went to different places based on my setup. I had to look in here sometimes: C:\Users\\AppData\Roaming\npm

Round 1:

I have pasted the tutorial example code in to my editor:


// spec.js

describe('Protractor Demo App', function() {
  it('should add one and two', function() {
    browser.get('http://juliemr.github.io/protractor-demo/');
    element(by.model('first')).sendKeys(1);
    element(by.model('second')).sendKeys(2);

    element(by.id('gobutton')).click();

    expect(element(by.binding('latest')).getText()).toEqual('5'); // This is wrong!
  });
});

My skin is crawling a bit, because even if I think specification by example is a useful idea (marginally, sometimes), I want to describe the general behaviour first. To do this, I need to write my test so it’s about the general case (Adding integers) rather than the specific case of adding one and two.:


// spec.js

describe('Protractor Demo App', function() {
   it('should add two integers', function() {
     browser.get('http://juliemr.github.io/protractor-demo/');
     element(by.model('first')).sendKeys(Math.random()*1000);
     element(by.mode l('second')).sendKeys(Math.random()*1000 );
    element(by.id('gobutton')).click();

     expect(element(by.binding('latest')).getText()).toEqual('5');
   });
});

I get distracted by some random thoughts around reporting and cross-browser testing and how have:


// conf.js

var reporters = require('jasmine-reporters');
var junitReporter = new reporters.JUnitXmlReporter({
   savePath: 'c:/js_test/protractor/',
   consolidateAll: false
});

exports.config = {
   seleniumAddress: 'http://localhost:4444/wd/hub',
   specs: ['spec.js'],
   capabilities: {
     browserName: 'chrome'
  },
   onPrepare: function() {
     jasmine.getEnv().addReporter(junitReporter)}
}

Round two:

I have some duplication in the random function and know I will need to learn how to externalise libraries and such, so I attempt to create a random function:


// spec.js
helper= require('./libs/helper.js');

describe('Protractor Demo App', function() {
  it('should add two integers', function() {
    browser.get('http://juliemr.github.io/protractor-demo/');
    first=helper.randomInt(1000);
    second=helper.randomInt(1000);
    expected=Math.floor(first+second)+'';
    
    element(by.model('first')).sendKeys(first);
    element(by.model('second')).sendKeys(second);
    element(by.id('gobutton')).click();

    expect(element(by.binding('latest')).getText()).toEqual(expected);
  });
});

In the helper.js file, I tried a few different ways to export, and settled on this as my first version because it works and is less typing.


//helper.js
module.exports = {
    randomInt: function (max) {
     return Math.floor(Math.random() * (max)) + 1;
    }
}

Round 3 – Second test

I start by copying and pasting the add spec as subtract. I pull the initial duplicated browser.get out into the beforeEach function, and then I have to figure out how to click on the dropdown. Like most frameworks designed to let developers think in terms of their UI building tools (I’m looking at you Geb), it is a pain to do some simple things because accessing individual elements of a dropdown list is something you probably never have to do when you are building a UI. The actual browser automation framework is kind of an afterthought. Eventually, I stumble across a tidy solution, and I now have two specs. I also factor out the common ‘calculate’ action and make it part of a calculator model.


// spec.js

helper= require('./libs/helper.js');
calculator=require('./libs/calculator.js');

describe('Protractor Demo App', function() {
    
  beforeEach(function() {
    browser.get('http://juliemr.github.io/protractor-demo/');
  });
    
  it('should add two integers', function() {
    first=helper.randomInt(1000);
    second=helper.randomInt(1000);
    expected=Math.floor(first+second)+'';
    
    element(by.model('first')).sendKeys(first);
    element(by.model('second')).sendKeys(second);
    calculator.calculate();

    expect(element(by.binding('latest')).getText()).toEqual(expected);
  });
    
  it('should subtract two integers', function() {
    first=helper.randomInt(1000);
    second=helper.randomInt(1000);
    expected=Math.floor(first-second)+'';
    
    element(by.model('first')).sendKeys(first);
    element(by.model('second')).sendKeys(second);
    operators=element(by.model('operator')).$('[value="SUBTRACTION"]').click();
    calculator.calculate();

    expect(element(by.binding('latest')).getText()).toEqual(expected+1);
  });
});


//calculator.js
var calculate = function () {
     return element(by.id('gobutton')).click();
    }

module.exports.calculate = calculate;

I get on a bit of a roll at this point and keep adding methods to the calculator:


// spec.js
helper= require('./libs/helper.js');
calculator=require('./libs/calculator.js');

describe('Protractor Demo App', function() {
    
  beforeEach(function() {
    browser.get('http://juliemr.github.io/protractor-demo/');
  });
    
  it('should add two integers', function() {
    first=helper.randomInt(1000);
    second=helper.randomInt(1000);
    expected=Math.floor(first+second)+'';
    calculator.add(first,second);
    expect(calculator.last_calculation()).toEqual(expected);
  });
    
  it('should subtract two integers', function() {
    first=helper.randomInt(1000);
    second=helper.randomInt(1000);
    expected=Math.floor(first-second)+'';
    calculator.subtract(first,second);
    expect(calculator.last_calculation()).toEqual(expected);
  });
});

I also refactor somewhat prematurely and drive out duplication in the new calculator methods:


var calculate = function () {
     return element(by.id('gobutton')).click();
    }

var set_first = function (first) {
     element(by.model('first')).sendKeys(first);
    }

var set_second = function (second) {
     element(by.model('second')).sendKeys(second);
    }

var operation = function (first, second, operation) {
     set_first(first);
     set_second(second);
     element(by.model('operator')).$('[value="' + operation + '"]').click();
     calculate();
    }

var add = function (first, second) {
    operation(first,second,'ADDITION');
}

var subtract = function (first, second) {
    operation(first,second,'SUBTRACTION');
}
    
var last_calculation = function () {
     return element(by.binding('latest')).getText();
    }

module.exports.calculate = calculate;
module.exports.set_first = set_first;
module.exports.set_second = set_second;
module.exports.add = add;
module.exports.subtract = subtract;
module.exports.last_calculation = last_calculation;

Cleaning up

I move the ‘browser.get’ in the beforeEach function into the calculator app model. With the refactoring I made to remove duplication in add and subtract, it’s trivial to add all of the other calculator tests. I also factor out the data into a calculation object in order to remove duplication. The helper file vanishes from the main spec as it is only used by the data functions.


// spec.js

calculator=require('./libs/calculator.js');
data = require('./libs/calc_data.js');

describe('Integer math', function() {
    
  beforeEach(function() {
    calculator.open();
  });

  it('should add two integers', function() {
    addition= new data.addition;
    calculator.add(addition.first,addition.second);
    expect(calculator.last_calculation()).toEqual(addition.result);
  });
    
  it('should subtract two integers', function() {
    subtraction= new data.subtraction;
    calculator.subtract(subtraction.first,subtraction.second);
    expect(calculator.last_calculation()).toEqual(subtraction.result);
  });
    
  it('should multiply two integers', function() {
    multiplication = new data.multiplication;
    calculator.multiply(multiplication.first,multiplication.second);
    expect(calculator.last_calculation()).toEqual(multiplication.result);
  });

  it('should divide two integers', function() {
    division = new data.division;
    calculator.divide(division.first,division.second);
    expect(calculator.last_calculation()).toEqual(division.result);
  });

  it('should get modulus of two integers', function() {
    modulo = new data.modulo;
    calculator.modulo(modulo.first,modulo.second);
    expect(calculator.last_calculation()).toEqual(modulo.result);
  });
    
});

Now that I’ve factored out common code, adding all five operations is trivial.

//calculator.js

var open = function () {
     browser.get('http://juliemr.github.io/protractor-demo/');
    }

var calculate = function () {
     return element(by.id('gobutton')).click();
    }

var set_first = function (first) {
     element(by.model('first')).sendKeys(first);
    }

var set_second = function (second) {
     element(by.model('second')).sendKeys(second);
    }

var operation = function (first, second, operation) {
     set_first(first);
     set_second(second);
     element(by.model('operator')).$('[value="' + operation + '"]').click();
     calculate();
    }

var add = function (first, second) {
    operation(first,second,'ADDITION');
}

var subtract = function (first, second) {
    operation(first,second,'SUBTRACTION');
}

var multiply = function (first, second) {
    operation(first,second,'MULTIPLICATION');
}

var divide = function (first, second) {
    operation(first,second,'DIVISION');
}

var modulo = function (first, second) {
    operation(first,second,'MODULO');
}

var last_calculation = function () {
     return element(by.binding('latest')).getText();
    }

module.exports.open = open
module.exports.calculate = calculate;
module.exports.set_first = set_first;
module.exports.set_second = set_second;
module.exports.add = add;
module.exports.subtract = subtract;
module.exports.multiply = multiply;
module.exports.divide = divide;
module.exports.modulo = modulo;
module.exports.last_calculation = last_calculation;


//calc_data.js

helper = require('./helper.js');

var addition = function () {
    this.first = helper.randomInt(1000);
    this.second = helper.randomInt(1000);
    this.result = Math.floor(this.first+this.second)+'';
    }

var subtraction = function () {
    this.first = helper.randomInt(1000);
    this.second = helper.randomInt(1000);
    this.result = Math.floor(this.first-this.second)+'';
    }

var multiplication = function () {
    this.first = helper.randomInt(1000);
    this.second = helper.randomInt(1000);
    this.result = Math.floor(this.first*this.second)+'';
    }

var division = function () {
    this.first = helper.randomInt(1000);
    this.second = helper.randomInt(1000);
    this.result = (this.first/this.second)+'';
    }

var modulo = function () {
    this.first = helper.randomInt(1000);
    this.second = helper.randomInt(1000);
    this.result = (this.first%this.second)+'';
    }

module.exports.addition = addition;
module.exports.subtraction = subtraction;
module.exports.multiplication = multiplication;
module.exports.division = division;
module.exports.modulo = modulo;

And I stop here, with a few thoughts (and the source code on github if you want it).

I’m building a parallel implementation of a calculator

A big part of testing is about bringing two (or more) models into alignment. One is the implementation. The other is the model (or models), formal or informal that you use to check the product against. For something like a calculator, specification by example is a pretty poor approach, so comparing it to some reference implementation seems like a better idea, which is where this test implementation seems to be heading.

Specification by example isn’t useful everywhere

If you are building a mathematical function, specifying the result by example probably isn’t a great approach compared to a complete test against a reference model for all inputs (if feasible) or a high-volume randomised test. Imagine trying to specify a sine function by example…

Page objects are overused

There’s very little duplication of UI implementation in these tests, which is what normally happens unconsciously when I build a framework around a goal or activity-oriented way. I tend to do this because I am looking for a stable model to align my test artefacts to, and implementation is rarely so.

Examples can help though…

Once the suite runs, it feels like it would be helpful to know what values were provided to the test if, for example, I wanted to see if a particular case had ever been executed by the for framework. At some point I will try and hack the reporter to report the actual values executed.

Attention to design should help you go faster

As you build more and more useful abstractions, you should find it easier to add new tests over time. You only get this through mercilessly refactoring, which is slower at the beginning.

Cucumber probably still sucks
It feels like doing this in cucumber (as opposed to something code-y like RSpec) would’ve made a lot of things more difficult and led to many more layers, especially the need to abstract data and pass it around.

Things to do

  • See if I can get the Jasmine Tagged library working to get a richer reporting model. A single test hierarchy is pretty limiting, and SerenityBDD style reports are a step up from xUnit.
  • These tests will need to run in multiple environments with various configurations of real and fake components, as well as with different datasets. At some point I need to build this.’
  • Fix the duplication in the calculator data by creating a more generic calculation object (two numbers plus an operation)
  • Implement the history tests.
  • Implement the weird behaviours around Infinity that I found through exploratory testing.
  • Move it to something non-javascript so it’s a real independent oracle.
  • Fix my wordpress site to format code properly. Sorry it’s so ugly!
Tags: , , ,

One comment on “Better BDD – a Javascript Protractor experiment – Building a test framework organically”

  1. Pingback: Testing Bits – 8/14/16 – 8/20/16 | Testing Curator Blog

Leave a Reply

Your email address will not be published. Required fields are marked *

About me

I’m Jared Quinert, a testing consultant located in Melbourne, Australia. With thirty years of experience, I specialise in agile testing, context-driven testing and intelligent toolsmithing with a focus on business outcomes over process. As one of the most experienced agile testers in Australia, I’ve been diving in hands-on since 2003 to discover how to build successful whole-team approaches to software development.

If you’re a tester and struggling to find a job in the current market, I may be able to help you improve your chances (as I’ve already done for others). Get in touch!

Contact Me