AUTOMATIC REGRESSION TESTING IN JAVASCRIPT

3 FEBRUARY, 2012 — MENLO PARK, CA

I don’t like writing test cases, and at Proxino I don’t write them — my browser does. I built a framework to automate our testing, and so to generate 800 test cases, I do this:

AngelJS.testcases = 200
// I want to test add_all, my_prop_is_true, fullname, 
// and join_char
AngelJS.test = 
  [{ func: add_all, 
    args: [AngelJS.arr("number")], 
    ret: "number"
  },
  { func: my_prop_is_true, 
    args: [{name:"string", valid:"boolean"}], 
    ret: "boolean"
  },
  { func: fullname, 
    args: [{first:"string", last:"string"}], 
    ret: "string"
  },
  { func: join_char,
    args: ["char", "char"],
    ret: "char"
  }];

The property AngelJS.testcases determines how many test cases to generate for each tested function. Then I assign AngelJS.test an array of objects, each of which contains three properties:

  1. func references the function I want to test
  2. args contains a typed “argument template”
  3. ret defines the desired return type.

This argument template is a type signature [1] which I use to generate reasonable test cases. The test framework repeatedly constructs parameters from the template, and on each repetition, passes these parameters to the function to form a test case. For instance:

[{name:"string", valid:"boolean"}]
// might generate {name:"string1", valid:true}
// or {name:"string2", valid:false} 
// or {name:"string3", valid:true} 
// and so on.

Of course I don’t actually generate “string[n].” The framework builds strings randomly (unless told otherwise). Another example:

["char", "char"]
// might generate ["a","b"]
// or ["5","y"]
// or ["/","l"]
// and so on.

These argument templates can define arbitrarily deep types.

[{name: "string", 
  contacts: AngelJS.arr({
    name: "string", 
    address: "string", 
    phone: "number"})}]
// Might generate
[{ name: "string1", 
  contacts: [
    { name: "string2", 
      address: "string3", 
      number: 7036985777},
    { name: "string4", 
      address: "string5", 
      number: 999899999 }
    ]}]

I use AngelJS.arr(type) to generate arbitrarily long parameter arrays of a given type (whereas the JavaScript array notation [] specifies fixed-length tuples). Array length is pseudo-random, and can be adjusted in the framework.

AngelJS.arr("string")
// Might generate ["string1","string2"]
// or ["string1","string2","string3","string4"]
// or []

Similarly:

[AngelJS.arr("string"),AngelJS.arr("number")]
// might generate [["string1", "string2"], [3456,34,0,45]]

This framework catches two classes of JavaScript errors. If a tested function throws an exception, it will report that (obviously). Otherwise, if a function return value does not match its expected return type, then it records a type error.

Here’s some sample output:

Type Error: add_all: Expected "number" and returned 
  "undefined" on input: [[]] 
Type Error: my_prop_is_true: Expected "boolean" and returned 
  "string" on input: [{"name":"?^|r)z+>>u","valid":true}]
...
Ran 400 cases. Failed 48.
Functions which failed >1 test case: ["add_all", "my_prop_is_true"]

These tests are not be as good as the hand-crafted variety, but they will catch a number of the simplest (and stupidest) bugs. And they’re certainly better than nothing. [2] I’ll probably open source the framework, after I clean up a few things.

1 A type signature in the sense that it defines the type of the expected arguments. I did not add proper typing to JavaScript. The types I support are: “number”, “string”, “char”, “array”, and “object”. Arrays and objects can be nested within on another, to form types like [[“string”],[“number”]] or {name:[“string”]}.

2 I’ve found that “no testing” is something of a convention among startups.