When it comes to testing workflows with SystemJS, most test frameworks and libraries don't actually come with existing instructions for how to get started with testing using SystemJS.

This post runs through a simple method for using Mocha with SystemJS test files in Node and the browser. We then demonstrate how to apply Istanbul coverage instrumentation to SystemJS to get full coverage reports on original sources back from tests running in both Node and the browser.

The methods shown here should help you to create testing workflows that work in a SystemJS way. All the examples below come with separate Vanilla SystemJS and jspm 0.17-beta configuration sections depending on whether or not you're using jspm as well. If not using jspm, you can easily skip the jspm sections. If using jspm it's advisable to read the Vanilla SystemJS sections anyway.

This post as well as the development of the coverage workflow was made possible by sponsorship from One.com.

Using Mocha with SystemJS

For the example app, let's say we have a super interesting counter module -

counter.js

export class Counter {
  constructor() {
    this.value = 0;
  }
  increment() {
    return ++this.value;
  }
}

We want to be able to test this code with tests that are themselves written as ES6 modules taking advantage of language features like async functions:

tests.js

import expect from 'unexpected';

export let SimpleTestSuite = {
  'First Test': async function() {
    let Counter = (await SystemJS.import('counter.js')).Counter;
    let c = new Counter();
    expect(c.increment(), 'to be', 1);
    expect(c.increment(), 'to be', 2);
  }
};

In the above example, we dynamically import the module being tested within the test itself with SystemJS.import('counter.js'). The use of async functions allows the test code running this dynamic import promise chain to be written succinctly.

So, how can we run the above tests.js file with full SystemJS support in Mocha?

Vanilla SystemJS Configuration

In order to get the tests to run we will need to install the following npm dependencies:

npm install mocha systemjs systemjs-plugin-babel unexpected

To wire these up, we create a SystemJS configuration file setting up our transpiler and module paths:

system.config.js

SystemJS.config({
  transpiler: 'plugin-babel',
  map: {
    'plugin-babel': 'node_modules/systemjs-plugin-babel/plugin-babel.js',
    'systemjs-babel-build': 'node_modules/systemjs-plugin-babel/systemjs-babel-browser.js',
    'unexpected': 'node_modules/unexpected/unexpected.js',
  }
});

Finally we can create our main test runner code:

runner.js

var SystemJS = require('systemjs');
var Mocha = require('mocha');

SystemJS.import('./system.config.js')
.then(function() {
  return SystemJS.import('./tests.js');
})
.then(function(tests) {
  var runner = new Mocha({ ui: 'exports' });

  runner.suite.emit('require', tests);

  return new Promise((resolve, reject) => {
    runner.run((failures) => {
      if (failures)
        reject(failures);
      else
        resolve();
    });
  });
})
.catch(console.error.bind(console));

The above uses the NodeJS require system to load SystemJS and Mocha, and then uses SystemJS to load further its own configuration and the tests.

The important thing here is that we are loading the tests themselves through SystemJS, which we then pipe into Mocha using the exports ui.

This is the UI that allows us to write the tests.js above as an exports object where the suites are export names and the object properties tests.

The full tests can then be run in Node via:

node runner.js

giving the output:

  SimpleTestSuite
    ✓ First Test (46ms)

  1 passing (54ms)

BDD Tests

We're writing these tests with an exports interface instead of a bdd describe(...) interface in Mocha because the bdd interface assumes the existence of global describe and it variables in the execution scope. To use the bdd style, use the alternative approach: 

var SystemJS = require('systemjs');
var Mocha = require('mocha');

SystemJS.import('./system.config.js')
.then(function() {
  var runner = new Mocha({ ui: 'bdd' });

  // set up the global variables
  runner.suite.emit('pre-require', global, 'global-mocha-context', runner);

  return SystemJS.import('./tests.js')
  .then(function(tests) {
    return new Promise((resolve, reject) => {
      runner.run((failures) => {
        if (failures)
          reject(failures);
        else
          resolve();
      });
    });
  });
})
.catch(console.error.bind(console));

jspm Configuration (0.17-beta)

In a new jspm project, we can get straight to installing mocha and unexpected:

jspm install mocha unexpected --dev

jspm then auto-generates the configuration file, so we just need to write our runner.js file now. This can now be written as an ES module directly as everything is now running through the SystemJS import mechanism:

runner.js

import Mocha from 'mocha';

SystemJS.import('./tests.js')
.then(function(tests) {
  var runner = new Mocha({
    ui: 'exports',
    // this line is what will allow this runner to work in both the browser and Node
    reporter: typeof window != 'undefined' ? 'html' : 'spec'
  });

  runner.suite.emit('require', tests);

  return new Promise((resolve, reject) => {
    runner.run((failures) => {
      if (failures)
        reject(failures);
      else
        resolve();
    });
  });
})
.catch(console.error.bind(console));

We could have equally used import tests from './tests.js' in the above, but the use of the dynamic SystemJS.import will make the code ready for adding code coverage shortly.

To execute this module in jspm, use:

jspm run runner.js

which runs the code as a top-level SystemJS.import itself with all jspm resolution rules applied, giving the same test output as expected above.

Running Mocha tests in the Browser

We can then take these same tests and run them directly in the browser.

Vanilla SystemJS

To run the vanilla SystemJS tests, we load Mocha and its CSS into the page, as well as SystemJS and its configuration file:

test-browser.html

<!doctype html>
<link rel="stylesheet" href="node_modules/mocha/mocha.css" />
<body>
  <div id="mocha"></div>
<script src="node_modules/systemjs/dist/system.src.js"></script>
<script src="node_modules/mocha/mocha.js"></script>
<script src="system.config.js"></script>
<script>
  SystemJS.import('./tests.js').then(function(tests) {
    var runner = new Mocha({ ui: 'exports', reporter: 'html' });
    runner.suite.emit('require', tests);
  });
</script>

The tests are imported by SystemJS and run the same as we did in the runner.js file, but now adjusted for the browser.

jspm

The jspm tests are already universal - they can be directly imported in the browser with the standard jspm template:

test-browser.html

<!doctype html>
  <script src="jspm_packages/system.src.js"></script>
  <script src="jspm.config.js"></script>
  <div id="mocha"></div>
  <script>
    SystemJS.import('./runner.js');
  </script>

Generating Istanbul Coverage Reports

Now that all the tests are running through SystemJS, we can look at adding Istanbul coverage into SystemJS to generate coverage reports.

The way we handle this is via extending the SystemJS translate hook to add the Istanbul instrumentation as the final step after all other SystemJS compilations.

There is some subtlety to this process, so there's an experimental SystemJS helper we will use, systemjs-istanbul-hook, which provides the custom SystemJS translate hook for us.

systemjs-istanbul-hook supports the ability to get full source maps support back to the original files loaded by SystemJS, regardless of transpilation or plugins using the remap-istanbul project.

Vanilla SystemJS Coverage

Continuing from the previous vanilla setup with Mocha for testing, we install the hook:

npm install systemjs-istanbul-hook

This helper can then be used to extend SystemJS with the following code in the previous runner.js (the commented lines are the ones that have changed):

runner.js

var SystemJS = require('systemjs');
var Mocha = require('mocha');

// extend the SystemJS translate hook with coverage instrumentation
var systemIstanbul = require('systemjs-istanbul-hook');
systemIstanbul.hookSystemJS(SystemJS);

var fs = require('fs');

SystemJS.import('./system.config.js')
.then(function() {
  return SystemJS.import('./tests.js');
})
.then(function(tests) {
  var runner = new Mocha({ ui: 'exports' });

  runner.suite.emit('require', tests);

  return new Promise((resolve, reject) => {
    runner.run((failures) => {
      if (failures)
        reject(failures);
      else
        resolve();
    });
  });
})
.then(function() {
  // after running the tests, save the coverage file
  var fs = require('fs');
  fs.writeFileSync('coverage.json', JSON.stringify(systemIstanbul.remapCoverage(__coverage__)));
})
.catch(console.error.bind(console));

Having extended SystemJS with coverage instrumentation, we then run the tests as before. After running the tests, the __coverage__ global contains the reported coverage for all code imported by SystemJS. We remap the source maps in this generated global coverage object to get coverage reports on the original source files, which is saved locally.

The report for this coverage can then be generated by running (after an npm install -g istanbul):

istanbul report

and viewed at coverage/lcov-report/index.html.

Coverage generation takes time, so to exclude files from coverage instrumentation, an optional second argument can be added to the hook containing a filter:

systemIstanbul.hookSystemJS(SystemJS, function exclude(address) {
  // exclude adding coverage to anything not in the tests folder
  if (!address.startsWith(SystemJS.baseURL + 'tests/'))
    return false;
});

jspm Coverage

With jspm, it is still best to install systemjs-istanbul-hook via npm:

npm install systemjs-istanbul-hook

While Istanbul can be made to work correctly through jspm install, remap-istanbul can't work this way due to the use of unsupported AMD define plugin patterns.

We can then instrument runner.js in a similar way:

import Mocha from 'mocha';

// extend the SystemJS translate hook with coverage instrumentation
import systemIstanbul from '@node/systemjs-istanbul-hook';
systemIstanbul.hookSystemJS(SystemJS);

import fs from '@node/fs';

SystemJS.import('./tests.js')
.then(function(tests) {
  var runner = new Mocha({
    ui: 'exports',
    reporter: typeof window != 'undefined' ? 'html' : 'spec'
  });

  runner.suite.emit('require', tests);

  return new Promise((resolve, reject) => {
    runner.run((failures) => {
      if (failures)
        reject(failures);
      else
        resolve();
    });
  });
})
// after running the tests, save the coverage file
.then(function() {
  fs.writeFileSync('coverage.json', JSON.stringify(systemIstanbul.remapCoverage(__coverage__)));
})
.catch(console.error.bind(console));

We use the @node/fs and @node/systemjs-istanbul-hook syntax to load a NodeJS module with the NodeJS module resolution through SystemJS (although note that adding these imports to our code makes it no longer compatible with running in the browser).

To view the coverage report:

jspm run runner.js && istanbul report && open coverage/lcov-report/index.html

again assuming npm install -g istanbul to allow report generation.

Browser Coverage Reports

To support browser-based coverage generation, the easiest method is to use SystemJS Builder to create a bundle that we run in the browser, with the istanbul instrumentation hook applied to SystemJS Builder before bundling.

This way we can get the exact same source maps support to original files for the coverage, provided we wire the coverage JSON back for remapping.

The wiring here is a little involved depending on the exact workflow, so an outline is given that can be adapted depending on your exact use case.

Instrumenting Browser Bundles

1. Instrumenting the SystemJS Builder loader at bundling time with coverage:

// var Builder = require("jspm").Builder // for jspm
var Builder = require('systemjs-builder');

var builder = new Builder();

// hook the builder loader with the coverage instrumentation
var systemIstanbul = require('systemjs-istanbul-hook');
systemIstanbul.hookSystemJS(builder.loader);

  // unnecessary with jspm
  builder.loadConfigSync('./system.config.js');

// having hooked the coverage, bundle the app and tests
builder.bundle('./tests.js + counter.js', 'browser-coverage-bundle.js')
.then(function() {
  // save the original source mapping data for remapping in a separate process
  // if remapping happens in the same process this is unnecessary of course
  var originalSources = systemIstanbul.originalSources;
  fs.writeFileSync('original-source-data.json', JSON.stringify(originalSources));
});

2. Having run the bundled code in the browser, we retrieve the window.__coverage__ global object and have it sent back to the server for remapping:

var systemIstanbul = require('systemjs-istanbul-hook');

// this implementation detail is left out
return retrieveBrowserCoverageObject()
.then(function(browserCoverage) {
  var originalSources = JSON.parse(fs.readFileSync('original-source-data.json'));

  // remap the coverage to the original sources
  var mappedCoverage = systemIstanbul.remap(browserCoverage, originalSources);
  fs.writeFileSync('browser-coverage.json', JSON.stringify(mappedCoverage));
});

We then generate the report from the coverage.json with istanbul report just as before.

The originalSources object of systemIstanbul is a local cache of the source and source map information that it picks up during execution of the SystemJS translate hook. It is the combination of this originalSources and and the __coverage__ object in the browser that is remapped by the remapCoverage function back into coverage data on the original sources loaded by SystemJS.

If we'd instrumented the bundle directly without a hook mechanism like this we'd still get coverage, but on the generated bundle code instead, so this provides the highest parity method for obtaining flexible coverage reports regardless of what we load through SystemJS.

For more information on the approach, see the systemjs-istanbul-hook project page.

Compile server alternative

There is another possible workflow here and that is to send precompiled module files to the browser that are individually instrumented. Instead of creating a single bundle, we create a server that sends compiled modules to SystemJS.

For an example of how a precompilation server approach can work, see the example server, although note that this approach is highly experimental.

Avoiding Catches with Paths configuration

One catch that is worth mentioning is that in most real world examples you would have a test folder containing the tests that is contained within the main project.

When this happens, we want to reference node_modules and app from the directory below the test folder. This simple pathing change can make life very complicated without knowing the right way to handle pathing configuration for SystemJS in this case.

As a general rule, when running vanilla SystemJS configurations, always configure paths for folder locations, even if they are identity maps:

SystemJS.config({
  paths: {
    'app/': 'app/',
    'node_modules/': 'node_modules/'
  }
});

This will help when bundling as it avoids the Unable to calculate canonical name to ... error, and also makes it much simpler when we want to move paths around.

When moving all the test files into a test folder, we need then only change:

SystemJS.config({
  paths: {
    'app/': '../app/',
    'node_modules/': '../node_modules/'
  }
});

when running in Node.

Paths configuration is environment-specific configuration. So in the vanilla SystemJS configuration case, we don't want to necessarily include this paths configuration in the system.config.js file as it may not be the same between the browser and Node environments.

Instead this is something that can be worth including with a script tag before loading the SystemJS configuration in the browser:

<script>
SystemJS.config({
  paths: {
    'app/': '/served/app/',
    'node_modules/': '/served/node_modules/'
  }
});
</script>

and similarly a separate configuration call in NodeJS.

That's all

Hopefully the above gives some helpful directions on how to approach integration between SystemJS, Mocha and Istanbul.

Questions and feedback are welcome in the comments. If you build something with it please do share what you come up with!