Cypress Automated Tests for Toph

Automated tests are like the brakes of your car. Without brakes, you wouldn’t feel comfortable driving at higher speeds.

The year 2021 started great for Toph in terms of development. Aside from launching some of the community’s favorite features, a lot of work has been done to improve the codebase and internal frameworks.

During this time, it was becoming increasingly infeasible to test Toph manually before deploying updates. Although Toph already has unit tests covering the critical parts (something to be covered in a future engineering blog post), they are not replacements for well-written end-to-end tests.

Testing with Cypress

Cypress seemed like a perfect fit for the job.

It was easy to get started with Cypress. And, once a few tests were written to get a feel of how Cypress works, it was integrated with Toph’s continuous integration (CI) service. Now, the entire suite of automated tests is run before a new version of Toph is deployed.

Seeding MongoDB

To make sure that the tests are run on a known state of the database, the mongo-seeding package has been used. This allows one to seed MongoDB with a predefined set of collections and documents.

With the documents to be seeded stored under the fixtures directory, it takes only a few lines of code to initiate the seeding process.

fixtures
└── seed
    └── mongodb
        ├── 1-configurations
        │   └── index.js
        ├── 1-contestFormats
        │   └── index.js
        ├── 1-languages
        │   ├── bash_5.0.js
        │   ├── c++17_gcc9.2.js
        │   └── java_1.8.js
        ├── 1-quotas
        │   └── index.js
        ├── 2-accounts
        │   ├── beifong.js
        │   └── blookz.js
        ├── 2-contests
        │   ├── ended.js
        │   ├── future.js
        │   ├── private.js
        │   └── running.js
        ├── 2-problems
        │   ├── hello-world.js
        │   └── lorem-ipsum.js
        ├── 3-challenges
        │   └── index.js
        ├── 3-participants
        │   └── index.js
        └── 3-submissions
            └── index.js

As an example, here is what 1-languages/bash_5.0.js looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

const { getObjectId } = require('mongo-seeding')

module.exports = {
	_id: getObjectId('language:c++17_gcc9.2'),
	label: 'C++17 GCC 9.2',
	family: 'C++',
	label_lower: 'c++17 gcc 9.2',
	stack: 'g++9',
	flags: {},
	exts: ['.cpp', '.cc'],
	needs_build: true,
	needs_filename: false,
	syntax: 'cpp',
	code_mirror_mode: 'text/x-c++src',
	initial_template: '#include <iostream>\n\nusing namespace std;\n\nint main() {\n	\n	return 0;\n}\n',
	initial_filename: '',
	scanspec_generator: 'cpp14',
	language_server: 'clangd11/cpp',
	enabled: true,
	notes: '',
	notes_html: '',
	suite: 'GCC 9.2.0',
	hello_toph: '#include <iostream>\n\nusing namespace std;\n\nint main() {\n	cout << "Hello Toph!" << endl;\n	return 0;\n}',
	command: 'g++ -static -s -x c++ -O2 -std=c++17 -D ONLINE_JUDGE hello.cpp -lm\n./a.out\n',
	website: 'https://gcc.gnu.org/',
	deprecated: false,
	experimental: false,
	created_at: new Date(),
	modified_at: new Date()
}

Notice how the document has been defined as a plain JavaScript object.

The getObjectId function makes it easy to generate predictable MongoDB ObjectIDs. For a given string, it will always produce the same ObjectID.

The following few lines of code are wrapped up neatly into a Cypress task so that it can be triggered from anywhere in the test code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

const seeder = new Seeder({
	database: {
		host: process.env.MONGO_HOST || 'localhost',
		port: 27017,
		name: 'toph-arena',
	},
	dropDatabase: true,
	mongoClientOptions: {
		w: 'majority'
	}
})
const collections = seeder.readCollectionsFromPath(path.resolve('cypress/fixtures/seed/mongodb'))
seeder.import(collections)

Writing Tests

One thing that Cypress does best is it gives the feel of synchronous programming language in a naturally asynchronous one. The test code is, thus, readable and easy to follow.

In the following example, you can see how the integrated code editor is tested:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

describe('CodePanel', () => {
	beforeEach(() => {
		cy.task('db:seed')
		faker.seed(0)

		cy.clearLocalStorage()

		cy.login('blookz', '[REDACTED]')
	})

	context('Problem Archive', () => {
		beforeEach(() => {
			cy.visit('/p/lorem-ipsum')
			cy.waitForResources(/codepanel(\.[0-9a-f]+)?\.js/)
			cy.get('.btn-codepanel').click()

			cy.get('.codepanel').should('have.class', 'open')
		})

		// ...

		it('can submit code', () => {
			cy.get('.codepanel select[name="languageId"]').select('Bash 5.0').wait(25)
			const source = faker.lorem.word()
			cy.get('.codepanel .cm-editor .cm-content').focus().type(`echo "${source}"{enter}`, {delay: 25})
			cy.get('.codepanel .btn-submit').click()

			cy.get('#mdlSubmissionContent').should('be.visible')
			cy.get('#mdlSubmissionContent h4').should('contain', 'Submission')
		})
	})
})

For each of the integrated code editor tests, the database is seeded, the random data generator is also set to a known seed, the web browser’s localStorage is cleared, and a user session is started (i.e. logged in).

The test here is to ensure if the user can submit their code through the integrated code editor while solving a problem from the archive.

The test code instructs Cypress to visit a problem page, wait for the integrated code editor to load, click on the button that opens the editor, enter some arbitrary text in the editor, and then click on the submit button.

The behavior that is expected in this test is that once the code is submitted, a modal should open.

Network Requests

For some tests, it makes sense to let network requests go through to the actual backend instance being run for automated tests. In other cases, it makes more sense to intercept those network requests and act based on them.

For example, to test whether the integrated editor can handle programming languages where the user can change the filename for the submission, all that had to be checked is if the API call being made had the right contents. The following code tests exactly that:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

it('can submit code with filename', () => {
	cy.intercept('POST', /\/api\/(problems|challenges)\/[0-9a-f]{24}\/submissions/, req => {
		req.reply(400, 'ErrThrottled')
	}).as('postSubmission')

	cy.get('.codepanel select[name="languageId"]').select('Java 1.8').wait(25)
	const filename = faker.lorem.word() + '.java'
	cy.get('.codepanel .cm-editor .cm-content').focus().type('{selectall}class Hello {\n  static public void main(String args[]) {\n    System.out.println("Hello Toph!");\n  }\n}', {delay: 25})
	cy.get('.codepanel input[name=filename]').should('be.visible').clear().type(filename)
	cy.get('.codepanel .btn-submit').click()

	cy.wait('@postSubmission').its('request.body').should('include', `filename="${filename}"`)
})

In the first three lines of the test, API calls made to the “create submission” endpoint are being intercepted and stubbed with a predefined response. In the very last line of the test, it is being asserted whether the API call was made at all and if it was, does the request body contain the filename as entered through the integrated code editor.

GitLab CI/CD

Making Cypress tests a part of GitLab CI was fairly easy:

• A Docker image was built based on Cypress’s official Docker image, combined with the dependencies needed to start the tophd (webserver) process.
• A test job was added to the test stage of Toph’s existing GitLab CI configuration.
  • Cypress tests are run before deploys and when the pipeline is invoked manually.
  • Screenshots taken and videos captured by Cypress are stored as artifacts to make it easy to debug issues.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

test:cypress:
  image:
    name: # REDACTED
  stage: test
  only:
    - tags
    - web
  services:
    - redis
    - mongo
    - rabbitmq
  script:
    - echo "$CYPRESS_ENV" > .env
    - ./tophd &
    - MONGO_HOST=mongo make cypress SKIP_DOCKER=true
  needs: ['build:binaries', 'build:assets']
  artifacts:
    when: always
    expire_in: 1 day
    paths:
      - cypress/screenshots
      - cypress/videos
  cache:
    key: "assets"
    paths:
      - .npm
      - .cypress-cache
      - node_modules
    policy: pull
  interruptible: true

What’s Next?

One of the short-term goals is to write tests to cover the most important user interactions. This includes everything from a user registering on Toph to solving problems from the archive to hosting or participating in a programming contest on Toph.

That more comprehensive suite of tests will greatly speed up how Toph is developed in the future.