Creators guide

Codility provides a way to verify developers' skills in a completely automated manner. Tasks are meant to verify a particular skill or set of skills, by

  • setting the expectations to the programmer solving it (task description and starting code),

  • running automated checks against a given solution,

  • building a report with feedback and the score.

Tools

This documentation

Local runner

Local runner aims to mimic the Codility production runtime to provide a way to validate a given task locally, without the need to deploy it to Codility in the first place.

Installation

Local runner is packaged with all its dependencies with Docker and released to Docker Hub.

Since it uses Docker itself, it needs to control the OS Docker, which might be achieved by linking the socket file to the container.

A common way of automating it is to create a simple script:

./build_and_run_task
#!/bin/bash

PARAMS_TO_PASS=""

while (( "$#" )); do
  case "$1" in
    -t|--task)
      TASK_PATH=$2
      shift 2
      ;;
    *) # preserve other arguments
      PARAMS_TO_PASS="$PARAMS_TO_PASS $1"
      shift
      ;;
  esac
done

if [[ ! -d $TASK_PATH ]]; then
    TASK_PATH="."
fi

TASK_PATH="$(cd "$TASK_PATH"; pwd)" # Calculate absolute path

# Run Codility task runner utility with all the remaining args
docker run \
    --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v $TASK_PATH:/task \
    codility/local_runner:0.10.1 local_runner -t /task --host-task-path $TASK_PATH $PARAMS_TO_PASS

The downside is you need to be updating the version yourself.

Task specification

Everything needed

Structure

While Codility doesn’t enforce a specific structure, use this as a default:

Task structure
task
├── Dockerfile      # defines the runtime environment
├── codility.yml    # describes how to execute the task
├── descriptions    # descriptions visible to candidates
│   └── en.md
├── solutions
│   ├── golden.js   # golden proves 100% points is possible
│   ├── initial.js  # the programmers' starting code
│   ├── wrong_missing_null_check.js  # additional wrong solutions verify the task quality
│   └── wrong_no_type_checks.js
└── src             # contains tests and setup code

Every element is described in detail below.

Sometimes it is simpler to work on multiple tasks (sharing common traits) inside of a single repository. In this case feel free to use the following structure:

Multi-task project
tasks/
├── Dockerfile
├── src  # shared source
├── task-a
│   ├── codility.yml
│   ├── descriptions
│   │   └── en.md
│   ├── solutions
│   │   ├── golden.js
│   │   ├── initial.js
│   │   └── wrong_missing_null_check.js
│   └── src # task-specific code, e.g. test cases
└── task-b
    ├── codility.yml
    ├── descriptions
    │   └── en.md
    ├── solutions
    │   ├── golden.js
    │   ├── initial.js
    │   └── wrong_no_type_checks.js
    └── src

Docker image

We have chosen Docker to encapsulate the runtime environment due to it’s massive popularity and big community.

You may choose one of the provided images, or build your own. The

TODO: List of Codility pre-defined images.

Description

A text description of the task which is shown in the interface. Describe what precisely is expected from the candidate and what factors are important to get a maximum number of points. You can use Markdown formatting here.

Codility YAML

The entire task is described by a YAML file containing information like

  • description file location,

  • how to run the task code on a condidate’s solution,

  • author, title, etc.

Below you may find more information over the YAML file format.

Examples

The minimal file only requires a few fields:

minimal.yml
format: '0.1'

id: minimal
title: Minimal task
version: '1.12'

tags:
  - javascript

ui:
  description: descriptions.md

solutions:
  golden: "golden.js"
  initial: "initial.js"

execution:
  image: "minimal:1.12"

  assessment:
    cases:
      - name: All tests
        script:
          - echo running tests

    volumes:
      solution: /solution.js
      execution_output: /report

The more complex yaml shows most of the available functions.

complex.yml
format: '0.1'

id: frontend_todolist

title: Frontend todo-list
version: '1.12'

authors: john_doe

tags:
  - frontend
  - javascript
  - react
  - angular

variants:
  framework:
    title: Front-end framework
    select: one
    values:
      - angular4:
          image: angular
          preview_path: a_preview/index.html

      - react16:
          image: react
          preview_path: r_preview/index.html

  component_id:
    select: random
    values:
      - task_spec-counter
      - counter
      - counter-id

  done_class:
    select: random
    values:
      - is-done
      - done
      - checked
      - finished

ui:
  description: descriptions/{{ language }}.md
  preview: "preview/{{ framework.preview_path }}"
  folds:
    - regex: '^describe\('
  highlight: "js"
  languages:
    - en
  user_test_cases:
    example_input: |
      Multiline
      Example
      Input
    widget_type: "dialog"


solutions:
  golden: "solutions/golden_{{ framework }}"
  initial: "solutions/begin_{{ framework }}"

execution:
  image: "frontend_todolist_{{ framework.image }}:1.12"

  build:
    script:
      - echo "{{ done_class }}"
      - mkdir ./src
      - cp -R ./solution/project_data/src/* ./src
      - cp -R ./solution/project_data/src/* ./dist
      - cp -R ./solution/solution.ts ./src
      - ./node_modules/typescript/bin/tsc -p ./tsconfig.json
      - cp -R ./dist/* ./execution_output/
      - rm -R ./execution_output/node_modules

    volumes:
      solution: /solution
      build_output: /compilation_output/{{ component_id }}

  assessment:
    # `name` is optional.
    # `script` should end up with a report. Multiple results per report are allowed.
    # output evaluator flattens reports into a single one.
    # if a case has a name, all tests from it's report have the name *overridden*.
    cases:
      - name: "The component should have a like button"
        obligatory: true
        dir: "testcase/01"
        kind: example
        script:
          - echo "{{ framework }}"
          - cp -R solution/* dist
          - MOCHA_FILE=./execution_output/junitresults.xml yarn mocha tests/test.js --grep "should have a like button" --timeout 5000 --reporter mocha-junit-reporter

      - name: "The component should have a proper format"
        dir: "testcase/02"
        kind: final
        script:
          - cp -R solution/* dist
          - MOCHA_FILE=./execution_output/junitresults.xml yarn mocha tests/test.js --grep "should have a proper format" --timeout 5000 --reporter mocha-junit-reporter

    volumes:
      task_data: /task/tests
      build_output: /execution_output
      execution_output: /report/{{ framework }}
      custom:
        - source: data/{{ case.dir }}
          target: /task_data

    output:
      evaluator:
        type: junit

Local runner will ensure it is compatible with a specified version of the YAML file.

Format version

The format string is there to make sure that the file format is consistent. Codility may be changing the format before 1.0 in a non-backawrds compatible way. The latest version is 0.1.

format: '0.1'
Task id
codility.yml:id
id: frontend_todolist

Task id should be unique, and follow the pattern of [_a-zA-Z]{0,50} (underscores and base alphabet chars up to a length of 256).

A good id should relate to the described problem rather than a particular technology. If a particular technology is essential to the task (e.g. REST, shell), it might be a part of the name.

Table 1. Task ids
Fair Better Best

react_difficult

todolist

frontend_todolist

django_post

webapp_post

webapp_login

java_selenium_submit

login_page

qa_login_page
Title
codility.yml:title
title: Frontend todo-list

Task name should be unique, and contain up to 256 characters.

name should reflect the id, but more readable.

Table 2. Task ids
id name

frontend_todolist

Front-end ToDo list

qa_login_page

QA login page
Task version
codility.yml:version
version: '1.12'

Tasks are versioned for a few reasons:

  • to preserve the version during programmer’s solving session,

  • to have a cycle of reviewing and publishing.

Simply start with any version, and bump it whenever you’re releasing a new version of the same task.

Authors

Specifying an author is optional, but preferred. Author field is a simple one (string). In the future this will be connected to the creator’s account.

codility.yml:autors
authors: john_doe
Tags

Tags are used for the task discovery in the task library.

codility.yml:autors(full)
tags:
  - frontend
  - javascript
  - react
  - angular
Available tags
"android",
"angular",
"blockchain",
"codelive",
"coding",
"distributed-programming",
"django",
"dotnet",
"ethereum",
"frontend",
"groovy",
"java",
"javascript",
"jvm",
"kotlin",
"labs",
"laravel",
"orm",
"python",
"qa",
"react",
"rest-assured",
"robot-framework",
"scala",
"sql",
"clojure",
"selenium",
"sequence-processing",
"solidity",
"spring",
"text-processing",
"web",
Variants

Variants the the powerful mechanism for making tasks more flexible and dynamic.

Potential use cases

  • providing multiple programming languages,

  • providing several libraries to choose from,

  • making the task dynamic (more resilient to plagiarism).

UI

TODO

Solutions

TODO

Execution

TODO

Build

TODO

Assessment

TODO

Output

TODO

Task code

TODO