GitLab CI

Table of Contents

This document is hosted on the hpclib gitlab project, cf. https://gitlab.inria.fr/sed-bso/hpclib, see the file tuto/gitlab-ci-nojenkins.org (you should be logged in https://gitlab.inria.fr/).

1 Introduction

GitLab CI is directly integrated in GitLab, contrary to Jenkins for which you need to play with plugins.

Userdoc:

What for?

  • merge request or push triggers your CI pipeline
  • makes it easy to see whether a merge request/push caused any of the tests to fail before you even look to the code.
  • continuous delivery and continuous deployment allow to automatically deploy tested code to staging and production environments

2 Getting started

The steps needed to have a working CI can be summed up to:

  1. Add .gitlab-ci.yml, where you define your tests jobs, to the root directory of your repository
  2. Enable Pipelines in your project's settings
  3. Configure one or several runners which will actually run the tests

2.1 First config file .gitlab-ci.yml

You need to create a YAML file named .gitlab-ci.yml in the root directory of your repository.

The YAML file defines a set of jobs with constraints stating when they should be run. The jobs are defined as top-level elements with a name and always have to contain at least the script clause: 

job1:
  script: "execute-script-for-job1" # ex: ./configure;make;make install

job2:
  script: "execute-script-for-job2" # ex: test.sh

Jobs are picked up by Runners and executed within the environment of the Runner. What is important, is that each job is run independently from each other (different build directory for example). To create dependent jobs, see pipelines.

Create the file, for example here we define a job "test" that execute the commands hostname and whoami in a shell on the runner 

test:
  script:
    - hostname
    - whoami

Then push it to your repository 

git add .gitlab-ci.yml
git commit -m "Add .gitlab-ci.yml"
git push origin master

Recall that we don't have defined any runners yet such that the job is not executed anywhere for now.

2.2 Enable Pipelines

To be able to configure runners, you have to enable Pipeline first in the configuration of your projet.

  • go to the General Settings of your project
  • go to the Permissions section
  • enable Pipelines (Build, test, and deploy your changes)
  • clic on "Save changes"

2.3 First runner configuration

See https://docs.gitlab.com/runner/

In GitLab, Runners run the jobs that you define in .gitlab-ci.yml. A Runner can be a virtual machine, a VPS, a bare-metal machine, a docker container or even a cluster of containers. GitLab and the Runners communicate through an API, so the only requirement is that the Runner's machine has Internet access.

Now, you should be able to register a runner. Please look at the instructions about the runners configuration in your project settings

  • go to the Settings of your project then, CI/CD, then Runners Settings
  • read the instructions at the left side Specific Runners, notice
    1. the URL that you will use during the runner registration https://gitlab.inria.fr/
    2. and the project registration token

Let's give an example with a Linux virtual machine. Let's consider we have previously configured an VM on ci.inria.fr, see https://ci.inria.fr/.

Log in, e.g. 

ssh ci@testgitlab-ubuntu1604.ci

Follow the instruction here http://docs.gitlab.com/runner/install/. 

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo su -
cat > /etc/apt/preferences.d/pin-gitlab-runner.pref <<EOF
Explanation: Prefer GitLab provided packages over the Debian native ones
Package: gitlab-runner
Pin: origin packages.gitlab.com
Pin-Priority: 1001
EOF
exit
sudo apt-get install gitlab-runner

Read the instruction to register the runner here http://docs.gitlab.com/runner/register/index.html.

Register the runner. An example 

sudo gitlab-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.inria.fr/
Please enter the gitlab-ci token for this runner:
# copy/paste the project's secret token here
Please enter the gitlab-ci description for this runner:
[ubuntu1604]: testgitlab-ubuntu1604
Please enter the gitlab-ci tags for this runner (comma separated):
ci.inria.fr, linux, ubuntu 16.04.3 amd64, vm, heat
Whether to run untagged builds [true/false]:
[false]: true
Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded                     runner=4Ds6gXde
Please enter the executor: virtualbox, docker+machine, docker-ssh+machine, docker, docker-ssh, parallels, shell, ssh, kubernetes:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

If you go back in the Runners settings of your project, you should see a newly exposed runner in the Runners activated for this project part with a green circle.

Try a first connexion to gitlab from the runner. The user will be "gitlab-runner" so login as "gitlab-runner" user 

sudo su gitlab-runner

Then try to reach gitlab, for example try 

git clone git@gitlab.inria.fr:toto.git
Cloning into 'toto'...
The authenticity of host 'gitlab.inria.fr (128.93.193.8)' can't be established.
ECDSA key fingerprint is SHA256:Taz+3BXCos4TzUaFEJPnlqox9DBmiDVKKJXLMsPe5tc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitlab.inria.fr,128.93.193.8' (ECDSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

It is normal not to be able to clone if it is a private project. Indeed, the user "gitlab-runner" does not have the rights here. But it is not a real problem because, when a build will be triggered by gitlab, a specific token will be automatically generated for the runner to checkout the project.

Now you can test a build by commit+push something or directly by hand in the gitlab interface, see Pipelines -> Run pipeline green button.

2.3.1 gitlab-runner + Docker

Install docker on your machine, e.g. 

sudo apt-get update && apt-get install -y curl
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt install -y software-properties-common
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt install -y docker-ce
sudo usermod -aG docker ${USER}
newgrp docker

When registering, http://docs.gitlab.com/runner/register/index.html, select docker as executor.

Then, in your .gitlab-ci.yml specify either globally the image to use for every jobs, e.g. 

image: ubuntu

job1:
  script: ls -alrt

job2:
  script: whoami

or for some specific jobs only 

job1:
  script: ls -alrt

job2:
  image: ubuntu
  script: whoami

2.3.2 Additional information about runners

  1. Shared vs. Specific runners

    A Runner that is specific only runs for the specified project. A shared Runner can run jobs for every project that has enabled the option Allow shared Runners. You can only register a Shared Runner if you have admin access to the GitLab instance. We are not admin so that we do not consider shared Runners.

  2. Specific runners can be shared between projects

    You can set up a specific Runner to be used by multiple projects. The difference with a shared Runner is that you have to enable each project explicitly for the Runner to be able to run its jobs.

  3. Lock a specific Runner from being enabled for other projects

    You can configure a Runner to assign it exclusively to a project. When a Runner is locked this way, it can no longer be enabled for other projects. This setting is available on each Runner in Project Settings > CI/CD > Runners settings.

  4. Are the runners shared when project is forked?

    Specific Runners do not get shared with forked projects automatically. A fork does copy the CI settings (jobs, allow shared, etc) of the cloned repository.

  5. Limit jobs to be run concurently on a runner

    See variables "concurrent" and "limit" https://docs.gitlab.com/runner/configuration/advanced-configuration.html

  6. Note about the gitlab build permission model

    https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html

  7. Working with Git Submodules

    https://docs.gitlab.com/ee/ci/git_submodules.html

3 Configuration of the jobs with the file .gitlab-ci.yml

https://gitlab.inria.fr/help/ci/yaml/README.md

Note: by default all jobs are considered independent and can be run in parallel if several runners are available. To define jobs that are dependent from each others, see "pipelines".

3.3 execute on different systems (runner/docker image)

In https://gitlab.inria.fr/help/ci/yaml/README.md#jobs see keyword tags and image.

3.4 configuration matrix

Sadly, this feature is currently missing.

What is existing is not convenient, just alias and not real configuration matrix feature, see https://gitlab.inria.fr/help/ci/yaml/README.md#special-yaml-features

Why it is very limited, illustration https://gitlab.com/gitlab-org/gitlab-ce/issues/19199 https://gitlab.com/StanfordLegion/legion/blob/master/.gitlab-ci.yml#L167-302

4 Unregister runners

Connect to the machine where registering occurs 

# list runners registered
sudo gitlab-runner list
# check runners registered
sudo gitlab-runner verify
# unregister one specific runner
sudo gitlab-runner unregister -t ...
# unregister all runners
sudo gitlab-runner unregister --all-runners

Or directly edit the /etc/gitlab-runner/config.toml file.

Created: 2019-03-28 og. 17:14

Validate