Speeding Up GitLab CI With Docker

Posted Jan 16, 2019

We use GitLab CI extensively to build and deploy BeyondTracks. However we have some heavy dependencies, which we used to build and install every time our CI process ran. This added an extra 5 minutes at the start of every build, which is just wasteful.

To speed things up we wanted to follow best practice by installing our dependencies into a Docker image and reused that in our build.

It was important that this Docker image had a well document build process, which didn’t depend on any local configuration, meaning it needed to be built by GitLab CI. So we created a new repository, beyondtracks-docker with just two files, .gitlab-ci.yml and Dockerfile.

.gitlab-ci.yml

GitLab makes this dead easy. Under Repository > Files select New > New file. Then choose the Template .gitlab-ci.yml, applying a GitLab CI Yaml template for Docker. This provides us a base GitLab CI configuration to build and deploy our Docker image.

image: docker:latest

services:
  - docker:dind

before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY

build-master:
  stage: build
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  only:
    - master

build:
  stage: build
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
  except:
    - master

DockerFile

We just need one more file, a Dockerfile which documents how to build our dependencies. This is what ours looks like.

RUN apt-get -y update && apt-get -y install curl gnupg openssh-client rsync
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN curl -sL https://deb.nodesource.com/setup_11.x | bash -
RUN apt-get -y update && apt-get -y install nodejs yarn
RUN apt-get -y install imagemagick webp postgresql postgis postgresql-contrib osm2pgsql wget osmium-tool gdal-bin python-pip pgtap s3cmd parallel git unzip build-essential libsqlite3-dev zlib1g-dev
RUN pip install mapboxcli
RUN npm install -g @mapbox/geojsonhint
RUN git clone --depth=1 https://github.com/mapbox/tippecanoe.git && cd tippecanoe && make -j && make install && cd ..

This builds a Docker image straight to our Container Registry which we can then reference in all our other GitLab CI with

image: "registry.gitlab.com/beyondtracks/beyondtracks-docker:latest"

Finally, to ensure our dependencies stay fresh with new software versions we can add a CI Schedule to rebuild our docker image at a set frequency.