diff options
Diffstat (limited to 'doc/ci/docker')
-rw-r--r-- | doc/ci/docker/README.md | 4 | ||||
-rw-r--r-- | doc/ci/docker/using_docker_build.md | 112 | ||||
-rw-r--r-- | doc/ci/docker/using_docker_images.md | 203 |
3 files changed, 319 insertions, 0 deletions
diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md new file mode 100644 index 00000000000..84eaf29efd1 --- /dev/null +++ b/doc/ci/docker/README.md @@ -0,0 +1,4 @@ +# Docker integration + ++ [Using Docker Images](using_docker_images.md) ++ [Using Docker Build](using_docker_build.md)
\ No newline at end of file diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md new file mode 100644 index 00000000000..702a6c6b587 --- /dev/null +++ b/doc/ci/docker/using_docker_build.md @@ -0,0 +1,112 @@ +# Using Docker Build + +GitLab CI can allows you to use Docker Engine to build and test docker-based projects. + +**This also allows to you to use `docker-compose` and other docker-enabled tools.** + +This is one of new trends in Continuous Integration/Deployment to: + +1. create application image, +1. run test against created image, +1. push image to remote registry, +1. deploy server from pushed image + +It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image: +```bash +$ docker build -t my-image dockerfiles/ +$ docker run my-docker-image /script/to/run/tests +$ docker tag my-image my-registry:5000/my-image +$ docker push my-registry:5000/my-image +``` + +However, this requires special configuration of GitLab Runner to enable `docker` support during build. +**This requires running GitLab Runner in privileged mode which can be harmful when untrusted code is run.** + +There are two methods to enable the use of `docker build` and `docker run` during build. + +## 1. Use shell executor + +The simplest approach is to install GitLab Runner in `shell` execution mode. +GitLab Runner then executes build scripts as `gitlab-runner` user. + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. During GitLab Runner installation select `shell` as method of executing build scripts or use command: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://ci.gitlab.com \ + --token RUNNER_TOKEN \ + --executor shell + --description "My Runner" + ``` + +2. Install Docker on server. + + For more information how to install Docker on different systems checkout the [Supported installations](https://docs.docker.com/installation/). + +3. Add `gitlab-runner` user to `docker` group: + + ```bash + $ sudo usermod -aG docker gitlab-runner + ``` + +4. Verify that `gitlab-runner` has access to Docker: + + ```bash + $ sudo -u gitlab-runner -H docker info + ``` + + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +5. You can now use `docker` command and install `docker-compose` if needed. + +6. However, by adding `gitlab-runner` to `docker` group you are effectively granting `gitlab-runner` full root permissions. +For more information please checkout [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). + +## 2. Use docker-in-docker executor + +Second approach is to use special Docker image with all tools installed (`docker` and `docker-compose`) and run build script in context of that image in privileged mode. +In order to do that follow the steps: + +1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). + +1. Register GitLab Runner from command line to use `docker` and `privileged` mode: + + ```bash + $ sudo gitlab-runner register -n \ + --url http://ci.gitlab.com \ + --token RUNNER_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "gitlab/dind:latest" \ + --docker-privileged + ``` + + The above command will register new Runner to use special [gitlab/dind](https://registry.hub.docker.com/u/gitlab/dind/) image which is provided by GitLab Inc. + The image at the start runs Docker daemon in [docker-in-docker](https://blog.docker.com/2013/09/docker-can-now-run-within-docker/) mode. + +1. You can now use `docker` from build script: + + ```yaml + before_script: + - docker info + + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +1. However, by enabling `--docker-privileged` you are effectively disables all security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. +For more information you could be interested in checking out [Runtime privilege](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration). + diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md new file mode 100644 index 00000000000..ef449cd45bc --- /dev/null +++ b/doc/ci/docker/using_docker_images.md @@ -0,0 +1,203 @@ +# Using Docker Images +GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. + +Docker is an open-source project that allows to use predefined images to run applications +in independent "containers" that are run within a single Linux instance. +[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. + +Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. +It makes it easier to have simple and reproducible build environment that can also be run on your workstation. +This allows you to test all commands from your shell, rather than having to test them on a CI server. + +### Register Docker runner +To use GitLab Runner with Docker you need to register new runner to use `docker` executor: + +```bash +gitlab-ci-multi-runner register \ + --url "https://ci.gitlab.com/" \ + --registration-token "PROJECT_REGISTRATION_TOKEN" \ + --description "docker-ruby-2.1" \ + --executor "docker" \ + --docker-image ruby:2.1 \ + --docker-postgres latest \ + --docker-mysql latest +``` + +**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** + +### What is image? +The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). +For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). + +### What is service? +Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. +The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. +It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. + +#### How is service linked to the build? +There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). +To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. +The service container for MySQL will be accessible under hostname `mysql`. +So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. + +### How to use other images as services? +You are not limited to have only database services. +You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). +Look for `[runners.docker]` section: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest"] +``` + +For example you need `wordpress` instance to test some API integration with `Wordpress`. +You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). +This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: +``` +[runners.docker] + image = "ruby:2.1" + services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] +``` + +Next time when you run your application the `tutum/wordpress` will be started +and you will have access to it from your build container under hostname: `tutum_wordpress`. + +Alias hostname for the service is made from the image name: +1. Everything after `:` is stripped, +2. '/' is replaced to `_`. + +### Configuring services +Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. + +1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, +you need to set POSTGRES_DB. + + ```yaml + services: + - postgres + + variables: + POSTGRES_DB: gitlab + ``` + +1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, +you need to set MYSQL_ALLOW_EMPTY_PASSWORD. + + ```yaml + services: + - mysql + + variables: + MYSQL_ALLOW_EMPTY_PASSWORD: yes + ``` + +For other possible configuration variables check the +https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ +or README page for any other Docker image. + +**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** + +### Overwrite image and services +It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. +If you add to your YAML the `image` and the `services` these parameters +be used instead of the ones that were specified during runner's registration. +``` +image: ruby:2.2 +services: + - postgres:9.3 +before_install: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +It's possible to define image and service per-job: +``` +before_install: + - bundle install + +test:2.1: + image: ruby:2.1 + services: + - postgres:9.3 + script: + - bundle exec rake spec + +test:2.2: + image: ruby:2.2 + services: + - postgres:9.4 + script: + - bundle exec rake spec +``` + +#### How to enable overwriting? +To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). +You can do that by hand modifying runner configuration: `config.toml`. +Please go to section where is `[runners.docker]` definition for your runner. +Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["ruby:*", "python:*"] + allowed_services = ["mysql:*", "redis:*"] +``` +This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. +You will be able to pick only `ruby` and `python` images. +The same rule can be applied to limit services. + +If you are courageous enough, you can make it fully open and accept everything: +``` +[runners.docker] + image = "ruby:2.1" + allowed_images = ["*", "*/*"] + allowed_services = ["*", "*/*"] +``` + +**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** + +### How Docker integration works +1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. +1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create build container and link any service container to build container. +1. Start build container and send build script to the container. +1. Run build script. +1. Checkout code in: `/builds/group-name/project-name/`. +1. Run any step defined in `.gitlab-ci.yml`. +1. Check exit status of build script. +1. Remove build container and all created service containers. + +### How to debug a build locally +1. Create a file with build script: +```bash +$ cat <<EOF > build_script +git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner +cd /builds/gitlab-org/gitlab-ci-multi-runner +make <- or any other build step +EOF +``` + +1. Create service containers: +``` +$ docker run -d -n service-mysql mysql:latest +$ docker run -d -n service-postgres postgres:latest +``` +This will create two service containers (MySQL and PostgreSQL). + +1. Create a build container and execute script in its context: +``` +$ cat build_script | docker run -n build -i -l mysql:service-mysql -l postgres:service-postgres ruby:2.1 /bin/bash +``` +This will create build container that has two service containers linked. +The build_script is piped using STDIN to bash interpreter which executes the build script in container. + +1. At the end remove all containers: +``` +docker rm -f -v build service-mysql service-postgres +``` +This will forcefully (the `-f` switch) remove build container and service containers +and all volumes (the `-v` switch) that were created with the container creation. |