diff options
-rw-r--r-- | doc/README.md | 12 | ||||
-rw-r--r-- | doc/ci/README.md | 12 | ||||
-rw-r--r-- | doc/ci/docker/using_docker_images.md | 285 | ||||
-rw-r--r-- | doc/ci/languages/README.md | 7 | ||||
-rw-r--r-- | doc/ci/languages/php.md | 284 | ||||
-rw-r--r-- | doc/ci/services/README.md | 9 | ||||
-rw-r--r-- | doc/ci/services/docker-services.md | 5 | ||||
-rw-r--r-- | doc/ci/services/mysql.md | 118 | ||||
-rw-r--r-- | doc/ci/services/postgres.md | 114 | ||||
-rw-r--r-- | doc/ci/services/redis.md | 69 | ||||
-rw-r--r-- | doc/ci/ssh_keys/README.md | 109 |
11 files changed, 910 insertions, 114 deletions
diff --git a/doc/README.md b/doc/README.md index 58ab5dd08e0..a3098094210 100644 --- a/doc/README.md +++ b/doc/README.md @@ -24,9 +24,21 @@ - [Using Docker Images](ci/docker/using_docker_images.md) - [Using Docker Build](ci/docker/using_docker_build.md) - [Using Variables](ci/variables/README.md) +- [Using SSH keys](ci/ssh_keys/README.md) - [User permissions](ci/permissions/README.md) - [API](ci/api/README.md) +### CI Languages + ++ [Testing PHP](ci/languages/php.md) + +### CI Services + ++ [Using MySQL](ci/services/mysql.md) ++ [Using PostgreSQL](ci/services/postgres.md) ++ [Using Redis](ci/services/redis.md) ++ [Using Other Services](ci/docker/using_docker_images.md#how-to-use-other-images-as-services) + ### CI Examples - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md) diff --git a/doc/ci/README.md b/doc/ci/README.md index 97325069ceb..5d9d7a81db3 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -9,6 +9,18 @@ + [Using Docker Images](docker/using_docker_images.md) + [Using Docker Build](docker/using_docker_build.md) + [Using Variables](variables/README.md) ++ [Using SSH keys](ssh_keys/README.md) + +### Languages + ++ [Testing PHP](languages/php.md) + +### Services + ++ [Using MySQL](services/mysql.md) ++ [Using PostgreSQL](services/postgres.md) ++ [Using Redis](services/redis.md) ++ [Using Other Services](docker/using_docker_images.md#how-to-use-other-images-as-services) ### Examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 1feae62b1c7..8d4bd44053e 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,19 +1,29 @@ # Using Docker Images -GitLab CI can use [Docker Engine](https://www.docker.com/) to build projects. -Docker is an open-source project that allows to use predefined images to run applications -in independent "containers" that are run within a single Linux instance. -[Docker Hub](https://registry.hub.docker.com/) have rich database of built images that can be used to build applications. +GitLab CI in conjuction with [GitLab Runner](../runners/README.md) can use +[Docker Engine](https://www.docker.com/) to test and build any application. -Docker when used with GitLab CI runs each build in separate and isolated container using predefined image and always from scratch. -It makes it easier to have simple and reproducible build environment that can also be run on your workstation. -This allows you to test all commands from your shell, rather than having to test them on a CI server. +Docker is an open-source project that allows you to use predefined images to +run applications in independent "containers" that are run within a single Linux +instance. [Docker Hub][hub] has a rich database of prebuilt images that can be +used to test and build your applications. -### Register Docker runner -To use GitLab Runner with Docker you need to register new runner to use `docker` executor: +Docker, when used with GitLab CI, runs each build in a separate and isolated +container using the predefined image that is set up in +[`.gitlab-ci.yml`](../yaml/README.md). + +This makes it easier to have a simple and reproducible build environment that +can also run on your workstation. The added benefit is that you can test all +the commands that we will explore later from your shell, rather than having to +test them on a dedicated CI server. + +## Register docker runner + +To use GitLab Runner with docker you need to register a new runner to use the +`docker` executor: ```bash -gitlab-ci-multi-runner register \ +gitlab-runner register \ --url "https://gitlab.com/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ --description "docker-ruby-2.1" \ @@ -23,101 +33,79 @@ gitlab-ci-multi-runner register \ --docker-mysql latest ``` -**The registered runner will use `ruby:2.1` image and will run two services (`postgres:latest` and `mysql:latest`) that will be accessible for time of the build.** +The registered runner will use the `ruby:2.1` docker image and will run two +services, `postgres:latest` and `mysql:latest`, both of which will be +accessible during the build process. -### What is image? -The image is the name of any repository that is present in local Docker Engine or any repository that can be found at [Docker Hub](https://registry.hub.docker.com/). -For more information about the image and Docker Hub please read the [Docker Fundamentals](https://docs.docker.com/introduction/understanding-docker/). +## What is image -### What is service? -Service is just another image that is run for time of your build and is linked to your build. This allows you to access the service image during build time. -The service image can run any application, but most common use case is to run some database container, ie.: `mysql`. -It's easier and faster to use existing image, run it as additional container than install `mysql` every time project is built. +The `image` keyword is the name of the docker image that is present in the +local Docker Engine (list all images with `docker images`) or any image that +can be found at [Docker Hub][hub]. For more information about images and Docker +Hub please read the [Docker Fundamentals][] documentation. -#### How is service linked to the build? -There's good document that describes how Docker linking works: [Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -To summarize: if you add `mysql` as service to your application, the image will be used to create container that is linked to build container. -The service container for MySQL will be accessible under hostname `mysql`. -So, **to access your database service you have to connect to host: `mysql` instead of socket or `localhost`**. +In short, with `image` we refer to the docker image, which will be used to +create a container on which your build will run. -### How to use other images as services? -You are not limited to have only database services. -You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). -Look for `[runners.docker]` section: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest"] -``` +## What is service -For example you need `wordpress` instance to test some API integration with `Wordpress`. -You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). -This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in: -``` -[runners.docker] - image = "ruby:2.1" - services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"] -``` +The `services` keyword defines just another docker image that is run during +your build and is linked to the docker image that the `image` keyword defines. +This allows you to access the service image during build time. -Next time when you run your application the `tutum/wordpress` will be started -and you will have access to it from your build container under hostname: `tutum__wordpress`. +The service image can run any application, but the most common use case is to +run a database container, eg. `mysql`. It's easier and faster to use an +existing image and run it as an additional container than install `mysql` every +time the project is built. -Alias hostname for the service is made from the image name: -1. Everything after `:` is stripped, -2. '/' is replaced with `__`. +You can see some widely used services examples in the relevant documentation of +[CI services examples](../services/README.md). -### Configuring services -Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment. +### How is service linked to the build -GitLab Runner 0.5.0 and up passes all YAML-defined variables to created service containers. +To better understand how the container linking works, read +[Linking containers together](https://docs.docker.com/userguide/dockerlinks/). -1. To configure database name for [postgres](https://registry.hub.docker.com/u/library/postgres/) service, -you need to set POSTGRES_DB. +To summarize, if you add `mysql` as service to your application, the image will +then be used to create a container that is linked to the build container. - ```yaml - services: - - postgres - - variables: - POSTGRES_DB: gitlab - ``` +The service container for MySQL will be accessible under the hostname `mysql`. +So, in order to access your database service you have to connect to the host +named `mysql` instead of a socket or `localhost`. -1. To use [mysql](https://registry.hub.docker.com/u/library/mysql/) service with empty password for time of build, -you need to set MYSQL_ALLOW_EMPTY_PASSWORD. +## Overwrite image and services - ```yaml - services: - - mysql - - variables: - MYSQL_ALLOW_EMPTY_PASSWORD: "yes" - ``` +See [How to use other images as services](#how-to-use-other-images-as-services). -For other possible configuration variables check the -https://registry.hub.docker.com/u/library/mysql/ or https://registry.hub.docker.com/u/library/postgres/ -or README page for any other Docker image. +## How to use other images as services -**Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.** +You are not limited to have only database services. You can add as many +services you need to `.gitlab-ci.yml` or manually modify `config.toml`. +Any image found at [Docker Hub][hub] can be used as a service. -### Overwrite image and services -It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`. -If you add to your YAML the `image` and the `services` these parameters -be used instead of the ones that were specified during runner's registration. -``` +## Define image and services from `.gitlab-ci.yml` + +You can simply define an image that will be used for all jobs and a list of +services that you want to use during build time. + +```yaml image: ruby:2.2 + services: - postgres:9.3 -before_install: + +before_script: - bundle install - + test: script: - bundle exec rake spec ``` -It's possible to define image and service per-job: -``` -before_install: +It is also possible to define different images and services per job: + +```yaml +before_script: - bundle install test:2.1: @@ -135,34 +123,73 @@ test:2.2: - bundle exec rake spec ``` -#### How to enable overwriting? -To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). -You can do that by hand modifying runner configuration: `config.toml`. -Please go to section where is `[runners.docker]` definition for your runner. -Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`: +## Define image and services in `config.toml` + +Look for the `[runners.docker]` section: + ``` [runners.docker] image = "ruby:2.1" - allowed_images = ["ruby:*", "python:*"] - allowed_services = ["mysql:*", "redis:*"] + services = ["mysql:latest", "postgres:latest"] ``` -This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. -You will be able to pick only `ruby` and `python` images. -The same rule can be applied to limit services. -If you are courageous enough, you can make it fully open and accept everything: -``` -[runners.docker] - image = "ruby:2.1" - allowed_images = ["*", "*/*"] - allowed_services = ["*", "*/*"] +The image and services defined this way will be added to all builds run by +that runner. + +## Accessing the services + +Let's say that you need a Wordpress instance to test some API integration with +your application. + +You can then use for example the [tutum/wordpress][] image in your +`.gitlab-ci.yml`: + +```yaml +services: +- tutum/wordpress:latest ``` -**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.** +When the build is run, `tutum/wordpress` will be started and you will have +access to it from your build container under the hostname `tutum__wordpress`. + +The alias hostname for the service is made from the image name following these +rules: + +1. Everything after `:` is stripped +2. Backslash (`/`) is replaced with double underscores (`__`) + +## Configuring services + +Many services accept environment variables which allow you to easily change +database names or set account names depending on the environment. + +GitLab Runner 0.5.0 and up passes all YAML-defined variables to the created +service containers. + +For all possible configuration variables check the documentation of each image +provided in their corresponding Docker hub page. + +*Note: All variables will be passed to all services containers. It's not +designed to distinguish which variable should go where.* + +### PostgreSQL service example + +See the specific documentation for +[using PostgreSQL as a service](../services/postgres.md). + +### MySQL service example + +See the specific documentation for +[using MySQL as a service](../services/mysql.md). + +## How Docker integration works + +Below is a high level overview of the steps performed by docker during build +time. -### How Docker integration works 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`. -1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example). +1. Create cache container to store all volumes as defined in `config.toml` and + `Dockerfile` of build image (`ruby:2.1` as in above example). 1. Create build container and link any service container to build container. 1. Start build container and send build script to the container. 1. Run build script. @@ -171,33 +198,63 @@ If you are courageous enough, you can make it fully open and accept everything: 1. Check exit status of build script. 1. Remove build container and all created service containers. -### How to debug a build locally -1. Create a file with build script: +## How to debug a build locally + +*Note: The following commands are run without root privileges. You should be +able to run docker with your regular user account.* + +First start with creating a file named `build script`: + ```bash -$ cat <<EOF > build_script +cat <<EOF > build_script git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git /builds/gitlab-org/gitlab-ci-multi-runner cd /builds/gitlab-org/gitlab-ci-multi-runner -make <- or any other build step +make EOF ``` -1. Create service containers: +Here we use as an example the GitLab Runner repository which contains a +Makefile, so running `make` will execute the commands defined in the Makefile. +Your mileage may vary, so instead of `make` you could run the command which +is specific to your project. + +Then create some service containers: + ``` -$ docker run -d -n service-mysql mysql:latest -$ docker run -d -n service-postgres postgres:latest +docker run -d -n service-mysql mysql:latest +docker run -d -n service-postgres postgres:latest ``` -This will create two service containers (MySQL and PostgreSQL). -1. Create a build container and execute script in its context: +This will create two service containers, named `service-mysql` and +`service-postgres` which use the latest MySQL and PostgreSQL images +respectively. They will both run in the background (`-d`). + +Finally, create a build container by executing the `build_script` file we +created earlier: + ``` -$ docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script +docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script ``` -This will create build container that has two service containers linked. -The build_script is piped using STDIN to bash interpreter which executes the build script in container. -1. At the end remove all containers: +The above command will create a container named `build` that is spawned from +the `ruby:2.1` image and has two services linked to it. The `build_script` is +piped using STDIN to the bash interpreter which in turn executes the +`build_script` in the `build` container. + +When you finish testing and no longer need the containers, you can remove them +with: + ``` docker rm -f -v build service-mysql service-postgres ``` -This will forcefully (the `-f` switch) remove build container and service containers -and all volumes (the `-v` switch) that were created with the container creation. + +This will forcefully (`-f`) remove the `build` container, the two service +containers as well as all volumes (`-v`) that were created with the container +creation. + +[Docker Fundamentals]: https://docs.docker.com/engine/introduction/understanding-docker/ +[hub]: https://hub.docker.com/ +[linking-containers]: https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/ +[tutum/wordpress]: https://registry.hub.docker.com/u/tutum/wordpress/ +[postgres-hub]: https://registry.hub.docker.com/u/library/postgres/ +[mysql-hub]: https://registry.hub.docker.com/u/library/mysql/ diff --git a/doc/ci/languages/README.md b/doc/ci/languages/README.md new file mode 100644 index 00000000000..54b2343e08b --- /dev/null +++ b/doc/ci/languages/README.md @@ -0,0 +1,7 @@ +### Languages + +This is a list of languages you can test with GitLab CI. Each section has +comprehensive documentation and comes with a test repository hosted on +GitLab.com + ++ [Testing PHP](php.md) diff --git a/doc/ci/languages/php.md b/doc/ci/languages/php.md new file mode 100644 index 00000000000..dacb67fa3ff --- /dev/null +++ b/doc/ci/languages/php.md @@ -0,0 +1,284 @@ +# Testing PHP projects + +This guide covers basic building instructions for PHP projects. + +There are covered two cases: testing using the Docker executor and testing +using the Shell executor. + +## Test PHP projects using the Docker executor + +While it is possible to test PHP apps on any system, this would require manual +configuration from the developer. To overcome this we will be using the +official [PHP docker image][php-hub] that can be found in Docker Hub. + +This will allow us to test PHP projects against different versions of PHP. +However, not everything is plug 'n' play, you still need to onfigure some +things manually. + +As with every build, you need to create a valid `.gitlab-ci.yml` describing the +build environment. + +Let's first specify the PHP image that will be used for the build process +(you can read more about what an image means in the Runner's lingo reading +about [Using Docker images](../docker/using_docker_images.md#what-is-image)). + +Start by adding the image to your `.gitlab-ci.yml`: + +```yaml +image: php:5.6 +``` + +The official images are great, but they lack a few useful tools for testing. +We need to first prepare the build environment. A way to overcome this is to +create a script which installs all prerequisites prior the actual testing is +done. + +Let's create a `ci/docker_install.sh` file in the root directory of our +repository with the following content: + +```bash +#!/bin/bash + +# We need to install dependencies only for Docker +[[ ! -e /.dockerinit ]] && exit 0 + +set -xe + +# Install git (the php image doesn't have it) which is required by composer +apt-get update -yqq +apt-get install git -yqq + +# Install phpunit, the tool that we will use for testing +curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar +chmod +x /usr/local/bin/phpunit + +# Install mysql driver +# Here you can install any other extension that you need +docker-php-ext-install pdo_mysql +``` + +You might wonder what `docker-php-ext-install` is. In short, it is a script +provided by the official php docker image that you can use to easilly install +extensions. For more information read the the documentation at +<https://hub.docker.com/_/php/>. + +Now that we created the script that contains all prerequisites for our build +environment, let's add it in `.gitlab-ci.yml`: + +```yaml +... + +before_script: +- bash ci/docker_install.sh > /dev/null + +... +``` + +Last step, run the actual tests using `phpunit`: + +```yaml +... + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml + +... +``` + +Finally, commit your files and push them to GitLab to see your build succeeding +(or failing). + +The final `.gitlab-ci.yml` should look similar to this: + +```yaml +# Select image from https://hub.docker.com/_/php/ +image: php:5.6 + +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Test against different PHP versions in Docker builds + +Testing against multiple versions of PHP is super easy. Just add another job +with a different docker image version and the runner will do the rest: + +```yaml +before_script: +# Install dependencies +- ci/docker_install.sh > /dev/null + +# We test PHP5.6 +test:5.6: + image: php:5.6 + script: + - phpunit --configuration phpunit_myapp.xml + +# We test PHP7.0 (good luck with that) +test:7.0: + image: php:7.0 + script: + - phpunit --configuration phpunit_myapp.xml +``` + +### Custom PHP configuration in Docker builds + +There are times where you will need to customise your PHP environment by +putting your `.ini` file into `/usr/local/etc/php/conf.d/`. For that purpose +add a `before_script` action: + +```yaml +before_script: +- cp my_php.ini /usr/local/etc/php/conf.d/test.ini +``` + +Of course, `my_php.ini` must be present in the root directory of your repository. + +## Test PHP projects using the Shell executor + +The shell executor runs your builds in a terminal session on your server. +Thus, in order to test your projects you first need to make sure that all +dependencies are installed. + +For example, in a VM running Debian 8 we first update the cache, then we +install `phpunit` and `php5-mysql`: + +```bash +sudo apt-get update -y +sudo apt-get install -y phpunit php5-mysql +``` + +Next, add the following snippet to your `.gitlab-ci.yml`: + +```yaml +test:app: + script: + - phpunit --configuration phpunit_myapp.xml +``` + +Finally, push to GitLab and let the tests begin! + +### Test against different PHP versions in Shell builds + +The [phpenv][] project allows you to easily manage different versions of PHP +each with its own config. This is specially usefull when testing PHP projects +with the Shell executor. + +You will have to install it on your build machine under the `gitlab-runner` +user following [the upstream installation guide][phpenv-installation]. + +Using phpenv also allows to easily configure the PHP environment with: + +``` +phpenv config-add my_config.ini +``` + +*__Important note:__ It seems `phpenv/phpenv` + [is abandoned](https://github.com/phpenv/phpenv/issues/57). There is a fork + at [madumlao/phpenv](https://github.com/madumlao/phpenv) that tries to bring + the project back to life. [CHH/phpenv](https://github.com/CHH/phpenv) also + seems like a good alternative. Picking any of the mentioned tools will work + with the basic phpenv commands. Guiding you to choose the right phpenv is out + of the scope of this tutorial.* + +### Install custom extensions + +Since this is a pretty bare installation of the PHP environment, you may need +some extensions that are not currently present on the build machine. + +To install additional extensions simply execute: + +```bash +pecl install <extension> +``` + +It's not advised to add this to `.gitlab-ci.yml`. You should execute this +command once, only to setup the build environment. + +## Extend your tests + +### Using atoum + +Instead of PHPUnit, you can use any other tool to run unit tests. For example +you can use [atoum](https://github.com/atoum/atoum): + +```yaml +before_script: +- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar + +test:atoum: + script: + - php mageekguy.atoum.phar +``` + +### Using Composer + +The majority of the PHP projects use Composer for managing their PHP packages. +In order to execute Composer before running your tests, simply add the +following in your `.gitlab-ci.yml`: + +```yaml +... + +# Composer stores all downloaded packages in the vendor/ directory. +# Do not use the following if the vendor/ directory is commited to +# your git repository. +cache: + paths: + - vendor/ + +before_script: +# Install composer dependencies +- curl -sS https://getcomposer.org/installer | php +- php composer.phar install + +... +``` + +## Access private packages / dependencies + +If your test suite needs to access a private repository, you need to configure +[the SSH keys](../ssh_keys/README.md) in order to be able to clone it. + +## Use databases or other services + +Most of the time you will need a running database in order for your tests to +run. If you are using the Docker executor you can leverage Docker's ability to +link to other containers. In GitLab Runner lingo, this can be achieved by +defining a `service`. + +This functionality is covered in [the CI services](../services/README.md) +documentation. + +## Testing things locally + +With GitLab Runner 1.0 you can also test any changes locally. From your +terminal execute: + +```bash +# Check using docker executor +gitlab-runner exec docker test:app + +# Check using shell executor +gitlab-runner exec shell test:app +``` + +## Example project + +We have set up an [Example PHP Project][php-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[php-hub]: https://hub.docker.com/_/php/ +[phpenv]: https://github.com/phpenv/phpenv +[phpenv-installation]: https://github.com/phpenv/phpenv#installation +[php-example-repo]: https://gitlab.com/gitlab-examples/php diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md new file mode 100644 index 00000000000..1ebb0a4a250 --- /dev/null +++ b/doc/ci/services/README.md @@ -0,0 +1,9 @@ +## GitLab CI Services + +GitLab CI uses the `services` keyword to define what docker containers should be +linked with your base image. Below is a list of examples you may use. + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) ++ [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services) diff --git a/doc/ci/services/docker-services.md b/doc/ci/services/docker-services.md new file mode 100644 index 00000000000..df36ebaf7d4 --- /dev/null +++ b/doc/ci/services/docker-services.md @@ -0,0 +1,5 @@ +## GitLab CI Services + ++ [Using MySQL](mysql.md) ++ [Using PostgreSQL](postgres.md) ++ [Using Redis](redis.md) diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md new file mode 100644 index 00000000000..c66d77122b2 --- /dev/null +++ b/doc/ci/services/mysql.md @@ -0,0 +1,118 @@ +# Using MySQL + +As many applications depend on MySQL as their database, you will eventually +need it in order for your tests to run. Below you are guided how to do this +with the Docker and Shell executors of GitLab Runner. + +## Use MySQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - mysql:latest + +variables: + # Configure mysql environment variables (https://hub.docker.com/_/mysql/) + MYSQL_DATABASE: el_duderino + MYSQL_ROOT_PASSWORD: mysql_strong_password +``` + +And then configure your application to use the database, for example: + +```yaml +Host: mysql +User: root +Password: mysql_strong_password +Database: el_duderino +``` + +If you are wondering why we used `mysql` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-mysql]. +For example, to use MySQL 5.5 the service becomes `mysql:5.5`. + +The `mysql` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-mysql]. + +## Use MySQL with the Shell executor + +You can also use MySQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the MySQL server: + +```bash +sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev +``` + +Pick a MySQL root password (can be anything), and type it twice when asked. + +*Note: As a security measure you can run `mysql_secure_installation` to +remove anonymous users, drop the test database and disable remote logins with +the root user.* + +The next step is to create a user, so login to MySQL as root: + +```bash +mysql -u root -p +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*Note: Do not type `mysql>`, this is part of the MySQL prompt.* + +```bash +mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; +``` + +Create the database: + +```bash +mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; +``` + +Grant the necessary permissions on the database: + +```bash +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost'; +``` + +If all went well you can now quit the database session: + +```bash +mysql> \q +``` + +Now, try to connect to the newly created database to check that everything is +in place: + +```bash +mysql -u runner -p -D el_duderino +``` + +As a final step, configure your application to use the database, for example: + +```bash +Host: localhost +User: runner +Password: $password +Database: el_duderino +``` + +## Example project + +We have set up an [Example MySQL Project][mysql-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-mysql]: https://hub.docker.com/_/mysql/ +[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md new file mode 100644 index 00000000000..17d21dbda1c --- /dev/null +++ b/doc/ci/services/postgres.md @@ -0,0 +1,114 @@ +# Using PostgreSQL + +As many applications depend on PostgreSQL as their database, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use PostgreSQL with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - postgres:latest + +variables: + POSTGRES_DB: nice_marmot + POSTGRES_USER: runner + POSTGRES_PASSWORD: "" +``` + +And then configure your application to use the database, for example: + +```yaml +Host: postgres +User: runner +Password: +Database: nice_marmot +``` + +If you are wondering why we used `postgres` for the `Host`, read more at +[How is service linked to the build](../docker/using_docker_images.md#how-is-service-linked-to-the-build). + +You can also use any other docker image available on [Docker Hub][hub-pg]. +For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`. + +The `postgres` image can accept some environment variables. For more details +check the documentation on [Docker Hub][hub-pg]. + +## Use PostgreSQL with the Shell executor + +You can also use PostgreSQL on manually configured servers that are using +GitLab Runner with the Shell executor. + +First install the PostgreSQL server: + +```bash +sudo apt-get install -y postgresql postgresql-client libpq-dev +``` + +The next step is to create a user, so login to PostgreSQL: + +```bash +sudo -u postgres psql -d template1 +``` + +Then create a user (in our case `runner`) which will be used by your +application. Change `$password` in the command below to a real strong password. + +*__Note:__ Do not type `template1=#`, this is part of the PostgreSQL prompt.* + +```bash +template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB; +``` + +*__Note:__ Notice that we created the user with the privilege to be able to +create databases (`CREATEDB`). In the following steps we will create a database +explicitly for that user but having that privilege can be useful if in your +testing framework you have tools that drop and create databases.* + +Create the database and grant all privileges on it for the user `runner`: + +```bash +template1=# CREATE DATABASE nice_marmot OWNER runner; +``` + +If all went well you can now quit the database session: + +```bash +template1=# \q +``` + +Now, try to connect to the newly created database with the user `runner` to +check that everything is in place. + +```bash +psql -U runner -h localhost -d nice_marmot -W +``` + +*__Note:__ We are explicitly telling `psql` to connect to localhost in order +to use the md5 authentication. If you omit this step you will be denied access.* + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +User: runner +Password: $password +Database: nice_marmot +``` + +## Example project + +We have set up an [Example PostgreSQL Project][postgres-example-repo] for your +convenience that runs on [GitLab.com](https://gitlab.com) using our publicly +available [shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-pg]: https://hub.docker.com/_/postgres/ +[postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md new file mode 100644 index 00000000000..b281e8f9f60 --- /dev/null +++ b/doc/ci/services/redis.md @@ -0,0 +1,69 @@ +# Using Redis + +As many applications depend on Redis as their key-value store, you will +eventually need it in order for your tests to run. Below you are guided how to +do this with the Docker and Shell executors of GitLab Runner. + +## Use Redis with the Docker executor + +If you are using [GitLab Runner](../runners/README.md) with the Docker executor +you basically have everything set up already. + +First, in your `.gitlab-ci.yml` add: + +```yaml +services: + - redis:latest +``` + +Then you need to configure your application to use the Redis database, for +example: + +```yaml +Host: redis +``` + +And that's it. Redis will now be available to be used within your testing +framework. + +You can also use any other docker image available on [Docker Hub][hub-redis]. +For example, to use Redis 2.8 the service becomes `redis:2.8`. + +## Use Redis with the Shell executor + +Redis can also be used on manually configured servers that are using GitLab +Runner with the Shell executor. + +In your build machine install the Redis server: + +```bash +sudo apt-get install redis-server +``` + +Verify that you can connect to the server with the `gitlab-runner` user: + +```bash +# Try connecting the the Redis server +sudo -u gitlab-runner -H redis-cli + +# Quit the session +127.0.0.1:6379> quit +``` + +Finally, configure your application to use the database, for example: + +```yaml +Host: localhost +``` + +## Example project + +We have set up an [Example Redis Project][redis-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[hub-redis]: https://hub.docker.com/_/redis/ +[redis-example-repo]: https://gitlab.com/gitlab-examples/redis diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md new file mode 100644 index 00000000000..210f9c3e849 --- /dev/null +++ b/doc/ci/ssh_keys/README.md @@ -0,0 +1,109 @@ +# Using SSH keys + +GitLab currently doesn't have built-in support for managing SSH keys in a build +environment. + +The SSH keys can be useful when: + +1. You want to checkout internal submodules +2. You want to download private packages using your package manager (eg. bundler) +3. You want to deploy your application to eg. Heroku or your own server +4. You want to execute SSH commands from the build server to the remote server +5. You want to rsync files from your build server to the remote server + +If anything of the above rings a bell, then you most likely need an SSH key. + +## Inject keys in your build server + +The most widely supported method is to inject an SSH key into your build +environment by extending your `.gitlab-ci.yml`. + +This is the universal solution which works with any type of executor +(docker, shell, etc.). + +### How it works + +1. Create a new SSH key pair with [ssh-keygen][] +2. Add the private key as a **Secret Variable** to the project +3. Run the [ssh-agent][] during build to load the private key. + +## SSH keys when using the Docker executor + +You will first need to create an SSH key pair. For more information, follow the +instructions to [generate an SSH key](../ssh/README.md). + +Then, create a new **Secret Variable** in your project settings on GitLab +following **Settings > Variables**. As **Key** add the name `SSH_PRIVATE_KEY` +and in the **Value** field paste the content of your _private_ key that you +created earlier. + +Next you need to modify your `.gitlab-ci.yml` with a `before_script` action. +Add it to the top: + +``` +before_script: + # Install ssh-agent if not already installed, it is required by Docker. + # (change apt-get to yum if you use a CentOS-based image) + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + + # Run ssh-agent (inside the build environment) + - eval $(ssh-agent -s) + + # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store + - ssh-add <(echo "$SSH_PRIVATE_KEY") + + # For Docker builds disable host key checking. Be aware that by adding that + # you are suspectible to man-in-the-middle attacks. + # WARNING: Use this only with the Docker executor, if you use it with shell + # you will overwrite your user's SSH config. + - mkdir -p ~/.ssh + - '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config` +``` + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +That's it! You can now have access to private servers or repositories in your +build environment. + +## SSH keys when using the Shell executor + +If you are using the Shell executor and not Docker, it is easier to set up an +SSH key. + +You can generate the SSH key from the machine that GitLab Runner is installed +on, and use that key for all projects that are run on this machine. + +First, you need to login to the server that runs your builds. + +Then from the terminal login as the `gitlab-runner` user and generate the SSH +key pair as described in the [SSH keys documentation](../ssh/README.md). + +As a final step, add the _public_ key from the one you created earlier to the +services that you want to have an access to from within the build environment. +If you are accessing a private GitLab repository you need to add it as a +[deploy key](../ssh/README.md#deploy-keys). + +Once done, try to login to the remote server in order to accept the fingerprint: + +```bash +ssh <address-of-my-server> +``` + +For accessing repositories on GitLab.com, the `<address-of-my-server>` would be +`git@gitlab.com`. + +## Example project + +We have set up an [Example SSH Project][ssh-example-repo] for your convenience +that runs on [GitLab.com](https://gitlab.com) using our publicly available +[shared runners](../runners/README.md). + +Want to hack on it? Simply fork it, commit and push your changes. Within a few +moments the changes will be picked by a public runner and the build will begin. + +[ssh-keygen]: http://linux.die.net/man/1/ssh-keygen +[ssh-agent]: http://linux.die.net/man/1/ssh-agent +[ssh-example-repo]: https://gitlab.com/gitlab-examples/ssh-private-key/ |