diff options
author | Marcel Amirault <mamirault@gitlab.com> | 2019-08-09 02:17:41 +0300 |
---|---|---|
committer | Evan Read <eread@gitlab.com> | 2019-08-09 02:17:41 +0300 |
commit | 8ee6c088aab8a6a05be3ebae3061b80050ce936d (patch) | |
tree | beb5060c13d9f3a6bf5d5c6c464cdda24d12419d /doc | |
parent | e7a997aa25145bfd09ff07009e6a45957dd36b8d (diff) |
Bring k8s cheat sheet to docs
Bring the support team k8s cheat sheet into the
main documentation
Diffstat (limited to 'doc')
-rw-r--r-- | doc/administration/index.md | 2 | ||||
-rw-r--r-- | doc/administration/troubleshooting/kubernetes_cheat_sheet.md | 251 |
2 files changed, 253 insertions, 0 deletions
diff --git a/doc/administration/index.md b/doc/administration/index.md index f7f9d753e58..91a4d5097f2 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -188,3 +188,5 @@ Learn how to install, configure, update, and maintain your GitLab instance. - Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab Support team. - [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch. +- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful + for troubleshooting Kubernetes-related issues. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md new file mode 100644 index 00000000000..95cdb1508fa --- /dev/null +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -0,0 +1,251 @@ +--- +type: reference +--- + +# Kubernetes, GitLab and You + +This is a list of useful information regarding Kubernetes that the GitLab Support +Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone +can make use of the Support team's collected knowledge + +CAUTION: **Caution:** +These commands **can alter or break** your Kubernetes components so use these at your own risk. + +If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how +to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) +and they will assist you with any issues you are having. + +## Generic kubernetes commands + +- How to authorize to your GCP project (can be especially useful if you have projects + under different GCP accounts): + + ```bash + gcloud auth login + ``` + +- How to access Kubernetes dashboard: + + ```bash + # for minikube: + minikube dashboard —url + # for non-local installations if access via kubectl is configured: + kubectl proxy + ``` + +- How to ssh to a Kubernetes node and enter the container as root + <https://github.com/kubernetes/kubernetes/issues/30656>: + + - For GCP, you may find the node name and run `gcloud compute ssh node-name`. + - List containers using `docker ps`. + - Enter container using `docker exec --user root -ti container-id bash`. + +- How to copy a file from local machine to a pod: + + ```bash + kubectl cp file-name pod-name:./destination-path + ``` + +- What to do with pods in `CrashLoopBackoff` status: + + - Check logs via Kubernetes dashboard. + - Check logs via `kubectl`: + + ```bash + kubectl logs <unicorn pod> -c dependencies + ``` + +- How to tail all Kubernetes cluster events in real time: + + ```bash + kubectl get events -w --all-namespaces + ``` + +- How to get logs of the previously terminated pod instance: + + ```bash + kubectl logs <pod-name> --previous + ``` + + NOTE: **Note:** + No logs are kept in the containers/pods themselves, everything is written to stdout. + This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) + for details. + +## Gitlab-specific kubernetes information + +- Minimal config that can be used to test a Kubernetes helm chart can be found + [here](https://gitlab.com/charts/gitlab/issues/620). + +- Tailing logs of a separate pod. An example for a unicorn pod: + + ```bash + kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn + ``` + +- It is not possible to get all the logs via `kubectl` at once, like with `gitlab-ctl tail`, + but a number of third-party tools can be used to do it: + + - [Kubetail](https://github.com/johanhaleby/kubetail) + - [kail: kubernetes tail](https://github.com/boz/kail) + - [stern](https://github.com/wercker/stern) + +- Check all events in the `gitlab` namespace (the namespace name can be different if you + specified a different one when deploying the helm chart): + + ```bash + kubectl get events -w --namespace=gitlab + ``` + +- Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner + pod. You may enter it and run commands inside or run them from the outside: + + ```bash + # find the pod + kubectl get pods | grep task-runner + + # enter it + kubectl exec -it <task-runner-pod-name> bash + + # open rails console + # rails console can be also called from other GitLab pods + /srv/gitlab/bin/rails console + + # source-style commands should also work + /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production + + # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart + /usr/local/bin/gitlab-rake gitlab:check + + # open console without entering pod + kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + + # check the status of DB migrations + kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + ``` + + You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. + +- Troubleshooting **Operations > Kubernetes** integration: + + - Check the output of `kubectl get events -w --all-namespaces`. + - Check the logs of pods within `gitlab-managed-apps` namespace. + - On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed + via helm chart, kubernetes.log can be found inside the sidekiq pod. + +- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: + + ```bash + # find the name of the secret containing the password + kubectl get secrets | grep initial-root + # decode it + kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo + ``` + +- How to connect to a GitLab postgres database: + + ```bash + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p + ``` + +- How to get info about helm installation status: + + ```bash + helm status name-of-installation + ``` + +- How to update GitLab installed using helm chart: + + ```bash + helm repo upgrade + + # get current values and redirect them to yaml file (analogue of gitlab.rb values) + helm get values <release name> > gitlab.yaml + + # run upgrade itself + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + + After <https://canary.gitlab.com/charts/gitlab/issues/780> is fixed, it should + be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#updating-gitlab-using-the-helm-chart) + for upgrades. + +- How to apply changes to GitLab config: + + - Modify the `gitlab.yaml` file. + - Run the following command to apply changes: + + ```bash + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + +## Installation of minimal GitLab config via minukube on macOS + +This section is based on [Developing for Kubernetes with Minikube](https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md) +and [Helm](https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md). Refer +to those documents for details. + +- Install kubectl via Homebrew: + + ```bash + brew install kubernetes-cli + ``` + +- Install minikube via Homebrew: + + ```bash + brew cask install minikube + ``` + +- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` + and repeat the steps: + + ```bash + minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work + minikube addons enable ingress + minikube addons enable kube-dns + ``` + +- Install helm via Homebrew and initialize it: + + ```bash + brew install kubernetes-helm + helm init --service-account tiller + ``` + +- Copy the file <https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> + to your workstation. + +- Find the IP address in the output of `minikube ip` and update the yaml file with + this IP address. + +- Install the GitLab helm chart: + + ```bash + helm repo add gitlab https://charts.gitlab.io + helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab + ``` + + If you want to modify some GitLab settings, you can use the above-mentioned config + as a base and create your own yaml file. + +- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. + The installation could take up to 20-30 minutes depending on the amount of resources + on your workstation. + +- When all the pods show either a `Running` or `Completed` status, get the GitLab password as + described in [Initial login](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#initial-login), + and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` + where `domain` is the value provided in the yaml file. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> |