diff options
Diffstat (limited to 'doc/user/clusters/agent')
| -rw-r--r-- | doc/user/clusters/agent/ci_cd_tunnel.md | 67 | ||||
| -rw-r--r-- | doc/user/clusters/agent/index.md | 62 | ||||
| -rw-r--r-- | doc/user/clusters/agent/repository.md | 11 |
3 files changed, 100 insertions, 40 deletions
diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md new file mode 100644 index 00000000000..1f794bac37f --- /dev/null +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -0,0 +1,67 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# CI/CD Tunnel + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327409) in GitLab 14.1. + +The CI/CD Tunnel enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network +connectivity between GitLab Runner and a cluster. GitLab Runner does not have to be running in the same cluster. + +Only CI/CD jobs set in the configuration project can access one of the configured agents. + +Prerequisites: + +- A running [`kas` instance](index.md#set-up-the-kubernetes-agent-server). +- A [configuration repository](index.md#define-a-configuration-repository) with an Agent config file + installed (`.gitlab/agents/<agent-name>/config.yaml`). +- An [Agent record](index.md#create-an-agent-record-in-gitlab). +- The agent is [installed in the cluster](index.md#install-the-agent-into-the-cluster). + +To access your cluster from a CI/CD job through the tunnel: + +1. In your `.gitlab-ci.yml` add a section that creates a `kubectl` compatible configuration file (`kubecontext`) and use it in one + or more jobs: + + ```yaml + variables: + AGENT_ID: 4 # agent id that you got when you created the agent record + KUBE_CFG_FILE: "$CI_PROJECT_DIR/.kubeconfig.agent.yaml" + + .kubectl_config: &kubectl_config + - | + cat << EOF > "$KUBE_CFG_FILE" + apiVersion: v1 + kind: Config + clusters: + - name: agent + cluster: + server: https://kas.gitlab.com/k8s-proxy/ + users: + - name: agent + user: + token: "ci:$AGENT_ID:$CI_JOB_TOKEN" + contexts: + - name: agent + context: + cluster: agent + user: agent + current-context: agent + EOF + + deploy: + image: + name: bitnami/kubectl:latest + entrypoint: [""] + script: + - *kubectl_config + - kubectl --kubeconfig="$KUBE_CFG_FILE" get pods + ``` + +1. Execute `kubectl` commands directly against your cluster with this CI/CD job you just created. + +We are working on [creating the configuration file automatically](https://gitlab.com/gitlab-org/gitlab/-/issues/324275) +to simplify the process. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 414ed8377db..83933524fd4 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -7,7 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Kubernetes Agent **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -> - [In GitLab 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/300960), KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. +> - [Introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6, `grpcs` is supported. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) in GitLab 13.10, KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. > - Introduced in GitLab 13.11, the GitLab Kubernetes Agent became available to every project on GitLab.com. The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) @@ -16,13 +17,14 @@ tasks in a secure and cloud-native way. It enables: - Integrating GitLab with a Kubernetes cluster behind a firewall or NAT (network address translation). -- Pull-based GitOps deployments by leveraging the - [GitOps Engine](https://github.com/argoproj/gitops-engine). +- Pull-based GitOps deployments. +- [Inventory object](../../infrastructure/clusters/deploy/inventory_object.md) to keep track of objects applied to your cluster. - Real-time access to API endpoints in a cluster. - Alert generation based on [Container network policy](../../application_security/threat_monitoring/index.md#container-network-policy). +- [CI/CD Tunnel](ci_cd_tunnel.md) that enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster. Many more features are planned. Please review [our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) -and [our development documentation](../../../development/agent/index.md). +and [our development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). ## GitLab Agent GitOps workflow @@ -37,7 +39,9 @@ sequenceDiagram participant M as Manifest repository participant K as Kubernetes Agent participant C as Agent configuration repository - K->C: Grab the configuration + loop Regularly + K-->>C: Grab the configuration + end D->>+A: Pushing code changes A->>M: Updating manifest loop Regularly @@ -246,12 +250,11 @@ example [`resources.yml` file](#example-resourcesyml-file) in the following ways - Specify the `grpc` scheme if both Agent and Server are installed in one cluster. In this case, you may specify `kas-address` value as `grpc://gitlab-kas.<your-namespace>:8150`) to use gRPC directly, where `gitlab-kas` - is the name of the service created by `gitlab-kas` chart, and `your-namespace` - is the namespace where the chart was installed. Encrypted gRPC is not supported yet. - Follow the - [Support TLS for gRPC communication issue](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) - for progress updates. - - When deploying KAS through the [GitLab chart](https://docs.gitlab.com/charts/), it's possible to customize the `kas-address` for `wss` and `ws` schemes to whatever you need. + is the name of the service created by `gitlab-kas` chart, and `<your-namespace>` + is the namespace where the chart was installed. + - Specify the `grpcs` scheme to use an encrypted gRPC connection. + - When deploying KAS through the [GitLab chart](https://docs.gitlab.com/charts/), it's possible to customize the + `kas-address` for `wss` and `ws` schemes to whatever you need. Check the [chart's KAS Ingress documentation](https://docs.gitlab.com/charts/charts/gitlab/kas/#ingress) to learn more about it. - In the near future, Omnibus GitLab intends to provision `gitlab-kas` under a sub-domain by default, instead of the `/-/kubernetes-agent/` path. Please follow [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5784) for details. @@ -447,42 +450,21 @@ There are several components that work in concert for the Agent to generate the - A working Kubernetes cluster. - Cilium integration through either of these options: - - Installation through [GitLab Managed Apps](../applications.md#install-cilium-using-gitlab-cicd). + - Installation through [cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium). - Enablement of [hubble-relay](https://docs.cilium.io/en/v1.8/concepts/overview/#hubble) on an existing installation. - One or more network policies through any of these options: - Use the [Container Network Policy editor](../../application_security/threat_monitoring/index.md#container-network-policy-editor) to create and manage policies. - Use an [AutoDevOps](../../application_security/threat_monitoring/index.md#container-network-policy-management) configuration. - Add the required labels and annotations to existing network policies. -- Use a configuration repository to inform the Agent through a `config.yaml` file, which - repositories can synchronize with. This repository might be the same, or a separate GitLab - project. +- A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab) The setup process follows the same steps as [GitOps](#get-started-with-gitops-and-the-gitlab-agent), with the following differences: -- When you define a configuration repository, you must do so with [Cilium settings](#define-a-configuration-repository-with-cilium-settings). +- When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). - You do not need to specify the `gitops` configuration section. -### Define a configuration repository with Cilium settings - -You need a GitLab repository to contain your Agent configuration. The minimal repository layout -looks like this: - -```plaintext -.gitlab/agents/<agent-name>/config.yaml -``` - -Your `config.yaml` file must specify the `host` and `port` of your Hubble Relay service. If your -Cilium integration was performed through [GitLab Managed Apps](../applications.md#install-cilium-using-gitlab-cicd), -you can use `hubble-relay.gitlab-managed-apps.svc.cluster.local:80`: - -```yaml -cilium: - hubble_relay_address: "<hubble-relay-host>:<hubble-relay-port>" - ... -``` - ## Management interfaces Users with at least the [Developer](../../permissions.md) can access the user interface @@ -516,9 +498,17 @@ This error is shown if there are some connectivity issues between the address specified as `kas-address`, and your Agent pod. To fix it, make sure that you specified the `kas-address` correctly. +```json +{"level":"error","time":"2021-06-25T21:15:45.335Z","msg":"Reverse tunnel","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""} +``` + +This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the +`wss` or `ws` URL ends with a training slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` +or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. + ### Agent logs - ValidationError(Deployment.metadata) -```plaintext +```json {"level":"info","time":"2020-10-30T08:56:54.329Z","msg":"Synced","project_id":"root/kas-manifest001","resource_key":"apps/Deployment/kas-test001/nginx-deployment","sync_result":"error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]"} ``` diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index cd40cc6810e..a3a3e4c29b0 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the Kubernetes Agent became available on GitLab.com. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -51,6 +51,7 @@ gitops: # in YAML or JSON format. - id: gitlab-org/cluster-integration/gitlab-agent # Namespace to use if not set explicitly in object manifest. + # Also used for inventory ConfigMap objects. default_namespace: my-ns # Paths inside of the repository to scan for manifest files. # Directories with names starting with a dot are ignored. @@ -84,13 +85,13 @@ gitops: # https://github.com/kubernetes/apimachinery/blob/44113beed5d39f1b261a12ec398a356e02358307/pkg/apis/meta/v1/types.go#L456-L470 # Can be: orphan, background, foreground prune_propagation_policy: foreground # 'foreground' by default - # InventoryPolicy defines if an inventory object can take over + # Inventory policy defines if an inventory object can take over # objects that belong to another inventory object or don't # belong to any inventory object. # This is done by determining if the apply/prune operation # can go through for a resource based on the comparison # the inventory-id value in the package and the owning-inventory - # annotation in the live object. + # annotation (config.k8s.io/owning-inventory) in the live object. # https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/inventory/policy.go#L12-L66 # Can be: must_match, adopt_if_no_inventory, adopt_all inventory_policy: must_match # 'must_match' by default @@ -157,7 +158,9 @@ cilium: hubble_relay_address: "<hubble-relay-host>:<hubble-relay-port>" ``` -If your Cilium integration was performed through GitLab Managed Apps, you can use `hubble-relay.gitlab-managed-apps.svc.cluster.local:80` as the address: +If your Cilium integration was performed through [GitLab Managed Apps](../applications.md#install-cilium-using-gitlab-cicd) or the +[cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium), +you can use `hubble-relay.gitlab-managed-apps.svc.cluster.local:80` as the address: ```yaml cilium: |