diff options
Diffstat (limited to 'doc/user/clusters/agent/gitops.md')
| -rw-r--r-- | doc/user/clusters/agent/gitops.md | 224 |
1 files changed, 108 insertions, 116 deletions
diff --git a/doc/user/clusters/agent/gitops.md b/doc/user/clusters/agent/gitops.md index 106f751555a..aff78ed477b 100644 --- a/doc/user/clusters/agent/gitops.md +++ b/doc/user/clusters/agent/gitops.md @@ -11,10 +11,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/346567) from GitLab Premium to GitLab Free in 15.3. > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/346585) to make the `id` attribute optional in GitLab 15.7. > - Specifying a branch, tag, or commit reference to fetch the Kubernetes manifest files [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4516) in GitLab 15.7. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/395364) in GitLab 16.1 to prioritize Flux for GitOps. -NOTE: -From GitLab 15.10, you should use [Flux](gitops/flux.md) for GitOps. For more information, see -[this announcement blog post](https://about.gitlab.com/blog/2023/02/08/why-did-we-choose-to-integrate-fluxcd-with-gitlab/). +GitLab integrates [Flux](https://fluxcd.io/flux/) for GitOps. +To get started with Flux, see the [Flux for GitOps tutorial](gitops/flux_tutorial.md). With GitOps, you can manage containerized clusters and applications from a Git repository that: @@ -25,163 +25,155 @@ By combining GitLab, Kubernetes, and GitOps, you can have: - GitLab as the GitOps operator. - Kubernetes as the automation and convergence system. -- GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment. +- GitLab CI/CD for Continuous Integration. +- The agent for Continuous Deployment and cluster observability. - Built-in automatic drift remediation. - Resource management with [server-side applies](https://kubernetes.io/docs/reference/using-api/server-side-apply/) for transparent multi-actor field management. +## Deployment sequence + This diagram shows the repositories and main actors in a GitOps deployment: ```mermaid sequenceDiagram participant D as Developer participant A as Application code repository - participant M as Manifest repository - participant K as GitLab agent + participant M as Deployment repository + participant R as OCI registry participant C as Agent configuration repository + participant K as GitLab agent + participant F as Flux loop Regularly K-->>C: Grab the configuration end D->>+A: Pushing code changes A->>M: Updating manifest - loop Regularly - K-->>M: Watching changes - M-->>K: Pulling and applying changes - end + M->>R: Build an OCI artifact + M->>K: Notify + K->>F: Notify and watch sync + R-->>F: Pulling and applying changes + K->>M: Notify after sync ``` -For details, view the [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture). +You should use both Flux and `agentk` for GitOps deployments. Flux keeps the cluster state synchronized with the source, while `agentk` simplifies the Flux setup, provides cluster-to-GitLab access management, and visualizes the cluster state in the GitLab UI. -## GitOps workflow steps +### OCI for source control -To update a Kubernetes cluster by using GitOps, complete the following steps. +You should use OCI images as a source controller for Flux, instead of a Git repository. The [GitLab container registry](../../packages/container_registry/index.md) supports OCI images. -1. Ensure you have a working Kubernetes cluster, and that the manifests or [Helm charts](gitops/helm.md) are in a GitLab project. -1. In the same project, [register and install the GitLab agent](install/index.md). -1. Configure the agent configuration file so that the agent monitors the project for changes to the Kubernetes manifests. - Use the [GitOps configuration reference](#gitops-configuration-reference) for guidance. +| OCI registry | Git repository | +| --- | --- | +| Designed to serve container images at scale. | Designed to version and store source code. | +| Immutable, supports security scans. | Mutable. | +| The default Git branch can store cluster state without triggering a sync. | The default Git branch triggers a sync when used to store cluster state. | -Any time you commit updates to your Kubernetes manifests, the agent updates the cluster. +## Repository structure -## GitOps configuration reference +To simplify configuration, use one delivery repository per team. +You can package the delivery repository into multiple OCI images per application. -The following snippet shows an example of the possible keys and values for the GitOps section of an [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). +For additional repository structure recommendations, see the [Flux documentation](https://fluxcd.io/flux/guides/repository-structure/). -```yaml -gitops: - manifest_projects: - - id: gitlab-org/cluster-integration/gitlab-agent - ref: # either `branch`, `tag` or `commit` can be specified - branch: production - # commit: <mysha> - # tag: v1.0 - default_namespace: my-ns - paths: - # Read all YAML files from this directory. - - glob: '/team1/app1/*.yaml' - # Read all .yaml files from team2/apps and all subdirectories. - - glob: '/team2/apps/**/*.yaml' - # If 'paths' is not specified or is an empty list, the configuration below is used. - - glob: '/**/*.{yaml,yml,json}' - reconcile_timeout: 3600s - dry_run_strategy: none - prune: true - prune_timeout: 3600s - prune_propagation_policy: foreground - inventory_policy: must_match -``` +## Immediate Git repository reconciliation -| Keyword | Description | -|--|--| -| `manifest_projects` | Projects where your Kubernetes manifests are stored. The agent monitors the files in the repositories in these projects. When manifest files change, the agent deploys the changes to the cluster. | -| `id` | Path to a Git repository that has Kubernetes manifests in YAML or JSON format. No authentication mechanisms are supported. Default is the agent configuration repository. | -| `ref` | Optional. Git reference in the configured Git repository to fetch the Kubernetes manifest files from. If not specified or empty, the default branch is used. If specified, it must contain either `branch`, `tag`, or `commit`. | -| `ref.branch` | Branch name in the configured Git repository to fetch the Kubernetes manifest files from. | -| `ref.tag` | Tag name in the configured Git repository to fetch the Kubernetes manifest files from. | -| `ref.commit` | Commit SHA in the configured Git repository to fetch the Kubernetes manifest files from. | -| `default_namespace` | Namespace to use if not set explicitly in object manifest. Also used for inventory `ConfigMap` objects. | -| `paths` | Repository paths to scan for manifest files. Directories with names that start with a dot `(.)` are ignored. | -| `paths[].glob` | Required. See [doublestar](https://github.com/bmatcuk/doublestar#about) and [the match function](https://pkg.go.dev/github.com/bmatcuk/doublestar/v2#Match) for globbing rules. | -| `reconcile_timeout` | Determines whether the applier should wait until all applied resources have been reconciled, and if so, how long to wait. Default is 3600 seconds (1 hour). | -| `dry_run_strategy` | Determines whether changes [should be performed](https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/common/common.go#L68-L89). Can be: `none`, `client`, or `server`. Default is `none`.| -| `prune` | Determines whether pruning of previously applied objects should happen after apply. Default is `true`. | -| `prune_timeout` | Determines whether to wait for all resources to be fully deleted after pruning, and if so, how long to wait. Default is 3600 seconds (1 hour). | -| `prune_propagation_policy` | The deletion propagation policy that [should be used for pruning](https://github.com/kubernetes/apimachinery/blob/44113beed5d39f1b261a12ec398a356e02358307/pkg/apis/meta/v1/types.go#L456-L470). Can be: `orphan`, `background`, or `foreground`. Default is `foreground`. | -| `inventory_policy` | Determines whether an inventory object can take over objects that belong to another inventory object or don't belong to any inventory object. This is done by determining if the apply/prune operation can go through for a resource based on comparison of the `inventory-id` value in the package and the `owning-inventory` annotation (`config.k8s.io/owning-inventory`) [in the live object](https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/inventory/policy.go#L12-L66). Can be: `must_match`, `adopt_if_no_inventory`, or `adopt_all`. Default is `must_match`. | +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/392852) in GitLab 16.1 with a [flag](../../../administration/feature_flags.md) named `notify_kas_on_git_push`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/126527) in GitLab 16.2. -## GitOps annotations +Usually, the Flux source controller reconciles Git repositories at configured intervals. +This can cause delays between a `git push` and the reconciliation of the cluster state, and results in +unnecessary pulls from GitLab. -The GitLab agent for Kubernetes has annotations you can use to: +The agent for Kubernetes automatically detects Flux `GitRepository` objects that +reference GitLab projects in the instance the agent is connected to, +and configures a [`Receiver`](https://fluxcd.io/flux/components/notification/receiver/) for the instance. +When the agent for Kubernetes detects a `git push`, the `Receiver` is triggered +and Flux reconciles the cluster with any changes to the repository. -- **Sort resources**: Apply or delete resources in a specific order. -- **Use apply-time mutation**: Dynamically substitute fields from one resource configuration to another. +To use immediate Git repository reconciliation, you must have a Kubernetes cluster that runs: -The agent has [default sorting](https://github.com/kubernetes-sigs/cli-utils/blob/d7d63f4b62897f584ca9e02b6faf4d2f327a9b09/pkg/ordering/sort.go#L74), -but with annotations, you can fine-tune the order and apply time-value injection. +- The agent for Kubernetes. +- Flux `source-controller` and `notification-controller`. -To provide the GitOps functionality, the GitLab agent for Kubernetes uses the [`cli-utils` library](https://github.com/kubernetes-sigs/cli-utils/), -a Kubernetes SIG project. For more information, see the available annotations in the [`cli-utils` documentation](https://github.com/kubernetes-sigs/cli-utils/blob/master/README.md). +Immediate Git repository reconciliation can reduce the time between a push and reconciliation, +but it doesn't guarantee that every `git push` event is received. You should still set +[`GitRepository.spec.interval`](https://fluxcd.io/flux/components/source/gitrepositories/#interval) +to an acceptable duration. -## Automatic drift remediation +### Custom webhook endpoints -Drift happens when the current configuration of an infrastructure resource differs from its desired configuration. -Typically, this is caused by manually editing resources directly rather than via the used infrastructure-as-code -mechanism. Minimizing the risk of drift helps to ensure configuration consistency and successful operations. +When the agent for Kubernetes calls the `Receiver` webhook, +the agent defaults to `http://webhook-receiver.flux-system.svc.cluster.local`, +which is also the default URL set by a Flux bootstrap installation. To configure a custom +endpoint, set `flux.webhook_receiver_url` to a URL that the agent can resolve. For example: -In GitLab, the agent for Kubernetes regularly compares the desired state from the `git` repository with -the actual state from the Kubernetes cluster. Deviations from the `git` state are fixed at every check. These checks -happen automatically every 5 minutes. They are not configurable. +```yaml +flux: + webhook_receiver_url: http://webhook-receiver.another-flux-namespace.svc.cluster.local +``` -The agent uses [server-side applies](https://kubernetes.io/docs/reference/using-api/server-side-apply/). -As a result, every field in a resource can have different managers. Only fields managed by `git` -are checked for drift. This facilitates the use of in-cluster controllers to modify resources like -[Horizontal Pod Autoscalers](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). +There is special handing for +[service proxy URLs](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster-services/) configured +in this format: `/api/v1/namespaces/[^/]+/services/[^/]+/proxy`. For example: -## Related topics +```yaml +flux: + webhook_receiver_url: /api/v1/namespaces/flux-system/services/http:webhook-receiver:80/proxy +``` -- [Deploying Helm charts with the GitOps workflow](gitops/helm.md) -- [GitOps working examples for training and demos](https://gitlab.com/groups/guided-explorations/gl-k8s-agent/gitops/-/wikis/home) -- [Self-paced classroom workshop](https://gitlab-for-eks.awsworkshop.io) (Uses AWS EKS, but you can use for other Kubernetes clusters) -- [Managing Kubernetes secrets in a GitOps workflow](gitops/secrets_management.md) -- [Application and manifest repository example](https://gitlab.com/gitlab-examples/ops/gitops-demo/hello-world-service-gitops) +In these cases, the agent for Kubernetes uses the available Kubernetes configuration +and context to connect to the API endpoint. +You can use this if you run an agent outside a cluster +and you haven't [configured an `Ingress`](https://fluxcd.io/flux/guides/webhook-receivers/#expose-the-webhook-receiver) +for the Flux notification controller. + +WARNING: +You should configure only trusted service proxy URLs. +When you provide a service proxy URL, +the agent for Kubernetes sends typical Kubernetes API requests which include +the credentials necessary to authenticate with the API service. -## Troubleshooting +## Token management -### Avoiding conflicts when you have multiple projects +To use certain Flux features, you might need multiple access tokens. Additionally, you can use multiple token types to achieve the same result. -The agent watches each glob pattern set under a project's `paths` section independently, and makes updates to the cluster concurrently. -If changes are found at multiple paths, when the agent attempts to update the cluster, -a conflict can occur. +This section provides guidelines for the tokens you might need, and provides token type recommendations where possible. -To prevent this from happening, consider storing a logical group of manifests in a single place and reference them only once to avoid overlapping globs. +### GitLab access by Flux -For example, both of these globs match `*.yaml` files in the root directory -and could cause conflicts: +To access the GitLab the container registry or Git repositories, Flux can use: -```yaml -gitops: - manifest_projects: - - id: project1 - paths: - - glob: '/**/*.yaml' - - glob: '/*.yaml' -``` +- A project or group deploy token. +- A project or group deploy key. +- A project or group access token. +- A personal access token. -Instead, specify a single glob that matches all `*.yaml` files recursively: +The token does not need write access. -```yaml -gitops: - manifest_projects: - - id: project1 - paths: - - glob: '/**/*.yaml' -``` +You should use project deploy tokens if `http` access is possible. +If you require `git+ssh` access, you should use deploy keys. +To compare deploy keys and deploy tokens, see [Deploy keys](../../project/deploy_keys/index.md). -### Use multiple agents or projects +Support for automating deploy token creation, rotation, and reporting is proposed in [issue 389393](https://gitlab.com/gitlab-org/gitlab/-/issues/389393). -If you store your Kubernetes manifests in separate GitLab projects, -update your agent configuration file with the location of these projects. +### Flux to GitLab notification -WARNING: -The project with the agent's -configuration file can be private or public. Other projects with Kubernetes manifests must be public. Support for private manifest projects is tracked -in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/7704). +If you configure Flux to synchronize from a Git source, [Flux can register an external job status](https://fluxcd.io/flux/components/notification/provider/#git-commit-status-updates) in GitLab pipelines. + +To get external job statuses from Flux, you can use: + +- A project or group deploy token. +- A project or group access token. +- A personal access token. + +The token requires `api` scope. To minimize the attack surface of a leaked token, you should use +a project access token. + +Integrating Flux into GitLab pipelines as a job is proposed in [issue 405007](https://gitlab.com/gitlab-org/gitlab/-/issues/405007). + +## Related topics + +- [GitOps working examples for training and demos](https://gitlab.com/groups/guided-explorations/gl-k8s-agent/gitops/-/wikis/home) +- [Self-paced classroom workshop](https://gitlab-for-eks.awsworkshop.io) (Uses AWS EKS, but you can use for other Kubernetes clusters) +- Managing Kubernetes secrets in a GitOps workflow + - [with SOPS built-in to Flux](https://fluxcd.io/flux/guides/mozilla-sops/) + - [with Sealed Secrets](https://fluxcd.io/flux/guides/sealed-secrets/) |