info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
# GitOps with the Kubernetes Agent **(PREMIUM SELF)**
This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md).
The [GitLab Kubernetes Agent](../../user/clusters/agent/index.md) supports the
<!-- This redirect file can be deleted after <2022-06-24>. -->
[pull-based version](https://www.gitops.tech/#pull-based-deployments) of
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
[GitOps](https://www.gitops.tech/). To be useful, the feature must be able to perform these tasks:
- Connect one or more Kubernetes clusters to a GitLab project or group.
- Synchronize cluster-wide state from a Git repository.
- Synchronize namespace-scoped state from a Git repository.
- Control the following settings:
- The kinds of objects an agent can manage.
- Enabling the namespaced mode of operation for managing objects only in a specific namespace.
- Enabling the non-namespaced mode of operation for managing objects in any namespace, and
managing non-namespaced objects.
- Synchronize state from one or more Git repositories into a cluster.
- Configure multiple agents running in different clusters to synchronize state
from the same repository.
## GitOps architecture
In this architecture, the Kubernetes cluster (`agentk`) periodically fetches
configuration from (`kas`), spawning a goroutine for each configured GitOps
repository. Each goroutine makes a streaming `GetObjectsToSynchronize()` gRPC call.
`kas` accepts these requests, then checks if this agent is authorized to access
this GitLab repository. If authorized, `kas` polls Gitaly for repository updates
and sends the latest manifests to the agent.
Before each poll, `kas` verifies with GitLab that the agent's token is still valid.
When `agentk` receives an updated manifest, it performs a synchronization using
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
# Kubernetes Agent identity and authentication **(PREMIUM SELF)**
This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md).
This page uses the word `agent` to describe the concept of the
<!-- This redirect file can be deleted after <2022-06-24>. -->
GitLab Kubernetes Agent. The program that implements the concept is called `agentk`.
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
The regex for names is: `/\A[a-z0-9]([-a-z0-9]*[a-z0-9])?\z/`.
## Multiple agents in a cluster
A Kubernetes cluster may have 0 or more agents running in it. Each agent likely
has a different configuration. Some may enable features A and B, and some may
enable features B and C. This flexibility enables different groups of people to
use different features of the agent in the same cluster.
For example, [Priyanka (Platform Engineer)](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#priyanka-platform-engineer)
may want to use cluster-wide features of the agent, while
a distinct Kubernetes identity, with a distinct set of permissions attached to it.
These permissions enable the agent administrator to follow the
[principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)
and minimize the permissions each particular agent needs.
## Kubernetes Agent authentication
When adding a new agent, GitLab provides the user with a bearer access token. The
agent uses this token to authenticate with GitLab. This token is a random string
and does not encode any information in it, but it is secret and must
be treated with care. Store it as a `Secret` in Kubernetes.
Each agent can have 0 or more tokens in a GitLab database. Having several valid
tokens helps you rotate tokens without needing to re-register an agent. Each token
record in the database has the following fields:
- Agent identity it belongs to.
- Token value. Encrypted at rest.
- Creation time.
- Who created it.
- Revocation flag to mark token as revoked.
- Revocation time.
- Who revoked it.
- A text field to store any comments the administrator may want to make about the token for future self.
Tokens can be managed by users with `maintainer` and higher level of
[permissions](../../user/permissions.md).
Tokens are immutable, and only the following fields can be updated:
- Revocation flag. Can only be updated to `true` once, but immutable after that.
- Revocation time. Set to the current time when revocation flag is set, but immutable after that.
- Comments field. Can be updated any number of times, including after the token has been revoked.
The agent sends its token, along with each request, to GitLab to authenticate itself.
For each request, GitLab checks the token's validity:
- Does the token exist in the database?
- Has the token been revoked?
This information may be cached for some time to reduce load on the database.
## Kubernetes Agent authorization
GitLab provides the following information in its response for a given Agent access token:
- Agent configuration Git repository. (The agent doesn't support per-folder authorization.)
- Agent name.
## Create an agent
You can create an agent by following the [user documentation](../../user/clusters/agent/index.md#create-an-agent-record-in-gitlab), or via Rails console:
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
# Kubernetes Agent development **(PREMIUM SELF)**
This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md).
This page contains developer-specific information about the GitLab Kubernetes Agent.
<!-- This redirect file can be deleted after <2022-06-24>. -->
[End-user documentation about the GitLab Kubernetes Agent](../../user/clusters/agent/index.md)
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
is also available.
The agent can help you perform tasks like these:
- Integrate a cluster, located behind a firewall or NAT, with GitLab. To
learn more, read [issue #212810, Invert the model GitLab.com uses for Kubernetes integration by leveraging long lived reverse tunnels](https://gitlab.com/gitlab-org/gitlab/-/issues/212810).
- Access API endpoints in a cluster in real time. For an example use case, read
[issue #218220, Allow Prometheus in K8s cluster to be installed manually](https://gitlab.com/gitlab-org/gitlab/-/issues/218220#note_348729266).
- Enable real-time features by pushing information about events happening in a cluster.
For example, you could build a cluster view dashboard to visualize changes in progress
in a cluster. For more information about these efforts, read about the
[Real-Time Working Group](https://about.gitlab.com/company/team/structure/working-groups/real-time/).
- Enable a [cache of Kubernetes objects through informers](https://github.com/kubernetes/client-go/blob/ccd5becdffb7fd8006e31341baaaacd14db2dcb7/tools/cache/shared_informer.go#L34-L183),
kept up-to-date with very low latency. This cache helps you:
- Reduce or eliminate information propagation latency by avoiding Kubernetes API calls
and polling, and only fetching data from an up-to-date cache.
- Lower the load placed on the Kubernetes API by removing polling.
- Eliminate any rate-limiting errors by removing polling.
- Simplify backend code by replacing polling code with cache access. While it's another
API call, no polling is needed. This example describes [fetching cached data synchronously from the front end](https://gitlab.com/gitlab-org/gitlab/-/issues/217792#note_348582537) instead of fetching data from the Kubernetes API.
## Architecture of the Kubernetes Agent
The GitLab Kubernetes Agent and the GitLab Kubernetes Agent Server use
This diagram describes how GitLab (`GitLab RoR`), the GitLab Kubernetes Agent (`agentk`), and the GitLab Kubernetes Agent Server (`kas`) work together.
```mermaid
graph TB
agentk -- gRPC bidirectional streaming --> kas
subgraph "GitLab"
kas[kas]
GitLabRoR[GitLab RoR] -- gRPC --> kas
kas -- gRPC --> Gitaly[Gitaly]
kas -- REST API --> GitLabRoR
end
subgraph "Kubernetes cluster"
agentk[agentk]
end
```
-`GitLab RoR` is the main GitLab application. It uses gRPC to talk to `kas`.
-`agentk` is the GitLab Kubernetes Agent. It keeps a connection established to a
`kas` instance, waiting for requests to process. It may also actively send information
about things happening in the cluster.
-`kas` is the GitLab Kubernetes Agent Server, and is responsible for:
- Accepting requests from `agentk`.
-[Authentication of requests](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md) from `agentk` by querying `GitLab RoR`.
- Fetching agent's configuration from a corresponding Git repository by querying Gitaly.
- Matching incoming requests from `GitLab RoR` with existing connections from
the right `agentk`, forwarding requests to it and forwarding responses back.
- (Optional) Sending notifications through ActionCable for events received from `agentk`.
- Polling manifest repositories for [GitOps support](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md) by communicating with Gitaly.
GitLab prefers to add logic into `kas` rather than `agentk`. `agentk` should be kept
streamlined and small to minimize the need for upgrades. On GitLab.com, `kas` is
managed by GitLab, so upgrades and features can be added without requiring you
to upgrade `agentk` in your clusters.
`agentk` can't be viewed as a dumb reverse proxy because features are planned to be built
[on top of the cache with informers](https://github.com/kubernetes/client-go/blob/ccd5becdffb7fd8006e31341baaaacd14db2dcb7/tools/cache/shared_informer.go#L34-L183).
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
# Run the Kubernetes Agent locally **(PREMIUM SELF)**
This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/local.md).
You can run `kas` and `agentk` locally to test the [Kubernetes Agent](index.md) yourself.
<!-- This redirect file can be deleted after <2022-06-24>. -->
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
1. Create a `cfg.yaml` file from the contents of
[`config_example.yaml`](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/pkg/kascfg/config_example.yaml), or this example:
[the agent you created](../../user/clusters/agent/index.md#create-an-agent-record-in-gitlab). This file must not contain a newline character. You can create the file with this command:
```shell
echo-n"<TOKEN>"> token.txt
```
1. Start the binaries with the following commands:
```shell
# Need GitLab to start
gdk start
# Stop GDK's version of kas
gdk stop gitlab-k8s-agent
# Start kas
bazel run //cmd/kas ----configuration-file="$(pwd)/cfg.yaml"
```
1. In a new terminal window, run this command to start `agentk`:
```shell
bazel run //cmd/agentk ----kas-address=grpc://127.0.0.1:8150 --token-file="$(pwd)/token.txt"
The `kas` QA tests currently have some limitations. You can run them manually on GDK, but they don't
run automatically with the nightly jobs against the live environment. See the section below
to learn how to run them against different environments.
### Prepare
Before performing any of these tests, if you have a `k3s` instance running, make sure to
stop it manually before running them. Otherwise, the tests might fail with the message
`failed to remove k3s cluster`.
You might need to specify the correct Agent image version that matches the `kas` image version. You can use the `GITLAB_AGENTK_VERSION` local environment for this.
### Against `staging`
1. Go to your local `qa/qa/service/cluster_provider/k3s.rb` and comment out
[this line](https://gitlab.com/gitlab-org/gitlab/-/blob/5b15540ea78298a106150c3a1d6ed26416109b9d/qa/qa/service/cluster_provider/k3s.rb#L8) and
1. Go to your `qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb` and comment out [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L27) and uncomment [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L28).
GDK's `kas` listens on `grpc`, not on `wss`.
1. Go to the GDK's root folder and `cd gitlab/qa`.
1. On the contrary to staging, run the QA test in GDK as admin, which is the default choice. To do so, use the default sandbox group and run the command below. Make sure to adjust your credentials if necessary, otherwise, the test might fail:
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
---
# Kubernetes Agent user stories **(PREMIUM SELF)**
This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/user_stories.md).
The [personas in action](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#user-personas)
<!-- This redirect file can be deleted after <2022-06-24>. -->
for the Kubernetes Agent are:
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
-[Sasha, the Software Developer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#sasha-software-developer).
-[Allison, the Application Operator](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#allison-application-ops).
-[Priyanka, the Platform Engineer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#priyanka-platform-engineer).
[Devon, the DevOps engineer](https://about.gitlab.com/handbook/marketing/strategic-marketing/roles-personas/#devon-devops-engineer)
is intentionally excluded here, as DevOps is more of a role than a persona.
There are various workflows to support, so some user stories might seem to contradict each other. They don't.
## Software Developer user stories
<!-- vale gitlab.FirstPerson = NO -->
- As a Software Developer, I want to push my code, and move to the next development task,
to work on business applications.
- As a Software Developer, I want to set necessary dependencies and resource requirements
together with my application code, so my code runs fine after deployment.
<!-- vale gitlab.FirstPerson = YES -->
## Application Operator user stories
<!-- vale gitlab.FirstPerson = NO -->
- As an Application Operator, I want to standardize the deployments used by my teams,
so I can support all teams with minimal effort.
- As an Application Operator, I want to have a single place to define all the deployments,
so I can assure security fixes are applied everywhere.
- As an Application Operator, I want to offer a set of predefined templates to
Software Developers, so they can get started quickly and can deploy to production
without my intervention, and I am not a bottleneck.
- As an Application Operator, I want to know exactly what changes are being deployed,
so I can fulfill my SLAs.
- As an Application Operator, I want deep insights into what versions of my applications
are running and want to be able to debug them, so I can fix operational issues.
- As an Application Operator, I want application code to be automatically deployed
to staging environments when new versions are available.
- As an Application Operator, I want to follow my preferred deployment strategy,
so I can move code into production in a reliable way.
- As an Application Operator, I want review all code before it's deployed into production,
so I can fulfill my SLAs.
- As an Application Operator, I want to be notified before deployment when new code needs my attention,
so I can review it swiftly.
<!-- vale gitlab.FirstPerson = YES -->
## Platform Engineer user stories
<!-- vale gitlab.FirstPerson = NO -->
- As a Platform Engineer, I want to restrict customizations to preselected values
for Operators, so I can fulfill my SLAs.
- As a Platform Engineer, I want to allow some level of customization to Operators,
so I don't become a bottleneck.
- As a Platform Engineer, I want to define all deployments in a single place, so
I can assure security fixes are applied everywhere.
- As a Platform Engineer, I want to define the infrastructure by code, so my
infrastructure management is testable, reproducible, traceable, and scalable.
- As a Platform Engineer, I want to define various policies that applications must
follow, so that I can fulfill my SLAs.
- As a Platform Engineer, I want approved tooling for log management and persistent storage,
so I can scale, secure, and manage them as needed.
- As a Platform Engineer, I want to be alerted when my infrastructure differs from
its definition, so I can make sure that everything is configured as expected.
@@ -23,7 +23,7 @@ tasks in a secure and cloud-native way. It enables:
...
@@ -23,7 +23,7 @@ tasks in a secure and cloud-native way. It enables:
-[CI/CD Tunnel](ci_cd_tunnel.md) that enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster.
-[CI/CD Tunnel](ci_cd_tunnel.md) that enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster.
Many more features are planned. Please review [our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329)
Many more features are planned. Please review [our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329)
and [our development documentation](../../../development/agent/index.md).
and [our development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc).
