GitLab on Kubernetes

null

Run GitLab on Kubernetes

GitLab is an open source DevSecOps platform for software engineering teams. There are two flavors available for deploying GitLab on your Kubernetes Cluster:

  1. GitLab Cloud native hybrid
  2. GitLab package (Omnibus)

GitLab Cloud native hybrid

Deploying the GitLab Cloud native hybrid architecture makes only sense in query specific use-cases. For example - as seen in the GitLab decision tree - if the GitLab instance need to serve at least 3,000 users. For this kind of deployment the GitLab components will be installed separately and in different docker containers. Resulting in minimum requirements of 10 nodes with a total of 19 vCPUs and 60 GB memory, which will result in a couple of thousand dollars cloud costs a month.

So in most cases GitLab Cloud native is not needed and overly complex.

So, how to deploy GitLab Omnibus on Kubernetes?

GitLab Omnibus on Kubernetes

GitLab publishes omnibus packaged containers on Docker Hub (hub.docker.com/r/gitlab/gitlab-ce), which are easily configurable with environment variables. Doing the configuration right makes it easy to deploy GitLab on Kubernetes.

Following important components should be configured correctly in order to archive a reasonable Kubernetes setup:

  1. PostgreSQL database
  2. SMTP configuration
  3. GitLab container registry on Kubernetes
  4. Sidekiq, GitLay & Puma configuration
  5. Prometheus metrics
  6. GitLab backups on Kubernetes

GitLab on Kubernetes: Configure an external PostgreSQL database

Creating a PostgreSQL can be done with the CloudNativePG PostgreSQL Operator. After the installation of the operator a new Postgres cluster can be deployed by applying a Kubernetes CR:

kind: Cluster
apiVersion: postgresql.cnpg.io/v1
metadata:
  name: gitlab
spec:
  enableSuperuserAccess: false
  instances: 2
  bootstrap:
    initdb:
      database: gitlabhq_production
      owner: gitlab
  storage:
    size: 20Gi

It is important that the database is called gitlabhq_production and the configured gitlab database user is the owner of the database. In this case we automatically created a postgres cluster with two replicas ensuring a HA setup if one node fails.

After the creation of the database the included database of the omnibus package must be disabled in the gitlab.rb file:

postgresql['enable'] = false

gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = 'gitlab-pg-rw'
gitlab_rails['db_password'] = ''

Nice, we already have cloud native database for our GitLab installation.

GitLab on Kubernetes: Configure SMTP credentials

In order to make sure your GitLab instance is able to send emails, you need to configure your SMTP server. This also can be done in the gitlab.rb file.

gitlab_rails['smtp_enable'] = ...
gitlab_rails['smtp_address'] = ...
gitlab_rails['smtp_port'] = ...
gitlab_rails['smtp_user_name'] = ...
gitlab_rails['smtp_password'] = ...
gitlab_rails['smtp_tls'] = ...
gitlab_rails['gitlab_email_from'] = ...

A free smtp server for transactional emails can be created for example on the Brevo platform.

GitLab on Kubernetes: Container registry

GitLab includes the reference implementation of the docker container registry, but it is disabled by default as it is quite a hassle to set it up correctly.

Important It is important to understand that all requests in the GitLab omnibus container will first be processed by the nginx server and then distributed to components.

As the container registry will only work with ssl connections we need to terminate the ssl traffic directly inside the GitLab container. If the nginx-ingress-controller is used an additional annotation to the ingress needs to be added in order to do not terminate the ssl connection on the nginx load balancer: nginx.ingress.kubernetes.io/ssl-passthrough: true

After that following gitlab.rb configurations must be applied:

registry['enable'] = true
gitlab_rails['registry_enabled'] = true
registry['token_realm'] = "https://" + ENV['GITLAB_HOST']
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = ENV['GITLAB_REGISTRY_HOST']
gitlab_rails['registry_api_url'] = "http://localhost:5000"

registry_external_url 'https://' + ENV['GITLAB_REGISTRY_HOST']
registry_nginx['redirect_http_to_https'] = true
registry_nginx['listen_port'] = 5443
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/tls.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/tls.key"
registry_nginx['real_ip_trusted_addresses'] = ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
registry_nginx['server_names_hash_bucket_size'] = 128

Additionally, a s3 bucket can be configured so all image layers will not be stored inside a volume, but in a scalable s3 compatible bucket.

GitLab on Kubernetes: Sidekiq, GitLay & Puma configuration

Puma (HTTP server for ruby), Sidekiq (job scheduler) and GitLay (GitLabs Git bridge) can all be started with a custom amount of workers / threads. The best configuration heavily depends on your use-case and the amount of users that need to be supported. A reasonable minimal configuration would be:

puma['worker_processes'] = 2
sidekiq['max_concurrency'] = 9

GitLab on Kubernetes: Prometheus Metrics

The GitLab omnibus package ships with its own prometheus instance inside the image. As a prometheus instance is already present in most Kubernetes clusters the included prometheus can safely be disabled.

prometheus_monitoring['enable'] = false

A ServiceMonitor can be used to grep the exposed metrics endpoints of GitLabs component on following ports:

  • 8082 (sidekiq)
  • 9168 (gitlab)
  • 9236 (gitlay)

GitLab on Kubernetes: Backups

Well backing up GitLab on Kubernetes could be tech guide on its own. The easiest way would be to back up the complete namespace using a backup solution like Velero.

Glasskube GitLab Kubernetes Operator

Meet the Glasskube GitLab Kubernetes Operator - a product of our developer team's frustration with the tedious configuration process, time-consuming setup, and constant troubleshooting associated with GitLab deployment. In response to these challenges, we crafted a solution that simplifies the entire experience. The Glasskube GitLab Kubernetes Operator deploys a fully configured GitLab instance with all features neatly abstracted through a Custom Resource Definition (CRD). Thanks to the Operator, the days of spending excessive time on setup and updates are over. Now, you can effortlessly spin up new GitLab instances within a mere 5 minutes. Try it out and give us feedback!

Install the Glasskube operator

The first step is to install Glasskube via its helm chart:

helm repo add glasskube https://charts.glasskube.eu/
helm repo update
helm install my-glasskube-operator glasskube/glasskube-operator

The full documentation can be found in our Getting started documentation.

Deploy GitLab

Important A DNS entry must be set on the LoadBalancer or the Ingress Host. SSL certificates are automatically generated by LoadBalancer or cert-manager if a ClusterIssuer is configured.

gitlab.yaml

apiVersion: "v1"
kind: "Secret"
metadata:
  name: "gitlab-smtp"
stringData:
  username: "..."
  password: "..."
---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-registry-secret
stringData:
  accessKey: "..."
  secretKey: "..."
---
apiVersion: "glasskube.eu/v1alpha1"
kind: "Gitlab"
metadata:
  name: "gitlab"
spec:
  host: "gitlab.mycompany.eu"
  sshEnabled: true
  sshHost: "ssh.gitlab.mycompany.eu"
  smtp:
    host: "..."
    port: 465
    fromAddress: "noreply+gitlab@mycompany.eu"
    authSecret:
      name: "gitlab-smtp"
    tlsEnabled: true
  registry:
    host: "registry.gitlab.mycompany.eu"
    storage:
      s3:
        bucket: gitlab-registry
        accessKeySecret:
          name: gitlab-registry-secret
          key: accessKey
        secretKeySecret:
          name: gitlab-registry-secret
          key: secretKey
        region: ...
kubectl apply -f gitlab.yaml

That's it! The complete custom resource documentation can be found in the Glasskube documentation about GitLab

Deploy GitLab Runner on Kubernetes

Runners on GitLab are execution agents that power continuous integration and continuous delivery (CI/CD) pipelines. They're responsible for running jobs, which are the individual steps or tasks within a pipeline.

The Glasskube Gitlab Kubernetes Operator makes it as simple as it gets to add runners to your Gitlab. First a new runner needs to be created via https://{{host}}/admin/runners/new. After that these runner tokens can just be added to gitlab.yaml spec and the Glasskube Kubernetes Operator will automatically spawn and connect these runners with the Gitlab instance.

  runners:
    - token: glrt-xxxxXX-xxxxxXXXXX # can be generated at https://{{host}}/admin/runners/new

The complete custom resource documentation can be found in the Glasskube documentation about GitLab runner

Now the installation is complete. Updates to the Kubernetes operator will automatically update GitLab.

Conclusion

Deploying a GitLab instance on Kubernetes for less than a thousand users shouldn't be done with GitLabs cloud native hybrid helm charts but rather with a special purposed installation of the GitLab Omnibus package in combination with cloud native infrastructure. The Glasskube Kubernetes operator takes care of that.

Support Glasskube
By leaving us a Star on GitHub
Star us
Glasskube Newsletter

Sign-Up to get the latest product updates and release notes!

Our solutions for reliable
and scalable infrastructure.

Easily and scale your IT infrastructure while deploying applications quickly and securely with our cloud native technology solutions.

Outdated software or technical debt?

Turn on autopilot