Nextcloud self-hosting on K8s

Jan 7, 2021 12 min read

Goal:

Self-host Nextcloud on Kubernetes and use it as server-side for the /e operating system

Nextcloud offers an on-premises content collaboration platform, which is open-source and free. The Nextcloud server is written in the PHP and JavaScript scripting languages.

The /e/ ROM is a fork of Android (more precisely from LineageOS).

See previous post to see how to install the OS on a LG G3 and my efforts to self-host the /e server beta version.

The /e self-hosting exercise turned to be quite cumbersome due to the installation script of the beta version, which is one of these do-everything-in-one scripts difficult to troubleshoot und which makes maintenance and updates overly complicated.

However, the /e server-side is in fact just a composition of services and their configuration:

Nextcloud, as file synchronization software
Postfix, to host the own mail server
OnlyOffice, to allow editing of documents

I was actually only intereeted in photos and files synchronization. Basing on this premise, I deciced to try to install Nextcloud directly and see if I could use it as the server side for the /e operating system on the phone. The nice guys in charge of /e development told me that this should be possible and advise me to do so.

As a hosting environment I wanted to use Kubernetes - as usual. This time - instead of deploying directly on either my K8s or my K3s cluster on premises - I wanted to try the Digital Ocean K8s services. If the experiment was succesfull, I would move the setup to my home network afterwards, to use nextcloud to sync my files and photos.

NOTE: another reason was that by the time of this writting I was on vacation with no access to my home network ;)

Pre-conditions

A DigitalOcean account to host the K8s cluster
A client device to test the installation
A hosted public domain (mydomain.com)
A DNS provider to register the necessary DNS entries

I had a DigitalOcean account and I was still enjoying the free credit I got when signing up for the first time. The client device will be the LG G3 where /e already was running. My public domain was hosted by GoDaddy and as DNS provider I would use Cloudfare as I always do.

Milestones

Set up a K8s cluster on DigitalOcean
Identify software components and write K8s manifests
Set up DNS
Ingress
Certmanager and cluster issuer
Deployment with Kustomize
Register /e client with nextcloud account

1. Set up a K8s cluster

1.1. Create and configure the new cluster

Setting up a K8s cluster on DigitalOcean was easy:

I selected the latest K8s version, which was the default
I selected the datacenter closest to my current location
I selected one single basic node with 2.5 GB RAM usable (4 GB Total)/2 vCPUs
As cluster name, I typed eramonk8s

NOTE: jumping a little ahead, I have to say this setup was painfully slow. More nodes and a little more RAM would have been more convenient.

1.2. Install and configure kubectl and doctl

As I waited for the provisioning of the cluster, I installed version 1.19.3 of kubectl:

eramon@pacharan:~/dev$ curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.19.3/bin/linux/amd64/kubectl

NOTE: the kubectl version must match the K8s version of the cluster

eramon@pacharan:~/dev$ chmod +x ./kubectl
eramon@pacharan:~/dev$ sudo mv ./kubectl /usr/local/bin/kubectl
eramon@pacharan:~/dev$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

I also downloaded the doctl tool as recommended by the DigitalOcean wizard:

eramon@pacharan:~/dev$ wget https://github.com/digitalocean/doctl/releases/download/v1.54.0/doctl-1.54.0-linux-amd64.tar.gz
eramon@pacharan:~/dev$ tar xvzf doctl-1.54.0-linux-amd64.tar.gz
eramon@pacharan:~/dev$ sudo mv doctl /usr/local/bin/

I used doctl for automated certificate management to access the K8s API with kubectl and download of the configuration file. Following the instructions, I first ran:

eramon@pacharan:~/dev$ doctl auth init
Please authenticate doctl for use with your DigitalOcean account. You can generate a token in the control panel at https://cloud.digitalocean.com/account/api/tokens

Enter your access token: 
Validating token... OK

I created the access token on the URL above and gave it the same name as the cluster. I copied it to the clipboard and pasted it on the console as instructed.

Then I used doctl to generate the configuration file:

eramon@pacharan:~/dev$ doctl kubernetes cluster kubeconfig save eramonk8s
Notice: Adding cluster credentials to kubeconfig file found in "/home/eramon/.kube/config"
Notice: Setting current-context to do-fra1-eramonk8s

After this I was able to connect to my cluster via kubectl:

eramon@pacharan:~/dev$ kubectl cluster-info
Kubernetes master is running at https://73c21301-5143-42fa-ab21-1c4c54e9b1f0.k8s.ondigitalocean.com
CoreDNS is running at https://73c21301-5143-42fa-ab21-1c4c54e9b1f0.k8s.ondigitalocean.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Continuing with the wizard on the browser, I installed the following 1-Click Apps:

NGINX Ingress Controller
Kubernetes Monitoring Stack

Over the control panel I was able to access the Kubernets Dashboard to monitor the status of the cluster :)

2. Software components, docker images and K8s manifests

Nextcloud is built upon the following components:

Persistent storage to save content
MariaDB for metadata storage
The nextcloud webapp: PHP + Apache

For persistent storage I would use the persistent volumes offered as service by DigitalOcean.

For both mariadb and nextcloud there are official docker images available in dockerhub.

With all this in mind, I was able to start writting the K8s manifests.

2.1 Secrets

Kubernetes Secrets let you store and manage sensitive information, such as passwords.

I started with the secrets I would need for the mariadb and nextcloud deployments later. Usually I generate the secrets manually, this time I used a manifest and included it as part of kustomization.yaml, just taking care not to commit the yaml file containing the passwords, but a example version of it with placeholders.

Kustomize provides a way to customize application configuration and it is built into kubectl as “apply -k”.

Following secrets are needed for the mariadb and nextcloud containers:

A MYSQL_DATABASE which I would set to nextcloud, needed for both mariadb and nextcloud.
A MYSQL_USER which I would set to nextcloud too, needed for both mariadb and nextcloud.
A MYSQL_PASSWORD which I would generate with pwgen and encode to have a base-64-encoded string, needed for both mariadb and nextcloud.
A MYSQL_ROOT_PASSWORD, which I would set - to simplify - with the same value as the MYSQL_PASSWORD

Install pwgen:

eramon@pacharan:~/dev/kubenextcloud$ sudo apt-get install pwgen

Use pwgen to generate a random password for mariadb:

eramon@pacharan:~/dev/kubenextcloud$ echo -n `pwgen -s -1 16`

Note the output.

As mentioned, I used this password for both the MYSQL_PASSWORD and the MYSQL_ROOT_PASSWORD.

For MYSQL_DATABASE and MYSQL_USER, I set nextcloud, use openssl to base64-encode “nextcloud”:

eramon@pacharan:~/dev/kubenextcloud$ echo -n 'nextcloud' | openssl base64
bmV4dGNsb3Vk

Having all the values, write a manifest for the secrets, using the generated passwords:

apiVersion: v1
kind: Secret
metadata:
  name: mariadb-secrets
type: Opaque
data:
  MYSQL_DATABASE: bmV4dGNsb3Vk 
  MYSQL_USER: bmV4dGNsb3Vk 
  MYSQL_PASSWORD: base64-encoded-pwgen-generated-password 
  MYSQL_ROOT_PASSWORD: base64-encoded-pwgen-generated-password

Create a new kustomize.yaml file and included the manifest for the secrets there.

2.2 Persistent Volumes

A PersistentVolumeClaim (PVC) is a request for storage by a user. A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned.

To see the possibilities for the creation of a persistent volume with DigitalOcean, I got to the Volumes section on the Manage menu.

NOTE: one persistent volume claim was already created - apparently it was automatically set for the use of the monitoring tool.

I went to the How to Add Block Storage Volumes to Kubernets Clusters tutorial. Taking the code from the example, I created a manifest for a 3GB block storage volume to persist the end user data managed by nextcloud:

nextcloud-pvc.yaml

I created an additional volume for the storage of the mariadb metadata with size 2GB, to persist the nextcloud metadata stored on mariadb:

mariadb-pvc.yaml

For managing all manifests and easing the deployment, I would use kustomize.

Add the pvc manifests to kustomization.yaml.

2.3 MariaDB

Create manifest file for the mariadb deployment:

mariadb-deployment.yaml

A Deployment provides declarative updates for Pods and ReplicaSets.

The mariadb docker image allows the creation of a database upon creation of the container. When you start the mariadb image, you can adjust the configuration of the MariaDB instance by passing one or more environment variables on the docker run command line. Do note that none of the variables below will have any effect if you start the container with a data directory that already contains a database: any pre-existing database will always be left untouched on container startup.

The deployment manifest requires configuring the following environment variables:

MYSQL_ROOT_PASSWORD: specifies the password that will be set for the MariaDB root superuser account (mandatory)
MYSQL_DATABASE: allows to specify the name of a database to be created on image startup (optional)
MYSQL_USER, MYSQL_PASSWORD: used in conjunction to create a new user and to set that user’s password (optional)

NOTE In order for mariadb to re-create the nextcloud database, the persistent volume must be deleted. It’s not enough with delete the container. The environment variables only work if the database is started for the first time.

Create manifest file for the mariadb service:

mariadb-service.yaml

A service is an abstract way to expose an application running on a set of pods as a network service.

Add the deployment and service manifest files to kustomization.yaml.

2.4 Redis

The performance of nextcloud server can be improved using memory caching. Redis provides local and distributed caching as well as transactional file locking.

Create the manifest file for the redis deployment:

redis-deployment.yaml

Create the manifest file for the redis service:

redis-service.yaml

For nextcloud to use redis, we’ll need to set the environment variable REDIS_HOST on the nextcloud container.

2.5 Nextcloud

Create the manifest file for the nextcloud deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nextcloud 
  labels:
    app: nextcloud 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nextcloud 
  template:
    metadata:
      labels:
        app: nextcloud
    spec:
      volumes:
      - name: nextcloud-storage
        persistentVolumeClaim: 
          claimName: nextcloud-pvc
      containers:
        - image: nextcloud:apache
          name: nextcloud 
          ports:
            - containerPort: 80
          env:
            - name: REDIS_HOST
              value: redis
            - name: MYSQL_HOST
              value: mariadb 
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  key: MYSQL_DATABASE
                  name: mariadb-secrets
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: MYSQL_PASSWORD
                  name: mariadb-secrets
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  key: MYSQL_USER
                  name: mariadb-secrets
            - name: NEXTCLOUD_ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: MYSQL_PASSWORD
                  name: mariadb-secrets 
            - name: NEXTCLOUD_ADMIN_USER
              value: "admin"
            - name: NEXTCLOUD_TRUSTED_DOMAINS 
              value: mydomain.com 
          volumeMounts:
            - mountPath: /var/www/html
              name: nextcloud-storage

The nextcloud image supports auto-configuration via environment variables. You can preconfigure everything that is asked on the install page on first run:

MYSQL_DATABASE: Name of the database using mysql / mariadb
MYSQL_USER: Username for the database using mysql / mariadb
MYSQL_PASSWORD: Password for the database user using mysql / mariadb
MYSQL_HOST: Hostname of the database server using mysql / mariadb

With a complete configuration by using all variables for your database type, you can additionally configure your Nextcloud instance by setting admin user and password:

NEXTCLOUD_ADMIN_USER: Name of the Nextcloud admin user. I used admin.
NEXTCLOUD_ADMIN_PASSWORD: Password for the Nextcloud admin user. For simplification - since this is just a experimental setup - I configured it to be the same password as the one used for the nextcloud database.

One or more trusted domains can be set through environment variable too:

NEXTCLOUD_TRUSTED_DOMAINS: optional space-separated list of IPs or domains. The IP of the loadbalancer or the domain mydomain.com must be configured here.

In order for nextcloud to see and use the redis service, we need to include an additional environment variable:

REDIS_HOST: name of the redis service which in this case is redis.

Create manifest file for the nextcloud service:

nextcloud-service.yaml

Add the nextcloud deployment and service manifest files to kustomization.yaml.

3. Ingress

Kubernetes Ingresses allow you to flexibly route traffic from outside your Kubernetes cluster to Services inside of your cluster. This is accomplished using Ingress Resources, which define rules for routing HTTP and HTTPS traffic to Kubernetes Services, and Ingress Controllers, which implement the rules by load balancing traffic and routing it to the appropriate backend Services.

I had already installed the nginx-ingress-controller as 1-click app when setting up the cluster at the beginning. To make sure it was running:

eramon@pacharan:~/dev/kubenextcloud$ kubectl get pods -n ingress-nginx
NAME                                                      READY   STATUS    RESTARTS   AGE
nginx-ingress-ingress-nginx-controller-7898d5969d-sgph4   1/1     Running   0          110m

Create manifest file for ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nextcloud-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    cert-manager.io/acme-challenge-type: http01
spec:
  tls:
  - hosts: 
    - nextcloud.mydomain.com 
    secretName: nextcloud-tls 
  rules:
  - host: nextcloud.mydomain.com 
    http: 
      paths: 
      - path: /
        backend:
          serviceName: nextcloud
          servicePort: 80

Don’t forget to include the tls directive and the certbot and letsencrypt anotations for the ssl server certificate to be automatically generated by cerbot.

Add the ingress manifest file to kustomization.yaml.

4. DNS

I have my domain mydomain.com registered and hosted by godaddy. I use Cloudflare to manage the DNS configuration for my projects.

I listed the services on namespace nginx-ingress to find out the public IP of the load balancer of the cluster:

eramon@pacharan:~/dev/kubenextcloud$ kubectl get service -n ingress-nginx
NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
nginx-ingress-ingress-nginx-controller             LoadBalancer   10.245.56.207   PUBLICIP	    80:32519/TCP,443:30323/TCP   111m

On Cloudflare, I added a new DNS entry:

Type: A record
Name: nextcloud.mydomain.com
Content: PUBLIC_IP

5. Certmanager and Cluster Issuer

Install certmanager and its custom resource definitions directly from manifest:

eramon@pacharan:~/dev/kubenextcloud$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager.yaml

Verify the installation:

eramon@pacharan:~/dev/kubenextcloud$ kubectl get pods --namespace cert-manager
NAME                                      READY   STATUS    RESTARTS   AGE
cert-manager-cainjector-fc6c787db-g9d5g   1/1     Running   0          67s
cert-manager-d994d94d7-7fzkd              1/1     Running   0          67s
cert-manager-webhook-845d9df8bf-d98vl     1/1     Running   0          66s

After installing certmanager, write a manifest for the cluster issuer:

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # EMail address used for ACME registration
    email: nextcloud@mydomain.com
    # Name of a secret to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod

    # Enable HTTP01 challenge provider using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Issuers (and ClusterIssuers) represent a certificate authority from which signed x509 certificates can be obtained, such as Let’s Encrypt.

Add the cluster issuer to kustomization.yaml.

6. Deployment

After completing all manifests and including them in kustomization.yaml, the file looked like this:

kustomization.yaml

Finally, to deploy everything:

eramon@pacharan:~/dev/kubenextcloud$ kubectl apply -k .

When accessing nextcloud.mydomain.com I was greeted with the login page. I logged in as Administrator and created a user eramon in the administration console, with e-mail eramon@mydomain.com (the e-mail is used as login name but it’s arbitrary otherwise, since we’re not managing an own mail server) and a password.

I logged out and logged in again, this time using the newly created user.