Resize disk

Last updated:

🌇 Sunsetting Kubernetes Deployments

This page covers our PostHog Kubernetes deployment, which we are currently in the process of sunsetting. Existing customers will receive support until May 31, 2023 and we will continue to provide security updates for the next year.

For existing customers
We highly recommend migrating to PostHog Cloud (US or EU). Take a look at this guide for more information on the migration process.
Looking to continue self-hosting?
We still maintain our Open-source Docker Compose deployment. Instructions for deploying can be found here.

Requirements

You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).

Details

PersistentVolumes can be configured to be expandable. This feature when set to true, allows the users to resize the volume by editing the corresponding PersistentVolumeClaims object.

This can become useful in case your storage usage grows and you want to resize the disk on-the-fly without having to resync data across PVCs.

To verify if your storage class allows volume expansion you can run:

Terminal
kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true

In case it returns false, you can enable volume expansion capabilities for your storage class by running:

Terminal
DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')
kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/gp2 patched

N.B:

  • expanding a persistent volume is a time consuming operation
  • some platforms have a per-volume quota of one modification every 6 hours
  • not all the volume types support this feature. Please take a look at the official docs for more info

How-to

  1. Connect to the Clickhouse container to verify the data directory filesystem size (in this example 20G)

    Terminal
    kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
    clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
    Filesystem Size Used Avail Use% Mounted on
    /dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c 20G 50M 19G 1% /var/lib/clickhouse
  2. In your Helm chart configuration, update the clickhouse.persistence.size value in value.yaml to the target size (40G in this example)

  3. Run a helm upgrade

  4. Connect to the ClickHouse container to verify the new filesystem size

    Terminal
    kubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bash
    clickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/
    Filesystem Size Used Avail Use% Mounted on
    /dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c 40G 186M 38G 1% /var/lib/clickhouse

Troubleshooting

If the resize didn't work check for errors in the ClickHouse operator pod.

This procedure doesn't work to decrease a volume. If you try, the disk won't be resized and the following errors can be seen in the ClickHouse operator pod

posthog/posthog:ERROR unable to reconcile PVC(posthog/data-volumeclaim-template-chi-posthog-posthog-0-0-0) err: PersistentVolumeClaim "data-volumeclaim-template-chi-posthog-posthog-0-0-0" is in valid: spec.resources.requests.storage: Forbidden: field can not be less than previous value

For increases double check what the storage class is that's used for ClickHouse and make sure it's expandable as mentioned in requirements above.

Terminal
kubectl get pvc data-volumeclaim-template-chi-posthog-posthog-0-0-0 -o json | jq '.spec.storageClassName'

If a storage class with that name doesn't exist (kubectl get storageclass), then it's probably because major upgrade steps from 9.x.x weren't completed. Set installCustomStorageClass: true & run a helm upgrade first (without the storage increase request).

Questions?

Was this page useful?

Next article

Restore

A backup is worthless if the restoration process hasn’t been tested. If you backed something up and never tried to restore it, chances are that restore will not work properly when you actually need it or it will take longer than business can tolerate. Perform regular test restores to ensure your data will be there when you need it! Automated using clickhouse-backup The tool provides an automatic restore operation that you can invoke by running: clickhouse-backup restore <BACKUP NAME> . For…

Read next article