🌇 Sunsetting Kubernetes Deployments
This page covers our PostHog Kubernetes deployment, which we are currently in the process of sunsetting. Existing customers will receive support until May 31, 2023 and we will continue to provide security updates for the next year.
For existing customers
We highly recommend migrating to PostHog Cloud (US or EU). Take a look at this guide for more information on the migration process.Looking to continue self-hosting?
We still maintain our Open-source Docker Compose deployment. Instructions for deploying can be found here.
Requirements
You need to run a Kubernetes cluster with the Volume Expansion feature enabled. This feature is supported on the majority of volume types since Kubernetes version >= 1.11 (see docs).
Details
PersistentVolumes
can be configured to be expandable. This feature when set to true
, allows the users to resize the volume by editing the corresponding PersistentVolumeClaims
object.
This can become useful in case your storage usage grows and you want to resize the disk on-the-fly without having to resync data across PVCs.
To verify if your storage class allows volume expansion you can run:
kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'true
In case it returns false
, you can enable volume expansion capabilities for your storage class by running:
DEFAULT_STORAGE_CLASS=$(kubectl get storageclass -o=jsonpath='{.items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")].metadata.name}')kubectl patch storageclass "$DEFAULT_STORAGE_CLASS" -p '{"allowVolumeExpansion": true}'storageclass.storage.k8s.io/gp2 patched
N.B:
- expanding a persistent volume is a time consuming operation
- some platforms have a per-volume quota of one modification every 6 hours
- not all the volume types support this feature. Please take a look at the official docs for more info
How-to
Connect to the Clickhouse container to verify the data directory filesystem size (in this example 20G)
Terminalkubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bashclickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/Filesystem Size Used Avail Use% Mounted on/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c 20G 50M 19G 1% /var/lib/clickhouseIn your Helm chart configuration, update the
clickhouse.persistence.size
value invalue.yaml
to the target size (40G in this example)Run a
helm
upgradeConnect to the ClickHouse container to verify the new filesystem size
Terminalkubectl -n posthog exec -it chi-posthog-posthog-0-0-0 -- /bin/bashclickhouse@chi-posthog-posthog-0-0-0:/$ df -h /var/lib/clickhouse/Filesystem Size Used Avail Use% Mounted on/dev/disk/by-id/scsi-0DO_Volume_pvc-f39035c1-c68c-4572-81f2-273de6eb088c 40G 186M 38G 1% /var/lib/clickhouse
Troubleshooting
If the resize didn't work check for errors in the ClickHouse operator pod.
This procedure doesn't work to decrease a volume. If you try, the disk won't be resized and the following errors can be seen in the ClickHouse operator pod
posthog/posthog:ERROR unable to reconcile PVC(posthog/data-volumeclaim-template-chi-posthog-posthog-0-0-0) err: PersistentVolumeClaim "data-volumeclaim-template-chi-posthog-posthog-0-0-0" is in valid: spec.resources.requests.storage: Forbidden: field can not be less than previous value
For increases double check what the storage class is that's used for ClickHouse and make sure it's expandable as mentioned in requirements above.
kubectl get pvc data-volumeclaim-template-chi-posthog-posthog-0-0-0 -o json | jq '.spec.storageClassName'
If a storage class with that name doesn't exist (kubectl get storageclass
), then it's probably because major upgrade steps from 9.x.x weren't completed. Set installCustomStorageClass: true
& run a helm upgrade
first (without the storage increase request).