Kafka Engine

Last updated:

ClickHouse provides a table engine called Kafka Engine to consume Kafka messages, convert the messages into table rows, and persist the table rows into the destination table. Although a Kafka engine can be configured with multiple topics, a Kafka engine can only have one table schema defined.

PostHog uses the Kafka engine type for several tables:

Terminal
clickhouse-client --query "SELECT name FROM system.tables WHERE database = 'posthog' AND engine = 'Kafka'"
kafka_events
kafka_events_dead_letter_queue
kafka_person
kafka_person_distinct_id
kafka_plugin_log_entries
kafka_session_recording_events

Note: SELECT is not particularly useful for reading messages (except for debugging), because each message can be read only once. You will lose data if you manually read from this table, as you’ll artificially increment the internal offset ClickHouse uses to read from this table.

Questions?

Was this page useful?

Next article

Resize disk

How-to Connect to the Clickhouse container to verify the data directory filesystem size (in this example 20G) In your Helm chart configuration, update the clickhouse.persistence.size value in value.yaml to the target size (40G in this example) Run a helm upgrade Connect to the ClickHouse container to verify the new filesystem size Troubleshooting If the resize didn't work check for errors in the ClickHouse operator pod. This procedure doesn't work to decrease a volume. If you try, the disk…

Read next article