ClickHouse provides a table engine called Kafka Engine to consume Kafka messages, convert the messages into table rows, and persist the table rows into the destination table. Although a Kafka engine can be configured with multiple topics, a Kafka engine can only have one table schema defined.
PostHog uses the Kafka
engine type for several tables:
clickhouse-client --query "SELECT name FROM system.tables WHERE database = 'posthog' AND engine = 'Kafka'"kafka_eventskafka_events_dead_letter_queuekafka_personkafka_person_distinct_idkafka_plugin_log_entrieskafka_session_recording_events
Note: SELECT
is not particularly useful for reading messages (except for debugging), because each message can be read only once. You will lose data if you manually read from this table, as you’ll artificially increment the internal offset ClickHouse uses to read from this table.