- 3 AM page 🚨 - Kafka broker is running out of disk (11.8TB out of 12TB used) 🚨 - You check everything: throughput, connections network. Nothing seems out of the ordinary 😥 - Retention settings are set to 1 day. It's one topic that hasn't been cleaned in weeks 🤷‍♂️ - You frantically Google, page, and ping team-mates 🙏 ... until you figure out ... the `log.message.timestamp.type` setting This setting defines what the timestamp on a Kafka record is for storage purposes on the broker. This timestamp is used, among other things, in the calculation of log retention. That is, when you have `log.retention.hours=24`, Kafka will delete a log file only when the LARGEST (latest) timestamp of any record in that log file is older than 24 hours. Back to the `timestamp.type`. There are two supported values: 🔸 LogAppendTime - the time at which the record was appended to the log (disk) 🔸CreateTime - the time that was provided by the Producer when constructing its message The DEFAULT is CreateTime. This means that if you have just one Producer client produce a message with a timestamp in the future -- e.g "01-01-2120" - your log file will never get deleted! This is a common gotcha and can cost you a night's (or a few) sleep. Ways to circumvent this: 💡- changing `log.message.timestamp.type` to AppendTime 💡- setting size-based retention limits too - `log.retention.bytes` This will ensure you have a maximum upper bound per topic that you'll never breach 👌