Kafka 0.11.0.0 has been running in production. We see that log compaction of the consumer offsets topic is not happening. In the consumer offset partitions, we see log segments remaining there for the last 3 months. Log cleaner logs showed that it failed building the map for compaction due to "CorruptRecordException".
Since there were a lot of segment files each of size 100mb in the partitions, instead of taking a DumpLogSegements and finding the bad segment, we decided to go ahead and delete the old segment files and keep only the ones from the last 3 days. After this, we restarted kafka and it seemed to work fine.
But in 2 days of doing this, we are seeing the logs getting built up again, just as it did before. We no longer see a corruptRecord Exception in the logs, but the offsets are not getting compacted and its been 7 days since.
None of the default values for compaction or retention were changed. preallocate is also set to false. Can anybody give me any insight of what could be going on here?
Edit: The CorruptRecordException that I was running into seems to originate from AbstractLegacyRecordBatch.java
long offset = offsetAndSizeBuffer.getLong(Records.OFFSET_OFFSET);
int size = offsetAndSizeBuffer.getInt(Records.SIZE_OFFSET);
if (size < LegacyRecord.RECORD_OVERHEAD_V0)
throw new CorruptRecordException(String.format("Record size is less than the minimum record overhead (%d)", LegacyRecord.RECORD_OVERHEAD_V0));
Any idea about when this can occur and why the compaction is not happeneing even after the old segments are deleted.