Last Commits Not Being Sent to Broker

[joewood]

I'm seeing a new issue since upgrading to 1.4.
A consumer group subscribing to a topic receives messages up to the final producer offset. The commit offsets are updated correctly on the broker. The issue seems to be that the final batch of messages are not committed - leaving a residual set of messages. If I query the commits directly from the broker I see that the commits and offsets on the partition never match. If I restart the consumers they will re-read the same last set of messages again. The consumers are set-up with fromOffset:"latest". I've tried with autoCommit and manually calling commit.

This is running version 1.4 against a 0.10.1.1 broker cluster (3 nodes). Connecting with zookeeper.

[joewood]

I ran the consumers with logging on and I don't see any errors. The code path is correctly going down the v2 branch in autoCommit, calling client.sendOffsetCommitV2Request.

[hyperlink]

Maybe caused by #568 ? Does 1.3.1 have this issue?

[joewood]

Yeah, 1.3.1 works fine.

[joewood]

Is the issue here when the call to autoCommit after updateOffsets fails because there's already a commit in progress. The next call to updateOffsets may not update the offset for that partition and therefore never get flushed to the broker. The only real way of doing this properly is to keep the previously successfully written offset and compare the two.