Clean up terminated shards before expiry

[istreeter]

The KinesisClientLibConfiguration takes a parameter cleanupTerminatedShardsBeforeExpiry, described here.

The default in the KinesisClientLibConfiguration is true. But the default in kinesis connector configuration is false. In the upgrade from version 1.0.0 to 2.1.0 we changed to use the kinesis connector default, instead of the kinesis client default.

I believe this explains an anomaly we have seen in cloudwatch metrics, in which it appears streams are lagging when they are not, especially after resharding events.

We should also check what other parameters got accidentally changed between versions 1.0.0 and 2.1.0.

[istreeter]

Update: I confirmed that using true for cleanupTerminatedShardsBeforeExpiry fixes the problem in cloudwatch metrics, in which it appears streams are lagging when they are not, especially after resharding events.

My description earlier was not completely correct though: we had it set to false even in version 1.0.0. However, the KCL libraries are vastly different between version 1.7 and 1.14, so it is not a surprise to me that we didn't see this problem in s3 loader 1.0.0.