HDDS-10338. Implement a Client Datanode API to stream a block

[chungen0126]

What changes were proposed in this pull request?

To reduce round trips between the Client and Datanode for reading a block, we nee a new API to read.

Client -> block(offset, length) -> Datanode
Client <- chunkN <- Datanode
Client <- chunkN+1 <- Datanode
..
Client <-chunkLast <- Datanode

This is using the ability of gRPC to send bidirectional traffic such that the server can pipeline the chunks to the client without waiting for ReadChunk API calls. This also avoids the client from creating multiple Chunk Stream Clients and should simplify the read path on the client side by a bit.

Please describe your PR in detail:

• Add a new logic at both client and server side to read block as streaming chunks.
• Add a new StreamBlockInput at client side called from KeyInputStream to read a block from the container.
• Add unit tests and integration tests for `StreamBlockInput.
• Add a new version in datanode for compatibilities, while new client reading blocks from old server, it will fallback and read blocks by BlockInputStream.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10338

How was this patch tested?

There are existed test for reading data.

[ChenSammi]

Could you add add an end to end case, to test read an empty file with an empty block?

[chungen0126]

Add testReadEmptyBlock in the TestStreamBlockInputStream to test read an empty block.

[ChenSammi]

@chungen0126 , thanks for the quick patch updating. I will try to finish the review this week.

[ptlrs]

Thanks for the PR @chungen0126. I have a few comments.

[ptlrs]

Can we add some tests which cover the conditions where BYTES_PER_CHECKSUM is equal-to and greater than CHUNK_SIZE?

[chungen0126]

I don't think we need to add this test case. Logically, BYTES_PER_CHECKSUM should be smaller than CHUNK_SIZE. Otherwise, a single chunk file cannot be verified.

[ptlrs]

Test conducted on our cluster(3DN / HDD / 10 Gigabit network) shows this improvement can boost read speed by at least 30%.

It can be seen that for 1GB file reading, streaming reading can reduce the reading time by about 30%.

Hi @chungen0126 @guohao-rosicky, do we know how much of an improvement is being seen for files smaller than 1GB?

[fenixjin]

Test conducted on our cluster(3DN / HDD / 10 Gigabit network) shows this improvement can boost read speed by at least 30%.

It can be seen that for 1GB file reading, streaming reading can reduce the reading time by about 30%.

Hi @chungen0126 @guohao-rosicky, do we know how much of an improvement is being seen for files smaller than 1GB?

@ptlrs
conducted test using freon on the latest version of this pr, hardware is 3DN / HDD / 10 Gigabit network, tested file sizes are 1GB 128MB 16MB 2MB 256KB. there is an index bug when verifying checksum, tests were conducted after fixing it.
here are the redunctions in mean read time given by freon：

• 1GB: 7%
• 128MB: 10%
• 16MB 8%
• 2MB 10%
• 256KB 8%

further tests were conducted with verifying checksum skipped, it turns out checksum have a substantial impact on read time.
when testing using freon on 1GB file, turning on checksum caused read time to increase from around 3700ms to 5000ms.
with checksum off, the read time reduction of stream block compared with current read method(checksum also off) is:

• 1GB: 30%
• 128MB: 21%
• 16MB: 11%
• 2MB: -5.6% (chunk size is 4MB)

[guohao-rosicky]

Can you resolve the code conflict? @chungen0126

[adoroszlai]

Temporarily converted to draft and assigned to myself, to resolve conflicts.

[chungen0126]

Thanks @adoroszlai for fixing the conflicts. I was just about to address it.