Skip to content
This repository was archived by the owner on Jan 5, 2021. It is now read-only.

Commit bf961c8

Browse files
MiloszKrajewskiMiloszKrajewski
committed
readme
1 parent dcb781d commit bf961c8

File tree

3 files changed

+270
-2
lines changed

3 files changed

+270
-2
lines changed

‎README.md

+180-2
Original file line numberDiff line numberDiff line change
@@ -56,14 +56,14 @@ So I ended up with 4 implementations:
5656
* Cons:
5757
* Requires VC++ 2010 Redistributable to be installed on target machine ([x86](http://www.microsoft.com/en-us/download/details.aspx?id=5555) and/or [x64](http://www.microsoft.com/en-us/download/details.aspx?id=14632)
5858
* Contains unmanaged code, may not be allowed in some environments
59-
* Does not have AnyCPU configuration (this can be solved though, see: [Automiatic loading of x86 or x64])
59+
* Does not have AnyCPU configuration (this can be solved though, see: [Automatic loading of x86 or x64])
6060
* **C++/CLI** - original C sources recompiled for CLR
6161
* Pros:
6262
* Almost as fast as Mixed Mode
6363
* Only managed code
6464
* Cons:
6565
* Contains unsafe code, may not be allowed in some environments
66-
* Does not have AnyCPU configuration (this can be solved though, see: [Automiatic loading of x86 or x64])
66+
* Does not have AnyCPU configuration (this can be solved though, see: [Automatic loading of x86 or x64])
6767
* **unsafe C#** - C# but still fast
6868
* Pros:
6969
* C# (more .NET-ish)
@@ -91,3 +91,181 @@ Plus class which chooses the best available implementation for the job: [One cla
9191
* (1) It looks like .NET Standard is picked anyway on Xamarin, so the "portable" version may be obsolete.
9292
* (2) Still experimental but seems to be working
9393
* (3) I've tested it on Android 6.0 (Nexus 7) and Android 2.3.5 (ancient HTC Desire HD)
94+
95+
## Use with streams
96+
97+
This LZ4 library can be used in two distinctive ways: to compress streams and packets. Compressing streams follow decorator pattern: `LZ4Stream` is-a `Stream` and takes-a `Stream`. Let's start with some imports as text we are going to compress:
98+
99+
```csharp
100+
using System;
101+
using System.IO;
102+
using LZ4;
103+
104+
const string LoremIpsum =
105+
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla sit amet mauris diam. " +
106+
"Mauris mollis tristique sollicitudin. Nunc nec lectus nec ipsum pharetra venenatis. " +
107+
"Fusce et consequat massa, eu vestibulum erat. Proin in lectus a lacus fermentum viverra. " +
108+
"Aliquam vel tellus aliquam, eleifend justo ultrices, elementum elit. " +
109+
"Donec sed ullamcorper ex, ac sagittis ligula. Pellentesque vel risus lacus. " +
110+
"Proin aliquet lectus et tellus tristique, eget tristique magna placerat. " +
111+
"Maecenas ut ipsum efficitur, lobortis mauris at, bibendum libero. " +
112+
"Curabitur ultricies rutrum velit, eget blandit lorem facilisis sit amet. " +
113+
"Nunc dignissim nunc iaculis diam congue tincidunt. Suspendisse et massa urna. " +
114+
"Aliquam sagittis ornare nisl, quis feugiat justo eleifend iaculis. " +
115+
"Ut pulvinar id purus non convallis.";
116+
```
117+
118+
Now, we can write this text to compressed stream:
119+
120+
```csharp
121+
static void WriteToStream()
122+
{
123+
using (var fileStream = new FileStream("lorem.lz4", FileMode.Create))
124+
using (var lz4Stream = new LZ4Stream(fileStream, LZ4StreamMode.Compress))
125+
using (var writer = new StreamWriter(lz4Stream))
126+
{
127+
for (var i = 0; i < 100; i++)
128+
writer.WriteLine(LoremIpsum);
129+
}
130+
}
131+
```
132+
133+
and read it back:
134+
135+
```csharp
136+
static void ReadFromStream()
137+
{
138+
using (var fileStream = new FileStream("lorem.lz4", FileMode.Open))
139+
using (var lz4Stream = new LZ4Stream(fileStream, LZ4StreamMode.Decompress))
140+
using (var reader = new StreamReader(lz4Stream))
141+
{
142+
string line;
143+
while ((line = reader.ReadLine()) != null)
144+
Console.WriteLine(line);
145+
}
146+
}
147+
```
148+
149+
`LZ4Stream` constructor requires inner stream and compression mode, plus takes some optional arguments, but their defaults are relatively sane:
150+
151+
```csharp
152+
LZ4Stream(
153+
Stream innerStream,
154+
LZ4StreamMode compressionMode,
155+
LZ4StreamFlags compressionFlags = LZ4StreamFlags.Default,
156+
int blockSize = 1024*1024);
157+
```
158+
159+
where:
160+
161+
```csharp
162+
enum LZ4StreamMode {
163+
Compress,
164+
Decompress
165+
};
166+
167+
[Flags] enum LZ4StreamFlags {
168+
None,
169+
InteractiveRead,
170+
HighCompression,
171+
IsolateInnerStream,
172+
Default = None
173+
}
174+
```
175+
176+
`compressionMode` configures `LZ4Stream` to either `Compress` or `Decompress`. `compressionFlags` is optional argument and allows to:
177+
178+
* use `HighCompression` mode, which provides better compression ratio for the price of performance. This is relevant on compression only.
179+
* use `IsolateInnerStream` mode to leave inner stream open after disposing `LZ4Stream`.
180+
* use `InteractiveRead` mode to read bytes as soon as they are available. This option may be useful when dealing with network stream, but not particularly useful with regular file streams.
181+
182+
`blockSize` is set 1MB by default but can be changed. Bigger `blockSize` allows better compression ratio, but uses more memory, stresses garbage collector and increases latency. It might be worth to experiment with it.
183+
184+
## Use with byte arrays
185+
186+
You can also compress byte arrays. It is useful when compressed chunks are relatively small and their size in known when compressing. `LZ4Codec.Wrap` compresses byte array and returns byte array:
187+
188+
```csharp
189+
static string CompressBuffer()
190+
{
191+
var text = Enumerable.Repeat(LoremIpsum, 5).Aggregate((a, b) => a + "\n" + b);
192+
193+
var compressed = Convert.ToBase64String(
194+
LZ4Codec.Wrap(Encoding.UTF8.GetBytes(text)));
195+
196+
return compressed;
197+
}
198+
```
199+
200+
In the example above we a little bit more, of course: first we concatenate multiple strings (`Enumerable.Repeat(...).Aggregate(...)`), then encode text as UTF8 (`Encoding.UTF8.GetBytes(...)`), then compress it (`LZ4Codec.Wrap(...)`) and then encode it with Base64 (`Convert.ToBase64String(...)`). On the end we have base64-encoded compressed string.
201+
202+
To decompress it we can use something like this:
203+
204+
```csharp
205+
static string DecompressBuffer(string compressed)
206+
{
207+
var lorems =
208+
Encoding.UTF8.GetString(
209+
LZ4Codec.Unwrap(Convert.FromBase64String(compressed)))
210+
.Split('\n');
211+
212+
foreach (var lorem in lorems)
213+
Console.WriteLine(lorem);
214+
}
215+
```
216+
217+
Which is a reverse operation: decoding base64 string (`Convert.FromBase64String(...)`), decompression (`LZ4Codec.Unwrap(...)`), decoding UTF8 (`Encoding.UTF8.GetString(...)`) and splitting the string (`Split('\n')`).
218+
219+
## Compatibility
220+
221+
Both `LZ4Stream` and `LZ4Codec.Wrap` is not compatible with original LZ4. It is an outstanding task to implement compatible streaming protocol and, to be honest, it does not seem to be likely in nearest future, but...
222+
223+
If you want to do it yourself, you can. It requires a little bit more understanding though, so let's look at "low level" compression. Let's create some compressible data:
224+
225+
```charp
226+
var inputBuffer =
227+
Encoding.UTF8.GetBytes(
228+
Enumerable.Repeat(LoremIpsum, 5).Aggregate((a, b) => a + "\n" + b));
229+
```
230+
231+
we also need to allocate buffer for compressed data.
232+
Please note, it might be actually more than input data (as not all data can be compressed):
233+
234+
```csharp
235+
var inputLength = inputBuffer.Length;
236+
var maximumLength = LZ4Codec.MaximumOutputLength(inputLength);
237+
var outputBuffer = new byte[maximumLength];
238+
```
239+
240+
Now, we can run actual compression:
241+
242+
```csharp
243+
var outputLength = LZ4Codec.Encode(
244+
inputBuffer, 0, inputLength,
245+
outputBuffer, 0, maximumLength);
246+
```
247+
248+
`Encode` method returns number of bytes which were actually used. It might be less or equal to `maximumLength`. It me be also `0` (or less) to indicate that compression failed. This happens when provided buffer is too small.
249+
250+
Buffer compressed this way can be decompressed with:
251+
252+
```csharp
253+
LZ4Codec.Decode(
254+
inputBuffer, 0, inputLength,
255+
outputBuffer, 0, outputLength,
256+
true);
257+
```
258+
259+
Last argument (`true`) indicates that we actually know output length. Alternatively we don't have to provide it, and use:
260+
261+
```csharp
262+
var actualOutputLength = LZ4Codec.Decode(
263+
inputBuffer, 0, inputLength,
264+
outputBuffer, 0, 0);
265+
```
266+
267+
but this will require guessing outputBuffer size which might be quite inefficient.
268+
269+
**Buffers compressed this way are fully compatible with original implementation if LZ4.**
270+
271+
Both `LZ4Stream` and `LZ4Codec.Wrap/Unwrap` use them internally.

‎src/LZ4.Tests/Demos.cs

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
using System;
2+
using System.Collections.Generic;
3+
using System.IO;
4+
using System.Linq;
5+
using System.Text;
6+
using NUnit.Framework;
7+
8+
namespace LZ4.Tests
9+
{
10+
[TestFixture]
11+
public class Demos
12+
{
13+
const string LoremIpsum =
14+
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla sit amet mauris diam. " +
15+
"Mauris mollis tristique sollicitudin. Nunc nec lectus nec ipsum pharetra venenatis. " +
16+
"Fusce et consequat massa, eu vestibulum erat. Proin in lectus a lacus fermentum viverra. " +
17+
"Aliquam vel tellus aliquam, eleifend justo ultrices, elementum elit. " +
18+
"Donec sed ullamcorper ex, ac sagittis ligula. Pellentesque vel risus lacus. " +
19+
"Proin aliquet lectus et tellus tristique, eget tristique magna placerat. " +
20+
"Maecenas ut ipsum efficitur, lobortis mauris at, bibendum libero. " +
21+
"Curabitur ultricies rutrum velit, eget blandit lorem facilisis sit amet. " +
22+
"Nunc dignissim nunc iaculis diam congue tincidunt. Suspendisse et massa urna. " +
23+
"Aliquam sagittis ornare nisl, quis feugiat justo eleifend iaculis. " +
24+
"Ut pulvinar id purus non convallis.";
25+
26+
void CompressToStream()
27+
{
28+
using (var fileStream = new FileStream("lorem.lz4", FileMode.Create))
29+
using (var lz4Stream = new LZ4Stream(fileStream, LZ4StreamMode.Compress))
30+
using (var writer = new StreamWriter(lz4Stream))
31+
{
32+
for (var i = 0; i < 100; i++)
33+
writer.WriteLine(LoremIpsum);
34+
}
35+
}
36+
37+
void DecompressFromStream()
38+
{
39+
using (var fileStream = new FileStream("lorem.lz4", FileMode.Open))
40+
using (var lz4Stream = new LZ4Stream(fileStream, LZ4StreamMode.Decompress))
41+
using (var reader = new StreamReader(lz4Stream))
42+
{
43+
string line;
44+
while ((line = reader.ReadLine()) != null)
45+
{
46+
Console.WriteLine(line);
47+
}
48+
}
49+
}
50+
51+
public string CompressToBuffer()
52+
{
53+
var text = Enumerable.Repeat(LoremIpsum, 5).Aggregate((a, b) => a + "\n" + b);
54+
55+
var compressed = Convert.ToBase64String(
56+
LZ4Codec.Wrap(Encoding.UTF8.GetBytes(text)));
57+
58+
return compressed;
59+
}
60+
61+
public void DecompressBuffer()
62+
{
63+
var compressed = CompressToBuffer();
64+
var lorems =
65+
Encoding.UTF8.GetString(
66+
LZ4Codec.Unwrap(Convert.FromBase64String(compressed)))
67+
.Split('\n');
68+
69+
foreach (var lorem in lorems)
70+
Console.WriteLine(lorem);
71+
}
72+
73+
public void LowLevelCompress()
74+
{
75+
var inputBuffer =
76+
Encoding.UTF8.GetBytes(
77+
Enumerable.Repeat(LoremIpsum, 5).Aggregate((a, b) => a + "\n" + b));
78+
79+
var inputLength = inputBuffer.Length;
80+
var maximumLength = LZ4Codec.MaximumOutputLength(inputLength);
81+
var outputBuffer = new byte[maximumLength];
82+
83+
var outputLength = LZ4Codec.Encode(
84+
inputBuffer, 0, inputLength,
85+
outputBuffer, 0, maximumLength);
86+
87+
}
88+
}
89+
}

‎src/LZ4.Tests/LZ4.Tests.csproj

+1
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
<Reference Include="System.Xml" />
6161
</ItemGroup>
6262
<ItemGroup>
63+
<Compile Include="Demos.cs" />
6364
<Compile Include="Utilities.cs" />
6465
<Compile Include="ConformanceTests.cs" />
6566
<Compile Include="PerformanceTests.cs" />

0 commit comments

Comments
 (0)