1
1
# Snowplow S3 Loader
2
2
3
3
[ ![ Build Status] [ travis-image ]] [ travis ]
4
- [ ![ Release] [ release-image ]] [ releases ]
4
+ [ ![ Release] [ release-image ]] [ releases ]
5
5
[ ![ License] [ license-image ]] [ license ]
6
6
7
7
## Overview
8
8
9
- The Snowplow S3 Loader consumes records from an [ Amazon Kinesis] [ kinesis ] stream, and writes them to S3.
9
+ The Snowplow S3 Loader consumes records from an [ Amazon Kinesis] [ kinesis ] stream or [ NSQ ] [ nsq ] topic , and writes them to S3.
10
10
11
11
There are 2 file formats supported:
12
12
* LZO
@@ -49,9 +49,9 @@ NOTE: These are already installed in the Vagrant quickstart environment.
49
49
The Snowplow S3 Loader has the following command-line interface:
50
50
51
51
```
52
- snowplow-s3-loader: Version 0.5 .0
52
+ snowplow-s3-loader: Version 0.6 .0
53
53
54
- Usage: java -jar snowplow-s3-loader.jar [options]
54
+ Usage: snowplow-s3-loader [options]
55
55
56
56
--config <filename>
57
57
```
@@ -62,17 +62,17 @@ Create your own config file:
62
62
63
63
``` bash
64
64
guest$ cd /vagrant
65
- guest$ cp src/main/resources /config.hocon.sample my.conf
65
+ guest$ cp examples /config.hocon.sample my.conf
66
66
```
67
67
68
68
You will need to edit all fields in the config. Consult [ this portion] [ config ] of the setup guide on how to fill in the fields.
69
69
70
70
Next, start the sink, making sure to specify your new config file:
71
71
72
72
``` bash
73
- guest$ sbt " run -- config my.conf"
73
+ guest$ java -jar snowplow-s3-loader-0.6.0.jar -- config my.conf
74
74
```
75
-
75
+
76
76
## Find out more
77
77
78
78
| Technical Docs | Setup Guide | Roadmap & Contributing |
@@ -96,13 +96,14 @@ limitations under the License.
96
96
[ travis-image ] : https://travis-ci.org/snowplow/snowplow-s3-loader.png?branch=master
97
97
[ travis ] : http://travis-ci.org/snowplow/snowplow-s3-loader
98
98
99
- [ release-image ] : http://img.shields.io/badge/release-0.5 .0-blue.svg?style=flat
99
+ [ release-image ] : http://img.shields.io/badge/release-0.6 .0-blue.svg?style=flat
100
100
[ releases ] : https://github.com/snowplow/snowplow-s3-loader/releases
101
101
102
102
[ license-image ] : http://img.shields.io/badge/license-Apache--2-blue.svg?style=flat
103
103
[ license ] : http://www.apache.org/licenses/LICENSE-2.0
104
104
105
105
[ kinesis ] : http://aws.amazon.com/kinesis/
106
+ [ nsq ] : http://nsq.io/
106
107
[ snowplow ] : http://snowplowanalytics.com
107
108
[ hadoop-lzo ] : https://github.com/twitter/hadoop-lzo
108
109
[ protobufs ] : https://github.com/google/protobuf/
@@ -113,9 +114,9 @@ limitations under the License.
113
114
[ vagrant-install ] : http://docs.vagrantup.com/v2/installation/index.html
114
115
[ virtualbox-install ] : https://www.virtualbox.org/wiki/Downloads
115
116
116
- [ setup ] : https://github.com/snowplow/snowplow/wiki/kinesis-lzo- s3-sink -setup
117
- [ config ] : https://github.com/snowplow/snowplow/wiki/kinesis-lzo- s3-sink -setup#configuration
118
- [ techdocs ] : https://github.com/snowplow/snowplow/wiki/kinesis-lzo- s3-sink
117
+ [ setup ] : https://github.com/snowplow/snowplow/wiki/snowplow- s3-loader -setup
118
+ [ config ] : https://github.com/snowplow/snowplow/wiki/snowplow- s3-loader -setup#configuration
119
+ [ techdocs ] : https://github.com/snowplow/snowplow/wiki/s3-loader
119
120
120
121
[ techdocs-image ] : https://d3i6fms1cm1j0i.cloudfront.net/github/images/techdocs.png
121
122
[ setup-image ] : https://d3i6fms1cm1j0i.cloudfront.net/github/images/setup.png
0 commit comments