logstash-output-kinesis does not work inside alpine linux

[bobzoller]

I'm going to open and then close this issue, because I'd like folks to find it and avoid the debugging hassle I just went through.

Short story: the logstash-output-kinesis plugin won't work inside an Alpine Linux container, because the KPL library is linked against glibc. See awslabs/amazon-kinesis-producer#86 for more info.

Before mucking around, all I could see in the logstash output was this error, immediately after startup:

22:47:49.812 [[main]>worker1] WARN  logstash.outputs.kinesis - Error writing event to Kinesis {:exception=>com.amazonaws.services.kinesis.producer.DaemonException: The child process has been shutdown and can no longer accept messages.}

This led me to #16, but in my case the credentials were fine. The root cause was an error at startup time which was not logged by default, making this even harder to track down. To enable logging internal to the KPL library, I downloaded the latest release of slf4j, and copied their "simple" logger jar into the java class path:

$ wget https://www.slf4j.org/dist/slf4j-1.7.25.tar.gz
$ tar xzvf slf4j-1.7.25.tar.gz
$ cp slf4j-1.7.25/slf4j-simple-1.7.25.jar  /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kinesis-5.0.0-java/vendor/jar-dependencies/runtime-jars/slf4j-simple.jar

then, restarting logstash, I finally saw on stderr:

[kpl-daemon-0000] ERROR com.amazonaws.services.kinesis.producer.KinesisProducer - Error in child process
com.amazonaws.services.kinesis.producer.IrrecoverableError: Error starting child process
	at com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:525)
	at com.amazonaws.services.kinesis.producer.Daemon.startChildProcess(Daemon.java:456)
	at com.amazonaws.services.kinesis.producer.Daemon.access$100(Daemon.java:63)
	at com.amazonaws.services.kinesis.producer.Daemon$1.run(Daemon.java:133)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Cannot run program "/tmp/amazon-kinesis-producer-native-binaries/kinesis_producer_d93825f806782576ef9f09eef67a2baeadfec35c": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at com.amazonaws.services.kinesis.producer.Daemon.startChildProcess(Daemon.java:454)
	... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 8 more

That led me to awslabs/amazon-kinesis-producer#86 👍