Problem

How to pack Logstash forwarder in Docker containers
How to pack Logstash forwarder in Docker containers

There are plenty of articles out there about how to create Logstash forwarder Docker image. For example here, here and here or use Google to find even more.

So why another one? The answer is very simple. I tried them all and there is a big problem with all of them! If Docker container gets recreated for any reason (system restart, image update or …), all log evens in monitored files will resend again to Logstash server.

Idea

By investigating a lot and reading Logstash forwarder source code, finally I found the missing piece of this puzzle. A quick look into the source code shows us here there is a special file with hardcoded name as .logtsash-forwarder which is being created in the program’s working directory, and actually stores all the information about last harvested points in monitored log files.

What happens is, when Docker container gets destroyed and recreated again these information is lost and forwarder has no clue where to start harvesting again. So what is does? Starts from the beginning.

Solution


Simply store that file in host file system by mounting is as a volume with Docker -v option. But there is another problem! If you look at here in source code again, you’ll notice that it calls os.Rename. What’s wrong with that? It turns out if you only mount this file into Docker image, this call fails. Because in Docker it’s not possible to move files from Docker filesystem into mounted filesystem due to system call restrictions.

Solution

Obviously the solution is to mount a directory instead of file itself. And here is a simple Docker file that does this:

FROM marvambass/oracle-java8

RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 \
    && echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | tee /etc/apt/sources.list.d/logstashforwarder.list \
    && apt-get update; apt-get install -y logstash-forwarder

ENV PATH /opt/logstash-forwarder/bin:$PATH

COPY entrypoint.sh /

VOLUME ["/logstash-forwarder-conf", "/certs", /home/logstash]

WORKDIR /home/logstash

ENTRYPOINT ["/entrypoint.sh"]
CMD ["logstash-forwarder", "-config=/etc/logstash-forwarder.conf"]

There are two important points here, in VOLUME you can see /logstash which is exposed and WORKDIR which is pointing to /logstash. Here is a complete code in github. This is actually fork of another forwarder project with these modifications.


How to use it

If you want to persist positions of harvested logs in files simply mount /logstash to host filesystem.

docker run -v /var/cache/logstash:/home/logstash docker-logstash-forwarder ...

And next time you update image or start a new container for forwarder it’ll start harvesting logs from the point it’s left them.

**Update: DOCKERFILE has been updated. See GITHUB commit for more info.

Continue on part II



comments powered by Disqus

ABOUT BOYNUX bLOG

My notes and experiments with different technologies and tools. All information here is solely my personal thougths and does not relate to my employer's point of view in anyways.


FOLLOW BOYNUX FOR UPDATES AND BLOGS


FORK ME!

© Copyright 2012-2017