So why another one? The answer is very simple. I tried them all and there is a big problem with all of them! If Docker container gets recreated for any reason (system restart, image update or …), all log evens in monitored files will resend again to Logstash server.
By investigating a lot and reading Logstash forwarder source code, finally I found the missing piece of this puzzle. A quick look into the source code shows us here there is a special file with hardcoded name as
.logtsash-forwarder which is being created in the program’s working directory, and actually stores all the information about last harvested points in monitored log files.
What happens is, when Docker container gets destroyed and recreated again these information is lost and forwarder has no clue where to start harvesting again. So what is does? Starts from the beginning.
Simply store that file in host file system by mounting is as a volume with Docker
-v option. But there is another problem! If you look at here in source code again, you’ll notice that it calls
os.Rename. What’s wrong with that? It turns out if you only mount this file into Docker image, this call fails. Because in Docker it’s not possible to move files from Docker filesystem into mounted filesystem due to system call restrictions.
Obviously the solution is to mount a directory instead of file itself. And here is a simple Docker file that does this:
FROM marvambass/oracle-java8 RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4 \ && echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | tee /etc/apt/sources.list.d/logstashforwarder.list \ && apt-get update; apt-get install -y logstash-forwarder ENV PATH /opt/logstash-forwarder/bin:$PATH COPY entrypoint.sh / VOLUME ["/logstash-forwarder-conf", "/certs", /home/logstash] WORKDIR /home/logstash ENTRYPOINT ["/entrypoint.sh"] CMD ["logstash-forwarder", "-config=/etc/logstash-forwarder.conf"]
There are two important points here, in
VOLUME you can see
/logstash which is exposed and
WORKDIR which is pointing to
/logstash. Here is a complete code in github. This is actually fork of another forwarder project with these modifications.
If you want to persist positions of harvested logs in files simply mount
/logstash to host filesystem.
docker run -v /var/cache/logstash:/home/logstash docker-logstash-forwarder ...
And next time you update image or start a new container for forwarder it’ll start harvesting logs from the point it’s left them.
**Update: DOCKERFILE has been updated. See GITHUB commit for more info.
ABOUT BOYNUX bLOG
My notes and experiments with different technologies and tools. All information here is solely my personal thougths and does not relate to my employer's point of view in anyways.
FOLLOW BOYNUX FOR UPDATES AND BLOGS