Preparation
OS: Scientific Linux 6.1 Fluentd: td-agent 0.10.25
type exec buffer_type file buffer_path /var/log/td-agent/buffer/td flush_interval 5s command 1>> /var/tmp/hoge.log 2>> /var/tmp/err.log cat keys k1,k2 retry_limit 4 type tail path /var/tmp/tail.log tag test.iwa format tsv pos_file /var/log/td-agent/tail.pos keys k1,k2
* confirm Fluentd configuration
$ wc -l tail.log 10000 tail.log $ wc -l hoge.log 10000 hoge.log
* logrotate.tdtest
/var/tmp/tail.log { size 100M rotate 200 missingok notifempty compress }
* confirm logrotate configuration
** prepare large file
$ cp tail.log tail.log.cp $ for i in {1..1100}; do cat tail.log.cp >> tail.log; done $ du -sm tail.log 106 tail.log
** confirm
$ wc -l tail.log 11110000 tail.log.1 $ sudo logrotate logrotate.tdtest $ ll -rw-r--r-- 1 m-iwamoto contentsuser 215534 6月 21 19:22 2013 tail.log.1.gz $ gunzip tail.log.1.gz $ wc -l tail.log.1 11110000 tail.log.1
Experiment
* Terminal 1
$ ll 合計 118808 -rw-rw-rw- 1 td-agent td-agent 110000 6月 21 19:13 2013 hoge.log -rw-r--r-- 1 m-iwamoto contentsuser 131 6月 21 18:54 2013 logrotate.tdtest -rw-r--r-- 1 m-iwamoto contentsuser 121527400 6月 21 19:13 2013 tail.log -rw-r--r-- 1 m-iwamoto contentsuser 242 6月 21 18:38 2013 tail.log.2.gz $ sudo rm -f hoge.log $ ll -rw-r--r-- 1 m-iwamoto contentsuser 131 6月 21 18:54 2013 logrotate.tdtest -rw-r--r-- 1 m-iwamoto contentsuser 121527400 6月 21 19:13 2013 tail.log -rw-r--r-- 1 m-iwamoto contentsuser 242 6月 21 18:38 2013 tail.log.2.gz # count line number of current target log file [m-iwamoto@sl6-64-mysqlgosh tmp]$ wc -l tail.log 12152740 tail.log* Terminal 2
$ for i in {1..100000}; do echo "hoge,fuga" >> /var/tmp/tail.log; done* Treminal 1
# run logrotate during processing on Terminal 2 $ sudo logrotate logrotate.tdtest $ ll 合計 1988 -rw-rw-rw- 1 td-agent td-agent 1100000 6月 21 19:14 2013 hoge.log -rw-r--r-- 1 m-iwamoto contentsuser 131 6月 21 18:54 2013 logrotate.tdtest -rw-r--r-- 1 m-iwamoto contentsuser 674200 6月 21 19:14 2013 tail.log -rw-r--r-- 1 m-iwamoto contentsuser 236387 6月 21 19:14 2013 tail.log.1.gz -rw-r--r-- 1 m-iwamoto contentsuser 242 6月 21 18:38 2013 tail.log.3.gz # confirm if the line number of file produced by Fluentd matches to process of Terminal 1 $ wc -l hoge.log 100000 hoge.log # check line number of current target log #=> (★) $ wc -l tail.log 67420 tail.log # check line number by decompressing archived file $ gunzip tail.log.1.gz $ wc -l tail.log.1 12185320 tail.log.1 # minus line number of previous file (before experiment) #=> (☆) $ expr 12185320 - 12152740 32580 # confirm sum of (★) and (☆) matches the number of process of Terminal 2 $ expr 67420 + 32580 100000
Conclusion
logrotate works pretty well with Fluentd :)
happy data aggregating!