does logrotate work well with Fluentd?

Preparation

OS: Scientific Linux 6.1
Fluentd: td-agent 0.10.25

    type exec
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    flush_interval 5s
    command 1>> /var/tmp/hoge.log 2>> /var/tmp/err.log cat
    keys k1,k2
    retry_limit 4
  

  
    type tail
    path /var/tmp/tail.log
    tag test.iwa
    format tsv
    pos_file /var/log/td-agent/tail.pos
    keys k1,k2
  

* confirm Fluentd configuration

  $ wc -l tail.log
  10000 tail.log
 
  $ wc -l hoge.log
  10000 hoge.log

* logrotate.tdtest

  /var/tmp/tail.log {
      size 100M
      rotate 200
      missingok
      notifempty
      compress
  }

* confirm logrotate configuration

** prepare large file

  $ cp tail.log tail.log.cp
 
  $ for i in {1..1100}; do cat tail.log.cp >> tail.log; done
 
  $ du -sm tail.log
  106     tail.log

** confirm

  $ wc -l tail.log
  11110000 tail.log.1
 
  $ sudo logrotate logrotate.tdtest
 
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser   215534  6月 21 19:22 2013 tail.log.1.gz
 
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  11110000 tail.log.1

Experiment

* Terminal 1

  $ ll
  合計 118808
  -rw-rw-rw-  1 td-agent  td-agent        110000  6月 21 19:13 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  $ sudo rm -f hoge.log
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  # count line number of current target log file
  [m-iwamoto@sl6-64-mysqlgosh tmp]$ wc -l tail.log
  12152740 tail.log

* Terminal 2

  $ for i in {1..100000}; do echo "hoge,fuga" >> /var/tmp/tail.log; done

* Treminal 1

  # run logrotate during processing on Terminal 2
  $ sudo logrotate logrotate.tdtest
  $ ll
  合計 1988
  -rw-rw-rw-  1 td-agent  td-agent     1100000  6月 21 19:14 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser     131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser  674200  6月 21 19:14 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser  236387  6月 21 19:14 2013 tail.log.1.gz
  -rw-r--r--  1 m-iwamoto contentsuser     242  6月 21 18:38 2013 tail.log.3.gz
 
  # confirm if the line number of file produced by Fluentd matches to process of Terminal 1
  $ wc -l hoge.log
  100000 hoge.log
 
  # check line number of current target log #=> (★)
  $ wc -l tail.log
  67420 tail.log
 
  # check line number by decompressing archived file
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  12185320 tail.log.1
 
  # minus line number of previous file (before experiment) #=> (☆)
  $ expr 12185320 - 12152740
  32580
 
  # confirm sum of (★) and (☆) matches the number of process of Terminal 2
  $ expr 67420 + 32580
  100000

Conclusion

logrotate works pretty well with Fluentd :)

happy data aggregating!