2013年6月21日金曜日

does logrotate work well with Fluentd?

Preparation

OS: Scientific Linux 6.1
Fluentd: td-agent 0.10.25
    type exec
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    flush_interval 5s
    command 1>> /var/tmp/hoge.log 2>> /var/tmp/err.log cat
    keys k1,k2
    retry_limit 4
  

  
    type tail
    path /var/tmp/tail.log
    tag test.iwa
    format tsv
    pos_file /var/log/td-agent/tail.pos
    keys k1,k2
  

* confirm Fluentd configuration

  $ wc -l tail.log
  10000 tail.log
 
  $ wc -l hoge.log
  10000 hoge.log

* logrotate.tdtest

  /var/tmp/tail.log {
      size 100M
      rotate 200
      missingok
      notifempty
      compress
  }

* confirm logrotate configuration

** prepare large file

  $ cp tail.log tail.log.cp
 
  $ for i in {1..1100}; do cat tail.log.cp >> tail.log; done
 
  $ du -sm tail.log
  106     tail.log

** confirm

  $ wc -l tail.log
  11110000 tail.log.1
 
  $ sudo logrotate logrotate.tdtest
 
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser   215534  6月 21 19:22 2013 tail.log.1.gz
 
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  11110000 tail.log.1

Experiment


* Terminal 1
  $ ll
  合計 118808
  -rw-rw-rw-  1 td-agent  td-agent        110000  6月 21 19:13 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  $ sudo rm -f hoge.log
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  # count line number of current target log file
  [m-iwamoto@sl6-64-mysqlgosh tmp]$ wc -l tail.log
  12152740 tail.log
* Terminal 2
  $ for i in {1..100000}; do echo "hoge,fuga" >> /var/tmp/tail.log; done
* Treminal 1
  # run logrotate during processing on Terminal 2
  $ sudo logrotate logrotate.tdtest
  $ ll
  合計 1988
  -rw-rw-rw-  1 td-agent  td-agent     1100000  6月 21 19:14 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser     131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser  674200  6月 21 19:14 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser  236387  6月 21 19:14 2013 tail.log.1.gz
  -rw-r--r--  1 m-iwamoto contentsuser     242  6月 21 18:38 2013 tail.log.3.gz
 
  # confirm if the line number of file produced by Fluentd matches to process of Terminal 1
  $ wc -l hoge.log
  100000 hoge.log
 
  # check line number of current target log #=> (★)
  $ wc -l tail.log
  67420 tail.log
 
  # check line number by decompressing archived file
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  12185320 tail.log.1
 
  # minus line number of previous file (before experiment) #=> (☆)
  $ expr 12185320 - 12152740
  32580
 
  # confirm sum of (★) and (☆) matches the number of process of Terminal 2
  $ expr 67420 + 32580
  100000

Conclusion

logrotate works pretty well with Fluentd :)
happy data aggregating!

2013年3月7日木曜日

set syntax highlight for specific files

It's next step of this article I wrote before.

Although I figured out that I enable syntax highlight for some files by vim's setf command,
its cumbersome that I type 'setf' each time when I open the file.

so I added the line bellow to my .vimrc.

au BufNewFile,BufRead php-fpm*.conf set filetype=dosini

As its just setting for php-fpm.conf, modify "php-fpm*.conf" and "filetype=xxx" properly if you want for other filetypes.

Thanks

http://beerpla.net/2008/04/02/how-to-add-a-vim-file-extension-to-syntax-highlighting/

:help au :au[tocmd] [group] {event} {pat} [nested] {cmd}

Add {cmd} to the list of commands that Vim will execute automatically on {event} for a file matching {pat}.
:help BufNewFile When starting to edit a file that doesn't exist.
:help BufRead When starting to edit a new buffer, after reading the file into the buffer.
:help filetype will actually tell this whole story in part B.

2013年2月9日土曜日

rails: rake aborted! uninitialized constant Movie


I got error when I try to use rake within Rails3 environment. (assume that there is a model named "Blog")


$ cat lib/tasks/test.rake
desc "mytest"
task :mytest do
  p Blog
end

$ rake mytest
rake aborted!
uninitialized constant Blog


the problem, was that I lack of ":environment" like bellow.


- task :mytest do
+ task :mytest => :environment do


I don't know why.. but anyway it works ^^;

2013年2月5日火曜日

moving gitolite server

old server



# backup repositories somehow.
user$ tar czvf repo.tgz /path/to/gitolite/repositories/


new server


user$ sudo yum install gitolite
user$ rpm -qa | grep gitolite
gitolite-2.3.1-1.el5

user$ sudo su -l gitolite


# initialize gitolite with dummy pub key
gitolite$ ssh-keygen -q -N '' -f dummy
gitolite$ gl-setup dummy.pub
creating gitolite-admin...
Initialized empty Git repository in /var/lib/gitolite/repositories/gitolite-admin.git/
creating testing...
Initialized empty Git repository in /var/lib/gitolite/repositories/testing.git/
[master (root-commit) a523b5b] gl-setup dummy.pub
 2 files changed, 8 insertions(+), 0 deletions(-)
 create mode 100644 conf/gitolite.conf
 create mode 100644 keydir/dummy.pub


# change it same as old server if you neeed
gitolite$ vim .gitolite.rc


# swap repositories directory with backup
gitolite$ rm -rf ./repositories

gitolite$ cd /path/to/backup/
gitolite$ tar xzvf repo.tgz
gitolite$ mv repositories /var/lig/gitolite/


# overwrite repository by empty commiting
gitolite$ cd /tmp
gitolite$ git clone /var/lib/gitolite/repositories/gitolite-admin.git
Cloning into gitolite-admin...
done.

gitolite$ cd gitolite-admin
gitolite$ git commit --allow-empty -m 'move gitolite server'
[master 1656534] move gitolite server

gitolite$ gl-admin-push -f
Counting objects: 1, done.
Unpacking objects: 100% (1/1), done.
Writing objects: 100% (1/1), 190 bytes, done.
Total 1 (delta 0), reused 0 (delta 0)
To /var/lib/gitolite/repositories/gitolite-admin.git
   d36c1e8..1656534  master -> master

gitolite$ exit


# confirm
user$ git clone ssh://gitolite@localhost/gitolite-admin
Cloning into gitolite-admin...
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (54/54), done.
Receiving objects: 100% (62/62), 8.95 KiB, done.
remote: Total 62 (delta 12), reused 0 (delta 0)
Resolving deltas: 100% (12/12), done.


done! :)

Thanks

http://sitaramc.github.com/gitolite/rare.html#existing

http://stackoverflow.com/questions/9835235/moving-gitolite-server

2013年2月2日土曜日

mongoid on rails: diff with Railscast tutorial for mongoid >= 3.0.0

Problem

there is a mongoid great tutorial:
http://railscasts.com/episodes/238-mongoid

but this tutorial doesn't work for mongoid >= 3.0.0.


you might see some error messages like:

undefined method `key?' for nil:NilClass

undefined method `referenced_in' for Article:Class

undefined method `references_many' for Author:Class

undefined method `article_comments_path' for

No route matches [POST] "/articles/.../comments"


it seems that there was some changes its specification.




Solution

all you need to modify for suits 3.0.0 are bellow:

diff --git a/app/models/article.rb b/app/models/article.rb
index a00e637..386536a 100644
--- a/app/models/article.rb
+++ b/app/models/article.rb
@@ -5,5 +5,5 @@ class Article
   field :published_on, :type => Date
   validates_presence_of :name
   embeds_many :comments
-  referenced_in :author
+  belongs_to :author
 end
diff --git a/app/models/author.rb b/app/models/author.rb
index 7741009..e52989e 100644
--- a/app/models/author.rb
+++ b/app/models/author.rb
@@ -1,6 +1,6 @@
 class Author
   include Mongoid::Document
   field :name, type: String
-  key :name
-  references_many :articles
+  field :_id, type: String, default: ->{ name }
+  has_many :articles
 end
diff --git a/config/routes.rb b/config/routes.rb
index 31cc8de..45ac1c4 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -2,7 +2,9 @@ Mongoid01::Application.routes.draw do
   resources :authors
 
 
-  resources :articles
+  resources :articles do
+    resources :comments
+  end
 
   # The priority is based upon order of creation:
@@ -54,7 +56,7 @@ Mongoid01::Application.routes.draw do
 
   # You can have the root of your site routed with "root"
   # just remember to delete public/index.html.
-  # root :to => 'welcome#index'
+  root :to => 'articles#index'
 
   # See how all your routes lay out with "rake routes"

happy coding ! :)

mongoDB backup and restore

Backup



-[28701]% mongodump --version

mongodump version 2.0.3

-[28680]% mongodump -d test -o /tmp/mongobackup/

connected to: 127.0.0.1
DATABASE: test   to     /tmp/mongobackup/test
        test.iwa to /tmp/mongobackup/test/masaki925.bson
                 1 objects
        test.system.indexes to /tmp/mongobackup/test/system.indexes.bson
                 3 objects

-[28699]% tree /tmp/mongobackup/

/tmp/mongobackup/
└── test
    ├── masaki925.bson
    └── system.indexes.bson

Restore



-[28702]% mongorestore --version

mongorestore version 2.0.3

-[28700]% mongorestore -d test_restore /tmp/mongobackup/test/

connected to: 127.0.0.1
Sat Feb  2 10:57:44 /tmp/mongobackup/test/masaki925.bson
Sat Feb  2 10:57:44      going into namespace [test_restore.masaki925]
1 objects found
Sat Feb  2 10:57:44 /tmp/mongobackup/test/system.indexes.bson
Sat Feb  2 10:57:44      going into namespace [test_restore.system.indexes]
Sat Feb  2 10:57:44 { key: { _id: 1 }, ns: "test_restore.masaki925", name: "_id_" }
1 objects found

More Info

Backup Strategies for MongoDB Systems — MongoDB Manual
http://docs.mongodb.org/manual/administration/backups/

2013年1月31日木曜日

vim: set syntax highlight


when I set up the php-fpm, I realized that I cannot get syntax highlight for vim to edit php-fpm.d/www.conf.

I looked around the file and recognized that the file syntax is similar to php.ini.

and I also realized that I can get syntax highlight for php.ini.

so at first, look at the filetype of the php.ini

:set filetype
#=> filetype=dosini

as I could get the filetype that I have to specify, let's try it.


:setf dosini


its work! :)

synonym is bellow:

:set filetype=language