2013年6月21日金曜日

does logrotate work well with Fluentd?

Preparation

OS: Scientific Linux 6.1
Fluentd: td-agent 0.10.25
    type exec
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    flush_interval 5s
    command 1>> /var/tmp/hoge.log 2>> /var/tmp/err.log cat
    keys k1,k2
    retry_limit 4
  

  
    type tail
    path /var/tmp/tail.log
    tag test.iwa
    format tsv
    pos_file /var/log/td-agent/tail.pos
    keys k1,k2
  

* confirm Fluentd configuration

  $ wc -l tail.log
  10000 tail.log
 
  $ wc -l hoge.log
  10000 hoge.log

* logrotate.tdtest

  /var/tmp/tail.log {
      size 100M
      rotate 200
      missingok
      notifempty
      compress
  }

* confirm logrotate configuration

** prepare large file

  $ cp tail.log tail.log.cp
 
  $ for i in {1..1100}; do cat tail.log.cp >> tail.log; done
 
  $ du -sm tail.log
  106     tail.log

** confirm

  $ wc -l tail.log
  11110000 tail.log.1
 
  $ sudo logrotate logrotate.tdtest
 
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser   215534  6月 21 19:22 2013 tail.log.1.gz
 
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  11110000 tail.log.1

Experiment


* Terminal 1
  $ ll
  合計 118808
  -rw-rw-rw-  1 td-agent  td-agent        110000  6月 21 19:13 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  $ sudo rm -f hoge.log
  $ ll
  -rw-r--r--  1 m-iwamoto contentsuser       131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser 121527400  6月 21 19:13 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser       242  6月 21 18:38 2013 tail.log.2.gz
 
  # count line number of current target log file
  [m-iwamoto@sl6-64-mysqlgosh tmp]$ wc -l tail.log
  12152740 tail.log
* Terminal 2
  $ for i in {1..100000}; do echo "hoge,fuga" >> /var/tmp/tail.log; done
* Treminal 1
  # run logrotate during processing on Terminal 2
  $ sudo logrotate logrotate.tdtest
  $ ll
  合計 1988
  -rw-rw-rw-  1 td-agent  td-agent     1100000  6月 21 19:14 2013 hoge.log
  -rw-r--r--  1 m-iwamoto contentsuser     131  6月 21 18:54 2013 logrotate.tdtest
  -rw-r--r--  1 m-iwamoto contentsuser  674200  6月 21 19:14 2013 tail.log
  -rw-r--r--  1 m-iwamoto contentsuser  236387  6月 21 19:14 2013 tail.log.1.gz
  -rw-r--r--  1 m-iwamoto contentsuser     242  6月 21 18:38 2013 tail.log.3.gz
 
  # confirm if the line number of file produced by Fluentd matches to process of Terminal 1
  $ wc -l hoge.log
  100000 hoge.log
 
  # check line number of current target log #=> (★)
  $ wc -l tail.log
  67420 tail.log
 
  # check line number by decompressing archived file
  $ gunzip tail.log.1.gz
  $ wc -l tail.log.1
  12185320 tail.log.1
 
  # minus line number of previous file (before experiment) #=> (☆)
  $ expr 12185320 - 12152740
  32580
 
  # confirm sum of (★) and (☆) matches the number of process of Terminal 2
  $ expr 67420 + 32580
  100000

Conclusion

logrotate works pretty well with Fluentd :)
happy data aggregating!

2013年3月7日木曜日

set syntax highlight for specific files

It's next step of this article I wrote before.

Although I figured out that I enable syntax highlight for some files by vim's setf command,
its cumbersome that I type 'setf' each time when I open the file.

so I added the line bellow to my .vimrc.

au BufNewFile,BufRead php-fpm*.conf set filetype=dosini

As its just setting for php-fpm.conf, modify "php-fpm*.conf" and "filetype=xxx" properly if you want for other filetypes.

Thanks

http://beerpla.net/2008/04/02/how-to-add-a-vim-file-extension-to-syntax-highlighting/

:help au :au[tocmd] [group] {event} {pat} [nested] {cmd}

Add {cmd} to the list of commands that Vim will execute automatically on {event} for a file matching {pat}.
:help BufNewFile When starting to edit a file that doesn't exist.
:help BufRead When starting to edit a new buffer, after reading the file into the buffer.
:help filetype will actually tell this whole story in part B.

2013年2月9日土曜日

rails: rake aborted! uninitialized constant Movie


I got error when I try to use rake within Rails3 environment. (assume that there is a model named "Blog")


$ cat lib/tasks/test.rake
desc "mytest"
task :mytest do
  p Blog
end

$ rake mytest
rake aborted!
uninitialized constant Blog


the problem, was that I lack of ":environment" like bellow.


- task :mytest do
+ task :mytest => :environment do


I don't know why.. but anyway it works ^^;

2013年2月5日火曜日

moving gitolite server

old server



# backup repositories somehow.
user$ tar czvf repo.tgz /path/to/gitolite/repositories/


new server


user$ sudo yum install gitolite
user$ rpm -qa | grep gitolite
gitolite-2.3.1-1.el5

user$ sudo su -l gitolite


# initialize gitolite with dummy pub key
gitolite$ ssh-keygen -q -N '' -f dummy
gitolite$ gl-setup dummy.pub
creating gitolite-admin...
Initialized empty Git repository in /var/lib/gitolite/repositories/gitolite-admin.git/
creating testing...
Initialized empty Git repository in /var/lib/gitolite/repositories/testing.git/
[master (root-commit) a523b5b] gl-setup dummy.pub
 2 files changed, 8 insertions(+), 0 deletions(-)
 create mode 100644 conf/gitolite.conf
 create mode 100644 keydir/dummy.pub


# change it same as old server if you neeed
gitolite$ vim .gitolite.rc


# swap repositories directory with backup
gitolite$ rm -rf ./repositories

gitolite$ cd /path/to/backup/
gitolite$ tar xzvf repo.tgz
gitolite$ mv repositories /var/lig/gitolite/


# overwrite repository by empty commiting
gitolite$ cd /tmp
gitolite$ git clone /var/lib/gitolite/repositories/gitolite-admin.git
Cloning into gitolite-admin...
done.

gitolite$ cd gitolite-admin
gitolite$ git commit --allow-empty -m 'move gitolite server'
[master 1656534] move gitolite server

gitolite$ gl-admin-push -f
Counting objects: 1, done.
Unpacking objects: 100% (1/1), done.
Writing objects: 100% (1/1), 190 bytes, done.
Total 1 (delta 0), reused 0 (delta 0)
To /var/lib/gitolite/repositories/gitolite-admin.git
   d36c1e8..1656534  master -> master

gitolite$ exit


# confirm
user$ git clone ssh://gitolite@localhost/gitolite-admin
Cloning into gitolite-admin...
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (54/54), done.
Receiving objects: 100% (62/62), 8.95 KiB, done.
remote: Total 62 (delta 12), reused 0 (delta 0)
Resolving deltas: 100% (12/12), done.


done! :)

Thanks

http://sitaramc.github.com/gitolite/rare.html#existing

http://stackoverflow.com/questions/9835235/moving-gitolite-server

2013年2月2日土曜日

mongoid on rails: diff with Railscast tutorial for mongoid >= 3.0.0

Problem

there is a mongoid great tutorial:
http://railscasts.com/episodes/238-mongoid

but this tutorial doesn't work for mongoid >= 3.0.0.


you might see some error messages like:

undefined method `key?' for nil:NilClass

undefined method `referenced_in' for Article:Class

undefined method `references_many' for Author:Class

undefined method `article_comments_path' for

No route matches [POST] "/articles/.../comments"


it seems that there was some changes its specification.




Solution

all you need to modify for suits 3.0.0 are bellow:

diff --git a/app/models/article.rb b/app/models/article.rb
index a00e637..386536a 100644
--- a/app/models/article.rb
+++ b/app/models/article.rb
@@ -5,5 +5,5 @@ class Article
   field :published_on, :type => Date
   validates_presence_of :name
   embeds_many :comments
-  referenced_in :author
+  belongs_to :author
 end
diff --git a/app/models/author.rb b/app/models/author.rb
index 7741009..e52989e 100644
--- a/app/models/author.rb
+++ b/app/models/author.rb
@@ -1,6 +1,6 @@
 class Author
   include Mongoid::Document
   field :name, type: String
-  key :name
-  references_many :articles
+  field :_id, type: String, default: ->{ name }
+  has_many :articles
 end
diff --git a/config/routes.rb b/config/routes.rb
index 31cc8de..45ac1c4 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -2,7 +2,9 @@ Mongoid01::Application.routes.draw do
   resources :authors
 
 
-  resources :articles
+  resources :articles do
+    resources :comments
+  end
 
   # The priority is based upon order of creation:
@@ -54,7 +56,7 @@ Mongoid01::Application.routes.draw do
 
   # You can have the root of your site routed with "root"
   # just remember to delete public/index.html.
-  # root :to => 'welcome#index'
+  root :to => 'articles#index'
 
   # See how all your routes lay out with "rake routes"

happy coding ! :)

mongoDB backup and restore

Backup



-[28701]% mongodump --version

mongodump version 2.0.3

-[28680]% mongodump -d test -o /tmp/mongobackup/

connected to: 127.0.0.1
DATABASE: test   to     /tmp/mongobackup/test
        test.iwa to /tmp/mongobackup/test/masaki925.bson
                 1 objects
        test.system.indexes to /tmp/mongobackup/test/system.indexes.bson
                 3 objects

-[28699]% tree /tmp/mongobackup/

/tmp/mongobackup/
└── test
    ├── masaki925.bson
    └── system.indexes.bson

Restore



-[28702]% mongorestore --version

mongorestore version 2.0.3

-[28700]% mongorestore -d test_restore /tmp/mongobackup/test/

connected to: 127.0.0.1
Sat Feb  2 10:57:44 /tmp/mongobackup/test/masaki925.bson
Sat Feb  2 10:57:44      going into namespace [test_restore.masaki925]
1 objects found
Sat Feb  2 10:57:44 /tmp/mongobackup/test/system.indexes.bson
Sat Feb  2 10:57:44      going into namespace [test_restore.system.indexes]
Sat Feb  2 10:57:44 { key: { _id: 1 }, ns: "test_restore.masaki925", name: "_id_" }
1 objects found

More Info

Backup Strategies for MongoDB Systems — MongoDB Manual
http://docs.mongodb.org/manual/administration/backups/

2013年1月31日木曜日

vim: set syntax highlight


when I set up the php-fpm, I realized that I cannot get syntax highlight for vim to edit php-fpm.d/www.conf.

I looked around the file and recognized that the file syntax is similar to php.ini.

and I also realized that I can get syntax highlight for php.ini.

so at first, look at the filetype of the php.ini

:set filetype
#=> filetype=dosini

as I could get the filetype that I have to specify, let's try it.


:setf dosini


its work! :)

synonym is bellow:

:set filetype=language


2013年1月18日金曜日

stty: standard input: Invalid argument

Back Ground

I got error "stty: standard input: Invalid argument" when I ssh to remote server.
It annoyed me for cron batch procedure that send mail STDERR of the batch.

Investigation

I wrote the command bellow for my favor.
stty werase undef
bind '"\C-w": backward-kill-word'
and the stty command was bad. if you put the snipet above, you can avoid get the error for first ssh login, but you will not enable the snipet when you start "screen".

Solusion

# enable only when login via interctive shell
if [ -n "$PS1" ] ; then
  stty werase undef
  bind '"\C-w": backward-kill-word'
fi
happy coding! :)

2013年1月17日木曜日

how does "desired capacity" work for autoscale?


test results



conclusion


  • "desired capacity" is flexible changing
  • "min size" and "max size" is stronger than I expected.