?
我想将WEB服务器日志通过管道处理然后写入数据库
?
$ cat match.d
import std.regex;
import std.stdio;
import std.string;
import std.array;
void main()
{
auto r = regex(`^(\S+) (\S+) (\S+) \[(.+)\] "([^"]+)" ([0-9]{3}) ([0-9]+) "([^"]+)" "([^"]+)" "([^"]+)"`);
foreach(line; stdin.byLine)
{
foreach(m; match(line, r)){
auto c = m.captures;
c.popFront();
auto value = join(c, "\",\"");
auto sql = format("insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value(\"%s\");", value );
writeln(sql);
}
}
}
?编译
$ dmd match.d 编译优化（减肥） $ strip match
测试
$ cat access.log | ./match
输出类似
insert into log(remote_addr,unknow,remote_user,time_local,request,status,body_bytes_sent,http_referer,http_user_agent,http_x_forwarded_for) value("192.168.2.76","-","-","19/Mar/2013:15:24:20 +0800","GET /favicon.ico HTTP/1.1","404","193","-","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.84 Safari/535.11 SE 2.X MetaSr 1.0","-");
??
用法
?
$ cat access.log | match | mysql -hlocalhost -ulog -p123456 logging
?
高级用法， 实时处理日志
- 首先创建一个管道
- 寻该日志文件写入管道中
- cat ?管道名?| match | mysql -hlocalhost -ulog -p123456 logging ?这样就可以实现实时日志插入。
上面程序稍加修改即可实现Hbase,?Hypertable?插入
?
?
?
?
?
?
