====== Parsing Apache Logs ====== Below are some simple awk/sed/etc command line scripts to parse apache logs and get quick statistics ===== Unique visitors per day ===== Where access.log is your combined log file with typical format as below: 69.175.xxx.yyy - - [13/Jul/2013:06:28:31 -0500] "GET /some/web/folder/somewebpage2 HTTP/1.0" 404 4212 "http://somesubdomain.example.org/some/web/folder/some_web_page1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1" 69.175.xxx.yyy - - [13/Jul/2013:06:28:35 -0500] "GET /some/web/folder/?do=register HTTP/1.0" 302 599 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1" 69.175.xxx.yyy - - [13/Jul/2013:06:28:36 -0500] "GET /some/web/folder/somewebpage2 HTTP/1.0" 404 4212 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1" 69.175.xxx.yyy - - [13/Jul/2013:06:28:41 -0500] "POST /some/web/folder/somewebpage2 HTTP/1.0" 200 2439 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1" 94.123.xxx.yyy - - [13/Jul/2013:06:32:49 -0500] "GET /some/web/folder/some_web_page1 HTTP/1.1" 200 5121 "http://somesubdomain.example.org/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" 94.123.xxx.yyy - - [13/Jul/2013:06:32:49 -0500] "GET /some/web/folder/?do=login HTTP/1.1" 302 599 "http://somesubdomain.example.org/some/web/folder/some_web_page1" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" 94.123.xxx.yyy - - [13/Jul/2013:06:32:50 -0500] "GET /some/web/folder/somewebpage2 HTTP/1.1" 404 4214 "http://somesubdomain.example.org/some/web/folder/some_web_page1" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" 94.123.xxx.yyy - - [13/Jul/2013:06:32:50 -0500] "GET /some/web/folder/?do=register HTTP/1.1" 302 599 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" 94.123.xxx.yyy - - [13/Jul/2013:06:32:50 -0500] "GET /some/web/folder/somewebpage2 HTTP/1.1" 404 4214 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" 94.123.xxx.yyy - - [13/Jul/2013:06:32:51 -0500] "POST /some/web/folder/somewebpage2 HTTP/1.1" 200 2530 "http://somesubdomain.example.org/some/web/folder/somewebpage2" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" Below is the command line script. This gets the unique hits per day. There is a grep at the very end to do a final filter for the Month and Year you may be looking for. cat access.log | awk '{print $1 " " $4}' | sed 's/\[//' | cut -d":" -f1 | awk '{print $2 " " $1}' | \ sort | uniq | awk '{print $1}' | uniq -c | grep "Feb/2013" The output is as below 52 01/Feb/2013 63 02/Feb/2013 47 03/Feb/2013 62 04/Feb/2013 59 05/Feb/2013 63 06/Feb/2013 ... etc. ==== Explanation of the command ==== This may be useful if you want to tweak it. === Break down 1 === Get IP and date (unformatted at this stage) cat access.log | awk '{print $1 " " $4}' Output of the above 69.175.xxx.yyy [13/Jul/2013:06:28:31 69.175.xxx.yyy [13/Jul/2013:06:28:35 69.175.xxx.yyy [13/Jul/2013:06:28:36 69.175.xxx.yyy [13/Jul/2013:06:28:41 94.123.xxx.yyy [13/Jul/2013:06:32:49 94.123.xxx.yyy [13/Jul/2013:06:32:49 94.123.xxx.yyy [13/Jul/2013:06:32:50 94.123.xxx.yyy [13/Jul/2013:06:32:50 94.123.xxx.yyy [13/Jul/2013:06:32:50 94.123.xxx.yyy [13/Jul/2013:06:32:51 === Break down 2 === Remove the **[** bracket with sed. Remove the time portion of the output with cut. cat access.log | awk '{print $1 " " $4}' | sed 's/\[//' | cut -d":" -f1 Output of the above 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 === Break down 3 === Swap IP and Date such that Date is 1st and IP is 2nd cat access.log | awk '{print $1 " " $4}' | sed 's/\[//' | cut -d":" -f1 | awk '{print $2 " " $1}' | Output of the above 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy 13/Jul/2013 94.123.xxx.yyy === Break down 4 === Remove duplicate IPs - to get unique IP hits per day cat access.log | awk '{print $1 " " $4}' | sed 's/\[//' | cut -d":" -f1 | awk '{print $2 " " $1}' | \ sort | uniq Output of the above 13/Jul/2013 69.175.xxx.yyy 13/Jul/2013 94.123.xxx.yyy === Break down 5 === Now the IPs are no longer interesting as we only need their count. So remove IP by printing only date. Then do a **uniq -c** on the output to get the counts for the dates cat access.log | awk '{print $1 " " $4}' | sed 's/\[//' | cut -d":" -f1 | awk '{print $2 " " $1}' | \ sort | uniq | awk '{print $1}' | uniq -c Output of the above. So there are only two unique hits in our example for the one date. 2 13/Jul/2013