Prior to version 2 of he log parser the logs were simply dumped into the database for the past 60 days every hour. It was planned that Users could view these logs instantly through a control panel. But unless you really knew what you were looking at this data was sort of useless.
So we came up with V2 of the Log Parser. This parser parses errors and website access into tables which gives a lot more information about the logs. For example I can pull up a number of how many 404 errors happened on any site hosted with CLDMV on any day in the past 60 days. As well as any redirect. If I want to view all the 404 or redirects I can do so. This functionality will be included in the control panel asap.
Why did we do this?
Simple answer really. Analytics services can provide you with how many visits your site gets and a ton of data about this. However they only work on pages where their code is present. Since we can track on the server level we can provide a lot more data. This data helps website owners make critical decisions and code fixes where they see trends in their software.
“Why am I not being indexed by [insert name of search engine]?”
We hear this question quite often. Simply put we do not control how search engines index websites. But we can show you (now) that your site is being crawled by their bots. V2 of the log parser is also logs if the visit was a bot or not.
To show an example:
| domain | year | month | day | bot_visits | bot_unique_visits | visits | unique_visits | fourohfours | redirects |
|---|---|---|---|---|---|---|---|---|---|
| blog.cldmv.net | 2014 | 5 | 27 | 20 | 3 | 30 | 5 | 0 | 0 |
The above results show this site’s basic stats for 2014-5-27. Visits and Unique visits are overall. So all we have to due to find human visits is subtract the bot visits from those numbers.
| year | month | day | domain | timestamp | ip | real_ip | ssl | bot | ajax | resource | url | response_code | visit | referer | bytes | gzip_ratio | log_entry | user_agent | request_type | request_version | remote_user |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2014 | 5 | 28 | example.com | 1401248811 | 66.249.73.71 | 1 | 1 | 0 | php | /exampleurl.php | 302 | 1 | 31 | 66.249.73.71 - - [28/May/2014:03:46:51 +0000] "GET /exampleurl.php HTTP/1.1" 302 31 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "-" "https" "-" | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | GET | HTTP/1.1 |
Above is an example of all the data available through our Log Parse V2.
