Log Parser V2

Prior to version 2 of he log parser the logs were simply dumped into the database for the past 60 days every hour. It was planned that Users could view these logs instantly through a control panel. But unless you really knew what you were looking at this data was sort of useless.

So we came up with V2 of the Log Parser. This parser parses errors and website access into tables which gives a lot more information about the logs. For example I can pull up a number of how many 404 errors happened on any site hosted with CLDMV on any day in the past 60 days. As well as any redirect. If I want to view all the 404 or redirects I can do so. This functionality will be included in the control panel asap.

Why did we do this?

Simple answer really. Analytics services can provide you with how many visits your site gets and a ton of data about this. However they only work on pages where their code is present. Since we can track on the server level we can provide a lot more data. This data helps website owners make critical decisions and code fixes where they see trends in their software.

“Why am I not being indexed by [insert name of search engine]?”

We hear this question quite often. Simply put we do not control how search engines index websites. But we can show you (now) that your site is being crawled by their bots. V2 of the log parser is also logs if the visit was a bot or not.

To show an example:

domainyearmonthdaybot_visitsbot_unique_visitsvisitsunique_visitsfourohfoursredirects
blog.cldmv.net201452720330500

The above results show this site’s basic stats for 2014-5-27. Visits and Unique visits are overall. So all we have to due to find human visits is subtract the bot visits from those numbers.

yearmonthdaydomaintimestampipreal_ipsslbotajaxresourceurlresponse_codevisitrefererbytesgzip_ratiolog_entryuser_agentrequest_typerequest_versionremote_user
2014528example.com140124881166.249.73.71110php/exampleurl.php30213166.249.73.71 - - [28/May/2014:03:46:51 +0000] "GET /exampleurl.php HTTP/1.1" 302 31 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "-" "https" "-"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)GETHTTP/1.1

Above is an example of all the data available through our Log Parse V2.

Basic Control Panel and Three of the Five Themes Completed

Earlier this week we rolled out the basic control panel. At this point there isn’t much. Pretty much just displays current bans on the server and invoice data.

Three of the five themes (Default, Orange, Light) have been completed. Blue and Dark are still to come. As well as additional features inside the control panel and a better look inside there.

Some things to come are also domain name management and SSL management. These are two services which we plan to provide on top of the hosting and email services we currently provide.

Developer Frustration on CLDMV

Recently I’ve been putting a lot of time into development of the control panel. Development of the control panel had been hindered though by one system that makes the sites on CLDMV’s network run faster.

OPCache was the problem. This module of PHP cache’s the results of the compiled PHP code and then serves that up if there is a cache rather than re-compiling the PHP code again. Seems rather simple and it was. For future reference for any developer working on a site in CLDMV’s network. All you have to do is add the below snippet of code to the first file that is being called to disable the cache during development stages.

 

Updates to Web Server

There was a bug affecting a few clients where they could not delete files in the root of their html folders. This has been fixed. I also have a process running every day now to ensure this doesn’t happen again. The following files will server up a default file if they are not found in the html root folder:

  • robots.txt
  • favicon.ico
  • favicon.png

The default robots.txt only had one rule which is to tell all bots to limit their crawl rate to 1 page per second. Any client can place their own custom robots.txt or favicon.ico in their html root and that file will be served up instead. Some additional cleanup and security was done to the FTP service. Namely when you log into the FTP, clients will only see folders which you can actually do stuff in now. Some of which are just backup or log folders which clients can only (most of them) download the files.

DNS Mail Settings

While most people never have an issue with their emails. One of the most basic ways to help prevent being black listed and prevent spam from being spoofed from your domain name is to set up SPF records.

Here is http://cldmv.net/ SPF record:

We’ll go over what each of the portions mean now.

v=spf1

This specifies what version the SPF is. Currently the only version supported is spf1.

mx

Specifies that email originating from MX records of the domain name may send email as well.

ptr

Allows any sub domain of the domain to send out email. This can be spoofed but not very easily. Generally someone would need to have access to your DNS records to change this. Also if you have an A Record with “*” pointing to your server this helps prevent this as well.

include:cldmv.net

Specifies that any domain ending in cldmv.net may send email for the domain as well. Generally this would be for your hosting provider to determine based upon their setup.

-all

Notice the negative sign. It tells servers which follow SPF to reject all emails which do not meet the previously set rules.

Anyone hosting email with CLDMV should set their MX record to mx.cldmv.net as well as add the following txt record in order to insure emails are sent and recieved following SPF.

Automatic Optimization of images

I was cleaning out my email folder yesterday and I came across this old email. While I have removed anything that would specify who this email is with the results are still proving of CLDMV’s systems. I’m also pretty sure this email was prior to the double compression PNG system we currently use now.

As you can see from above not only did our system compress the images as much as Google said we could. But we were able to compress 1-4% more. Here is a recent daily report of our image compression system.

Keeping in mind all the compression is done lossless. Meaning there is no loss in visual quality. The only downside to the compression is if the images store data such as where they were taken, that data is removed.

 

New Security Features

Previously we had been using the over-popular Fail2Ban to scan our logs as a temporary fix for the issue. While the anti-DDOS software written by CLDMV takes care of a ton of bans every day. The hacking attempts are still being attempted by some what smarter hackers.

Today we rolled out our first module for log scanning. With SSH probably being the #1 threat to servers out there that is what we chose to target for our scanner. Took several days but the results are amazing. While I can’t divulge the inner workings of the module. Let me just show you the first ban email we got after running it for the first time:

Keep in mind these numbers and bans are simply based upon the past 24 hours of logs.

Update:

This guy takes the cake for CLDMV’s Anti-DDOS software catching a potential intrusion. Just received this email:

Also the log processing module for SMTP hackers is in place as well now. Here’s the first email for the past 24 hours of attempts:

Why CLDMV for hosting?

On average our clients are seeing a 20-30% increase in page speeds from their old hosting environments to CLDMV’s Servers. Roughly a 40% decrease in the size of downloaded content as all CSS/JS is minified, users can minify their HTML if they choose to do so and all images are compresses to the maximin they can be (completely lossless compression of course).

So far all of our clients have came to us from their own dedicated servers where they were paying on average 300$/month for their servers. CLDMV’s clients pay on average 130$/month as of writing this post. While essentially our service is “Shared hosting on steroids” it can also be compared to a “no limit” VPS. Since most people who are running online websites grow out of their shared hosting environment within the first couple years of business (assuming traffic increases at a steady rate) they don’t want all the hassle of managing a server.

Often more times than not they pay 30-40$/month to have a commercial control panel installed and on top of that pay a system administrator to change things when needed. With CLDMV changes are only a call away and once the control panel rolls out many operations of the hosting plan will be available via the control panel (free of charge).

In our experience most clients running online websites simply need a faster website. They don’t need nor want an entire dedicated server. The industry has forced users to upgrade to that point due to over selling and over crowding servers. Here at CLDMV we monitor everything on the servers daily and apply upgrades or code changes to keep the system running smoothly. Client stats are tracked every hour, server stats every 15 minutes. Traffic is always instantly tracked (logged every 60 seconds). Our mission is to provide affordable hosting with all the benefits of a going out and hiring a optimization company to come and optimize your dedicated server. With CLDMV you don’t have to hire or pay for anything extra for your service.

With many options available (some not listed or posted about yet) the possibilities of hosting with CLDMV are endless. Need your own or want to use your own database server? We got your covered. Need load balancing? We can do that as well (though our biggest servers can handle over 1000 512MB php threads at any given instance and up to 10Gbps of transfer). Just want complete isolation? We offer dedicated servers as well where all the optimization and management of the server is done by us and you have a completely isolated part of our network.

Lets step back for a minute. Any hosting company can offer you the world but can they deliver? Do they have hard numbers which prove their service? Lets let the numbers talk for themselves. Below is a table of response time, load time, downloaded content (with a hard refresh to force all content to be downloaded), google page speed score and estimated cost if any idea of cost is available.

Site 1: http://beautyjoint.com

 

Site  Response Time  Load Time  Download (First Load)  Page Speed Score  Cost
Original  575ms  2.25s  900KB  66  1000$~
Compare  444ms  1.56s  627KB  86  200$~

Site 2: http://adrianas.1stwebstudio.com/en/

 

Site  Response Time  Load Time  Download (First Load)  Page Speed Score  Cost
Original1.61s7.00s4.9MB46??
Compare775ms3.97s3.6MB86130$~??

Site 3: http://www.extasybooks.com/

 

Site  Response Time  Load Time  Download (First Load)  Page Speed Score  Cost
Old945ms5.85s1.1MBKB  74600$~
Current772ms1.97s  627KB  88300$~

 

Notes:

Site speeds and scores ran on 2014/05/07 at 4pm PST (for site 1 and site 2)

Links may or may not be valid at time of viewing.

Sites listed above have given permission to post these stats as a comparison of our service.

Minify HTML as a service

While mileage may vary with minifying HTML it is not done automatically like minifying css and js. However there is a background service running for anyone wishing to use it on the server free of charge. Below is the php code to access the service.

Combined with the above code you can use the below code to retrieve the minified html body as well as the original bytes, minified bytes, and saved percent by minifying the html.

Server Upgrade

Server (s1) is scheduled to have another SSD hard drive added to it on:

5.4.2014 (Sun) 01:00 (GMT-0500) – 04:00 (GMT-0500)

Shouldn’t be any downtime as it’s simply a hard drive however there may be some downtime between 10pm PST and 1am PST.