In an effort to avoid this last weekends downtime again. The Database server was taken offline for a few minutes while the logs were moved to a separate hard drive. Previously there was approx 60GB of space available for the logs. This has been more than enough for nearly 2 years. However it was not enough this last weekend. So we have moved the logs into their own separate 500GB hard drive.
We will also be putting in a pruning system to avoid this issue even if the logs reach that 500GB size without auto pruning.
As with any downtime clients are compensated with much more time as a credit on their account for the downtime. For this last weekends downtime every client recieved a 1 week credit to their account for their subscription based services.