Forrst downtime post-mortem, 9/24
Forrst was down for approximately one hour today, 9/24 (between ~10:30 and 11:30am EST). During that time, we disabled logins to prevent database writes while we assessed the situation. The root cause was that our database server was slightly misconfigured and was allowing idle connections to sit around instead of flushing them. This caused us to quickly exceed our max connection limit, causing mysql to sporadically return “unavailable” error messages to the web application. This caused random white screens to appear on certain requests, as well as sessions to be inadvertently destroyed. The offending configurations have been fixed, and the issue should be resolved going forward.
I’d like to personally apologize for the downtime this morning, and you have my word that we’re doing our best to ensure that Forrst remains stable and available to the highest degree possible.
If you have any questions or concerns, please email me at kyle AT forrst DOT com. Please also get in touch if you still experience white screens or being randomly logged out, and we will do our very best to resolve it.
Thank you.
