We’re doing some server maintenance tonight from 6:00PM to midnight Pacific Time. You’ll still be able to use EEBA on your device, but won’t be able to sync or reach the website during that time. Thanks for your understanding!
Updated: October 24, 2011
We’d like to update you on last night’s unplanned maintenance activities. We know that you rely on EEBA to always be available for you. (We do too, because we’re users!) We’re sorry that that was not the case for 6 hours last night.
EEBA’s website and sync-to-server were unavailable from 10/23 6:05pm to 10/24 12:15am Pacific Time. Our staff was onsite within two hours, and a technician and parts arrived within the time agreed upon in our maintenance contract. (Kudos to them for working so quickly on a Sunday evening.) Service is now fully restored. In addition, because of the way that EEBA is built (supporting true offline use and synchronization) if you were already logged into the EEBA app on your Android device during this time, you were able to continue entering transactions there. Any work you did on your phone could be synced back up after the server came back live at 12:15am.
For those of you who were using EEBA on the web between 6:05pm and 7:05pm Pacific Time, you would have seen some glitchiness. We apologize for any confusion this may have caused you.
If you are interested in the technical details, keep reading:
- As a part of our commitment to keep our systems up-to-date and performing smoothly, we scheduled maintenance on Tuesday 10/18 to bring a brand-new Dell database server online with 10x the capacity of our previous server. The migration went smoothly and everything checked out.
- Last night, our automated monitoring systems detected that this new server was not responding. Our staff diagnosed the issue, determined there was a hardware problem, and arrived at our server co-location facility within 2 hours.
- Dell sent parts and a technician within the agreed timeframe according to our 24×7, “mission-critical system” service agreement — on a Sunday evening.
- Most of our parts for the servers we run for EEBA (including power supplies, RAID-configured hard disks, fan, network cards, etc.) are internally redundant. However, the problem in this case ran deeper. Our staff and the technician worked together to replace first a memory stick, then one of the CPU’s, and finally the whole motherboard. That did the trick.
- The disk array and data were all intact.
- Within 15 minutes of completing the hardware replacements and verifying the server’s health, we brought EEBA’s website and server live again.
It turns out that our new server was a bit of a lemon — but now it’s *really, really* new. To keep your data safe, we continue to remotely backup your data offsite with Amazon’s EC2 web service. We’re also evaluating other steps that we can take to reduce the impact of failures like this.
Thanks for your support and understanding. Please feel free to email firstname.lastname@example.org if you have any questions at all!