UPDATE 5:30 PM ET - This afternoon, we experienced a sudden, extreme high volume of requests to one of our customer databases that spiked our server resources, sending them beyond their ability to respond for a period of time. After attempting to resolve the spike itself directly to avoid any further issues, we ultimately took additional measures to take down and restart all of the services supporting customer databases and websites. We have put the following measures in place to prevent an additional spike in requests causing widespread issues:
- Blocking measures have been put in place for the specific source of these requests from this incident
- Increased total memory available for heavy loads on our SQL server
- Plans are in place to add additional rate limiting on requests to our servers from non-human traffic
We will continue to research this incident and work to prevent any future incidents with this type of impact.
-----------------------------------------------------
UPDATE 4:30 PM ET - All services have been restored for all customers, and we are investigating the cause of the outage.
------------------------------------------------------
3:35 PM ET
Starting around 3:20pm ET, we discovered all internal pages in the browser version of MC Trade are not loading correctly. Any browser-specific pages will not load, and some tabs or pages in the desktop that rely on the browser pages also may not load.
We are investigating this with utmost urgency and will update this article with more information as we have it.