Important: Email Service Update

Rackspace, our email technology partner, has been working through a disruption in service since yesterday morning (July 13th, 2017) making it a tough couple of days for a lot of you, and for us, too. Email is like oxygen these days, and we want you to know that we are right here with you in this and are feeling the frustration right alongside you.

We partner with Rackspace for our email server because they’re the best, most reliable solution 99.9% of the time. Right now, though, we’re experiencing the other 0.1% of the time as they sort through issues with the email server. Typically, these interruptions of service are little blips that are resolved quickly, however, this one seems to be a big one.

Rackspace is currently working hard to fix this problem, and we are hoping it will be up and running again soon. If you’d like to keep an eye on the progress Rackspace is making, you can take a look at their real-time updates on their status update page.

We wish we could offer more information as to when the issue will be resolved, but currently, we are passing along as much as we know from Rackspace.

We apologize for the inconvenience and appreciate your understanding.

Update as of 11:09 PM EDT 07/14/17

The official “all clear” has not been issued, but it appears the Rackspace email system is returning to normal. Here’s information presented on the Rackspace status update page:

System load is progressing towards normal levels and most users should find the email system operational. Until the system is fully restored, users who experience occasional timeouts and connection errors should refresh their connection.

[We] are staffed to work around the clock over the weekend to follow up on the issues from this week.

We will continue to provide updates as they become available.

Update as of 8:26 AM EDT 07/15/17

The Rackspace status update page indicates:

We have made progress in repairing the impacted mailboxes. Some users may experience sporadic delays today while the repair continues.

Update as of 03:05 PM EDT 07/15/17

 The Rackspace status update page indicates:

We have completed another phase of repairs and have initiated the next phase. During this time, some users may experience sporadic delays along with Webmail, POP & IMAP connectivity issues.

Update as of 06:36 PM EDT 07/16/17

The Rackspace status update page indicates:

Our efforts this weekend have been focused on system wide cleanup to address the consequences of high system load after expanding capacity. These efforts are nearing completion. We will soon begin a more detailed mailbox-level cleanup effort to address some issues that we have seen among some users. While those updates happen there may be very brief periods where accessing email is slow or gives an error; trying again a few minutes later should be successful.

Update as of 08:55 AM EDT 07/17/17

The Rackspace status update page indicates:

Over the weekend we have worked through a system-wide cleanup to address the after-effects of the high system load experienced last week. The load was the result of an unknown bug that impacted an effort to proactively double system capacity.

Our priority today is to maintain good access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. In an abundance of caution, we are limiting the number of simultaneous connections to a single mailbox to 2 POP connections and 4 IMAP connections. Mail programs that exceed the limit will usually retry and users are unlikely to see errors due to connection limits.

We are also rate limiting email delivery for mailboxes that receive hundreds or thousands of email messages per minute; that email will safely queue for delivery over a longer period of time. If a user normally receives less email than this, they should see normal delivery speeds.

After peak load hours today, usually finished before 2pm Eastern, we will consider a bulk mailbox level cleanup to address several issues we have identified proactively from system logs. These issues include:

  • some users have not yet been able to successfully access their mailbox
  • missing messages or folders; the information is present but not visible to email programs including Webmail
  • extraneous folders starting with “lost-“

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

Update as of 07:12 PM EDT 07/17/17

The Rackspace status update page indicates:

We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load.

Update as of 10:54 AM EDT 07/18/17

The Rackspace status update page indicates:

Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues.

Update as of 08:29 PM EDT 07/18/17

The Rackspace status update page indicates:

The initial passes to fix affected mailboxes were successful and the hosted email environment has stabilized.

Update as of 05:03 PM EDT 07/19/17

The Rackspace status update page indicates:

We continue to perform mailbox level cleanups to address the following issues, which we have identified proactively from system logs:

  • mailboxes with login/access issues: this issue may cause intermittent errors for users that connect via POP or IMAP.
  • extraneous folders prefixed with “lost-“
  • missing messages or folders: the information is present, but not visible to email programs, including Webmail.

We recommend that those users who experience intermittent errors connecting via POP or IMAP utilize Webmail instead.

While we are working to resolve all symptoms, our highest priority is ensuring all affected users have access to their mailbox.

Update as of 06:25 PM EDT 07/20/17

The Rackspace status update page indicates:

The primary focus early this week was to establish stable mail delivery and login access. While we have made significant progress, some work remains. Our current focus is on individual mailbox repairs due to issues indirectly caused when the system was under high load. Now that the system is stabilized, these mailbox repairs are processing more quickly. While we can automatically detect and remediate some issues, others require you to contact the support team.

There are certain issues which we are unable to detect If a user has not logged in. Because some users may go days or even weeks without logging in, we anticipate the symptoms described below will persist over a period of time. Administrators should contact the support team for the following issue:

  • if a user believes that there may be missing messages or folders, the administrator should contact support with that user’s email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

We auto detect the following issues by scanning system logs for errors that are generated when a user logs in to access their email:

  • for those users who have not logged in since the system load issue last week, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.
  • if a user has extraneous folders beginning with the word “lost-” these errors are automatically detected by our log scan. While this requires no action on your part to be cleaned, the user may also safely delete them if they do not contain any email. If a folder containing email is deleted, users may check the Trash or Recover Deleted Items folders to retrieve the email.

These processes will continue to run through the weekend. We expect to provide the next update on Monday, July 24, 2017.

Update as of 06:50 PM EDT 07/24/1

The Rackspace status update page indicates:

We have made progress in repairing the impacted mailboxes and will continue to run the processes listed below:

We auto detect the following issue by scanning system logs for errors that are generated when a user logs in to access their email:

  • for those users who have not logged in since the system load issue, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.

Users may clean-up affected folders for the following issue:

  • if a user has extraneous folders beginning with the word “lost-“, any email in these folders can be moved back to the original folder and the extraneous folder may be deleted.

Administrators should contact the support team for the following issue:

  • if a user believes that there may be missing messages or folders, the administrator should contact support with that user’s email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

We do not anticipate issuing further updates for this incident.

We will continue to provide updates from Rackspace as warranted.