tag:status.appen.com,2005:/historyAppen Status - Incident History2024-03-28T11:40:42-07:00Appentag:status.appen.com,2005:Incident/186589012023-10-01T10:13:29-07:002023-10-01T10:13:29-07:002023-10-01 Connect Emergency Maintenance<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>10:13</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>08:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>08:26</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>08:15</var> PDT</small><br><strong>Scheduled</strong> - We will be performing maintenance on Appen Connect - connect.appen.com, raterlabs.appen.com, facility.appen.com and raterqualification.com will be unavailable. Appen Data Annotation Platform (ADAP) will stay online.</p>tag:status.appen.com,2005:Incident/184246402023-08-31T01:30:00-07:002023-09-07T10:17:21-07:00Degraded platform performance<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>01:30</var> PDT</small><br><strong>Resolved</strong> - Between Aug 31 8:33 - 19:51 UTC, the Appen data annotation platform experienced service degradation resulting in problems with the API, job unit state and loss of submitted judgment data. The platform returned to fully operational status by Sept 1 2:03 UTC. We apologize for this inconvenience.</p>tag:status.appen.com,2005:Incident/183244912023-08-30T12:30:32-07:002023-08-30T12:30:32-07:00Increased error rate on platform<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>12:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>12:17</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.appen.com,2005:Incident/182672022023-08-25T15:00:00-07:002023-08-25T18:27:47-07:00Platform outage<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - We experienced a brief system outage. Issue has been identified and resolved. The outage lasted 8 minutes.</p>tag:status.appen.com,2005:Incident/175645612023-06-13T14:25:27-07:002023-06-13T14:25:28-07:00AWS outage affecting data retrieval and uploads<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:25</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>13:55</var> PDT</small><br><strong>Monitoring</strong> - We're seeing reduced error rates on ADAP. Still monitoring and watching AWS for updates.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>13:33</var> PDT</small><br><strong>Investigating</strong> - AWS is having an outage with Security Token Service in us-east-1. This is causing problems when uploading data into a job, loading data from a storage provider. We're monitoring the situation.</p>tag:status.appen.com,2005:Incident/174471622023-06-10T13:57:33-07:002023-06-10T13:57:34-07:002023-06-10 Maintenance Window<p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>13:57</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>13:50</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>10:02</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>15:14</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.appen.com,2005:Incident/172316422023-05-12T13:05:31-07:002023-05-12T13:05:31-07:00Some jobs are not reachable due to platform errors<p><small>May <var data-var='date'>12</var>, <var data-var='time'>13:05</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved and pages load as expected.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>12:52</var> PDT</small><br><strong>Monitoring</strong> - We are now monitoring the implemented solution.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>12:46</var> PDT</small><br><strong>Identified</strong> - We have identified and are fixing a problem causing jobs to be unreachable on the ADAP platform. A fix is being deployed and will be tested.</p>tag:status.appen.com,2005:Incident/169326692023-04-23T13:42:11-07:002023-04-23T13:42:12-07:002023-04-23 Maintenance Window<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>13:42</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>11:46</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>09:59</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>11:28</var> PDT</small><br><strong>Scheduled</strong> - ADAP and Appen Connect will be unavailable during the scheduled maintenance</p>tag:status.appen.com,2005:Incident/167421252023-04-03T16:02:28-07:002023-04-03T16:02:28-07:00client.appen.com is in a degraded state<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>16:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>15:54</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>15:42</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>15:38</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.appen.com,2005:Incident/166087412023-03-25T14:00:15-07:002023-03-25T14:02:27-07:002023-03-25 Maintenance Window<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>14:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>13:51</var> PDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>10:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>10:21</var> PDT</small><br><strong>Scheduled</strong> - ADAP and Appen Connect will be unavailable during the scheduled maintenance</p>tag:status.appen.com,2005:Incident/164809312023-03-13T14:14:27-07:002023-03-13T14:14:28-07:00Elevated Error Levels - api.appen.com<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:10</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:09</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:05</var> PDT</small><br><strong>Investigating</strong> - We are investigating this issue.</p>tag:status.appen.com,2005:Incident/164807292023-03-13T13:46:54-07:002023-03-13T13:46:55-07:00Elevated Error Levels<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>13:46</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>13:45</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>13:43</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>13:42</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.appen.com,2005:Incident/162305542023-02-21T21:12:51-08:002023-02-21T21:12:52-08:00Authentication system in degraded state<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>21:12</var> PST</small><br><strong>Resolved</strong> - The incident is resolved and capacity is restored. Authentication and log-in requests are functional.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>20:41</var> PST</small><br><strong>Monitoring</strong> - We have identified the issue and are implementing a fix to increase capacity.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>20:31</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue affecting some logins on Appen Connect and the data annotation platform</p>tag:status.appen.com,2005:Incident/159444692023-01-26T13:57:38-08:002023-01-26T13:57:38-08:00Degraded performance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>13:57</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:40</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:20</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>10:43</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an operational issue impacting job processing</p>tag:status.appen.com,2005:Incident/158389732023-01-21T14:00:42-08:002023-01-21T14:11:17-08:00Scheduled Maintenance on 21st January 2023<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>14:11</var> PST</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>14:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>13:17</var> PST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>10:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>12:49</var> PST</small><br><strong>Scheduled</strong> - ADAP and Appen Connect will be unavailable during the scheduled maintenance</p>tag:status.appen.com,2005:Incident/156834752022-12-20T19:04:13-08:002022-12-20T19:04:14-08:00Appen Connect failure to login<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>19:04</var> PST</small><br><strong>Resolved</strong> - We have addressed issues around users with failing logins and functionality is now restored for all users.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>18:40</var> PST</small><br><strong>Update</strong> - We have started the process to fix the issue observed on the remaining users.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>16:38</var> PST</small><br><strong>Monitoring</strong> - We are implementing a fix now for internal users first before applying for all users.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>15:41</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting Appen Connect.</p>tag:status.appen.com,2005:Incident/156833242022-12-20T15:08:46-08:002022-12-20T15:08:47-08:00Failure to log in to Appen Connect<p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>15:08</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>15:08</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>15:07</var> PST</small><br><strong>Identified</strong> - We have found and identified an issue blocking authentication to Appen Connect via connect.appen.com and are implementing a fix now.</p>tag:status.appen.com,2005:Incident/132140642022-11-16T10:52:20-08:002022-11-16T10:52:21-08:00Analytics temporarily down<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:52</var> PST</small><br><strong>Resolved</strong> - The issue has been identified and resolved. Analytics are now available. Please reach out to help@appen.com if you are still having issue accessing this feature.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>08:29</var> PST</small><br><strong>Investigating</strong> - Analytics feature is temporarily unavailable. We are working to restore this feature. Please contact help@appen.com if your work is being blocked</p>tag:status.appen.com,2005:Incident/131700882022-11-15T19:22:37-08:002022-11-15T19:22:38-08:00Copying jobs temporarily down<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>19:22</var> PST</small><br><strong>Resolved</strong> - The issue is now resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>18:26</var> PST</small><br><strong>Monitoring</strong> - Copying functionality has been restored and being monitoring. please reach out to help@appen.com if you are still experience any issues</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>17:23</var> PST</small><br><strong>Investigating</strong> - We are currently investigating the issue with copying jobs. We will have more updates shortly. Please reach out to help@appen.com if this is blocking your work.</p>tag:status.appen.com,2005:Incident/127706002022-11-12T14:00:07-08:002022-11-12T14:00:08-08:00Maintenance Window 11/12/2022<p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>14:00</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>13:45</var> PST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>10:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>15:20</var> PST</small><br><strong>Scheduled</strong> - We will be performing software updates to databases, caching and search systems as well as server migrations for some of these systems during this maintenance window. We'll provide updates as the work progresses to completion.</p>tag:status.appen.com,2005:Incident/128543542022-11-11T09:00:01-08:002022-11-11T09:00:02-08:00Increased error rate on ADAP<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Resolved</strong> - We have cleared the degraded performance issue. We will follow up on issues identified and continue monitoring internally.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>08:56</var> PST</small><br><strong>Monitoring</strong> - We've found issues and made adjustments to improve the situation</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>08:32</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>08:25</var> PST</small><br><strong>Investigating</strong> - We are finding increased error rates across our platform, we are investigating the source of the issue.</p>tag:status.appen.com,2005:Incident/127449582022-11-07T09:24:27-08:002022-11-07T09:24:27-08:00Platform outage<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:24</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:13</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>08:59</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>08:54</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.appen.com,2005:Incident/123705112022-10-20T07:01:34-07:002022-10-20T07:01:34-07:0010-20-22 instability<p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>07:01</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>07:01</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>06:40</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we're watching the results.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>06:30</var> PDT</small><br><strong>Identified</strong> - the issue has been identified and we're working on resolution</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>05:41</var> PDT</small><br><strong>Investigating</strong> - We're experiencing system instability and are investigating the issue.</p>tag:status.appen.com,2005:Incident/113197922022-10-01T13:26:18-07:002022-10-01T13:26:19-07:00Scheduled Maintenance 10/1/2022<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>13:26</var> PDT</small><br><strong>Completed</strong> - The maintenance has completed successfully</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>12:14</var> PDT</small><br><strong>Verifying</strong> - Verification is underway</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>10:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>16:08</var> PDT</small><br><strong>Scheduled</strong> - We will be performing software updates to databases, caching and search systems as well as server migrations for some of these systems during this maintenance window. We'll provide updates as the work progresses to completion.</p>tag:status.appen.com,2005:Incident/113011672022-09-28T08:43:32-07:002022-09-28T08:43:34-07:00Network Outage<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>08:43</var> PDT</small><br><strong>Resolved</strong> - We experienced a two minute outage this morning as a result of a network configuration change. The error was detected immediately and corrected. The platform is functioning normally at this time. Apologies for any inconvenience.</p>