OpenTrainTimes experienced a minor incident on November 11, 2021 affecting Train movement data, lasting 12h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 11, 2021, 08:57 AM UTC
A problem with our database server means trains are showing up at incorrect stations. We aim to have this fixed in the coming hours.
- identified Nov 11, 2021, 08:57 AM UTC
We are continuing to work on a fix for this issue.
- identified Nov 11, 2021, 01:50 PM UTC
We have tracked the problem down to a single missing character in one of the scripts that manages our databases. A fix will be in place in the next two hours.
- monitoring Nov 11, 2021, 03:41 PM UTC
We have restored the database and are waiting for a backlog of messages to be replayed
- resolved Nov 11, 2021, 09:35 PM UTC
This incident has been resolved.
- postmortem Nov 12, 2021, 08:58 AM UTC
We have to laugh about this - it’s the perfect case of a single missing character screwing everything up. On the evening of Wednesday 10th November, we had to restore schedule data to our database server after a problem importing it. These things happen from time to time, and there was due to be no issue. However, the process we carry out is to re-import the data on another server and then ship two SQL tables of data over to the database server. Here’s where the fun starts. When exporting the data, we have to specify the names of the tables on the command line - ‘schedules’ and ‘schedule\_locations’ in this case. However, we’d mis-typed them as ‘schedule’ and ‘schedule\_locations’, meaning that only one of the tables was exported, not the second. We failed to spot the problem until after the data was restored, when that night’s timetable file was imported - creating a set of schedules for Thursday 11th November which were also wrong. We ended up with a schedule database where all of the timing points of schedules were intact and grouped correctly, but the ‘header’ records were pointing at the wrong set of timing points. This is why you saw trains at stations destined for unfamiliar locations. Once we’d identified the problem and were sure of the cause, we re-extracted and imported the correct set of schedule data, and baby-sat the site to validate everything was fine - which it is now.