About yesterday’s file import hiccup

We had a technical issue yesterday afternoon. Some file imports created duplicated strings instead of aligning the translations.

This post is fairly technical. If you’re not technically versed, rest assured I fixed the problem yesterday and I am now making sure this doesn’t happen again. I’d like to sincerely apologise if you had any issue yesterday.

Please contact support@atelierconvivialite.com if we can make it up for you or if you spot any problem.

What happened

We have 3 distinct work environments:

• development, on my computer,

• staging, a separate environment on the production server, where I test things before going to production,

• and production, the live Web Translate It.

I am currently working on improving the importer’s speed. After testing my changes on my development environment, I deployed them on the staging environment.

Web Translate It’s importers are long-running tasks (sometimes up to 10 minutes for large files), so they have to run in the background. We have a queue system (delayed_job) and currently 4 workers working off jobs from the queue. We use god to monitor our workers and restart them automatically when they crash or eat too much memory or CPU.

When we deploy new code, the deploy script restarts the workers so they use the newer code. However, I didn’t expect the deployment script for the staging website to actually restart the production workers and replace them by the staging workers. I didn’t noticed it immediately, so for a few hours, we had staging workers working off jobs from the production database.

These workers were buggy and in some cases, were duplicating strings instead of aligning the translations. I’m really sorry about that.

This shouldn’t happen again

Lesson learnt. I’m moving the staging environment off to a separate server. Such mistakes should be really hard to make in the future.