Fixed 2 bugs on the language file update system

We’ve fixed 2 bugs on the language file update system. In order to be completely open, we’re sharing here what caused these bugs and what we plan to do so that it never happens again.

Last week we released a performance improvement to our language file system.

It contained 3 performance improvements:

1. On our LanguageFileHandler (our private library that contains the code to build your language files), we optimised the code to better handle the different file encodings and avoid unnecessary encoding conversions in order to improve performance. That change hasn’t caused any issues.
2. On the main app we changed the segments loading system to load segments in batches to save time and reduce memory usage. Again, all good there.
3. On the main app we also changed the translation eager loading system to only load relevant translations in order to avoid loading translations in all languages when we generate files in a specific language.

Unfortunately, this last change resulted in a bug in which projects using language inheritance would have their language files filled up with blank strings.

We have reverted that part of the change while we work on a better eager loading system that works with language inheritance, which is way more complicated than we first thought.

We’ve regenerated all the affected language files so everything should be correct now. We are also working on increasing the test coverage on the inheritance feature to make sure this kind of issue gets caught in the future. We’re really sorry about that.

We also found a fixed another bug due to a change on an internal tool on our end. We’ve currently trying a new queuing system for our background job (the system importing and updating your language files).

The new system (GoodJob) is about 30 times faster at scheduling jobs than our older system (DelayedJob). So fast, actually, that it exposed a race condition that was in our code since day 1.

When you upload a language file to WebTranslateIt, we originally added a File Import job to the queue and then flagged the file status to queued. The new queueing system is so fast that the job could sometimes finish before the parent process would had time to flag the file as queued, then the parent process would flag the file as queued although the import job was already done.

We’ve fixed this bug and fixed all the files affected by this issue.