The WebTranslateIt Blog

Hey there!

Last Thursday the 12th of December in the morning WebTranslateIt was hit with severe performance issues. The website and the API became unavailable at time.

Every 10 minutes, we received massive waves of requests (more than 400 requests per seconds on the API), which made the web server very slow.

Upon investigation, we found out that the performance issues were due to some users using automated tools performing requests on the String API. We always intended this API to be rate limited (in fact, our documentation has always stated that we limit requests up to 25 requests per seconds), but we noticed that some users were using the API way over 100 requests per seconds, so our rate-limiting was clearly not working.

We are now properly limiting this API up to 25 requests per seconds with bursts up to 30 requests per seconds. We think this is a very reasonable limit given the fact that this API serves up to 250 segments and all their translations, and requesting over 30 requests per seconds to one web service really is doing a lot of requests.

If you get a HTTP error code 429 Too Many requests it means that you are being hit by our rate-limiter. The only way to avoid this limit is by reducing the amount of requests you are doing to that endpoint. You can do that by implementing caching or by introducing a few milliseconds pause on the device performing the request. We limit requests by IP and by organization. If you cannot use caching nor can you pause the server doing these requests, are you using the correct API for this job? The File API for instance, lets you download all the segments of a file in one request.

Properly rate-limiting this API is essential. Having no rate-limit to this API results in a unreliable API and service in general.

If you need any help or have any issues with our API, don’t hesitate to contact us.

We released a new version of the web_translate_it gem, the open-source synchronization tool for Web Translate It.

Our synchronization tool is very stable now, so we do very few releases. When we do, it’s mostly tiny bug fixes and features requested by our customers.

New Feature

There have been several releases since the last release announcement (for web_translate_it 2.4.0) on the blog.

We implemented one new features: the ability to move files with the wti mv source target command.

Here’s what is new since that version from the changelog

Version 2.4.11

• New command: wti mv path/to/file.po newpath/to/file.po to move a language file both on your local working directory and on your WebTranslateIt project.

Version 2.4.9 / 2019-08-07

• Fix issue referencing old Trollop library. #149
• Fix issue with SSL certificates. If you encounter any issues you will need to update your CA certificates please visit this help: https://github.com/AtelierConvivialite/webtranslateit/wiki/Update-CA-certificates

Version 2.4.8 / 2018-09-06

• Update deprecated gem Trollop to Optimist.

Version 2.4.7 / 2018-03-28

• Add an option to .wti file to silence SSL errors.

Version 2.4.6 / 2018-02-28

• Fix SSL issue.

Version 2.4.5 / 2017-12-21

• Fix issue pulling multiple files. #142

Version 2.4.4 / 2017-11-27

• Fix issue on wti status command.

Version 2.4.3 / 2017-11-22

• Clearer error message when no files to push. #136
• New: ability to pull by file name and language. #133 Example: wti pull config/locales/app/* -l en
• New: wti pull [filepath] now pulls files matching a glob match on the files hosted on WebTranslateIt.com (instead of relying on shell’s list of files which might not exist on the first pull). Close #137. This shouldn’t change existing commands but allows typing something like: wti pull config/locales/*/en.yml to download only the en files.
• Fix: Report error messages when running commands such as:
  • wti rmlocale xxx
  • wti addlocale xxx
  • wti status #139

Version 2.4.2 / 2017-09-28

• Fixed an issue where a file would not be created if its content was empty.

Version 2.4.1 / 2016-02-03

• wti status now returns a status code when a project is not 100% translated or not 100% proofread. #127 Status codes are:
  • 100 if not 100% translated,
  • 101 if not 100% proofread,
  • 0 if project is 100% translated and proofread.
• wti init now returns a successful status code. #126

---

Install or Upgrade

To install web_translate_it, please refer to the gem documentation.

To upgrade web_translate_it to its latest version, type in a terminal: gem install web_translate_it.

DockerHub build

We released a DockerHub build for our wti client tool. So you shouldn’t have to install the whole ruby stack if you want to use wti.

To use it, simply install Docker and run: docker pull webtranslateit/webtranslateit

Hi there!

We haven’t been updating this blog regularly lately, because we’ve been working on compliances and features that can’t be seen and on larger features that can’t be released yet. We’ve also been adding smaller, new features and bug fixes. Here are a few of these features and changes that we released recently. In a nutshell: updated payment gateway with SCA support and SEPA Direct Debit for our European customers, updated machine translation APIs, updated file handlers and a new Team and Project cleanup feature.

Newer payment workflow and new payment method (SEPA Direct Debit)

First came the RGPD, and now the SCA.

SCA is another big thing for European web companies. It requires web companies based in Europe to verify their European customer’s bank cards using 3D Secure in order to verify their consent for payment. This extra mandatory step in the payment workflow required us to rewrite most of our payment pages. I’d like to think it’s for the best, because we are now using the latest version of Stripe’s tools and also because it allowed us to support payments by SEPA Direct Debit easily.

If you are a European customer and would like to pay by card, just update your payment method in your organization settings, right under the “Payment Method” tab.

Upgraded Microsoft Azure Translator to v3

We also upgraded our Microsoft Azure Translate from v2 to v3. This change should be seamless and invisible for you, however it is great to know that you are now running on the latest version of the API.

Microsoft .resx files now supports plural forms

We constantly deploy bug fixes or small improvements to our language files handler based on our customer’s feedback. Most of these changes are too small to be detailed here, but we released something new on our Microsoft .resx/.resw file handler.

Our Microsoft .resx file handler can now handle plural forms. Although they are not officially supported, you can use plural forms by using an external library such as PluralNet or ReswPlus. You will find in our documentation an example of .resx file with a plural form.

Users and Team cleanup

WebTranslateIt is around since 10 years now. This means that some of our users are simply not active anymore. We now monitor our e-mail bounce rate and flag some e-mail addresses as not deliverable, so we don’t send so many e-mails to unreachable recipients again.

We thought that knowing if a user can receive messages and is still active is interesting data for project managers, too. So we now show this information on the Team page or Project Users page for managers.

Users whose work e-mail address bounces back have probably left your company and won’t collaborate on your project anymore.

Note that invitations to join translation projects are already automatically cleaned up. Invitations that haven’t been accepted in 3 months are automatically deactivated.

Stickers anyone?

We ❤ our users, and we have some stickers for you if you want to grab them. Just let us know and we’ll send you a few sticky dogs with its jetpack on your way! 😃

---

I hope you will like all these changes. Don’t hesitate to get in touch with us if you have any questions, feedback or feature requests!

And don’t forget to follow us on Facebook, Twitter and LinkedIn.

If there’s one thing we’re all interested in, it’s free software - especially when it comes with no ads. Metatogger and Echosync both meet these expectations. Developed and launched by Sylvain Rougeaux, legal expert during the day and amateur programmer at night, you can find them on the Lumisoft Software website.
Lumisoft/Luminescence Software is not a company nor an association, it doesn’t have any legal existence. It’s just a non-official brand Sylvain uses as a « home » for the software he develops. He’s just a private individual sharing his software programs for free.

WTI: When did you start developing Metatogger and Echosync and what motivated you to do it?
S.R.: I have started developing Metatogger in 2005 and Echosync (ex-SyncToVHD) in 2010, on my own. It was a long time ago now, time flies!
I am not a professional developer, I have learnt how to code by myself. I’ve always been in awe of the potential of computer engineering: I find programming self-rewarding - in the way that you don’t have to limit yourself to what others have created for you and can create something that really fits your own needs - and also fascinating because of the constant innovations in this field allowing you to keep improving things all the time.

WTI: Can you tell us what kind of software are Metatogger and Echosync and how did you come up with idea of creating them?
S.R.: Metatogger is a tag editor for audio files (Ogg Vorbis, FLAC, M4A (AAC et ALAC) and WMA). These tags or metadata allow you to manage substantial music collections more easily. The problem is that many audio files are not coming along with metadata, while others are having wrong, useless or inconsistent data. Sometimes, even the file name doesn’t reflect the true content of the file. It is then difficult to identify the piece without playing it. This is where Metatogger makes it great.
Metatogger allows you to fix or complete these tags. Here are a few of its features:
* edit tags manually or using C# scripts
* retrieve the tags from the file name or path
* rename files according to their tags
* look for potentially duplicate files
* order files according to their tags (copy, move, symbolic links)
* retrieve tags from a local database holding up to 1,5 million music albums
* recover album covers or lyrics from the internet
* identify files after analyzing their acoustic fingerprint
* quickly clean out unwanted tags

Echosync is a software that doesn’t require any installation and allows the differential synchronization of two directories.
This way, only the modifications that have happened since the last synchronization has passed on. Echosync uses an algorithm to detect files that have been renamed and/or moved around which allows not to delete and then copy all these files in the destination directory. Echosync can copy locked files and supports file paths of over 255 characters.

Partial synchronizations are possible, like for instance copying only new files but not the modified ones.

Many amateur programmers spend a lot of time and energy on making clones of existing software, keeping on reinventing the wheel. I wouldn’t have created these two programs if I hadn’t thought I wasn’t able to make something better than what was already available.
I wanted them to different: free of charge so that anybody could use them, with a simple interface, a user-friendly design (I think a lot of developers built software only meant for other developers) and reliable. I hope I’ve met these objectives!

WTI: Who is maintaining and developing Metatogger and Echosync now and have you ever thought of selling these programs?
S.R.: I’m still doing it on my own. I’m not making a living developing software, it’s my hobby. Anyway, I don’t think these two specific programs have any sufficient market potential. And not selling them gives me total freedom as I don’t have to meet market expectations, I develop only features I want to develop and I do it my way.

WTI: Metatogger and Echosync are already available in French and English, do you plan on having them translated into other languages?
S.R.: I’m the one taking care of their localization in French and English. Metatogger is also available in Italian, Spanish, German, Russian and Greek, thanks to users volunteering to translate it. The same users just offered to translate Echosync so it will soon be available in these languages as well.

WTI: In how many languages do you intend to translate the website and how could people help out with the translation?
S.R.: We’re currently covering 14 of the European languages: Croatian; English, Finnish, French, German, Hungarian, Italian, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish, Swedish plus Ukrainian - and we’d still need help for Bulgarian, Czech, Danish, Dutch, Estonian,  Greek,  Irish, Latvian, Lithuanian, and Maltese.

WTI: Can you tell us how WebTranslateIt is helping you out with the localization process?
S.R.: WebTranslateIt is perfect for localizing .NET software because it natively supports .resx files. Downloading and uploading datas extracted from these files onto WebTranslateIt is a very smooth process and the translated segments are directly usable with Visual Studio.
WebTranslateIt being an online software makes collaborating with a team of remote translators very easy. Each translator gets an automatic notification when a new segment has been added or modified on the platform, which insures the continuity of the localization with up-to-date translations ready to be deployed.

WTI: How do you finance the development and translation of Metatogger and Echosync?
S.R.: Since neither of them generates revenue, I have to be careful to keep the development and maintenance costs under control because I finance everything myself.
Most of costs come from hosting the Luminescence Software website and WPF graphic components from DevExpress. I do get a few donations through PayPal but it covers only a very small part of the global costs.
Nevertheless, I’ve managed to keep them manageable: I’ve obtained an important discount from DevExpress after I explained both my projects are non-profit and I use as much as possible open source libraries and websites offering their services for free to non-profit organizations, like WebTranslateIt. Thanks guys!

WTI: Are you looking for volunteers to help with the translation process?
S.R.: Always! It’s thanks to volunteer users that Metatogger is now available in other languages than French and English. And thanks to the same people, Echosync will soon be too, I’m really thankful to them!
Make Metatogger et Echosync available in their users’ native languages is one of my priorities, their accessibility really matters to me. So, any help is welcome, to reinforce the already existing translation team or offer translation in a language that is not yet available.

Want to help out? Don’s hesitate to get in touch with Sylvain.

Want more? Follow us on Facebook, Twitter and LinkedIn.

Hello there!

In order to improve WebTranslateIt’s performance we will do an important hardware upgrade to WebTranslateIt’s servers.

In order to do this upgrade we will need to be unavailable on Sunday, March 24th 2019, 06:00 UTC+1 as we move our database to a new server. We expect the downtime to not exceed 2 hours. The website will be unavailable from Sunday, March 24th 2019, 06:00 UTC+1 to Sunday, March 24th 2019, 08:00 UTC+1

As always, we’ll keep you updated on this blog post and we’ll also post live updates on @webtranslateit on Twitter.

We apologize for this planned downtime and we hope you will appreciate the performance improvements this will allow.

PS: For the nitty-gritty details, we’re moving WebTranslateIt out of a large, single server to two smaller, newer servers with loads of RAM (we’ll have more than 4 times the amount of RAM that we currently have). This will allow for more background workers, which means faster file imports and file generations and faster suggestions. We’ll still be hosted by Leaseweb in the Netherlands which have been amazing in the past 3 years.