Add machine translation service#6179
Add machine translation service#6179mrcasals merged 51 commits intodecidim:feat/machine-translationfrom
Conversation
|
@anaghavl @mrcasals Awesome work ! Does this feature means we will have a saved in the db a translated version for each contribution ? Does the system targets a destination language everytime or does it translate to all the languages activated on the org ? Thanks in advance for your reply looking forward to testing this ! |
|
@virgile-dev the idea is that, when a resource is created or updated, we automatically machine-translate the specified fields. These machine translations will be persisted in the DB, and we will translate them using the organization available locales 😄 There's a missing piece that is the actual integration with a translation service. We'll write an example dummy translator, but real integrations will most probably live outside the main Decidim repo (eg. Google Translate, DeepL, Bing...) |
|
Nice job, @anaghavl!! I don't understand why that test is failing, though 😕 |
|
@decidim/core can you review that? Check #6127 for more context |
tramuntanal
left a comment
There was a problem hiding this comment.
Good job @anaghavl , I see the code is clear and well tested 😄
I just have left some questions, I know this goes into another PR but would like to clarify some decisions you are making here, thanks!
decidim-core/app/jobs/decidim/machine_translation_create_fields_job.rb
Outdated
Show resolved
Hide resolved
decidim-core/app/jobs/decidim/machine_translation_create_resource_job.rb
Outdated
Show resolved
Hide resolved
decidim-core/app/jobs/decidim/machine_translation_create_fields_job.rb
Outdated
Show resolved
Hide resolved
decidim-core/app/jobs/decidim/machine_translation_update_fields_job.rb
Outdated
Show resolved
Hide resolved
decidim-core/app/jobs/decidim/machine_translation_create_fields_job.rb
Outdated
Show resolved
Hide resolved
|
@anaghavl so this PR is still in WIP? |
Hi @tramuntanal , |
|
Hi @anaghavl , |
The old change made us change the |
4ae9f10 to
5cbf1c3
Compare
|
@anaghavl you have conflicting files, can you check please 😃 |
b01cf57 to
de56e04
Compare
Sorry again, resolved the conflicts 😄 |
5cbf1c3 to
6cd43aa
Compare
* Require confirmation on exit * Add specs * Use path instead of url * Fix changelog * Trigger build * Fix expected path on test * Fix method call * Take textareas and selects into account
aa55ca8 to
dc98640
Compare
decidim-core/app/jobs/decidim/machine_translation_fields_job.rb
Outdated
Show resolved
Hide resolved
|
@decidim/core tests are green now! Can you review it, please? 😄 |
|
@decidim/core sorry again, can you review this PR please? |
Co-authored-by: Marc Riera Casals <mrc2407@gmail.com> Co-authored-by: Marc Riera <mrc2407@gmail.com>
Co-authored-by: Marc Riera Casals <mrc2407@gmail.com> Co-authored-by: Marc Riera <mrc2407@gmail.com>
* Base branch * remove file * Base branch * remove file * Identify translatable resources (#6145) * Base branch * remove file * Require confirmation on exiting a survey mid-answering (#6118) * Require confirmation on exit * Add specs * Use path instead of url * Fix changelog * Trigger build * Fix expected path on test * Fix method call * Take textareas and selects into account * WIP adding concern * Adding concern in all the models which have translatable fields * removed :extended_data as translatable field * WIP adding concern * Adding concern in all the models which have translatable fields * Revert "Require confirmation on exiting a survey mid-answering (#6118)" This reverts commit bdeb933. * Revert "remove file" This reverts commit 2565dbb. * Revert "Base branch" This reverts commit 2a09cc4. Co-authored-by: Marc Riera Casals <mrc2407@gmail.com> * Add Decidim global and organization config for machine translation (#6128) * Adding setting to organizations table and creating global config * Adding config accessor to core.rb * Base branch * Added a check to display machine translation settings and changed initializer value * Fixing lint issue in migration file * Adding test and removing test file Co-authored-by: decidim-bot <decidim-bot@users.noreply.github.com> Co-authored-by: anagha <anagha1996@gmail.com> * Identifying translatable fields in meetings and comments (#6333) * Base branch * remove file * Idenifying translatable fields in meetings and comments Co-authored-by: Marc Riera Casals <mrc2407@gmail.com> * Identifying translatable fields for proposals (#6346) * Add machine translation service (#6179) Co-authored-by: Marc Riera Casals <mrc2407@gmail.com> Co-authored-by: Marc Riera <mrc2407@gmail.com> * Make some fields non-translatable * Improve spec * Don't run job if class is not defined * Improvee method naming * Machine translation display priority (#6385) * Add docs on how to enable the integration * Add docs on how to write a machine translation service * Improve code strength * Fix specs * Fix specs Co-authored-by: anagha vl <44900292+anaghavl@users.noreply.github.com> Co-authored-by: decidim-bot <decidim-bot@users.noreply.github.com> Co-authored-by: anagha <anagha1996@gmail.com>
Related to #6127.
New Approach with machine translations, we add the translations to the same object. This way we don't need an extra table to store the translations.
ex.
title = { :en => "title", :machine_translations{ :ca => "ca-title", :en => "en-title"}}Why?
The old change made us change the translatable_attribute method. And so it forced us to change a lot of the existing code to make it compatible and some points can't use that (eg component settings).
The new approach allows us to keep human and machine translations identifiable, and lets us use the previous translatable_attribute helper as it is and only change the internals of the method.
Changes in the PR:
MachineTranslationResourceJob
That being said, there are a few other cases that are taken care of:
Let's assume the default locale is
enCase 1:
ie. In case machine_translations exists for a particular locale,
ex.
title = { :en => "title", :machine_translations{ :ca => "ca-title", :en => "en-title"}}and we update the title with
title = { :en => "title", :ca => "ca- title" }Then the duplicate
catranslation is removed frommachine_translations.title = { :en => "title", :machine_translations{:en => "en-title"}}Case 2:
We identify the locale that has been set to "" and trigger a job to get the machine translation for that.
You can see how it looks in each case,

MachineTranslationFieldsJob
Decidim.machine_translation_servicewhich in this case is the DummyTranslator.DummyTranslator: We have a dummy translator that updates the field with
machine_translations => { translation_locale =>"#{translation_locale} - #{field_value}"}This is just to test the flow of machine translation and hence the storing operations are done inside it.
Jobs can be used to achieve the same if a different translation service is integrated.
Finally, incase there isn't human translation available, we look for machine translations in the
translated_attributeto use it in the views.