build: Add i18n machinery

This adds an initial version of internationalisation support. It doesn’t
support Transifex (translations have to be committed manually) for now.

See `docs/i18n.md` for a guide to what commands to use and what the
files do.

Signed-off-by: Philip Withnall <[email protected]>

Fixes: #760

Author: pwithnall

.gitignore

@@ -36,3 +36,7 @@ kolibri_explore_plugin/static/collections
 # pytest files
 .pytest_cache/
 pytestdebug.log
+
+# Translations
+kolibri_explore_plugin/locale/**/LC_MESSAGES/*.csv
+kolibri_explore_plugin/locale/en/LC_MESSAGES/profiles/*

docs/i18n.md

@@ -0,0 +1,74 @@
+Internationalisation of kolibri-explore-plugin
+===
+
+Overview
+---
+
+Internationalisation (i18n) of kolibri-explore-plugin works the same way as i18n
+of Kolibri, to the extent that the build rules are named the same. It’s done
+using tooling from kolibri-tools, and an overview of the system is given
+[in the Kolibri documentation](https://kolibri-dev.readthedocs.io/en/develop/i18n.html).
+
+An overview of i18n in Django is [here](https://docs.djangoproject.com/en/1.11/topics/i18n/),
+which covers the backend of kolibri-explore-plugin.
+
+At a high level, translatable strings are extracted from Python, JavaScript and
+Vue files, and are collated into several files:
+ * `kolibri_explore_plugin/locale/en/LC_MESSAGES/django.po`
+ * `kolibri_explore_plugin/locale/en/LC_MESSAGES/kolibri_explore_plugin-messages.csv`
+ * `kolibri_explore_plugin/locale/en/LC_MESSAGES/welcome-screen-messages.csv`
+
+This extraction process is done by running `yarn i18n-extract`.
+
+They are in several files because they are extracted from the Django backend and
+the different node frontend modules separately.
+
+These files are checked into git, even though they are generated mechanically
+from the source code, so that translators or external translation tools can pick
+them up at any time, without having to run kolibri-tools on the code.
+
+Translations are done (currently manually), and the results committed to git as:
+ * `kolibri_explore_plugin/locale/${locale}/LC_MESSAGES/django.po`
+ * `kolibri_explore_plugin/locale/${locale}/LC_MESSAGES/kolibri_explore_plugin.app-messages.json`
+ * `kolibri_explore_plugin/locale/${locale}/LC_MESSAGES/welcome-screen-messages.json`
+
+CSV versions of the JSON files are also available for review.
+
+The translated files for Django now need to be compiled to a machine readable
+form. This is done by `yarn i18n-download-translations`. (The naming is because,
+in Kolibri, this step downloads updated translations from a translation website
+before compiling them --- we do not currently do that in kolibri-explore-plugin.)
+
+It compiles the Django messages to the following file:
+ * `kolibri_explore_plugin/locale/${locale}/LC_MESSAGES/django.mo`
+
+This is loaded by Django to provide backend translations.
+
+It also generates the following two files, which contain information about the
+locale (such as its name and currency and date formats). They are for use with
+the node [`intl`](https://www.npmjs.com/package/intl) and
+[`vue-intl`](https://www.npmjs.com/package/vue-intl) packages:
+ * `kolibri_explore_plugin/assets/src/intl-locale-data.js`
+ * `kolibri_explore_plugin/assets/src/vue-intl-locale-data.js`
+
+The frontend loads the installed source JSON files (such as
+`kolibri_explore_plugin/locale/${locale}/LC_MESSAGES/kolibri_explore_plugin.app-messages.json`)
+at runtime. It does this by looking for them using a well-known name in
+`LOCALE_PATHS`, then the `WebpackBundleHook.frontend_messages()` function
+injects the contents of the JSON files into the frontend JavaScript with a call
+to `kolibriCoreAppGlobal.registerLanguageAssets()` to register them with the
+translation functions.
+
+Configuration
+---
+
+Internationalisation configuration is in two places:
+ * `kolibri_explore_plugin/locale/language_info.json`
+ * `setup.cfg`
+
+The JSON file contains the list of languages which we want kolibri-explore-plugin
+translated to. It’s in a format needed by kolibri-tools documented
+[here](https://kolibri-dev.readthedocs.io/en/develop/i18n.html#adding-a-newly-supported-language).
+
+`setup.cfg` is needed to point kolibri-tools at the right directories for
+extracting and compiling translatable strings.

kolibri_explore_plugin/locale/README.md

@@ -0,0 +1 @@
+../../docs/i18n.md

kolibri_explore_plugin/locale/language_info.json

@@ -0,0 +1,16 @@
+[
+  {
+    "crowdin_code": "en",
+    "intl_code": "en",
+    "language_name": "English",
+    "english_name": "English",
+    "default_font": "NotoSans"
+  },
+  {
+    "crowdin_code": "es-ES",
+    "intl_code": "es-es",
+    "language_name": "Español (España)",
+    "english_name": "Spanish (Spain)",
+    "default_font": "NotoSans"
+  }
+]

package.json

@@ -2,14 +2,18 @@
   "name": "kolibri-explore-plugin",
   "description": "Monorepo with kolibri's Discovery page and custom channel presentations.",
   "private": true,
+  "config": {
+    "kolibri_tools_i18n_modules": "--plugins kolibri_explore_plugin --namespace ek-components --searchPath ./packages/ek-components --namespace template-ui --searchPath ./packages/template-ui --namespace welcome-screen --searchPath ./packages/welcome-screen"
+  },
   "scripts": {
-    "build": "yarn build:version && yarn build:packages && yarn deploy:welcome && yarn build:plugin && yarn build:info",
+    "build": "yarn build:version && yarn build:packages && yarn deploy:welcome && yarn build:plugin && yarn build:info && yarn build:i18n",
     "build:plugin": "./scripts/set_override.py default && kolibri-tools build prod --plugins kolibri_explore_plugin --transpile",
     "build:packages": "yarn build:libs",
     "build:libs": "lerna run build --ignore template-ui",
     "build:apps": "lerna run build --scope template-ui",
     "build:info": "./scripts/build_info.sh > kolibri_explore_plugin/static/build-info.json",
     "build:version": "python -m setuptools_scm",
+    "build:i18n": "yarn run i18n-extract && yarn run i18n-django-compilemessages",
     "build-dist": "./scripts/clean.sh && yarn clean && yarn build && python -m build",
     "deploy:apps": "./scripts/bundle_bundles.py",
     "clean": "kolibri-tools build clean --plugins kolibri_explore_plugin",
@@ -20,7 +24,13 @@
     "bump-version": "./scripts/bump_version.py",
     "deploy:welcome": "cp -rf packages/welcome-screen/dist kolibri_explore_plugin/welcomeScreen && rm -f kolibri_explore_plugin/welcomeScreen/js/*map",
     "release": "./scripts/check_can_relese.sh && twine upload -s dist/*",
-    "test:python": "python -m pytest"
+    "test:python": "python -m pytest",
+    "makemessages": "kolibri-tools i18n-extract-messages $npm_package_config_kolibri_tools_i18n_modules",
+    "i18n-extract-backend": "cd kolibri_explore_plugin && python -m kolibri manage makemessages --skip-update -l en --ignore 'node_modules/*' --ignore 'kolibri_explore_plugin/dist/*'",
+    "i18n-extract-frontend": "yarn run makemessages",
+    "i18n-extract": "yarn run i18n-extract-frontend && yarn run i18n-extract-backend",
+    "i18n-django-compilemessages": "cd kolibri_explore_plugin && PYTHONPATH=\"..:$$PYTHONPATH\" python -m kolibri manage compilemessages --skip-update",
+    "i18n-download-translations": "kolibri-tools i18n-code-gen --output-dir ./kolibri_explore_plugin/assets/src --lang-info ./kolibri_explore_plugin/locale/language_info.json && yarn run i18n-django-compilemessages && yarn exec kolibri-tools i18n-create-message-files -- $npm_package_config_kolibri_tools_i18n_modules"
   },
   "workspaces": [
     "kolibri_explore_plugin/assets",
@@ -44,8 +54,10 @@
   },
   "dependencies": {
     "@babel/preset-env": "^7.22.10",
+    "intl": "^1.2.4",
     "kolibri-design-system": "https://github.com/learningequality/kolibri-design-system#269b294bf562dd7d0e3a616aaace97bf0d3433c3",
-    "node-sass": "file:dependency-stub"
+    "node-sass": "file:dependency-stub",
+    "vue-intl": "3.1.0"
   },
   "browserslist": [
     "extends browserslist-config-kolibri"

setup.cfg

@@ -0,0 +1,8 @@
+# FIXME: This has to live in setup.cfg rather than pyproject.toml for now,
+# because kolibri-tools explicitly loads setup.cfg.
+# See https://github.com/learningequality/kolibri/issues/11146
+[kolibri:i18n]
+project = kolibri_explore_plugin
+locale_data_folder = kolibri_explore_plugin/locale
+# Glob to exclude node_modules and static folders
+ignore = **/node_modules/**,**/static/**

yarn.lock

@@ -10046,6 +10046,35 @@ interpret@^3.1.1:
   resolved "https://registry.yarnpkg.com/interpret/-/interpret-3.1.1.tgz#5be0ceed67ca79c6c4bc5cf0d7ee843dcea110c4"
   integrity sha512-6xwYfHbajpoF0xLW+iwLkhwgvLoZDfjYfoFNu8ftMoXINzwuymNLd9u/KmwtdT2GbR+/Cz66otEGEVVUHX9QLQ==
 
+intl-format-cache@^2.0.5:
+  version "2.2.9"
+  resolved "https://registry.yarnpkg.com/intl-format-cache/-/intl-format-cache-2.2.9.tgz#fb560de20c549cda20b569cf1ffb6dc62b5b93b4"
+  integrity sha512-Zv/u8wRpekckv0cLkwpVdABYST4hZNTDaX7reFetrYTJwxExR2VyTqQm+l0WmL0Qo8Mjb9Tf33qnfj0T7pjxdQ==
+
+[email protected]:
+  version "1.2.0"
+  resolved "https://registry.yarnpkg.com/intl-messageformat-parser/-/intl-messageformat-parser-1.2.0.tgz#5906b7f953ab7470e0dc8549097b648b991892ff"
+  integrity sha512-2qnIxfkrrzKAXR9m+rUqWb2yzeRX78k1l7+MU4ji7i4TIUyAIOo3KGV4hPKElRNyD40AAoAQdw1XXs5umB/usA==
+
+[email protected], intl-messageformat@^1.3.0:
+  version "1.3.0"
+  resolved "https://registry.yarnpkg.com/intl-messageformat/-/intl-messageformat-1.3.0.tgz#f7d926aded7a3ab19b2dc601efd54e99a4bd4eae"
+  integrity sha512-dtj6VJ0Ul/DFyAksbEtOzsslkfuEu7r/d/oi8XltZgDLH3aUzxNkcb6SbguhHTi1RyPGtdWsfx6Q7qJGUySpow==
+  dependencies:
+    intl-messageformat-parser "1.2.0"
+
+intl-relativeformat@^1.3.0:
+  version "1.3.0"
+  resolved "https://registry.yarnpkg.com/intl-relativeformat/-/intl-relativeformat-1.3.0.tgz#893dc7076fccd380cf091a2300c380fa57ace45b"
+  integrity sha512-pDbaJb3lh9IFyeY/bLZucvwcQgiUY6iqJ+ltMLi7FxMxIwzjHZrlSnvQbH3D0gQj1e1N9vpngh1kXMdfEbhPpA==
+  dependencies:
+    intl-messageformat "1.3.0"
+
+intl@^1.2.4:
+  version "1.2.5"
+  resolved "https://registry.yarnpkg.com/intl/-/intl-1.2.5.tgz#82244a2190c4e419f8371f5aa34daa3420e2abde"
+  integrity sha512-rK0KcPHeBFBcqsErKSpvZnrOmWOj+EmDkyJ57e90YWaQNqbcivcqmKDlHEeNprDWOsKzPsh1BfSpPQdDvclHVw==
+
 [email protected], invariant@^2.2.2:
   version "2.2.4"
   resolved "https://registry.yarnpkg.com/invariant/-/invariant-2.2.4.tgz#610f3c92c9359ce1db616e538008d23ff35158e6"
@@ -17095,6 +17124,16 @@ vue-hot-reload-api@^2.3.0:
   resolved "https://registry.yarnpkg.com/vue-hot-reload-api/-/vue-hot-reload-api-2.3.4.tgz#532955cc1eb208a3d990b3a9f9a70574657e08f2"
   integrity sha512-BXq3jwIagosjgNVae6tkHzzIk6a8MHFtzAdwhnV5VlvPTFxDCvIttgSiHWjdGoTJvXtmRu5HacExfdarRcFhog==
 
+[email protected]:
+  version "3.1.0"
+  resolved "https://registry.yarnpkg.com/vue-intl/-/vue-intl-3.1.0.tgz#707f1f7406595c9b4afc6049254b333093be37be"
+  integrity sha512-0v3S5gspuYnt6j1G+KLfPUsNnjRdbMOcYrWYoSd1gYk6rX8VuOyh7NLztPrSIJt+NLs/qzLOZXxI1LORukEiqA==
+  dependencies:
+    babel-runtime "^6.26.0"
+    intl-format-cache "^2.0.5"
+    intl-messageformat "^1.3.0"
+    intl-relativeformat "^1.3.0"
+
 vue-jest@^3.0.0:
   version "3.0.7"
   resolved "https://registry.yarnpkg.com/vue-jest/-/vue-jest-3.0.7.tgz#a6d29758a5cb4d750f5d1242212be39be4296a33"