localized components containing i18n tags will produce identical chunk hashes when building with --localize

[nas90]

🐞 Bug report

Command (mark with an x)

• new
• build
• serve
• test
• e2e
• generate
• add
• update
• lint
• xi18n
• run
• config
• help
• version
• doc

Description

I have an application that is being translated in several (21 at the moment) languages using the Angular i18n mechanism. I'm compiling the app AOT, therefore I need a separate build for each locale. To achieve this I setup my Azure pipeline to build the app using multiple agents in parallel, one for each locale, and it works like a charm. However, this is very resource-consuming, so I'm updating to Angular 9 in order to use the --localize option, which perfectly fits my needs.
After the update I can build the app once and translate it to all languages in a few seconds, which is amazing.
I am now facing this issue: when I build my application for all locales, the resulting chunk hashes are identical. As a result, browser cache is not invalidated and files are not re-requested, so the app is not translated.

🔬 Minimal Reproduction

• clone this repo
• run npm install
• run ng build --localize
• check the dist folder: file content for main.js is different for each locale (template text is translated), while the hashes are the same

🌍 Your Environment

Angular CLI: 9.1.0
Node: 10.19.0
OS: win32 x64

Angular: 9.1.0
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.901.0
@angular-devkit/build-angular     0.901.0
@angular-devkit/build-optimizer   0.901.0
@angular-devkit/build-webpack     0.901.0
@angular-devkit/core              9.1.0
@angular-devkit/schematics        9.1.0
@ngtools/webpack                  9.1.0
@schematics/angular               9.1.0
@schematics/update                0.901.0
rxjs                              6.5.5
typescript                        3.8.3
webpack                           4.42.0

Anything else relevant?
In my scenario, the user language preference is stored in the cookies, and a shell app serves the corresponding folder. I can't add the locale to the URL.
Therefore in my angular.json I set baseHref="/" for all locales.

I am aware of this issue and the PR mentioned there, however the issue is still not resolved.

[alan-agius4]

Duplicate of #16526

[leomendoza123]

Seems like it was supposed to be fixed with #16817 but I am having the same issue with@angular/[email protected]

[pjm0616]

I'm also having this issue.

My repro: https://github.com/pjm0616/pastebin/tree/202004-angular-localize-outputhash
@angular and @angular-devkit version: 9.1.3
I've also included the build output in the repo.
The dist-localize folder contains the build output with localize=true, and the dist folder was built separately for each locale without localize=true.

$ npm run buildall
$ sha1sum dist{,-localize}/*/main-es5*.js
6e0407f00e27d877cb8bbcf0d7b069dfcc42cb8b  dist/en/main-es5.27d2d00de67fb6cace7d.js
bc09ba481672f092a359949c6d55b14e319fc63a  dist/ko/main-es5.54a148db9b0b5fb29549.js
6e0407f00e27d877cb8bbcf0d7b069dfcc42cb8b  dist-localize/en/main-es5.54a148db9b0b5fb29549.js
bc09ba481672f092a359949c6d55b14e319fc63a  dist-localize/ko/main-es5.54a148db9b0b5fb29549.js

[bridzius]

#16526 was never addressed and was closed without resolution. #16817 fixes a separate issue, where the hashes would remain the same when rebuilding after changing translations and not changing code.
This exact case of the file hashes being the same for different locales is still present. It was considered a regression in #16526, but has since been reclassified as a feature.
Quite an annoying regression, that prevents having flat file structures.

[H--o-l]

I agree with @bridzius that #16817 fix only a symptom of the more deep issue of #16526.

I also have the same hash for multiple main.hash.js files that don't have the same content due to Angular i18n post compilation change.

(Side note, I think that, even if I'm not 100% sure, it mess with Angular service worker itself when I switch between translations of my app).

@clydin in #16526 (comment) you say I quote:

Removing the use of webpack’s file hashing and moving the file name creation to a later stage of the build process is currently planned. However, such a change is out of scope within the current release candidate period.

It seems like the right things to do to fix this issue.
But you didn't post any link. Can you tell where we could upvote for this solution, or even help?

(@nas90 thanks for (re-)opening this issue!)

[nas90]

@alan-agius4 any updates on this one?

[dmitry-urenev]

Recently migrated to Angular 9 and faced this issue. Was really disappointing to see this. Even the different configurations for different languages (for example --configuration=production,de and --configuration=production,fr, plus properly set angular.json as Documentation advises) lead to the same hashes. This means that there is only a one way to get the flat files deployment working – to use the deprecated build parameters --i18nLocale, --i18nFile and --i18nFormat 😞

[nas90]

Just tried on latest Angular (10.0.5) and I'm still facing this issue 😢

[alan-agius4]

This issue is still open hence it hasn’t been addressed yet.

While this is on our radar, there are issues which have a higher priority.

PRs to implement this feature are welcome.

[tujlaky]

I am having the same issue with angular 11.1.0.

[azerafati]

I confirm this is still an issue in v13, the workaround to build the different locales separately can become faster using run-p of npm-run-all.

[bartcorremans]

As a different workaround that doesn't involve running a separate build for each language, I wrote a shell script that adds language codes to the file hashes and their occurrences after the build finishes. This ensures uniqueness between different languages.

There is probably a more efficient way of doing this and I haven't tested it on different setups, but here it is in case it helps anyone:

Edit May 6, 2022: Improve performance and make compatible with macOS

locale-utils/fix-hashes.sh

#!/bin/bash -eu

# Appends language code to file hashes for js files
# (and references to those files) in dist/
# Motivation: https://github.com/angular/angular-cli/issues/17416

shopt -s extglob nullglob

# https://stackoverflow.com/questions/19242275/re-error-illegal-byte-sequence-on-mac-os-x
LC_CTYPE=C

# https://stackoverflow.com/a/40777793
suffixArg=()
sed --version 2>/dev/null | grep -q GNU || suffixArg=( '' )

cd ./dist/

LOCALES=($(ls))

cd "./${LOCALES[0]}"

# Find js files | extract hash | filter duplicates
# Hashes are the same for every language (this is the problem we are trying to solve)
# So we only need to do this once.
HASHES=($(find . -maxdepth 1 -name '*.js' | sed -E 's|.*\.([a-z0-9]*)\.js|\1|' | sort -u))

cd ..

for LOCALE in ${LOCALES[@]}
do
    echo "Updating file hashes for $LOCALE"
    cd "./${LOCALE}"

    # Replace occurrences of each hash.
    for HASH in ${HASHES[@]}
    do
        HASH_SUFFIXED=${HASH}_${LOCALE}
        # echo "Replacing occurrences of hash: $HASH with $HASH_SUFFIXED"
        sed -i "${suffixArg[@]}" -E "s|$HASH|$HASH_SUFFIXED|g" *.@(js|html)
    done

    # Rename files to update their hashes too.
    for FILE in *.js*
    do
        # echo "Renaming $FILE"
        mv "$FILE" "${FILE/.js/_${LOCALE}.js}"
    done

    cd ..
done

I added it as a postbuild script in package.json so it runs automatically after each build:

        "postbuild": "./locale-utils/fix-hashes.sh",

[azerafati]

I wrote a shell script that adds language codes to the file hashes and their occurrences after the build finishes. This ensures uniqueness between different languages.

I want to point out that this won't fix the problem of source maps not working properly
#16526 (comment)

[matrium0]

Have the same problem with Angular 13 and didn't find a workaround. No matter what I do I always end up with different main-bundles that have the same hash..

[kertof]

Is there is an update on this issue?

Having the same hash for different file contents is counter-intuituve, especially as this breaks cache busting, and make it impossible to do a location.reload() when the language is set by a cookie, which is the case for Firebase.

This issue makes it impossible to deploy Angular localized apps on Firebase

[bjorkegeek]

The way I solved this was to use a post build step to change the file names and the references. I serve with nginx and use cookies to select language. I'm considering a write-up of my solution if there is some interest.

[kertof]

That would be great.
I still don't quite understand though why this is not considered a bug: when building an angular app with multiple locales, the command outputs a main.js file for each locale, containing locale-specific data, so files with different content, but all these files have the same hash.

Could the locale be appended to the filename, as a quick fix, basically producing main.en.[hash].js, main.fr.[hash].js,...

[bartcorremans]

I have updated the workaround script in my previous comment to improve performance and provide support for macOS.

[dgp1130]

Looking over the comments quickly, it sounds like most of the motivation here comes down to wanting to put multiple locale outputs into a single flat directory, is that the ask here? Why exactly do folks want to do this, does it make some server configuration or URL pathing easier? I think a better understanding of the motivation here would help us identify the right solution.

[ryansheehan]

Looking over the comments quickly, it sounds like most of the motivation here comes down to wanting to put multiple locale outputs into a single flat directory, is that the ask here? Why exactly do folks want to do this, does it make some server configuration or URL pathing easier? I think a better understanding of the motivation here would help us identify the right solution.

For our projects we do path based routing in our ingress, so we opted to do localization with other methods (ie cookies). Without different hashing on the localized files, the browser cache will return the same locale even though the user switched locales, and a different index file is being served.

We dont need to flatten anything. Our server-side picks up the cookie and serves the right files.

[dgp1130]

For our projects we do path based routing in our ingress, so we opted to do localization with other methods (ie cookies). Without different hashing on the localized files, the browser cache will return the same locale even though the user switched locales, and a different index file is being served.

We dont need to flatten anything. Our server-side picks up the cookie and serves the right files.

That sounds like the Cache-Control header is wrong then? If the response is contingent on a cookie which the browser doesn't key its cache by, then I think it's not safe to let the browser cache that URL.

I can see that marking those URLs as uncacheable could reduce performance for users who don't change languages, but I think that's a consequence of not putting the locale in the URL. Using a different hash is just another form of making the URL locale-dependent so the locale is taken into account in the browser cache key. Whether that's done via a locale-dependent hash, or an explicit locale path / query parameter is an implementation detail to the browser cache.

[matrium0]

I don't think this is related to caching at all. You can reproduce it like explained above with just a regular "ng build --translate" - you end with 2 folders, one for each locale - so far so good!

But if you deep-dive it you notice that your main.js bundles (you have one of those for each language) is messed up. You end up with different bundles because there are differences due to language. But the HASH of all these main.ts bundles is the same.

Now depending on your solution you might end up with rewrites that (for the users browser) appear to be from the same path - that would be no problem, but since your ENGLISH-main.ts and your GERMAN-main.ts have the same hash you end up with issues when you switch the language.

Given, there may be ways to host it differently - but still: how could that NOT be a bug? The Hash of my german main.ts file is obviously based on the content of my english main.ts file. That's just wrong, it should be based on the content of my german main.ts, right?

[dgp1130]

Given, there may be ways to host it differently - but still: how could that NOT be a bug? The Hash of my german main.ts file is obviously based on the content of my english main.ts file. That's just wrong, it should be based on the content of my german main.ts, right?

As I understand it, I believe we use Webpack's [hash] replacement here, not [contenthash]. The hash value is not based on file content, it's only guaranteed to be unique within a build. This SO answer seems to have a good breakdown of the differences. @clydin is that understanding correct?

Given that generated resources shouldn't be loaded into the same page (you shouldn't be loading a French chunk into a German application), then I don't think it's necessarily wrong to see duplicate hashes across different locales.

[matrium0]

Thanks for explaining and the SO link! I think I understand, but it still feels wrong and there might be some bugs with that implementation - if I build my project while only changing the german translation file the hash would still be unchanged, right? That would lead to people not getting the newest version of my german main.ts file! So basically only changing translations without touching a source file is not supported with the current build?

The question is: is there a good reason NOT to use [contenthash] or maybe add an optional flag for that? This puts a strong opinion/restriction on how files are served. For me it also seems a bit also unintuitive (because of what I believe a file-hash should do)

Example

will try to explain my approach while hosting my mini-homepage https://budisoft.at that lead to this problem:

• on my server there is a folder de_DE and another folder en_US, containing the two localized applications
• I set some rules on my reverse proxy (based on "Accept-Language" and a Custom Cookie) to rewrite all URLs to /de or /en_US accordingly
• the basehref is just set to "/" - my rules ensure that the correct bundles are loaded

This was a naive - but I think reasonable - approach and would work perfectly with [contenthash], but fails with [hash] (wrong language bundle get's loaded from cache)

Solution

Thanks to your response I think I finally understand what I'd have to do instead of dirty workaround:

• rewrite requests just for the index.html.
• keep the baseHrefs like default, e.g. "en_US" and "de"
• now my english index.html would request a different file location (e.g. https://budisoft/**en_US**/main.03b15c0004fb5dfc.js) than my german version, even when the hash is the same.

I will implement it that way, but still hope that there can be a solution that enables some flexibility while serving the bundles sometime.

[bartcorremans]

Looking over the comments quickly, it sounds like most of the motivation here comes down to wanting to put multiple locale outputs into a single flat directory, is that the ask here? Why exactly do folks want to do this, does it make some server configuration or URL pathing easier? I think a better understanding of the motivation here would help us identify the right solution.

Thanks for following up! I'll try to elaborate the use case in our application, please let me know if my explanation is not clear.

Our application is typically gated by a user login. To determine the language of the application, it will look at a user's language preferences, saved in their user account. This ensures that they are served the correct language wherever they log in. If the user is not yet logged in, or the user's language is not yet set, there is logic to determine the most preferable fallback language based on the environment in which the application is deployed, as well as the user's Accept-Language list.

Once the correct language is served, we store it in the browser's cookies, ensuring that these checks and redirects do not happen again until a different user - with a different preferred language - signs in.

Because of this process, we do not use the browser URL to determine which language should be served.

This does not mean that we can't have the language in the URL - we could determine the user's preferred language and then redirect to the correct URL once we know, this is true.

However, there are other features of the application that complicate this:

• Users can bookmark direct links within the application - or, more relevant in this case, administrators might provide their users with these bookmarks. These links should not contain language info in the URL, because they could cause the application to be served in the wrong language. We would require additional steps to detect that the user's preferred locale does not match the URL, and redirect accordingly. Unlike our current process, this is something that would have to be done every time the user accesses the bookmark.
• Users can also share file links with one another, even with anonymous users. Similar to bookmarks, having a language code in the URL could cause the file to be served in the wrong language. Admittedly, this could probably be resolved by stripping languages codes from share links.

While none of these individual problems might be insurmountable, it makes much more sense for our use case to simply not use the browser URL for the language. It is there that we run into this issue - switching languages is unreliable when the file hashes for each language are the same, when these files are served with the same URL. We currently use the workaround I mentioned in an earlier comment of adding language codes to file hashes, but this is somewhat slow and likely to break in the future.

[matrium0]

Hey Bart, I think you are a bit in the same "mental trap" as I was. Yes, URLs to a specific content-page should not contain a language url, to ensure deep-linkability across languages.
Example: https://your-page/order/123456 <= this is a link to one specific order and should work, regardless of language.

But this is only important for your own pages/routes! This doesn't matter for JavaScript bundles though. Let me explain:

If you give the link above to a german user your reverse-proxy will probably use your cookie (+ fallback) to determine the correct index.html and retrieve it from a file system like this:
/opt/data/your-page/de/index.html for german OR
/opt/data/your-page/en-US/index.html for english

In each index.html you have <script> tags, linking to your bundles. And these links could absolutely contain the language, because once the user has the correct index.html does it really matter how he loads the bundle?
https://your-page/main.03b15c0004fb5dfc.js VS
https://your-page/en_US/main.03b15c0004fb5dfc.js

Sadly it seems as of now you NEED to use the second variant though. You end up with two main.03b15c0004fb5dfc.js files (same hash, but different content), but since you load them from different URLs you don't have caching issues.

Hope this is helpful

[thduttonuk]

This issue causes problems when we upload source maps to Sentry. We have to create a release for each Language in Sentry. This then breaks resolved in the next release working correctly.

[matrium0]

Hey, I just rewoved the script-workaround (thank you for that @bartcorremans !) from my homepage build https://budisoft.at and now everything is working fine. What I did was completely removing "baseHref": "/" in my angular.json - so the baseHrefs stay at default. I end up with the following after build:

• german index.xml contains base href="/de/"
• english index.html contains base href="/en_US/"

I changed my rewrite-rules to only rewrite for my index.html - all other urls/files can be loaded as-is

Now my bundles (e.g. main.1234567.js) are loaded from different URLs and it doesn't matter any longer that they have the same filename.

One small issue I had with that, was that routing did include the base-href at first. (e.g. routing to /articles would bring me to https://budisoft.at/en_US/articles). After consulting the documentation I added a provider in my app-module to bring the context back to /:
{provide: APP_BASE_HREF, useValue: ''}

With that everything seems to work as expected and I am happy

[dgp1130]

Thanks for explaining and the SO link! I think I understand, but it still feels wrong and there might be some bugs with that implementation - if I build my project while only changing the german translation file the hash would still be unchanged, right? That would lead to people not getting the newest version of my german main.ts file! So basically only changing translations without touching a source file is not supported with the current build?

The hash probably should be dependent on the translation files, as I think it's reasonable to expect these hashes to update on any text content changes. If that's not the case, then that sounds like a separate issue we can look at.

Since all the hashes are currently tied together across locales, that would mean that your French build is actually dependent on all the other translations, which is a bit weird and bad for build caching. It would make more sense that the French build only depends on French translations. @alan-agius4 would there be any build improvements from better isolating languages this way? In theory I think it would be better, but in practice everything is done via a Webpack compilation which probably preprocesses everything for this hash, so I'm not sure this would actually have a meaningful effect?

Thanks to your response I think I finally understand what I'd have to do instead of dirty workaround:

• rewrite requests just for the index.html.
• keep the baseHrefs like default, e.g. "en_US" and "de"
• now my english index.html would request a different file location (e.g. https://budisoft/**en_US**/main.03b15c0004fb5dfc.js) than my german version, even when the hash is the same.

Yes, I think that is the right approach. If you want "language-agnostic" URLs, those should be limited to the URLs actually visible to users (such as HTML pages). JS bundles aren't typically user-visible or stable (hashes change on each deployment), so putting them in a locale-specific URL doesn't negatively impact the user.

This feels like it also addresses @bartcorremans' issue.

This issue causes problems when we upload source maps to Sentry. We have to create a release for each Language in Sentry. This then breaks resolved in the next release working correctly.

I'm not very familiar with Sentry, but it sounds like they're keying sourcemaps by file name? If so, that makes it a subset of the flattening locale resources use case. Given that Sentry has direct Angular support I wonder if that might be better addressed on their end? Seems to me like a sourcemapping utility should be able to handle multiple files with the same name in different directories, but I'm not an expert in that area and I'm sure there's more nuance to it than that.

The question is: is there a good reason NOT to use [contenthash] or maybe add an optional flag for that? This puts a strong opinion/restriction on how files are served. For me it also seems a bit also unintuitive (because of what I believe a file-hash should do)

I agree that this can be unintuitive. Devs have a strong bias to interpret a hash in a file name as a hash of that file's contents. That's not always true (as in this case), but it is often the initial expectation. One other advantage to consider about a [contenthash]-based approach is that files can be cached across deployments for any chunk that didn't change. This can result in some performance improvements, but it's hard to measure the exact impact of that.

I think the main reason not to change the current behavior is that Webpack chunking happens before localization as I understand it. So just changing to [contenthash] doesn't solve the problem because subsequent edits to these files to merge translations would alter the file and change its hash. The Angular CLI then needs to manually update all the files (and their chunk dependencies on each other) to use the new hashes. I think this also doesn't solve the problem of file conflicts, since a JS file without translations will be duplicated across locales, though in that case it is at least the same for each locale. That basically means we'd be implementing our own hashing system on top of Webpack's.

[bartcorremans]

Hey Bart, I think you are a bit in the same "mental trap" as I was. Yes, URLs to a specific content-page should not contain a language url, to ensure deep-linkability across languages. Example: https://your-page/order/123456 <= this is a link to one specific order and should work, regardless of language.

But this is only important for your own pages/routes! This doesn't matter for JavaScript bundles though. Let me explain:

If you give the link above to a german user your reverse-proxy will probably use your cookie (+ fallback) to determine the correct index.html and retrieve it from a file system like this: /opt/data/your-page/de/index.html for german OR /opt/data/your-page/en-US/index.html for english

In each index.html you have <script> tags, linking to your bundles. And these links could absolutely contain the language, because once the user has the correct index.html does it really matter how he loads the bundle? https://your-page/main.03b15c0004fb5dfc.js VS https://your-page/en_US/main.03b15c0004fb5dfc.js

Sadly it seems as of now you NEED to use the second variant though. You end up with two main.03b15c0004fb5dfc.js files (same hash, but different content), but since you load them from different URLs you don't have caching issues.

Hope this is helpful

Hi matrium0,

Thank you for the thoughtful reply! After some tinkering I was able to get our setup to work this way. The main struggle came from angular-router redirecting us to the baseHref, as you mentioned in your later comment - which I wrongly thought was being caused by our server configuration. After figuring that out it was pretty straightforward!

Thank you again! This sees our use case covered without the need for unique file hashes per locale.

[dgp1130]

Following up with this conversation, really appreciate everyone's feedback here. It looks like we've mostly figured out how to meet these needs with the CLI's current behavior. I think the recommendation for anyone finding this issue in the future:

1. Hashes are not keyed by the locale. That means if you want to cache the same file across different locales, that locale must appear in the URL.
2. This only applies to JS files generated by the CLI, not user-visible links like HTML files. If you don't want user-visible links to have locales (like /my_home_page.html), you can automatically infer the user locale via cookies or backend state, and then link to /${locale}/main.abc123.js to load your application. That way user-facing links are locale-independent, but still load localized and cacheable JS sources.

I agree this behavior can be confusing, and in a perfect world asset hashes would probably be locale dependent. However, there is a lot of complexity and nuance to make that work and maintain it over time. Given that this can be worked around, I don't think changing this is the best use of our limited time and energy. So I'll close this issue for now, and we can reevaluate if it becomes a bigger challenge in the future.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}