Skip to content

Commit f36331c

Browse files
srl295MylesBorins
srl295
authored and
MylesBorins
committed
doc,deps: document how to maintain ICU in Node.js
- update v8 guide to mention ICU - move content from the tools/icu/README.md but leave a pointer Fixes: #26108 Co-Authored-By: Vse Mozhet Byt <[email protected]> PR-URL: #30607 Reviewed-By: Ujjwal Sharma <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Myles Borins <[email protected]>
1 parent 7f94fe0 commit f36331c

File tree

3 files changed

+255
-115
lines changed

3 files changed

+255
-115
lines changed

‎doc/guides/maintaining-V8.md

+4
Original file line numberDiff line numberDiff line change
@@ -313,6 +313,10 @@ Node.js keeps a vendored copy of V8 inside of the deps/ directory. In addition,
313313
Node.js may need to float patches that do not exist upstream. This means that
314314
some care may need to be taken to update the vendored copy of V8.
315315

316+
V8 builds against the version of ICU supplied by Node.js,
317+
see [maintaining-icu.md](./maintaining-icu.md) for special considerations.
318+
Specifically, a V8 update may necessitate an ICU update.
319+
316320
### Minor updates (patch level)
317321

318322
Because there may be floating patches on the version of V8 in Node.js, it is

‎doc/guides/maintaining-icu.md

+233
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
# Maintaining ICU in Node.js
2+
3+
## Background
4+
5+
International Components for Unicode ([ICU4C][ICU]) is used both by V8
6+
and also by Node.js directly to provide internationalization
7+
functionality. To quote from icu-project.org:
8+
9+
> ICU is a mature, widely used set of C/C++ and Java libraries providing
10+
> Unicode and Globalization support for software applications. ICU is
11+
> widely portable and gives applications the same results on all platforms
12+
> and between C/C++ and Java software.
13+
14+
## Data dependencies
15+
16+
ICU consumes and includes:
17+
18+
* Extracted locale data from [CLDR][]
19+
* Extracted [Unicode][] data.
20+
* Time zone ([tz][]) data
21+
22+
The current versions of these items can be viewed for node with `node -p process.versions`:
23+
24+
```shell
25+
$ node -p process.versions
26+
27+
{
28+
…
29+
cldr: '35.1',
30+
icu: '64.2',
31+
tz: '2019a',
32+
unicode: '12.1'
33+
}
34+
```
35+
36+
## Release Schedule
37+
38+
ICU typically has >1 release a year, particularly coinciding with a major
39+
release of [Unicode][]. The current release schedule is available on the [ICU][]
40+
website on the left sidebar.
41+
42+
### V8 depends on ICU
43+
44+
V8 will aggressively upgrade to a new ICU version, due to requirements for
45+
features/bugfixes needed for [Ecma402][] support. The minimum required version
46+
of ICU is specified within the V8 source tree. If the ICU version is too old,
47+
V8 will not compile.
48+
49+
```c
50+
// deps/v8/src/objects/intl-objects.h
51+
#define V8_MINIMUM_ICU_VERSION 64
52+
```
53+
54+
V8 in Node.js depends on the ICU version supplied by Node.js.
55+
56+
The file `tools/icu/icu_versions.json` contains the current minimum
57+
version of ICU that Node.js is known to work with. This should be
58+
_at least_ the same version as V8, so that users will find out
59+
earlier that their ICU is too old. A test case validates this when
60+
Node.js is built.
61+
62+
## How to upgrade ICU
63+
64+
* Make sure your Node.js workspace is clean (`git status`
65+
should be sufficient).
66+
* Configure Node.js with the specific [ICU version](http://icu-project.org/download)
67+
you want to upgrade to, for example:
68+
69+
```shell
70+
./configure \
71+
--with-intl=full-icu \
72+
--with-icu-source=http://download.icu-project.org/files/icu4c/58.1/icu4c-58_1-src.tgz
73+
make
74+
```
75+
76+
> _Note_ in theory, the equivalent `vcbuild.bat` commands should work also,
77+
> but the commands below are makefile-centric.
78+
79+
* If there are ICU version-specific changes needed, you may need to make changes
80+
in `tools/icu/icu-generic.gyp` or add patch files to `tools/icu/patches`.
81+
* Specifically, look for the lists in `sources!` in the `tools/icu/icu-generic.gyp` for
82+
files to exclude.
83+
84+
* Verify the Node.js build works:
85+
86+
```shell
87+
make test-ci
88+
```
89+
90+
Also running
91+
92+
<!-- eslint-disable strict -->
93+
94+
```js
95+
new Intl.DateTimeFormat('es', { month: 'long' }).format(new Date(9E8));
96+
```
97+
98+
…Should return `enero` not `January`.
99+
100+
* Now, copy `deps/icu` over to `deps/icu-small`
101+
102+
> :warning: Do not modify any source code in `deps/icu-small` !
103+
> See section below about floating patches to ICU.
104+
105+
```shell
106+
python tools/icu/shrink-icu-src.py
107+
```
108+
109+
* Now, do a clean rebuild of Node.js to test:
110+
111+
```shell
112+
make -k distclean
113+
./configure
114+
make
115+
```
116+
117+
* Test this newly default-generated Node.js
118+
119+
<!-- eslint-disable strict -->
120+
121+
```js
122+
process.versions.icu;
123+
new Intl.DateTimeFormat('es', { month: 'long' }).format(new Date(9E8));
124+
```
125+
126+
(This should print your updated ICU version number, and also `January` again.)
127+
128+
You are ready to check in the updated `deps/icu-small`. This is a big commit,
129+
so make this a separate commit from the smaller changes.
130+
131+
> :warning: Do not modify any source code in `deps/icu-small` !
132+
> See section below about floating patches to ICU.
133+
134+
* Now, rebuild the Node.js license.
135+
136+
```shell
137+
# clean up - remove deps/icu
138+
make clean
139+
tools/license-builder.sh
140+
```
141+
142+
* Update the URL and hash for the full ICU file in `tools/icu/current_ver.dep`.
143+
It should match the ICU URL used in the first step. When this is done, the
144+
following should build with small ICU.
145+
146+
```shell
147+
# clean up
148+
rm -rf out deps/icu deps/icu4c*
149+
./configure --with-intl=small-icu --download=all
150+
make
151+
make test-ci
152+
```
153+
154+
* commit the change to `tools/icu/current_ver.dep` and `LICENSE` files.
155+
156+
* Note: To simplify review, I often will “pre-land” this patch, meaning that
157+
I run the patch through `curl -L https://github.com/nodejs/node/pull/xxx.patch
158+
| git am -3 --whitespace=fix` per the collaborator’s guide… and then push that
159+
patched branch into my PR's branch. This reduces the whitespace changes that
160+
show up in the PR, since the final land will eliminate those anyway.
161+
162+
## Floating patches to ICU
163+
164+
Floating patches are applied at `configure` time. The "patch" files
165+
are used instead of the original source files. The patch files are
166+
complete `.cpp` files replacing the original contents.
167+
168+
Patches are tied to a specific ICU version. They won’t apply to a
169+
future ICU version. We assume that you filed a bug against [ICU][] and
170+
upstreamed the fix, so the patch won’t be needed in a later ICU
171+
version.
172+
173+
### Example
174+
175+
For example, to patch `source/tools/toolutil/pkg_genc.cpp` for
176+
ICU version 63:
177+
178+
```shell
179+
# go to your Node.js source directory
180+
cd <node>
181+
182+
# create the floating patch directory
183+
mkdir -p tools/icu/patches/63
184+
185+
# create the subdirectory for the file(s) to patch:
186+
mkdir -p tools/icu/patches/63/source/tools/toolutil/
187+
188+
# copy the file to patch
189+
cp deps/icu-small/source/tools/toolutil/pkg_genc.cpp \
190+
tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp
191+
192+
# Make any changes to this file:
193+
(edit tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp )
194+
195+
# test
196+
make clean && ./configure && make
197+
```
198+
199+
You should see a message such as:
200+
201+
```shell
202+
INFO: Using floating patch "tools/icu/patches/63/source/tools/toolutil/pkg_genc.cpp" from "tools/icu"
203+
```
204+
205+
### Clean Up
206+
207+
Any patches older than the minimum version given in `tools/icu/icu_versions.json`
208+
ought to be deleted, because they will never be used.
209+
210+
### Why not just modify the ICU source directly?
211+
212+
Especially given the V8 dependencies above, there may be times when a floating
213+
patch to ICU is required. Though it seems expedient to simply change a file in
214+
`deps/icu-small`, this is not the right approach for the following reasons:
215+
216+
1. **Repeatability.** Given the complexity of merging in a new ICU version,
217+
following the steps above in the prior section of this document ought to be
218+
repeatable without concern for overriding a patch.
219+
220+
2. **Verifiability.** Given the number of files modified in an ICU PR,
221+
a floating patch could easily be missed — or dropped altogether next time
222+
something is landed.
223+
224+
3. **Compatibility.** There are a number of ways that ICU can be loaded into
225+
Node.js (see the top level README.md). Only modifying `icu-small` would cause
226+
the patch not to be landed in case the user specifies the ICU source code
227+
another way.
228+
229+
[ICU]: http://icu-project.org
230+
[Unicode]: https://unicode.org
231+
[tz]: https://www.iana.org/time-zones
232+
[CLDR]: https://unicode.org/cldr
233+
[Ecma402]: https://github.com/tc39/ecma402

‎tools/icu/README.md

+18-115
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
# Notes about the `tools/icu` subdirectory
22

3-
This directory contains tools, data, and information about the [ICU](http://icu-project.org)
4-
(International Components for Unicode) integration. ICU is used to provide
5-
internationalization functionality.
3+
This directory contains tools, data, and information about the
4+
International Components for Unicode integration. [ICU][] is used
5+
both by V8 and also by
6+
Node.js itself to provide internationalization functionality.
67

78
* `patches/` are one-off patches, actually entire source file replacements,
89
organized by ICU version number.
@@ -18,119 +19,21 @@ internationalization functionality.
1819
* `README.md` — you are here
1920
* `shrink-icu-src.py` — this is used during upgrade (see guide below)
2021

21-
## How to upgrade ICU
22+
Note:
23+
> The files in this directory were written for the Node.js v0.12 effort.
24+
> The original intent was to merge the tools such as `icutrim.py` and `iculslocs.cc`
25+
> back into ICU. ICU has gained its own “data slicer” tool.
26+
> There is an issue open, https://github.com/nodejs/node/issues/25136
27+
> for replacing `icutrim.py` with the [ICU data slicer][].
2228
23-
* Make sure your Node.js workspace is clean (clean `git status`) should be
24-
sufficient.
25-
* Configure Node.js with the specific [ICU version](http://icu-project.org/download)
26-
you want to upgrade to, for example:
29+
## See Also
2730

28-
```shell
29-
./configure \
30-
--with-intl=full-icu \
31-
--with-icu-source=http://download.icu-project.org/files/icu4c/58.1/icu4c-58_1-src.tgz
32-
make
33-
```
31+
* [docs/guides/maintaining-icu.md](../../doc/guides/maintaining-icu.md) for
32+
information on maintaining ICU in Node.js
3433

35-
> _Note_ in theory, the equivalent `vcbuild.bat` commands should work also,
36-
> but the commands below are makefile-centric.
34+
* [docs/api/intl.md](../../doc/api/intl.md) for information on the
35+
internationalization-related APIs in Node.js
36+
* [The ICU Homepage][ICU]
3737

38-
* If there are ICU version-specific changes needed, you may need to make changes
39-
in `icu-generic.gyp` or add patch files to `tools/icu/patches`.
40-
* Specifically, look for the lists in `sources!` in the `icu-generic.gyp` for
41-
files to exclude.
42-
43-
* Verify the Node.js build works:
44-
45-
```shell
46-
make test-ci
47-
```
48-
49-
Also running
50-
51-
<!-- eslint-disable strict -->
52-
53-
```js
54-
new Intl.DateTimeFormat('es', {month: 'long'}).format(new Date(9E8));
55-
```
56-
57-
…Should return `enero` not `January`.
58-
59-
* Now, copy `deps/icu` over to `deps/icu-small`
60-
61-
```shell
62-
python tools/icu/shrink-icu-src.py
63-
```
64-
65-
* Now, do a clean rebuild of Node.js to test:
66-
67-
```shell
68-
make -k distclean
69-
./configure
70-
make
71-
```
72-
73-
* Test this newly default-generated Node.js
74-
75-
<!-- eslint-disable strict -->
76-
77-
```js
78-
process.versions.icu;
79-
new Intl.DateTimeFormat('es', {month: 'long'}).format(new Date(9E8));
80-
```
81-
82-
(This should print your updated ICU version number, and also `January` again.)
83-
84-
You are ready to check in the updated `deps/small-icu`. This is a big commit,
85-
so make this a separate commit from the smaller changes.
86-
87-
* Now, rebuild the Node.js license.
88-
89-
```shell
90-
# clean up - remove deps/icu
91-
make clean
92-
tools/license-builder.sh
93-
```
94-
95-
* Update the URL and hash for the full ICU file in `tools/icu/current_ver.dep`.
96-
It should match the ICU URL used in the first step. When this is done, the
97-
following should build with small ICU.
98-
99-
```shell
100-
# clean up
101-
rm -rf out deps/icu deps/icu4c*
102-
./configure --with-intl=small-icu --download=all
103-
make
104-
make test-ci
105-
```
106-
107-
* commit the change to `tools/icu/current_ver.dep` and `LICENSE` files.
108-
109-
* Note: To simplify review, I often will “pre-land” this patch, meaning that
110-
I run the patch through `curl -L https://github.com/nodejs/node/pull/xxx.patch
111-
| git am -3 --whitespace=fix` per the collaborator’s guide… and then push that
112-
patched branch into my PR's branch. This reduces the whitespace changes that
113-
show up in the PR, since the final land will eliminate those anyway.
114-
115-
-----
116-
117-
## Postscript about the tools
118-
119-
The files in this directory were written for the Node.js effort.
120-
It was the intent of their author (Steven R. Loomis / srl295) to
121-
merge them upstream into ICU, pending much discussion within the
122-
ICU-TC.
123-
124-
`icu_small.json` is somewhat node-specific as it specifies a "small ICU"
125-
configuration file for the `icutrim.py` script. `icutrim.py` and
126-
`iculslocs.cpp` may themselves be superseded by components built into
127-
ICU in the future. As of this writing, however, the tools are separate
128-
entities within Node.js, although theyare being scrutinized by interested
129-
members of the ICU-TC. The “upstream” ICU bugs are given below.
130-
131-
* [#10919](http://bugs.icu-project.org/trac/ticket/10919)
132-
(experimental branch - may copy any source patches here)
133-
* [#10922](http://bugs.icu-project.org/trac/ticket/10922)
134-
(data packaging improvements)
135-
* [#10923](http://bugs.icu-project.org/trac/ticket/10923)
136-
(rewrite data building in python)
38+
[ICU data slicer]: https://github.com/unicode-org/icu/blob/master/docs/userguide/icu_data/buildtool.md
39+
[ICU]: http://icu-project.org

0 commit comments

Comments
 (0)