build: i18n: improve Intl build, add "--with-intl"

Note: This is a rebase on master/v0.12 (as of 57ed3da)

The two main goals of this change are:
 - to make it easier to build the Intl option using ICU
   (particularly, using a newer ICU than v8/Chromium's version)
 - to enable a much smaller ICU build with only English support
   The goal here is to get node.js binaries built this way by
   default so that the Intl API can be used. Additional data
   can be added at execution time (see Readme and wiki)

More details are in the PR nodejs#7719

In particular, this change adds the --with-intl= configure option to
 provide more ways of building "Intl":
 - full-icu picks up an ICU from deps/icu
 - small-icu is similar, but builds only English
 - system-icu uses pkg-config to find an installed ICU
 - none does nothing (no Intl)

For Windows builds, the "full-icu" or "small-icu" options are added to
vcbuild.bat.

Note that the existing "--with-icu-path" option is not removed from
configure, but may not be used alongside the new option.

Wiki changes have already been made on
 https://github.com/joyent/node/wiki/Installation
and a new page created at
 https://github.com/joyent/node/wiki/Intl
(marked as provisional until this change lands.)

Now, a summary of the changes:

* README.md : doc updates

* .gitignore : added "deps/icu" as this is the location where ICU is
  unpacked to.

* Makefile : added the tools/icu/* files to cpplint, but excluded a
  problematic file.

* configure : added the --with-intl option mentioned above.
  Calculate at config time the list of ICU source files to use and data
  packaging options.

* node.gyp : add the new files src/node_i18n.cc/.h as well as ICU
  linkage.

* src/node.cc : add call into
  node::i18n::InitializeICUDirectory(icu_data_dir) as well as new
  --icu-data-dir option and NODE_ICU_DATA env variable to configure ICU
  data loading. This loading is only relevant in the "small"
  configuration.

* src/node_i18n.cc: new source file for the above Initialize.. function,
  to setup ICU as needed.

* tools/icu : new directory with some tools needed for this build.

* tools/icu/icu-generic.gyp : new .gyp file that builds ICU in some new
  ways, both on unix/mac and windows.

* tools/icu/icu-system.gyp : new .gyp file to build node against a
  pkg-config detected ICU.

* tools/icu/icu_small.json : new config file for the "English-only" small
  build.

* tools/icu/icutrim.py : new tool for trimming down ICU data. Reads the
  above .json file.

* tools/icu/iculslocs.cc : new tool for repairing ICU data manifests
  after trim operation.

* tools/icu/no-op.cc : dummy file to force .gyp into using a C++ linker.

* vcbuild.bat : added small-icu and full-icu options, to call into
  configure.

* fixed toolset dependencies, see nodejs#7719 (comment)

* Made clang work

* wrapped the `--icu-data-dir=` param in `#if defined(NODE_HAVE_I18N_SUPPORT)` as requested

Author: srl295

.gitignore

@@ -45,6 +45,7 @@ ipch/
 /test/addons/doc-*/
 email.md
 deps/v8-*
+deps/icu
 ./node_modules
 .svn/
 

Makefile

@@ -406,7 +406,7 @@ CPPLINT_EXCLUDE += src/queue.h
 CPPLINT_EXCLUDE += src/tree.h
 CPPLINT_EXCLUDE += src/v8abbr.h
 
-CPPLINT_FILES = $(filter-out $(CPPLINT_EXCLUDE), $(wildcard src/*.cc src/*.h src/*.c))
+CPPLINT_FILES = $(filter-out $(CPPLINT_EXCLUDE), $(wildcard src/*.cc src/*.h src/*.c tools/icu/*.h tools/icu/*.cc))
 
 cpplint:
 	@$(PYTHON) tools/cpplint.py $(CPPLINT_FILES)

README.md

@@ -19,17 +19,6 @@ make
 make install
 ```
 
-With libicu i18n support:
-
-```sh
-svn checkout --force --revision 214189 \
-   http://src.chromium.org/svn/trunk/deps/third_party/icu46 \
-   deps/v8/third_party/icu46
-./configure --with-icu-path=deps/v8/third_party/icu46/icu.gyp
-make
-make install
-```
-
 If your python binary is in a non-standard location or has a
 non-standard name, run the following instead:
 
@@ -47,7 +36,9 @@ Prerequisites (Windows only):
 
 Windows:
 
-    vcbuild nosign
+```sh
+vcbuild nosign
+```
 
 You can download pre-built binaries for various operating systems from
 [http://nodejs.org/download/](http://nodejs.org/download/).  The Windows
@@ -92,6 +83,35 @@ make doc
 man doc/node.1
 ```
 
+### To build `Intl` (ECMA-402) support:
+
+*Note:* more docs, including how to reduce disk footprint, are on
+[the wiki](https://github.com/joyent/node/wiki/Intl).
+
+#### Use existing installed ICU (Unix/Macintosh only):
+
+```sh
+pkg-config --modversion icu-i18n && ./configure --with-intl=system-icu
+```
+
+#### Build ICU from source:
+
+First: Unpack latest ICU
+  [icu4c-**##.#**-src.tgz](http://icu-project.org/download) (or `.zip`)
+  as `deps/icu` (You'll have: `deps/icu/source/...`)
+
+Unix/Macintosh:
+
+```sh
+./configure --with-intl=full-icu
+```
+
+Windows:
+
+```sh
+vcbuild full-icu
+```
+
 Resources for Newcomers
 ---
   - [The Wiki](https://github.com/joyent/node/wiki)

configure

@@ -241,6 +241,11 @@ parser.add_option('--with-icu-path',
     dest='with_icu_path',
     help='Path to icu.gyp (ICU i18n, Chromium version only.)')
 
+parser.add_option('--with-intl',
+    action='store',
+    dest='with_intl',
+    help='Intl mode: none, full-icu, small-icu (default is none)')
+
 parser.add_option('--with-perfctr',
     action='store_true',
     dest='with_perfctr',
@@ -686,13 +691,138 @@ def configure_winsdk(o):
   print('ctrpp not found in WinSDK path--using pre-gen files '
         'from tools/msvs/genfiles.')
 
-
-def configure_icu(o):
+def glob_to_var(dir_base, dir_sub):
+  list = []
+  dir_all = os.path.join(dir_base, dir_sub)
+  files = os.walk(dir_all)
+  for ent in files:
+    (path, dirs, files) = ent
+    for file in files:
+      if file.endswith('.cpp') or file.endswith('.c') or file.endswith('.h'):
+        list.append('%s/%s' % (dir_sub, file))
+    break
+  return list
+
+
+def configure_intl(o):
+  # small ICU is off by default.
+  # always set icu_small, node.gyp depends on it being defined.
+  o['variables']['icu_small'] = b(False)
+
+  with_intl = options.with_intl
   have_icu_path = bool(options.with_icu_path)
-  o['variables']['v8_enable_i18n_support'] = int(have_icu_path)
-  if have_icu_path:
+  if have_icu_path and with_intl:
+    print 'Error: Cannot specify both --with-icu-path and --with-intl'
+    sys.exit(1)
+  elif have_icu_path:
+    # Chromium .gyp mode: --with-icu-path
+    o['variables']['v8_enable_i18n_support'] = 1
+    # use the .gyp given
     o['variables']['icu_gyp_path'] = options.with_icu_path
-
+    return
+  # --with-intl=<with_intl>
+  if with_intl == 'none' or with_intl is None:
+    o['variables']['v8_enable_i18n_support'] = 0
+    return  # no Intl
+  elif with_intl == 'small-icu':
+    # small ICU (English only)
+    o['variables']['v8_enable_i18n_support'] = 1
+    o['variables']['icu_small'] = b(True)
+  elif with_intl == 'full-icu':
+    # full ICU
+    o['variables']['v8_enable_i18n_support'] = 1
+  elif with_intl == 'system-icu':
+    # ICU from pkg-config.
+    o['variables']['v8_enable_i18n_support'] = 1
+    pkgicu = pkg_config('icu-i18n')
+    if not pkgicu:
+      print 'Error: could not load pkg-config data for "icu-i18n".'
+      print 'See above errors or the README.md.'
+      sys.exit(1)
+    (libs, cflags) = pkgicu
+    o['libraries'] += libs.split()
+    o['cflags'] += cflags.split()
+    # use the "system" .gyp
+    o['variables']['icu_gyp_path'] = 'tools/icu/icu-system.gyp'
+    return
+  else:
+    print 'Error: unknown value --with-intl=%s' % with_intl
+    sys.exit(1)
+  # Note: non-ICU implementations could use other 'with_intl'
+  # values.
+
+  # ICU mode. (icu-generic.gyp)
+  byteorder = sys.byteorder
+  o['variables']['icu_gyp_path'] = 'tools/icu/icu-generic.gyp'
+  # ICU source dir relative to root
+  icu_full_path = os.path.join(root_dir, 'deps/icu')
+  o['variables']['icu_path'] = icu_full_path
+  if not os.path.isdir(icu_full_path):
+    print 'Error: ICU path is not a directory: %s' % (icu_full_path)
+    sys.exit(1)
+  # Now, what version of ICU is it? We just need the "major", such as 54.
+  # uvernum.h contains it as a #define.
+  uvernum_h = os.path.join(icu_full_path, 'source/common/unicode/uvernum.h')
+  if not os.path.isfile(uvernum_h):
+    print 'Error: could not load %s - is ICU installed?' % uvernum_h
+    sys.exit(1)
+  icu_ver_major = None
+  matchVerExp = r'^\s*#define\s+U_ICU_VERSION_SHORT\s+"([^"]*)".*'
+  match_version = re.compile(matchVerExp)
+  for line in open(uvernum_h).readlines():
+    m = match_version.match(line)
+    if m:
+      icu_ver_major = m.group(1)
+  if not icu_ver_major:
+    print 'Could not read U_ICU_VERSION_SHORT version from %s' % uvernum_h
+    sys.exit(1)
+  icu_endianness = sys.byteorder[0];  # TODO(srl295): EBCDIC should be 'e'
+  o['variables']['icu_ver_major'] = icu_ver_major
+  o['variables']['icu_endianness'] = icu_endianness
+  icu_data_file_l = 'icudt%s%s.dat' % (icu_ver_major, 'l')
+  icu_data_file = 'icudt%s%s.dat' % (icu_ver_major, icu_endianness)
+  # relative to configure
+  icu_data_path = os.path.join(icu_full_path,
+                               'source/data/in',
+                               icu_data_file_l)
+  # relative to dep..
+  icu_data_in = os.path.join('../../deps/icu/source/data/in', icu_data_file_l)
+  if not os.path.isfile(icu_data_path) and icu_endianness != 'l':
+    # use host endianness
+    icu_data_path = os.path.join(icu_full_path,
+                                 'source/data/in',
+                                 icu_data_file)
+    # relative to dep..
+    icu_data_in = os.path.join('icu/source/data/in',
+                               icu_data_file)
+  # this is the input '.dat' file to use .. icudt*.dat
+  # may be little-endian if from a icu-project.org tarball
+  o['variables']['icu_data_in'] = icu_data_in
+  # this is the icudt*.dat file which node will be using (platform endianness)
+  o['variables']['icu_data_file'] = icu_data_file
+  if not os.path.isfile(icu_data_path):
+    print 'Error: ICU prebuilt data file %s does not exist.' % icu_data_path
+    print 'See the README.md.'
+    # .. and we're not about to build it from .gyp!
+    sys.exit(1)
+  # map from variable name to subdirs
+  icu_src = {
+    'stubdata': 'stubdata',
+    'common': 'common',
+    'i18n': 'i18n',
+    'io': 'io',
+    'tools': 'tools/toolutil',
+    'genccode': 'tools/genccode',
+    'genrb': 'tools/genrb',
+    'icupkg': 'tools/icupkg',
+  }
+  # this creates a variable icu_src_XXX for each of the subdirs
+  # with a list of the src files to use
+  for i in icu_src:
+    var  = 'icu_src_%s' % i
+    path = '../../deps/icu/source/%s' % icu_src[i]
+    o['variables'][var] = glob_to_var('tools/icu', path)
+  return  # end of configure_intl
 
 # determine the "flavor" (operating system) we're building for,
 # leveraging gyp's GetFlavor function
@@ -717,7 +847,7 @@ configure_libuv(output)
 configure_v8(output)
 configure_openssl(output)
 configure_winsdk(output)
-configure_icu(output)
+configure_intl(output)
 configure_fullystatic(output)
 
 # variables should be a root level element,

node.gyp

@@ -103,6 +103,7 @@
         'src/node_stat_watcher.cc',
         'src/node_watchdog.cc',
         'src/node_zlib.cc',
+        'src/node_i18n.cc',
         'src/pipe_wrap.cc',
         'src/signal_wrap.cc',
         'src/smalloc.cc',
@@ -134,6 +135,7 @@
         'src/node_version.h',
         'src/node_watchdog.h',
         'src/node_wrap.h',
+        'src/node_i18n.h',
         'src/pipe_wrap.h',
         'src/queue.h',
         'src/smalloc.h',
@@ -164,6 +166,17 @@
       ],
 
       'conditions': [
+        [ 'v8_enable_i18n_support==1', {
+          'defines': [ 'NODE_HAVE_I18N_SUPPORT=1' ],
+          'dependencies': [
+            '<(icu_gyp_path):icui18n',
+            '<(icu_gyp_path):icuuc',
+          ],
+          'conditions': [
+            [ 'icu_small=="true"', {
+              'defines': [ 'NODE_HAVE_SMALL_ICU=1' ],
+          }]],
+        }],
         [ 'node_use_openssl=="true"', {
           'defines': [ 'HAVE_OPENSSL=1' ],
           'sources': [

src/node.cc

@@ -35,6 +35,10 @@
 #include "node_crypto.h"
 #endif
 
+#if defined(NODE_HAVE_I18N_SUPPORT)
+#include "node_i18n.h"
+#endif
+
 #if defined HAVE_DTRACE || defined HAVE_ETW
 #include "node_dtrace.h"
 #endif
@@ -135,6 +139,11 @@ static node_module* modpending;
 static node_module* modlist_builtin;
 static node_module* modlist_addon;
 
+#if defined(NODE_HAVE_I18N_SUPPORT)
+// Path to ICU data (for i18n / Intl)
+static const char* icu_data_dir = NULL;
+#endif
+
 // used by C++ modules as well
 bool no_deprecation = false;
 
@@ -2953,6 +2962,14 @@ static void PrintHelp() {
          "  --trace-deprecation  show stack traces on deprecations\n"
          "  --v8-options         print v8 command line options\n"
          "  --max-stack-size=val set max v8 stack size (bytes)\n"
+#if defined(NODE_HAVE_I18N_SUPPORT)
+         "  --icu-data-dir=dir   set ICU data load path to dir\n"
+         "                         (overrides NODE_ICU_DATA)\n"
+#if !defined(NODE_HAVE_SMALL_ICU)
+         "                       Note: linked-in ICU data is\n"
+         "                       present.\n"
+#endif
+#endif
          "\n"
          "Environment variables:\n"
 #ifdef _WIN32
@@ -2964,6 +2981,13 @@ static void PrintHelp() {
          "NODE_MODULE_CONTEXTS   Set to 1 to load modules in their own\n"
          "                       global contexts.\n"
          "NODE_DISABLE_COLORS    Set to 1 to disable colors in the REPL\n"
+#if defined(NODE_HAVE_I18N_SUPPORT)
+         "NODE_ICU_DATA          Data path for ICU (Intl object) data\n"
+#if !defined(NODE_HAVE_SMALL_ICU)
+         "                       (will extend linked-in data)\n"
+#endif
+         "\n"
+#endif
          "\n"
          "Documentation can be found at http://nodejs.org/\n");
 }
@@ -3054,6 +3078,10 @@ static void ParseArgs(int* argc,
     } else if (strcmp(arg, "--v8-options") == 0) {
       new_v8_argv[new_v8_argc] = "--help";
       new_v8_argc += 1;
+#if defined(NODE_HAVE_I18N_SUPPORT)
+    } else if (strncmp(arg, "--icu-data-dir=", 15) == 0) {
+      icu_data_dir = arg + 15;
+#endif
     } else {
       // V8 option.  Pass through as-is.
       new_v8_argv[new_v8_argc] = arg;
@@ -3410,6 +3438,18 @@ void Init(int* argc,
     }
   }
 
+#if defined(NODE_HAVE_I18N_SUPPORT)
+  if (icu_data_dir == NULL) {
+    // if the parameter isn't given, use the env variable.
+    icu_data_dir = getenv("NODE_ICU_DATA");
+  }
+  // Initialize ICU.
+  // If icu_data_dir is NULL here, it will load the 'minimal' data.
+  if (!i18n::InitializeICUDirectory(icu_data_dir)) {
+    FatalError(NULL, "Could not initialize ICU "
+                "(check NODE_ICU_DATA or --icu-data-dir parameters)");
+  }
+#endif
   // The const_cast doesn't violate conceptual const-ness.  V8 doesn't modify
   // the argv array or the elements it points to.
   V8::SetFlagsFromCommandLine(&v8_argc, const_cast<char**>(v8_argv), true);
@@ -3592,7 +3632,7 @@ Environment* CreateEnvironment(Isolate* isolate,
 
 
 int Start(int argc, char** argv) {
-  const char* replaceInvalid = getenv("NODE_INVALID_UTF8");
+  const char* replaceInvalid = getenv("NODE_ISOLATEINVALID_UTF8");
 
   if (replaceInvalid == NULL)
     WRITE_UTF8_FLAGS |= String::REPLACE_INVALID_UTF8;