| blob | 74556402e472a6ccb214044cbbd1f2389abbb234 |
2 Issues (and their resolutions) when using gettext for message translation
4 Contents
5 ========
7 * Windows issues
8 * Automatic characterset conversion
9 * Translations on the client
10 * No translations on the server
11 * Translating plural forms (ngettext() support)
14 \f
15 Windows issues
16 ==============
18 On Windows, Subversion is linked against a modified version of GNU gettext.
19 This resolves several issues:
21 - Eliminated need to link against libiconv (which would be the second
22 iconv library, since we already link against apr-iconv)
23 - No automatic charset conversion (guaranteed UTF-8 strings returned by
24 gettext() calls without performance penalties)
26 More in the paragraphs below...
28 \f
29 Automatic characterset conversion
30 =================================
32 Some gettext implementations automatically convert the strings in the
33 message catalogue to the active system characterset. The source encoding
34 is stored in the "" message id. The message string looks somewhat like
35 a mime header and contains a "Content-Encoding" line. It's typically GNU's
36 gettext which does this.
38 Subversion uses UTF-8 to encode strings internally, which may not be the
39 systems default character encoding. To prevent internal corruption,
40 libsvn_subr:svn_cmdline_init2() explicitly tells gettext to return UTF-8
41 encoded strings if it has bind_textdomain_codeset().
43 Some gettext implementations don't contain automatic string recoding. In
44 order to work with both recoding and non-recoding implementations, the
45 source strings must be UTF-8 encoded. This is achieved by requiring .po
46 files to be UTF-8 encoded. [Note: a pre-commit hook has been installed to
47 ensure this.]
49 On Windows Subversion links against a version of GNU gettext, which has
50 been modified not to do character conversions. This eliminates the
51 requirement to link against libiconv which would mean Subversion being
52 linked against 2 iconv libraries (apr_iconv as well as libiconv).
54 \f
55 Translations on the client
56 ==========================
58 The translation effort is to translate all error messages generated on
59 the system on which the user has invoked his subversion command (svnadmin,
60 svnlook, svndumpfilter, svnversion or svn).
62 This means that in all layers of the libraries strings have been marked for
63 translation, either with _() or N_().
65 Parameters are sprintf-ed straight into errorstrings at the time they are
66 added to the error structure, so most strings are marked with _() and
67 translated directly into the language for which the client was set up.
68 [Note: The N_() macro marks strings for delayed translation.]
71 \f
72 Translations on the server
73 ==========================
75 On systems which define the LC_MESSAGES constant, setlocale() can be used
76 to set string translation for all (error) strings even those outside
77 the Subversion domain.
79 Windows doesn't define LC_MESSAGES. Instead GNU gettext uses the environ-
80 ment variables LANGUAGE, LC_ALL, LC_MESSAGES and LANG (in that order) to
81 find out what language to translate to. If none of these are defined, the
82 system and user default locales are queried. Though setting one of
83 the aforementioned variables before starting the server will avoid
84 localization by Subversion to the default locale, messages generated
85 by the system itself are likely to still be in its default locale
86 (they are on Windows).
88 While systems which have the LC_MESSAGES flag (or setenv() - of which
89 Windows has neither) allow languages to be switched at run time, this cannot
90 be done portably.
92 Any attempt to use setlocale() in an Apache environment may conflict
93 with settings other modules expect to be setup (even when using a
94 prefork MPM). On the svnserve side, having no portable way to change
95 languages dynamically means that the environment has to be set up
96 correctly from the start. Futhermore, the svnserve protocol doesn't
97 yet support content negotiation.
99 In other words, there is no way -- programmatically -- to ensure that
100 messages are served in any specific language using a traditional
101 gettext implementation. Current consensus is that gettext must be
102 replaced on the server side with a more flexible implementation.
104 Server requirement(s):
105 - Language negotiation on a per-client session basis. For a
106 stateless protocol like HTTP, this means per-request. For a
107 stateful protocol like the one used by svnserve, this means
108 per-connection.
109 - Avoid contamination of environment used by other code (e.g. other
110 Apache modules running in the same server as mod_dav_svn).
111 - Allow for propogation of the language to use to hook scripts.
112 - Continue to inter-op with generic HTTP/DAV clients, and stay
113 compatible with SVN clients of various versions (as per existing
114 compatibility rules).
116 I18N module requirement(s):
117 - Cross-platform.
118 - Interoperable with gettext tools (e.g. for .po files).
119 - Non-viral license which allows for any necessary modifications.
120 - gettext-like API (needn't be an exact match).
122 Implementation guidelines:
123 - The L10N API will be uniform across all libraries, clients, and
124 servers. Server-negotiated language will be recorded in either a
125 context baton (e.g. apr_pool_t.userdata), or in thread-local
126 storage (TLS).
127 - Implemented on top of a new gettext-like module with per-struct or
128 per-thread locale mutator functions and storage for name/value
129 pairs (a glorified apr_hash_t). (See implementation from Nicolás
130 Lichtmaier noted below.)
131 - Language chosen by the server will be negotiated based on a ranked
132 list of preferences provided by the client.
133 - Language used by httpd/mod_dav_svn will be derived from the
134 Accept-Language HTTP header, and setup by mod_negotiation (when
135 available), or by mod_dav_svn on a per-request basis.
136 - Language used by svnserve derived from additions to the protocol
137 which allow for HTTP-style content negotiation on a per-connection
138 basis. The protocol extension would use the same sort of q-value
139 list found in the Accept-Language header to specify user language
140 preferences.
142 Investigation: A brief canvasing of developers (on IRC) indicated that
143 no thorough investigation of existing solutions which might meet the
144 above requirements has been done. This incomplete canvasing may not
145 paint an accurate picture, however.
147 A branch <https://svn.collab.net/repos/svn/branches/server-l10n> has
148 been created to explore a solution to the above requirements. While
149 the L10N module is important, how that module is applied to both the
150 server-side and client-side is possibly even more so; an
151 implementation which meets the requirements should not dramatically
152 impact the solution used across the code base for the general L10N
153 API, nor the necessary server-side machinations.
155 Nicolás Lichtmaier wrote something along the lines of the module
156 referenced in the "Possible implementation" section
157 <http://svn.haxx.se/dev/archive-2004-04/0788.shtml>, which has been
158 committed to the server-l10n branch. However, it depends upon the GNU
159 gettext .mo format, and the GNU implementation may not be available on
160 all platforms (unless re-implemented). This module will need to be
161 enhanced or replaced, ideally completely obviating the need for
162 linkage against a platform's own gettext implementation.
164 Whether to use TLS or a context baton for the L10N API is under
165 discussion. TLS can provide a more friendly API (albeit somewhat
166 underhanded), while use of a context baton more resilient to change
167 (e.g. if httpd someday allowed more than one thread to service a
168 request). Here's a sample:
169 - No localization: "A message to localize"
170 - Localization w/ TLS or gloabl preference: _("A message to localize")
171 - Localization w/ a context baton: _("A message to localize", pool)
173 Historical note: Original consensus indicated that messages from the
174 server side should stay untranslated for transmission to the client.
175 However, client side localization is not an option, because by then
176 the parameter values have been inserted into the string, meaning that
177 it can't be looked up in the messages catalogue anymore. So any
178 localization must occur on the server, or significantly increase the
179 complexity of marshalling messages from the server as
180 unlocalized/unformatted data structures and localizing them on the
181 client side using some additional wrapper APIs to handle the
182 unmarshalling and message formatting. Additionally, client and server
183 versions may not match up, meaning that message keys and format string
184 values provided by the server may not correspond to what's available
185 on the client.
187 Paul Querna suggested a variation on this scheme involving requesting
188 (once) and caching the localizations (to the local disk) for each
189 server version, along with sending the message key (for lookup of
190 localized text) and an already formatted text (to use as the default
191 when no localization bundle is available). In addition to the
192 complications mentioned previously, this has the downside of crippling
193 the localization of server-generated messages when no write access to
194 the local disk is available to the client.
197 \f
198 Translating plural forms (ngettext() support)
199 =============================================
201 The code below works in english and can be translated to a number of
202 languages. However in some languages more than 2 forms are required
203 to do a correct translation. The ngettext() function takes care of
204 grabbing the right translation for those languages. Unfortunately,
205 the function is a GNU extention and thus non-portable.
207 message = (n > 1) ? _("1 File found") :
208 apr_sprintf (pool, _("%d Files found"), n);
210 Because of this limitation, some strings in the client have not been
211 marked for translation.
213 *** We're looking for good suggestions to work around this.