trunk/doc/ltnews24.tex

   1 % \iffalse meta-comment
   2 %
   3 % Copyright 2015,2016
   4 % The LaTeX3 Project and any individual authors listed elsewhere
   5 % in this file.
   6 %
   7 % This file is part of the LaTeX base system.
   8 % -------------------------------------------
   9 %
  10 % It may be distributed and/or modified under the
  11 % conditions of the LaTeX Project Public License, either version 1.3c
  12 % of this license or (at your option) any later version.
  13 % The latest version of this license is in
  14 %    http://www.latex-project.org/lppl.txt
  15 % and version 1.3c or later is part of all distributions of LaTeX
  16 % version 2005/12/01 or later.
  17 %
  18 % This file has the LPPL maintenance status "maintained".
  19 %
  20 % The list of all files belonging to the LaTeX base distribution is
  21 % given in the file `manifest.txt'. See also `legal.txt' for additional
  22 % information.
  23 %
  24 % The list of derived (unpacked) files belonging to the distribution
  25 % and covered by LPPL is defined by the unpacking scripts (with
  26 % extension .ins) which are part of the distribution.
  27 %
  28 % \fi
  29 % Filename: ltnews24.tex
  30 %
  31 % This is issue 24 of LaTeX News.
  32
  33 \documentclass{ltnews}
  34 \usepackage[T1]{fontenc}
  35
  36 \usepackage{lmodern,url,hologo}
  37
  38 \makeatletter % -- provide command introduced in new release
  39               %    so this typesets with an old format
  40
  41 % Check we are not in the preamble of a composite document
  42 \def\@tempa{\@latex@error{Can be used only in preamble}\@eha}
  43 \ifx\DeclareTextCommandDefault\@tempa
  44 \else
  45   \DeclareTextCommandDefault\textcommaabove[1]{%
  46     \hmode@bgroup
  47     \ooalign{%
  48       \hidewidth
  49       \raise.7ex\hbox{%
  50         \check@mathfonts\fontsize\ssf@size\z@\math@fontsfalse\selectfont`%
  51       }%
  52      \hidewidth\crcr
  53      \null#1\crcr
  54     }%
  55     \egroup
  56   }
  57 \fi
  58
  59 \makeatother
  60
  61 \publicationmonth{February}
  62 \publicationyear{2016}
  63
  64 \publicationissue{24}
  65
  66 \begin{document}
  67
  68 \maketitle
  69
  70 \tableofcontents
  71
  72 \section{\hologo{LuaTeX} support}
  73
  74 This release refines the \hologo{LuaTeX} support introduced in the
  75 2015/10/01 release. A number of patches have been added to improve the
  76 behavior of \package{ltluatex} (thanks largely to code review by Philipp Gesang).
  77 The kernel code has been adjusted to
  78 allow for changes in \hologo{LuaTeX} v0.85--v0.88. Most notably, newer
  79 \hologo{LuaTeX} releases allow more than $16$ write streams and these are now
  80 enabled for use by \verb|\newwrite|, but also the experimental
  81 \texttt{newtoken} Lua library has been renamed back to \texttt{token}
  82 which required small adjustments in the  \hologo{LuaTeX} setup.
  83
  84 The biggest change in  \hologo{LuaTeX} v0.85--v0.87 compared to previous
  85 versions is that all the primitives (originally defined in
  86 \hologo{pdfTeX}) dealing with the PDF ``back end''
  87 are no longer defined, being replaced by a new much smaller set of primitives.
  88 This does not directly affect the core \LaTeX{}
  89 files in this release but has required major changes to the
  90 \texttt{.ini} files used by \TeX{} Live and similar distributions to
  91 set up the format files. These changes in the  \hologo{LuaTeX} engine
  92 will affect any packages using these back end commands (packages such
  93 as \package{graphics},  \package{color},  \package{hyperref}, etc.).
  94 Until all contributed packages are updated to the new syntax users may
  95 need to add aliases for the old \hologo{pdfTeX} commands. A new
  96 \package{luapdftexalias} package has been contributed to CTAN (not part of the
  97 core \LaTeX\ release) that may be used for this purpose.
  98
  99 See also the sections below for related changes in the
 100 \textsf{tools} and \textsf{graphics} bundles.
 101
 102
 103 \section{Unicode data}
 104
 105 As noted in \LaTeX{} News~22, the 2015/01/01 release of \LaTeX{} introduced
 106 built-in support for extended \TeX{} systems. In particular, the kernel now
 107 loads appropriate data from the Unicode Consortium to set \verb|\lccode|,
 108 \verb|\uccode|, \verb|\catcode| and \verb|\sfcode| values in an automated
 109 fashion for the entire Unicode range.
 110
 111 The initial approach taken by the team was to incorporate the existing model
 112 used by (plain) \hologo{XeTeX} and to pre-process the ``raw'' Unicode data into
 113 a ready-to-use form as \verb|unicode-letters.def|. However, the relationship
 114 between the Unicode Consortium files and \TeX{} data structures is non-trivial and still
 115 being explored. As such, it is preferable to directly parse the original
 116 (\verb|.txt|) files at point of use. The team has therefore ``spun-out'' both
 117 the data and the loading to a new generic package, \package{unicode-data}. This
 118 package makes the original Unicode Consortium data files available in the
 119 \verb|texmf| tree (in \verb|tex/generic/unicode-data|) and provides generic
 120 loaders suitable for reading this data into the plain, \LaTeXe{}, and other,
 121 formats.
 122
 123 At present, the following data files are included in this new package:
 124 \begin{itemize}\parskip=0pt
 125   \item \verb|CaseFolding.txt|
 126   \item \verb|EastAsianWidth.txt|
 127   \item \verb|LineBreak.txt|
 128   \item \verb|MathClass.txt|
 129   \item \verb|SpecialCasing.txt|
 130   \item \verb|UnicodeData.txt|
 131 \end{itemize}
 132 These files are used either by \LaTeXe{} or by \package{expl3}
 133 (i.e.~they represent the set currently required by the team). The
 134 Unicode Consortium provide various other data files and we would be happy to add
 135 these to the generic package, as it is intended to provide a single place
 136 to collect this material in the \verb|texmf| tree. Such requests can be
 137 mailed to the team as usual or logged at the package home page:
 138 \url{https://github.com/latex3/unicode-data}.
 139
 140 The new approach extends use of Unicode data in setting \TeX{} information in
 141 two ways. First, the \verb|\sfcode| of all end-of-quotation/closing punctuation
 142 is now set to $0$ (transparent to \TeX{}). Second, \verb|\Umathcode| values are
 143 now set using \verb|MathClass.txt| rather than setting up only letters (which
 144 was done using an arbitrary plane~$0$/plane~$1$ separation). There are also
 145 minor refinements to the existing code setting, particularly splitting the
 146 concepts of case and letter/non-letter category codes.
 147
 148 For \hologo{XeTeX}, users should note that \verb|\xtxHanGlue| and
 149 \verb|\xtxHanSpace| are \emph{no longer defined}, that no assignments are made
 150 to \verb|\XeTeXinterchartoks| and that no \verb|\XeTeXintercharclass| data is
 151 loaded into the format. The values which were previously inherited from
 152 the plain \hologo{XeTeX} set up files are \emph{not} suitable for properly
 153 typesetting East Asian text. There are third-party packages addressing this
 154 area well, notably those in the \package{CTeX} bundle. Third-party packages
 155 may need adjustment to load the data themselves: see the \textsf{unicode-data}
 156 package for one possible loader.
 157
 158 \section{More Support for east European Accents}
 159
 160 As noted in \LaTeX{} News~23, comma accent support was added for \texttt{s} and
 161 \texttt{t} in the 2015/10/01 release. In this release a matching
 162 \verb|\textcommaabove| accent has been added for U+0123 (\verb|\c{g}|,
 163 \textcommaabove{g}) which is the lower case of U+0122 (\verb|\c{G}|,
 164 \textcommabelow{G}).  In the OT1 and T1 encodings the combinations are
 165 declared as composites with the \verb|\c| command, which matches the
 166 Unicode names ``\textsf{latin (capital|small) letter g with cedilla}'' and
 167 also allows \verb|\MakeUppercase{\c{g}}| to produce \verb|\c{G}|, as
 168 required.  In T1 encoding, the composite of \verb|\c| with \texttt{k}, \texttt{l},
 169 \texttt{n} and \texttt{r} are also
 170 declared to use the comma below accent rather than cedilla to match the
 171 conventional use of these letters.
 172
 173 The UTF-8 \texttt{inputenc} option \texttt{utf8} has been extended to
 174 support all latin combinations that can be reasonably constructed with a
 175 (single) accent command an a base character for the T1 encoding so
 176 \textcommaabove{g}, \k{u} and similar characters may be directly input
 177 using UTF-8 encoding.
 178
 179 \section{Changes in Graphics}
 180
 181 The changes in \hologo{LuaTeX}~v0.87 mean that the \package{color} and
 182 \package{graphics} packages no longer share the \texttt{pdftex.def} file
 183 between \hologo{LuaTeX} and \hologo{pdfTeX}. A separate file
 184 \texttt{luatex.def} (distributed separately) has been produced, and
 185 distributions are encouraged to modify \texttt{graphics.cfg} and
 186 \texttt{color.cfg} configuration files to default to the \texttt{luatex}
 187 option if \hologo{LuaTeX}~v0.87 or later is being used. The team has
 188 contributed suitable \texttt{.cfg} files to \textsc{ctan} to be used
 189 as models.
 190
 191 Normally it is best to let the local \texttt{graphics.cfg}
 192 automatically supply the right option depending on the \TeX\ engine
 193 being used; however the \package{color} and \package{graphics} (and so
 194 \package{graphicx}) packages have been extended to have an explicit
 195 \texttt{luatex} option comparable to the existing \texttt{pdftex} and
 196 \texttt{xetex} options.
 197
 198 The \package{trig} package has been updated so that pre-computed values
 199 such as $\sin(90)$ now expand to digits (\texttt{1} rather than the
 200 internal token \verb|\@one| in this case). This allows them to be used
 201 directly in PDF literal strings.
 202
 203 \section{Changes in Tools}
 204
 205 \hologo{LuaTeX} from version v0.87 no longer supports the
 206 \verb|\write18| syntax to access system commands. A new package
 207 \package{shellesc} has been added to \textsf{tools} that defines a new
 208 command \verb|\ShellEscape| that may be used in all \TeX\ variants to
 209 provide a consistent access to system commands. The package also
 210 defines \verb|\write18| in  \hologo{LuaTeX} so that it does access
 211 system commands as before; see the package documentation for details.
 212
 213 \section{Improving support for Unicode engines}
 214
 215 Stability concerns are always paramount when considering any change to
 216 the \LaTeXe{} kernel. At the same time, it is important that the format
 217 remains usable and gives reliable results for users. For the Unicode
 218 \TeX{} engines \hologo{XeTeX} and \hologo{LuaTeX} there are important
 219 differences in behaviour from classical ($8$-bit) \TeX{} engines which
 220 mean that identical default behaviors are not appropriate. Over the
 221 past 18 months the team has addressed the most pressing of these
 222 considerations (as detailed above and in \LaTeX{} News~22 and 23),
 223 primarily by integrating existing patches into the kernel. There are,
 224 though, important areas which still need consideration, and which
 225 \emph{may} result in refinements to kernel support in this area in
 226 future releases.
 227
 228 The default font set up in \LaTeXe{} at present is to use the \texttt{OT1}
 229 encoding. This assumes that hyphenation patterns have been read using
 230 appropriate codes: the \texttt{T1} encoding is assumed. The commonly-used
 231 hyphenation patterns today, \package{hyph-utf8}, are set up in this
 232 way for $8$-bit engines (\hologo{pdfTeX}) but for Unicode engines use
 233 Unicode code points. This means that hyphenation will be incorrect
 234 with Unicode engines unless a Unicode font is loaded. This requires
 235 a concept of a Unicode font encoding, which is currently provided by
 236 the \package{fontspec} package in two versions, \texttt{EU1} and
 237 \texttt{EU2}. The team is working to fully understand what is meant
 238 by a ``Unicode font encoding'', as unlike a classical \TeX{} encoding
 239 it is essentially impossible to know what glyphs will be provided
 240 (though each slot is always defined with the same meaning).  There
 241 is also an overlap between this area and ideas of language and writing
 242 system, most obviously in documents featuring mixed scripts (for example
 243 Latin and Cyrillic).
 244
 245 As well as these font considerations, the team is also exploring
 246 to what extent it is possible to allow existing ($8$-bit)
 247 documents to compile directly with Unicode engines without requiring
 248 changes in the sources. Whether this is truly possible remains an open
 249 question.
 250
 251 It is important to stress that changes will only be made in this area where
 252 they do \emph{not} affect documents processed with
 253 \hologo{eTeX}/\hologo{pdfTeX} (i.e.~documents which are written for
 254 ``classical'' $8$-bit \TeX{} engines). Changes will also be made only where
 255 they clearly address deficiencies in the current set up for Unicode engines
 256 (i.e.~where current behaviors are wrong).
 257
 258 \end{document}