Tidy up char 127

[latex2e.git] / trunk / doc / ltnews24.tex

% \iffalse meta-comment
%
% Copyright 2015,2016
% The LaTeX3 Project and any individual authors listed elsewhere
% in this file.
%
% This file is part of the LaTeX base system.
% -------------------------------------------
%
% It may be distributed and/or modified under the
% conditions of the LaTeX Project Public License, either version 1.3c
% of this license or (at your option) any later version.
% The latest version of this license is in
%    http://www.latex-project.org/lppl.txt
% and version 1.3c or later is part of all distributions of LaTeX
% version 2005/12/01 or later.
%
% This file has the LPPL maintenance status "maintained".
%
% The list of all files belonging to the LaTeX base distribution is
% given in the file `manifest.txt'. See also `legal.txt' for additional
% information.
%
% The list of derived (unpacked) files belonging to the distribution
% and covered by LPPL is defined by the unpacking scripts (with
% extension .ins) which are part of the distribution.
%
% \fi
% Filename: ltnews24.tex
%
% This is issue 24 of LaTeX News.

\documentclass{ltnews}
\usepackage[T1]{fontenc}

\usepackage{lmodern,url,hologo}

\makeatletter % -- provide command introduced in new release
              %    so this typesets with an old format

% Check we are not in the preamble of a composite document
\def\@tempa{\@latex@error{Can be used only in preamble}\@eha}
\ifx\DeclareTextCommandDefault\@tempa
\else
  \DeclareTextCommandDefault\textcommabelow[1]
    {\hmode@bgroup\ooalign{\null#1\crcr\hidewidth\raise-.31ex
     \hbox{\check@mathfonts\fontsize\ssf@size\z@
     \math@fontsfalse\selectfont,}\hidewidth}\egroup}
  \DeclareTextCommandDefault\textcommaabove[1]{%
      \hmode@bgroup
      \ooalign{%
        \hidewidth
        \raise.7ex\hbox{%
          \check@mathfonts\fontsize\ssf@size\z@\math@fontsfalse\selectfont`%
        }%
       \hidewidth\crcr
       \null#1\crcr
      }%
      \egroup
    }
\fi

\makeatother

\publicationmonth{February}
\publicationyear{2016}

\publicationissue{24}

\begin{document}

\maketitle

\tableofcontents

\section{\hologo{LuaTeX} support}

This release refines the \hologo{LuaTeX} support introduced in the
2015/10/01 release. A number of patches have been added to improve the
behavior of \package{ltluatex} (thanks largely to code review by Philipp Gesang).
The kernel code has been adjusted to
allow for changes in \hologo{LuaTeX} v0.85--v0.88. Most notably, newer
\hologo{LuaTeX} releases allow more than $16$ write streams and these are now
enabled for use by \verb|\newwrite|, but also the experimental
\texttt{newtoken} Lua library has been renamed back to \texttt{token}
which required small adjustments in the  \hologo{LuaTeX} setup.

The biggest change in  \hologo{LuaTeX} v0.85--v0.87 compared to previous
versions is that all the primitives (originally defined in
\hologo{pdfTeX}) dealing with the PDF ``back end''
are no longer defined, being replaced by a much smaller set of new primitives.
This does not directly affect the core \LaTeX{}
files in this release but has required major changes to the
\texttt{.ini} files used by \TeX{} Live and similar distributions to
set up the format files. These changes in the  \hologo{LuaTeX} engine
will affect any packages using these back end commands (packages such
as \package{graphics},  \package{color},  \package{hyperref}, etc.).
Until all contributed packages are updated to the new syntax users may
need to add aliases for the old \hologo{pdfTeX} commands. A new
\package{luapdftexalias} package has been contributed to CTAN (not part of the
core \LaTeX\ release) that may be used for this purpose. 

See also the sections below for related changes in the
\textsf{tools} and \textsf{graphics} bundles.


\section{Unicode data}

As noted in \LaTeX{} News~22, the 2015/01/01 release of \LaTeX{} introduced
built-in support for extended \TeX{} systems. In particular, the kernel now
loads appropriate data from the Unicode Consortium to set \verb|\lccode|,
\verb|\uccode|, \verb|\catcode| and \verb|\sfcode| values in an automated
fashion for the entire Unicode range.

The initial approach taken by the team was to incorporate the existing model
used by (plain) \hologo{XeTeX} and to pre-process the ``raw'' Unicode data into
a ready-to-use form as \verb|unicode-letters.def|. However, the relationship
between the Unicode Consortium files and \TeX{} data structures is non-trivial and still
being explored. As such, it is preferable to directly parse the original
(\verb|.txt|) files at point of use. The team has therefore ``spun-out'' both
the data and the loading to a new generic package, \package{unicode-data}. This
package makes the original Unicode Consortium data files available in the
\verb|texmf| tree (in \verb|tex/generic/unicode-data|) and provides generic
loaders suitable for reading this data into the plain, \LaTeXe{}, and other,
formats.

At present, the following data files are included in this new package:
\begin{itemize}\parskip=0pt
  \item \verb|CaseFolding.txt|
  \item \verb|EastAsianWidth.txt|
  \item \verb|LineBreak.txt|
  \item \verb|MathClass.txt|
  \item \verb|SpecialCasing.txt|
  \item \verb|UnicodeData.txt|
\end{itemize}
These files are used either by \LaTeXe{} or by \package{expl3}
(i.e.~they represent the set currently required by the team). The
Unicode Consortium provides various other data files and we would be happy to add
these to the generic package, as it is intended to provide a single place
to collect this material in the \verb|texmf| tree. Such requests can be
mailed to the team as~usual or logged at the package home page:
\url{https://github.com/latex3/unicode-data}.

The new approach extends use of Unicode data in setting \TeX{} information in
two ways. First, the \verb|\sfcode| of all end-of-quotation/closing punctuation
is now set to $0$ (transparent to \TeX{}). Second, \verb|\Umathcode| values are
now set using \verb|MathClass.txt| rather than setting up only letters (which
was done using an arbitrary plane~$0$/plane~$1$ separation). There are also
minor refinements to the existing code setting, particularly splitting the
concepts of case and letter/non-letter category codes.

For \hologo{XeTeX}, users should note that \verb|\xtxHanGlue| and
\verb|\xtxHanSpace| are \emph{no longer defined}, that no assignments are made
to \verb|\XeTeXinterchartoks| and that no \verb|\XeTeXintercharclass| data is
loaded into the format. The values which were previously inherited from
the plain \hologo{XeTeX} setup files are \emph{not} suitable for properly
typesetting East Asian text. There are third-party packages addressing this
area well, notably those in the \package{CTeX} bundle. Third-party packages
may need adjustment to load the data themselves; see the \textsf{unicode-data}
package for one possible loader.

\section{More support for east European accents}

As noted in \LaTeX{} News~23, comma accent support was added for \texttt{s} and
\texttt{t} in the 2015/10/01 release. In this release a matching
\verb|\textcommaabove| accent has been added for U+0123 (\verb|\c{g}|,
\textcommaabove{g}) which is the lower case of U+0122 (\verb|\c{G}|,
\textcommabelow{G}).  In the OT1 and T1 encodings the combinations are
declared as composites with the \verb|\c| command, which matches the
Unicode names ``\textsf{latin (capital|small) letter g with cedilla}'' and
also allows \verb|\MakeUppercase{\c{g}}| to produce \verb|\c{G}|, as
required.  In T1 encoding, the composite of \verb|\c| with \texttt{k}, \texttt{l},
\texttt{n} and \texttt{r} are also
declared to use the comma below accent rather than cedilla to match the
conventional use of these letters.

The UTF-8 \texttt{inputenc} option \texttt{utf8} has been extended to
support all latin combinations that can be reasonably constructed with a
(single) accent command an a base character for the T1 encoding so
\textcommaabove{g}, \k{u} and similar characters may be directly input
using UTF-8 encoding.

\section{Changes in Graphics}

The changes in \hologo{LuaTeX}~v0.87 mean that the \package{color} and
\package{graphics} packages no longer share the \texttt{pdftex.def} file
between \hologo{LuaTeX} and \hologo{pdfTeX}. A separate file
\texttt{luatex.def} (distributed separately) has been produced, and
distributions are encouraged to modify \texttt{graphics.cfg} and
\texttt{color.cfg} configuration files to default to the \texttt{luatex}
option if \hologo{LuaTeX}~v0.87 or later is being used. The team has
contributed suitable \texttt{.cfg} files to CTAN to be used
as models.

Normally it is best to let the local \texttt{graphics.cfg}
automatically supply the right option depending on the \TeX\ engine
being used; however the \package{color} and \package{graphics} (and so
\package{graphicx}) packages have been extended to have an explicit
\texttt{luatex} option comparable to the existing \texttt{pdftex} and
\texttt{xetex} options.

The \package{trig} package has been updated so that pre-computed values
such as $\sin(90)$ now expand to digits (\texttt{1} rather than the
internal token \verb|\@one| in this case). This allows them to be used
directly in PDF literal strings. 

\section{Changes in Tools}

\hologo{LuaTeX} from version v0.87 no longer supports the
\verb|\write18| syntax to access system commands. A new package
\package{shellesc} has been added to \textsf{tools} that defines a new
command \verb|\ShellEscape| that may be used in all \TeX\ variants to
provide a consistent access to system commands. The package also
defines \verb|\write18| in  \hologo{LuaTeX} so that it continues to access
system commands as before; see the package documentation for details.

\section{Improving support for Unicode engines}

\looseness=-1
Stability concerns are always paramount when considering any change to
the \LaTeXe{} kernel. At the same time, it is important that the format
remains usable and gives reliable results for users. For the Unicode
\TeX{} engines \hologo{XeTeX} and \hologo{LuaTeX} there are important
differences in behavior from classical ($8$-bit) \TeX{} engines which
mean that identical default behaviors are not appropriate. Over the
past 18 months the team has addressed the most pressing of these
considerations (as detailed above and in \LaTeX{} News~22 and 23),
primarily by integrating existing patches into the kernel. There are,
though, important areas which still need consideration, and which
\emph{may} result in refinements to kernel support in this area in
future releases.

The default font setup in \LaTeXe{} at present is to use the \texttt{OT1}
encoding. This assumes that hyphenation patterns have been read using
appropriate codes: the \texttt{T1} encoding is assumed. The commonly-used
hyphenation patterns today, \package{hyph-utf8}, are set up in this
way for $8$-bit engines (\hologo{pdfTeX}) but for Unicode engines use
Unicode code points. This means that hyphenation will be incorrect
with Unicode engines unless a Unicode font is loaded. This requires
a concept of a Unicode font encoding, which is currently provided by
the \package{fontspec} package in two versions, \texttt{EU1} and
\texttt{EU2}. The team is working to fully understand what is meant
by a ``Unicode font encoding'', as unlike a classical \TeX{} encoding
it is essentially impossible to know what glyphs will be provided
(though each slot is always defined with the same meaning).  There
is also an overlap between this area and ideas of language and writing
system, most obviously in documents featuring mixed scripts (for example
Latin and Cyrillic).

As well as these font considerations, the team is also exploring
to what extent it is possible to allow existing ($8$-bit)
documents to compile directly with Unicode engines without requiring
changes in the sources. Whether this is truly possible remains an open
question.

It is important to stress that changes will only be made in this area where
they do \emph{not} affect documents processed with
\hologo{eTeX}/\hologo{pdfTeX} (i.e.~documents which are written for
``classical'' $8$-bit \TeX{} engines). Changes will also be made only where
they clearly address deficiencies in the current setup for Unicode engines
(i.e.~where current behaviors are wrong).

\end{document}