It makes no sense to have a left pointing arrow as the desktop icon.

[wine/testsucceed.git] / documentation / winedev-kernel.sgml

  <chapter>
    <title>Kernel modules</title>
    <para>
      This section covers the kernel modules.  As already stated, Wine
      implements the NT architecture, hence provides <filename>NTDLL</filename>
      for the core kernel functions, and <filename>KERNEL32</filename>, which is
      the implementation of the basis of the Win32 subsystem, on top of
      <filename>NTDLL</filename>.
    </para>
    <para>
      This chapter is made of two types of material (depending of their point of
      view).  Some items will be tackled from a global point of view and then,
      when needed, explaining the split of work between
      <filename>NTDLL</filename> and <filename>KERNEL32</filename>; some others
      will be tackled from a DLL point of view (<filename>NTDLL</filename> or
      <filename>KERNEL32</filename>).  The choice is made so that the output is
      more readable and understantable.  At least, that's the intend (sigh).
    </para>

    <sect1 id="initialization">

      <title>The Wine initialization process</title>

      <para>
        Wine has a rather complex startup procedure, so unlike many programs the
        best place to begin exploring the code-base is
        <emphasis>not</emphasis> in fact at the <function>main()</function>
        function but instead at some of the  more straightforward DLLs that
        exist on the periphery such as MSI, the widget library (in
        <filename>USER</filename> and <filename>COMCTL32</filename>) etc. The
        purpose of this section is to document and explain how Wine starts up
        from the moment the user runs "<command>wine myprogram.exe</command>" to
        the point at which <filename>myprogram</filename> gets control.
      </para>

      <sect2>
        <title>First Steps</title>

        <para>
          The actual wine binary that the user runs does not do very much, in
          fact it is only responsible for checking the threading model in use
          (NPTL vs LinuxThreads) and then invoking a new binary which performs
          the next stage in the startup sequence. See the beginning of this
          chapter for more information on this check and why it's necessary. You
          can find this code in <filename>loader/glibc.c</filename>. The result
          of this check is an exec of either <command>wine-pthread</command> or
          <command>wine-kthread</command>, potentially (on Linux) via the
          <emphasis>preloader</emphasis>. We need to use separate binaries here
          because overriding the native pthreads library requires us to exploit
          a property of ELF symbol fixup semantics: it's not possible to do this
          without starting a new process.
        </para>

        <para>
          The Wine preloader is found in
          <filename>loader/preloader.c</filename>, and is required in order to
          impose a Win32 style address space layout upon the newly created Win32
          process. The details of what this does is covered in the address space
          layout chapter. The preloader is a statically linked ELF binary which
          is passed the name of the actual Wine binary to run (either
          <command>wine-kthread</command> or <command>wine-pthread</command>)
          along with the arguments the user passed in from the command line. The
          preloader is an unusual program: it does not have a
          <function>main()</function> function. In standard ELF applications,
          the entry point is actually at a symbol named
          <function>_start()</function>: this is provided  by the
          standard <command>gcc</command> infrastructure and normally jumps to
          <function>__libc_start_main()</function> which initializes glibc before
          passing control to the main function as defined by the programmer.
        </para>

        <para>
          The preloader takes control direct from the entry point for a few
          reasons. Firstly, it is required that glibc is not initialized twice:
          the result of such behaviour is undefined and subject to change
          without notice. Secondly, it's possible that as part of initializing
          glibc, the address space layout could be changed - for instance, any
          call to <function>malloc()</function> will initialize a heap arena
          which modifies the VM mappings. Finally, glibc does not return to
          <function>_start()</function> at any point, so by reusing it we avoid
          the need to recreate the ELF bootstrap stack
          (<varname>env</varname>, <varname>argv</varname>, auxiliary array etc).
        </para>

        <para>
          The preloader is responsible for two things: protecting important
          regions of the address space so the dynamic linker does not map shared
          libraries into them, and once that is done loading the real Wine
          binary off disk, linking it and starting it up. Normally all this is 
          automatically by glibc and the kernel but as we intercepted this
          process by using a static binary it's up to us to restart the
          process. The bulk of the code in the preloader is about loading
          <command>wine-[pk]thread</command> and
          <filename>ld-linux.so.2</filename> off disk, linking them together,
          then starting the dynamic linking process.
        </para>

        <para>
          One of the last things the preloader does before jumping into the
          dynamic linker is scan the symbol table of the loaded Wine binary and
          set the value of a global variable directly: this is a more efficient
          way of passing information to the main Wine program than flattening
          the data structures into an environment variable or command line
          parameter then unpacking it on the other side, but it achieves pretty
          much the same thing. The global variable set points to the preload
          descriptor table, which contains the VMA regions protected by the
          preloader. This allows Wine to unmap them once the dynamic linker has
          been run, so leaving gaps we can initialize properly later on.
        </para>

      </sect2>
    
      <sect2>
        <title>Starting the emulator</title>

        <para>
          The process of starting up the emulator itself is mostly one of
          chaining through various initializer functions defined in the core
          libraries and DLLs: <filename>libwine</filename>, then
          <filename>NTDLL</filename>, then <filename>KERNEL32</filename>.
        </para>

        <para>
          Both the <command>wine-pthread</command> and
          <command>wine-kthread</command> binaries share a common
          <function>main()</function> function, defined in
          <filename>loader/main.c</filename>, so no matter which binary is
          selected after the preloader has run we start here. This passes the
          information provided by the preloader into
          <filename>libwine</filename> and then calls
          <function>wine_init()</function>, defined in
          <filename>libs/wine/loader.c</filename>. This is where the emulation
          really starts: 
          <function>wine_init()</function> can, with the correct preparation,
          be called from programs other than the wine loader itself.
        </para>

        <para>
          <function>wine_init()</function> does some very basic setup tasks such
          as initializing the debugging infrastructure, yet more address space
          manipulation (see the information on the 4G/4G VM split in the address
          space chapter), before loading <filename>NTDLL</filename> - the core
          of both Wine and the Windows NT series - and jumping to the
          <function>__wine_process_init()</function> function defined
          in <filename>dlls/ntdll/loader.c</filename>
        </para>

        <para>
          This function is responsible for initializing the primary Win32
          environment. In <function>thread_init()</function>, it sets up the
          TEB, the <command>wineserver</command> connection for the main thread
          and the process heap. See the beginning of this chapter for more
          information on this.

        </para>

        <para>
          Finally, it loads and jumps to
          <function>__wine_kernel_init()</function> in
          <filename>KERNEL32.DLL</filename>: this is defined in
          <filename>dlls/kernel32/process.c</filename>. This is where the bulk
          of the work is done. The <filename>KERNEL32</filename> initialization
          code  retrieves the startup info for the process from the server,
          initializes the registry, sets up the drive mapping system and locale
          data, then begins loading the requested application itself. Each
          process has a <structname>STARTUPINFO</structname> block that can be
          passed into <function>CreateProcess</function> specifying various
          things like how the first window should be displayed: this is sent to
          the new process via the <command>wineserver</command>.
        </para>

        <para>
          After determining the type of file given to Wine by the user (a Win32
          EXE file, a Win16 EXE, a Winelib app etc), the program is loaded into
          memory (which may involve loading and initializing other DLLs, the
          bulk of Wines startup code), before control reaches the end of
          <function>__wine_kernel_init()</function>. This function ends with the
          new process stack being initialized, and start_process being called on
          the new stack. Nearly there!
        </para>

        <para>
          The final element of initializing Wine is starting the newly loaded
          program itself. <function>start_process()</function> sets up the SEH
          backstop handler, calls <function>LdrInitializeThunk()</function>
          which performs the last part of the process initialization (such as
          performing relocations and calling the <function>DllMain()</function>
          with <constant>PROCESS_ATTACH</constant>), grabs the entry point of
          the executable and then on this line:
        </para>

        <programlisting>
ExitProcess( entry( peb ) );
        </programlisting>

        <para>
          ... jumps to the entry point of the program. At this point the users
          program is running and the API provided by Wine is ready to be
          used. When entry returns, the <function>ExitProcess()</function> API
          will be used to initialize a graceful shutdown.
        </para>
      </sect2>
    </sect1>

    <sect1>
      <title>Detailed memory management</title>
      <para>
        As already explained in previous chapter (see <xref linkend="arch-mem">
        for the details), Wine creates every 32-bit Windows process in its own
        32 address space. Wine also tries to map at the relevant addresses what
        Windows would do. There are however a few nasty bits to look at.
      </para>

      <sect2>
        <title>Implementation</title>
        <para>
          Wine (with a bit of black magic) is able to map the main module at
          it's desired address (likely <constant>0x400000</constant>), to create
          the process heap, its stack (as a Windows executable can ask for a
          specific stack size), Wine simply use the initial stack of the ELF
          executable for its initialisation, but creates a new stack (as a Win32
          one) for the main thread of the executable. Wine also tries to map all
          native DLLs at their desired address, so that no relocation has to be
          performed.
        </para>
        <para>
          Wine also implements the shared heap so native win9x DLLs can be
          used. This heap is always created at the
          <constant>SYSTEM_HEAP_BASE</constant> address or
          <constant>0x80000000</constant> and defaults to 16 megabytes in size.
        </para>
        <para>
          There are a few other magic locations. The bottom 64k of memory is
          deliberately left unmapped to catch null pointer dereferences. The
          region from 64k to 1mb+64k are reserved for DOS compatibility and
          contain various DOS data structures. Finally, the address space also
          contains mappings for the Wine binary itself, any native libaries
          Wine is using, the glibc malloc arena and so on.
        </para>
      </sect2>

      <sect2 id="address-space">
        <title>Laying out the address space</title>

        <para>
          Up until about the start of 2004, the Linux address space very much
          resembled the Windows 9x layout: the kernel sat in the top gigabyte,
          the bottom pages were unmapped to catch null pointer dereferences, and
          the rest was free. The kernels mmap algorithm was predictable: it
          would start by mapping files at low addresses and work up from there.
        </para>

        <para>
          The development of a series of new low level patches violated many of
          these assumptions, and resulted in Wine needing to force the Win32
          address space layout upon the system. This section looks at why and
          how this is done.
        </para>

        <para>
          The exec-shield patch increases security by randomizing the kernels
          mmap algorithms. Rather than consistently choosing the same addresses
          given the same sequence of requests, the kernel will now choose
          randomized addresses. Because the Linux dynamic linker
          (<filename>ld-linux.so.2</filename>) loads DSOs into memory by using
          mmap, this means that DSOs are no longer loaded at predictable
          addresses, so making it harder to attack software by using buffer
          overflows. It also attempts to relocate certain binaries into a
          special low area of memory known as the ASCII armor so making it
          harder to jump into them when using string based attacks.
        </para>

        <para>
          Prelink is a technology that enhances startup times by precalculating
          ELF global offset tables then saving the results inside the native
          binaries themselves. By grid fitting each DSO into the address space,
          the dynamic linker does not have to perform as many relocations so
          allowing applications that heavily rely on dynamic linkage to be
          loaded into memory much quicker. Complex C++ applications such as
          Mozilla, OpenOffice and KDE can especially benefit from this
          technique.
        </para>

        <para>
          The 4G VM split patch was developed by Ingo Molnar. It gives the Linux
          kernel its own address space, thereby allowing processes to access the
          maximum addressable amount of memory on a 32-bit machine: 4
          gigabytes. It allows people with lots of RAM to fully utilise that in
          any given process at the cost of performance: the reason behind giving
          the kernel a part of each processes address space was to avoid the
          overhead of switching on each syscall.
        </para>

        <para>
          Each of these changes alter the address space in a way incompatible
          with Windows. Prelink and exec-shield mean that the libraries Wine
          uses can be placed at any point in the address space: typically this
          meant that a library was sitting in the region that the EXE you wanted
          to run had to be loaded (remember that unlike DLLs, EXE files cannot
          be moved around in memory). The 4G VM split means that programs could
          receive pointers to the top gigabyte of address space which some are
          not prepared for (they may store extra information in the high bits of
          a pointer, for instance). In particular, in combination with
          exec-shield this one is especially deadly as it's possible the process
          heap could be allocated beyond
          <constant>ADDRESS_SPACE_LIMIT</constant> which causes Wine
          initialization to fail.
        </para>

        <para>
          The solution to these problems is for Wine to reserve particular parts
          of the address space so that areas that we don't want the system to
          use will be avoided. We later on (re/de)allocate those areas as
          needed. One problem is that some of these mappings are put in place
          automatically by the dynamic linker: for instance any libraries that
          Wine is linked to (like <filename>libc</filename>,
          <filename>libwine</filename>, <filename>libpthread</filename> etc)
          will be mapped into memory before Wine even gets control. In order to
          solve that, Wine overrides the default ELF initialization sequence at
          a low level and reserves the needed areas by using direct syscalls
          into the kernel (ie without linking against any other code to do it)
          before restarting the standard initialization and letting the dynamic
          linker continue. This is referred to as the preloader and is found in
          <filename>loader/preloader.c</filename>.
        </para>

        <para>
          Once the usual ELF boot sequence has been completed, some native
          libraries may well have been mapped above the 3gig limit: however,
          this doesn't matter as 3G is a Windows limit, not a Linux limit. We
          still have to prevent the system from allocating anything else above
          there (like the heap or other DLLs) though so Wine performs a binary
          search over the upper gig of address space in order to iteratively
          fill in the holes with <constant>MAP_NORESERVE</constant> mappings so
          the address space is allocated but the memory to actually back it is
          not. This code can be found in
          <filename>libs/wine/mmap.c</filename>:<function>reserve_area</function>.
        </para>
      </sect2>
    </sect1>

    <sect1>
      <title>Multi-processing in Wine</title>
      <para>
        Let's take a closer look at the way Wine loads and run processes in memory.
      </para>
      <sect2>
        <title>Starting a process from command line</title>
        <para>
          When starting a Wine process from command line (we'll get later on to
          the differences between NE, PE and Winelib executables), there are a
          couple of things Wine need to do first. A first executable is run to
          check the threading model of the underlying OS (see
          <xref linkend="threading"> for the details) and will start the real
          Wine loader corresponding to the choosen threading model.
        </para>
        <para>
          Then Wine graps a few elements from the Unix world: the environment,
          the program arguments. Then the <filename>ntdll.dll.so</filename> is
          loaded into memory using the standard shared library dynamic
          loader. When loaded, <filename>NTDLL</filename> will mainly first
          create a decent Windows environment:
          <itemizedlist>
            <listitem>
              <para>
                create a PEB (Process Environment Block) and a TEB (Thread
                Environment Block).
              </para>
            </listitem>
            <listitem>
              <para>
                set up the connection to the Wine server - and eventually
                launching the Wine server if none runs
              </para>
            </listitem>
            <listitem>
              <para>create the process heap</para>
            </listitem>
          </itemizedlist>
        </para>
        <para>
          Then <filename>Kernel32</filename> is loaded (but now using the
          Windows dynamic loading capabilities) and a Wine specific entry point
          is called <function>__wine_kernel_init</function>. This function will
          actually handle all the logic of the process loading and execution,
          and will never return from it's call.
        </para>
        <para>
          <function>__wine_kernel_init</function> will undergo the following
          tasks:
          <itemizedlist>
            <listitem>
              <para>
                initialization of program arguments from Unix program arguments
              </para>
            </listitem>
            <listitem>
              <para>lookup of executable in the file system</para>
            </listitem>
            <listitem>
              <para>
                If the file is not found, then an error is printed and the Wine
                loader stops.
              </para>
            </listitem>
            <listitem>
              <para>
                We'll cover the non-PE file type later on, so assume for now
                it's a PE file. The PE module is loaded in memory using the
                same mechanisms as for a native DLLs (mainly mapping the file
                data and code sections into memory, and handling relocation if
                needed). Note that the dependencies on the module are not
                resolved at this point.
              </para>
            </listitem>
            <listitem>
              <para>
                A new stack is created, which size is given in the PE header,
                and this stack is made the one of the running thread (which is
                still the only one in the process). The stack used at startup
                will no longer be used.
              </para>
            </listitem>
            <listitem>
              <para>
                Which this new stack,
                <function>ntdll.LdrInitializeThunk</function> is called which
                performs the remaining initialization parts, including resolving
                all the DLL imports on the PE module, and doing the init of the
                TLS slots.
              </para>
            </listitem>
            <listitem>
              <para>
                Control can now be passed to the <function>EntryPoint</function>
                of the PE module, which will let the executable run.
              </para>
            </listitem>
          </itemizedlist>
        </para>
      </sect2>
      <sect2>
        <title>Creating a child process from a running process</title>
        <para>
          The steps used are closely link to what is done in the previous case.
        </para>
        <para>
          There are however a few points to look at a bit more closely. The
          inner implementation creates the child process using the
          <function>fork()</function> and <function>exec()</function>
          calls. This means that we don't need to check again for the threading
          model, we can use what the parent (or the grand-parent process...)
          started from command line has found.
        </para>
        <para>
          The Win32 process creation allows to pass a lot of information between
          the parent and the child. This includes object handles, windows title,
          console parameters, environment strings... Wine makes use of both the
          standard Unix inheritance mechanisms (for environment for example) and
          the Wine server (to pass from parent to child a chunk of data
          containing the relevant information).
        </para>
        <para>
          The previously described loading mechanism will check in the Wine
          server if such a chunk exists, and, if so, will perform the relevant
          initialization.
        </para>
        <para>
          Some further synchronization is also put in place: a parent will wait
          until the child has started, or has failed. The Wine server is also
          used to perform those tasks.
        </para>
      </sect2>
      <sect2>
        <title>Starting a Winelib process</title>
        <para>
          Before going into the gory details, let's first go back to what a
          Winelib application is. It can be either a regular Unix executable, or
          a more specific Wine beast. This later form in fact creates two files
          for a given executable (say <filename>foo.exe</filename>). The first
          one, named <filename>foo</filename> will be a symbolic link to the
          Wine loader (<filename>wine</filename>). The second one, named
          <filename>foo.exe.so</filename>, is the equivalent of the
          <filename>.dll.so</filename> files we've already described for
          DLLs. As in Windows, an executable is, among other things, a module
          with its import and export information, as any DLL, it makes sense
          Wine uses the same mechanisms for loading native executables and
          DLLs.
        </para>
        <para>
          When starting a Winelib application from the command line (say with
          <command>foo arg1 arg2</command>), the Unix shell will execute
          <command>foo</command> as a Unix executable. Since this is in fact the
          Wine loader, Wine will fire up. However, will notice that it hasn't
          been started as <command>wine</command> but as <command>foo</command>,
          and hence, will try to load (using Unix shared library mechanism) the
          second file <filename>foo.exe.so</filename>. Wine will recognize a 32
          bit module (with its descriptor) embedded in the shared library, and
          once the shared library loaded, it will proceed the same path as when
          loading a standard native PE executable.
        </para>
        <para>
          Wine needs to implement this second form of executable in order to
          maintain the order of initialization of some elements in the
          executable. One particular issue is when dealing with global C++
          objects. In standard Unix executable, the call of the constructor to
          such objects is stored in the specific section of the executable
          (<function>.init</function> not to name it). All constructors in this
          section are called before the <function>main()</function> or
          <function>WinMain</function> function is called. Creating a Wine
          executable using the first form mentionned above will let those
          constructors being called before Wine gets a chance to initialize
          itself. So, any constructor using a Windows API will fail, because
          Wine infrastructure isn't in place. The use of the second form for
          Winelib executables ensures that we do the initialization using the
          following steps:
          <itemizedlist>
            <listitem>
              <para>
                initialize the Wine infrastructure
              </para>
            </listitem>
            <listitem>
              <para>
                load the executable into memory
              </para>
            </listitem>
            <listitem>
              <para>
                handle the import sections for the executable
              </para>
            </listitem>
            <listitem>
              <para>
                call the global object constructors (if any). They now can
                properly call the Windows APIs
              </para>
            </listitem>
            <listitem>
              <para>
                call the executable entry point
              </para>
            </listitem>
          </itemizedlist>
        </para>
        <para>
          The attentive reader would have noted that the resolution of imports
          for the executable is done, as for a DLL, when the executable/DLL
          descriptor is registered. However, this is done also by adding a
          specific constructor in the <function>.init</function> section. For
          the above describe scheme to function properly, this constructor must
          be the first constructor to be called, before all the other
          constructors, generated by the executable itself. The Wine build chain
          takes care of that, and also generating the executable/DLL descriptor
          for the Winelib executable.
        </para>
      </sect2>
    </sect1>

    <sect1 id="threading">
      <title>Multi-threading in Wine</title>

      <para>
        This section will assume you understand the basics of multithreading. If
        not there are plenty of good tutorials available on the net to get you
        started.
      </para>

      <para>
        Threading in Wine is somewhat complex due to several factors. The first
        is the advanced level of multithreading support provided by Windows -
        there are far more threading related constructs available in Win32 than
        the Linux equivalent (pthreads). The second is the need to be able to
        map Win32 threads to native Linux threads which provides us with
        benefits like having the kernel schedule them without our
        intervention. While it's possible to implement threading entirely
        without kernel support, doing so is not desirable on most platforms that
        Wine runs on.
      </para>

      <sect2>
        <title>Threading support in Win32</title>

        <para>
          Win32 is an unusually thread friendly API. Not only is it entirely
          thread safe, but it provides many different facilities for working
          with threads. These range from the basics such as starting and
          stopping threads, to the extremely complex such as injecting threads
          into other processes and COM inter-thread marshalling.
        </para>

        <para>
          One of the primary challenges of writing Wine code therefore is
          ensuring that all our DLLs are thread safe, free of race conditions
          and so on. This isn't simple - don't be afraid to ask if you aren't
          sure whether a piece of code is thread safe or not!
        </para>

        <para>
          Win32 provides many different ways you can make your code thread safe
          however the most common are <emphasis>critical section</emphasis> and
          the <emphasis>interlocked functions</emphasis>. Critical sections are
          a type of mutex designed to protect a geographic area of code. If you
          don't want multiple threads running in a piece of code at once, you
          can protect them with calls to
          <function>EnterCriticalSection()</function> and
          <function>LeaveCriticalSection()</function>. The first call to
          <function>EnterCriticalSection()</function> by a thread will lock the
          section and continue without stopping. If another thread calls it then
          it will block until the original thread calls
          <function>LeaveCriticalSection()</function> again.
        </para>

        <para>
          It is therefore vitally important that if you use critical sections to
          make some code thread-safe, that you check every possible codepath out
          of the code to ensure that any held sections are left. Code like this:
        </para>

        <programlisting>
if (res != ERROR_SUCCESS) return res;
        </programlisting>

        <para>
          is extremely suspect in a function that also contains a call to
          <function>EnterCriticalSection()</function>. Be careful.
        </para>

        <para>
          If a thread blocks while waiting for another thread to leave a
          critical section, you will see an error from the
          <function>RtlpWaitForCriticalSection()</function> function, along with
          a note of which thread is holding the lock. This only appears after a
          certain timeout, normally a few seconds. It's possible the thread
          holding the lock is just being really slow which is why Wine won't
          terminate the app like a non-checked build of Windows would, but the
          most common cause is that for some reason a thread forgot to call
          <function>LeaveCriticalSection()</function>, or died while holding the
          lock (perhaps because it was in turn waiting for another lock). This
          doesn't just happen in Wine code: a deadlock while waiting for a
          critical section could be due to a bug in the app triggered by a
          slight difference in the emulation.
        </para>
    
        <para>
          Another popular mechanism available is the use of functions like
          <function>InterlockedIncrement()</function>
          and <function>InterlockedExchange()</function>. These make use of native
          CPU abilities to execute a single instruction while ensuring any other
          processors on the system cannot access memory, and allow you to do
          common operations like add/remove/check a variable in thread-safe code
          without holding a mutex. These are useful for reference counting
          especially in free-threaded (thread safe) COM objects.
        </para>

        <para>
          Finally, the usage of TLS slots are also popular. TLS stands for
          thread-local storage, and is a set of slots scoped local to a thread
          which you can store pointers in. Look on MSDN for the
          <function>TlsAlloc()</function> function to learn more about the Win32
          implementation of this. Essentially, the contents of a given slot will
          be different in each thread, so you can use this to store data that is
          only meaningful in the context of a single thread. On recent versions
          of Linux the __thread keyword provides a convenient interface to this
          functionality - a more portable API is exposed in the pthread
          library. However, these facilities are not used by Wine, rather, we
          implement Win32 TLS entirely ourselves.
        </para>
      </sect2>

      <sect2>
        <title>POSIX threading vs. kernel threading</title>

        <para>
          Wine runs in one of two modes: either pthreads (posix threading) or
          kthreads (kernel threading). This section explains the differences
          between them. The one that is used is automatically selected on
          startup by a small test program which then execs the correct binary,
          either <command>wine-kthread</command> or 
          <command>wine-pthread</command>. On NPTL-enabled systems pthreads
          will be used, and on older non-NPTL systems kthreads is selected.
        </para>

        <para>
          Let's start with a bit of history. Back in the dark ages when Wine's
          threading support was first implemented a problem was faced - Windows
          had much more capable threading APIs than Linux did. This presented a
          problem - Wine works either by reimplementing an API entirely or by
          mapping it onto the underlying systems equivalent. How could Win32
          threading be implemented using a library which did not have all the
          needed features? The answer, of course, was that it couldn't be.
        </para>

        <para>
          On Linux the pthreads interface is used to start, stop and control
          threads. The pthreads library in turn is based on top of so-called
          "kernel threads" which are created using the
          <function>clone(2)</function> syscall. Pthreads provides a nicer (more
          portable) interface to this functionality and also provides APIs for
          controlling mutexes. There is a <ulink
            url="http://www.llnl.gov/computing/tutorials/pthreads/"> good
            tutorial on pthreads</ulink> available if you want to learn more.
        </para>

        <para>
          As pthreads did not provide the necessary semantics to implement Win32
          threading, the decision was made to implement Win32 threading on top
          of the underlying kernel threads by using syscalls like
          <function>clone()</function> directly. This provided maximum
          flexibility and allowed a correct implementation but caused some bad
          side effects. Most notably, all the userland Linux APIs assumed that
          the user was utilising the pthreads library. Some only enabled thread
          safety when they detected that pthreads was in use - this is true of
          glibc, for instance. Worse, pthreads and pure kernel threads had
          strange interactions when run in the same process yet some libraries
          used by Wine used pthreads internally. Throw in source code porting
          using Winelib - where you have both UNIX and Win32 code in the same
          process - and chaos was the result.
        </para>

        <para>
          The solution was simple yet ingenious: Wine would provide its own
          implementation of the pthread library <emphasis>inside</emphasis> its
          own binary. Due to the semantics of ELF symbol scoping, this would
          cause Wine's own implementation to override any implementation loaded
          later on (like the real libpthread.so). Therefore, any calls to the
          pthread APIs in external libraries would be linked to Wine's instead
          of the system's pthreads library, and Wine implemented pthreads by
          using the standard Windows threading APIs it in turn implemented
          itself.
        </para>

        <para>
          As a result, libraries that only became thread-safe in the presence of
          a loaded pthreads implementation would now do so, and any external
          code that used pthreads would actually end up creating Win32 threads
          that Wine was aware of and controlled. This worked quite nicely for a
          long time, even though it required doing some extremely un-kosher
          things like overriding internal libc structures and functions. That
          is, it worked until NPTL was developed at which point the underlying
          thread implementation on Linux changed dramatically.
        </para>

        <para>
          The fake pthread implementation can be found in
          <filename>loader/kthread.c</filename>, which is used to
          produce the <command>wine-kthread</command> binary. In contrast,
          <filename>loader/pthread.c</filename> produces the
          <command>wine-pthread</command> binary which is used on newer NPTL
          systems.
        </para>

        <para>
          NPTL is a new threading subsystem for Linux that hugely improves its
          performance and flexibility. By allowing threads to become much more
          scalable and adding new pthread APIs, NPTL made Linux competitive with
          Windows in the multi-threaded world. Unfortunately it also broke many
          assumptions made by Wine (as well as other applications such as the
          Sun JVM and RealPlayer) in the process.
        </para>

        <para>
          There was, however, some good news. NPTL made Linux threading powerful
          enough that Win32 threads could now be implemented on top of pthreads
          like any other normal application. There would no longer be problems
          with mixing win32-kthreads and pthreads created by external libraries,
          and no need to override glibc internals. As you can see from the
          relative sizes of the <filename>loader/kthread.c</filename> and
          <filename>loader/pthread.c</filename> files, the difference in code
          complexity is considerable. NPTL also made several other semantic
          changes to things such as signal delivery so changes were required in
          many different places in Wine.
        </para>

        <para>
          On non-Linux systems the threading interface is typically not powerful
          enough to replicate the semantics Win32 applications expect and so
          kthreads with the pthread overrides are used.
        </para>
      </sect2>

      <sect2>
        <title>The Win32 thread environment</title>

        <para>
          All Win32 code, whether from a native EXE/DLL or in Wine  itself,
          expects certain constructs to be present in its environment. This
          section explores what those constructs are and how Wine sets them
          up. The lack of this environment is one thing that makes it hard to
          use Wine code directly from standard Linux applications - in order to
          interact with Win32 code a thread must first be
          "adopted" by Wine.
        </para>

        <para>
          The first thing Win32 code requires is the
          <emphasis>TEB</emphasis> or "Thread Environment Block". This is an
          internal (undocumented) Windows structure associated with every thread
          which stores a variety of things such as TLS slots, a pointer to the
          threads message queue, the last error code and so on. You can see the
          definition of the TEB in <filename>include/thread.h</filename>, or at
          least what we know of it so far. Being internal and subject to change,
          the layout of the TEB has had to be reverse engineered from scratch.
        </para>

        <para>
          A pointer to the TEB is stored in the %fs register and can be accessed
          using <function>NtCurrentTeb()</function> from within Wine code. %fs
          actually stores a selector, and setting it therefore requires
          modifying the processes local descriptor table (LDT) - the code to do
          this is in <filename>lib/wine/ldt.c</filename>. 
        </para>

        <para>
          The TEB is required by nearly all Win32 code run in the Wine
          environment, as any <command>wineserver</command> RPC will use it,
          which in turn implies that any code which could possibly block for
          instance by using a critical section) needs it. The TEB also holds the
          SEH exception handler chain as the first element, so if disassembling
          you see code like this:
        </para>

        <programlisting>movl %esp, %fs:0</programlisting>

        <para>
          ... then you are seeing the program set up an SEH handler frame. All
          threads must have at least one SEH entry, which normally points to the
          backstop handler which is ultimately responsible for popping up the
          all-too-familiar  This program has performed an illegal operation and
          will be terminated" message. On Wine we just drop straight into the
          debugger. A full description of SEH is out of the scope of this
          section, however there are some good articles in MSJ if you are
          interested.
        </para>

        <para>
          All Win32-aware threads must have a <command>wineserver</command>
          connection. Many  different APIs require the ability to communicate
          with the <command>wineserver</command>. In turn, the
          <command>wineserver</command> must be aware of Win32 threads in order
          to be able to accurately report information to other parts of the
          program and do things like route inter-thread messages, dispatch APCs
          (asynchronous procedure calls) and so on. Therefore a part of thread
          initialization is initializing the thread server-side. The result is
          not only correct information in the server, but a set of file
          descriptors the thread can use to communicate with the server - the
          request fd, reply fd and wait fd (used for blocking).
        </para>
      </sect2>
    </sect1>

    <sect1 id="seh">
      <title>Structured Exception Handling</title>
      
      <para>
        Structured Exception Handling (or SEH) is an implementation of
        exceptions inside the Windows core. It allows code written in different
        languages to throw exceptions across DLL boundaries, and Windows reports
        various errors like access violations by throwing them. This section
        looks at how it works, and how it's implemented in Wine.
      </para>

      <sect2>
        <title>How SEH works</title>

        <para>
          SEH is based on embedding
          <structname>EXCEPTION_REGISTRATION_RECORD</structname> structures in
          the stack. Together they form a linked list rooted at offset zero in
          the TEB (see the threading section if you don't know what this is). A
          registration record points to a handler function, and when an
          exception is thrown the handlers are executed in turn. Each handler
          returns a code, and they can elect to either continue through the
          handler chain or it can handle the exception and then restart the
          program. This is referred to as unwinding the stack. After each
          handler is called it's popped off the chain.
        </para>

        <para>
          Before the system begins unwinding the stack, it runs vectored
          handlers. This is an extension to SEH available in Windows XP, and
          allows registered functions to get a first chance to watch or deal
          with any exceptions thrown in the entire program, from any thread.
        </para>

        <para>
          A thrown exception is represented by an
          <structname>EXCEPTION_RECORD</structname> structure. It consists of a
          code, flags, an address and an arbitrary number of <type>DWORD</type>
          parameters. Language runtimes can use these parameters to associate
          language-specific information with the exception.
        </para>
        
        <para>
          Exceptions can be triggered by many things. They can be thrown
          explicitly by using the RaiseException API, or they can be triggered
          by a crash (ie, translated from a signal). They may be used internally
          by a language runtime to implement language-specific exceptions. They
          can also be thrown across DCOM connections.
        </para>

        <para>
          Visual C++ has various extensions to SEH which it uses to implement,
          eg, object destruction on stack unwind as well as the ability to throw
          arbitrary types. The code for this is in
          <filename>dlls/msvcrt/except.c</filename>
        </para>

      </sect2>
      
      <sect2>
        <title>Translating signals to exceptions</title>

        <para>
          In Windows, compilers are expected to use the system exception
          interface, and the kernel itself uses the same interface to
          dynamically insert exceptions into a running program. By contrast on
          Linux the exception ABI is implemented at the compiler level
          (inside GCC and the linker) and the kernel tells a thread of
          exceptional events by sending <emphasis>signals</emphasis>. The
          language runtime may or may not translate these signals into native
          exceptions, but whatever happens the kernel does not care.
        </para>

        <para>
          You may think that if an app crashes, it's game over and it really
          shouldn't matter how Wine handles this. It's what you might
          intuitively guess, but you'd be wrong. In fact some Windows programs
          expect to be able to crash themselves and recover later without the
          user noticing, some contain buggy binary-only components from third
          parties and use SEH to swallow crashes, and still others execute
          priviledged (kernel-level) instructions and expect it to work. In
          fact, at least one set of APIs (the <function>IsBad*Ptr()</function>
          series) can only be implemented by performing an operation that may
          crash and returning <constant>TRUE</constant> if it does, and
          <constant>FALSE</constant> if it doesn't! So, Wine needs to not only
          implement the SEH infrastructure but also translate Unix signals into
          SEH exceptions.
        </para>

        <para>
          The code to translate signals into exceptions is a part of
          <filename>NTDLL</filename>, and can be found in
          <filename>dlls/ntdll/signal_i386.c</filename>. This file sets up
          handlers for various signals during Wine startup, and for the ones
          that indicate exceptional conditions translates them into
          exceptions. Some signals are used by Wine internally and have nothing
          to do with SEH.
        </para>

        <para>
          Signal handlers in Wine run on their own stack. Each thread has its
          own signal stack which resides 4k after the TEB. This is important for
          a couple of reasons. Firstly, because there's no guarantee that the
          app thread which triggered the signal has enough stack space for the
          Wine signal handling code. In Windows, if a thread hits the limits of
          its stack it triggers a fault on the stack guard page. The language
          runtime can use this to grow the stack if it wants to.

          <!-- fixme: is it really the language runtime that does this? i
               can't find any code in Wine to reallocate the stack on
               STATUS_GUARD_PAGE_VIOLATION -->

          However, because a guard page violation is just a regular segfault to
          the kernel, that would lead to a nested signal handler and that gets
          messy really quick so we disallow that in Wine. Secondly, setting up
          the exception to throw requires modifying the stack of the thread
          which triggered it, which is quite hard to do when you're still
          running on it.
        </para>

        <para>
          Windows exceptions typically contain more information than the Unix
          standard APIs provide. For instance, a
          <constant>STATUS_ACCESS_VIOLATION</constant> exception
          (<constant>0xC0000005</constant>) structure contains the faulting
          address, whereas a standard Unix <constant>SIGSEGV</constant> just
          tells the app that it crashed. Usually this information is passed as
          an extra parameter to the signal handler, however its location and
          contents vary between kernels (BSD, Solaris, etc). This data is
          provided in a <structname>SIGCONTEXT</structname> structure, and on
          entry to the signal handler it contains the register state of the CPU
          before the signal was sent. Modifying it will cause the kernel to
          adjust the context before restarting the thread.
        </para>
      </sect2>
    </sect1>

    <sect1>
      <title>File management</title>
      <para>
        With time, Windows API comes closer to the old Unix paradigm "Everything
        is a file".  Therefore, this whole section dedicated to file management
        will cover firstly the file management, but also some other objects like
        directories, and even devices, which are manipulated in Windows in a
        rather coherent way.  We'll see later on some other objects fitting
        (more or less) in this picture (pipes or consoles to name a few).
      </para>

      <para>
        First of all, Wine, while implementing the file interface from Windows,
        needs to maps a file name (expressed in the Windows world) onto a file
        name in the Unix world.  This encompasses several aspects: how to map
        the file names, how to map access rights (both on files and
        directories), how to map physical devices (hardisks, but also other
        devices - like serial or parallel interfaces - and even VxDs).
      </para>

      <sect2>
        <title>Various Windows formats for file names</title>
        <para>
          Let's first review a bit the various forms Windows uses when it comes
          to file names.
        </para>
        
        <sect3>
          <title>The DOS inheritance</title>

          <para>
            At the beginning was DOS, where each file has to sit on a drive,
            called from a single letter.  For separating device names from
            directory or file names, a ':' was appended to this single letter,
            hence giving the (in)-famous <filename>C:</filename> drive
            designations.  Another great invention was to use some fixed names
            for accessing devices: not only where these named fixed, in a way
            you couldn't change the name if you'd wish to, but also, they were
            insensible to the location where you were using them.  For example,
            it's well known that <filename>COM1</filename> designates the first
            serial port, but it's also true that
            <filename>c:\foo\bar\com1</filename> also designates the first
            serial port.  It's still true today: on XP, you still cannot name a
            file <filename>COM1</filename>, whatever the directory!!!
          </para>
          <para>
            Well later on (with Windows 95), Microsoft decided to overcome some
            little details in file names: this included being able to get out of
            the 8+3 format (8 letters for the name, 3 letters for the
            extension), and so being able to use "long names" (that's the
            "official" naming; as you can guess, the 8+3 format is a short
            name), and also to use very strange characters in a file name (like
            a space, or even a '.').  You could then name a file
            <filename>My File V0.1.txt</filename>, instead of
            <filename>myfile01.txt</filename>.  Just to keep on the fun side of
            things, for many years the format used on the disk itself for
            storing the names has been the short name as the real one and to use
            some tricky aliasing techniques to store the long name.  When some
            newer disk file systems have been introduced (NTFS with NT), in
            replacement of the old FAT system (which had little evolved since
            the first days of DOS), the long name became the real name while the
            short name took the alias role.
          </para>
          <para>
            Windows also started to support mounting network shares, and see
            them as they were a local disk (through a specific drive letter).
            The way it has been done changed along the years, so we won't go
            into all the details (especially on the DOS and Win9x side).
          </para>
        </sect3>

        <sect3>
          <title>The NT way</title>
          <para>
            The introduction of NT allowed a deep change in the ways DOS had
            been handling devices:
            <itemizedlist>
              <listitem>
                <para>
                  There's no longer a forest of DOS drive letters (even if the
                  <command>assign</command> was a way to create symbolic links
                  in the forest), but a single hierarchical space.
                </para>
              </listitem>
              <listitem>
                <para>
                  This hierarchy includes several distinct elements.  For
                  example, <filename>\Device\Hardisk0\Partition0</filename>
                  refers to the first partition on the first physical hard disk
                  of the system.
                </para>
              </listitem>
              <listitem>
                <para>
                  This hierarchy covers way more than just the files and drives
                  related objects, but most of the objects in the system.  We'll
                  only cover here the file related part.
                </para>
              </listitem>
              <listitem>
                <para>
                  This hierarchy is not directly accessible for the Win32 API,
                  but only the <filename>NTDLL</filename> API.  The Win32 API
                  only allows to manipulate part of this hierarchy (the rest
                  being hidden from the Win32 API). Of course, the part you see
                  from Win32 API looks very similar to the one that DOS
                  provided.
                </para>
              </listitem>
              <listitem>
                <para>
                  Mounting a disk is performed by creating a symbol link in this
                  hierarchy from <filename>\Global??\C:</filename> (the name
                  seen from the Win32 API) to
                  <filename>\Device\Harddiskvolume1</filename> which determines
                  the partition on a physical disk where C: is going to be seen.
                </para>
              </listitem>
              <listitem>
                <para>
                  Network shares are also accessible through a symbol link.
                  However in this case, a symbol link is created from 
                  <filename>\Global??\UNC\host\share\</filename> for the share
                  <filename>share</filename> on the machine
                  <filename>host</filename>) to what's called a network
                  redirector, and which will take care of 1/ the connection to
                  the remote share, 2/ handling with that remote share the rest
                  of the path (after the name of the server, and the name of the
                  share on that server).
                </para>
                <note>
                  <para>
                    In NT naming convention, <filename>\Global??</filename> can
                    also be called <filename>\??</filename> to shorten the
                    access.
                  </para>
                </note>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            All of these things, make the NT system pretty much more flexible
            (you can add new types of filesystems if you want), you provide a
            unique name space for all objects, and most operations boil down to
            creating relationship between different objects.
          </para>
        </sect3>

        <sect3>
          <title>Wrap up</title>
          <para>
            Let's end this chapter about files in Windows with a review of the
            different formats used for file names:
            <itemizedlist>
              <listitem>
                <para><filename>c:\foo\bar</filename> is a full path.</para>
              </listitem>
              <listitem>
                <para>
                  <filename>\foo\bar</filename> is an absolute path; the full
                  path is created by appending the default drive (ie. the drive
                  of the current directory).
                </para>
              </listitem>
              <listitem>
                <para>
                  <filename>bar</filename> is a relative path; the full path is
                  created by adding the current directory.
                </para>
              </listitem>
              <listitem>
                <para>
                  <filename>c:bar</filename> is a drive relative path.  Note
                  that the case where <filename>c:</filename> is the drive of
                  the current directory is rather easy; it's implemented the
                  same way as the case just below (relative path). In the rest
                  of this chapter, drive relative path will only cover the case
                  where the drive in the path isn't the drive of the default
                  directory.  The resolution of this to a full pathname defers
                  according to the version of Windows, and some parameters.
                  Let's take some time browsing through these issues.  On
                  Windows 9x (as well as on DOS), the system maintains a process
                  wide set of default directories per drive.  Hence, in this
                  case, it will resolve <filename>c:bar</filename> to the
                  default directory on drive <filename>c:</filename> plus file
                  <filename>bar</filename>.  Of course, the default per drive
                  directory is updated each time a new current directory is set
                  (only the current directory of the drive specified is
                  modified).  On Windows NT, things differ a bit.  Since NT
                  implements a namespace for file closer to a single tree
                  (instead of 26 drives), having a current directory per drive
                  is a bit ackward.  Hence, Windows NT default behavior is to
                  have only one current directory across all drives (in fact, a
                  current directory expressed in the global tree) - this
                  directory is of course related to a given process -,
                  <filename>c:bar</filename> is resolved this way:
                  <itemizedlist>
                    <listitem>
                      <para>
                        If <filename>c:</filename> is the drive of the default
                        directory, the final path is the current directory plus
                        <filename>bar</filename>.
                      </para>
                    </listitem>
                    <listitem>
                      <para>
                        Otherwise it's resolved into
                        <filename>c:\bar</filename>.
                      </para>
                    </listitem>
                    <listitem>
                      <para>
                        In order to bridge the gap between the two
                        implementations (Windows 9x and NT), NT adds a bit of
                        complexity on the second case.  If the
                        <envar>=C:</envar> environment variable is defined, then
                        it's value is used as a default directory for drive
                        <filename>C:</filename>.  This is handy, for example,
                        when writing a DOS shell, where having a current drive
                        per drive is still implemented, even on NT.  This
                        mechanism (through environment variables) is implemented
                        on <command>CMD.EXE</command>, where those variables are
                        set when you change directories with the
                        <command>cd</command>.  Since environment variables are
                        inherited at process creation, the current directories
                        settings are inherited by child processes, hence
                        mimicing the behavior of the old DOS shell.  There's no
                        mechanism (in <filename>NTDLL</filename> or
                        <filename>KERNEL32</filename>) to set up, when current
                        directory changes, the relevant environment variables.
                        This behavior is clearly band-aid, not a full featured
                        extension of current directory behavior.
                      </para>
                    </listitem>
                  </itemizedlist>
                  Wine fully implements all those behaviors (the Windows 9x vs
                  NT ones are triggered by the version flag in Wine).
                </para>
              </listitem>
              <listitem>
                <para>
                  <filename>\\host\share</filename> is <firstterm>UNC</firstterm>
                  (Universal Naming Convention) path, ie. represents a file on a
                  remote share.
                </para>
              </listitem>
              <listitem>
                <para>
                  <filename>\\.\device</filename> denotes a physical device
                  installed in the system (as seen from the Win32 subsystem).  A
                  standard NT system will map it to the
                  <filename>\??\device</filename> NT path.  Then, as a standard
                  configuration, <filename>\??\device</filename> is likely to be
                  a link to in a physical device described and hooked into the
                  <filename>\Device\</filename> tree.  For example,
                  <filename>COM1</filename> is a link to
                  <filename>\Device\Serial0</filename>.
                </para>
              </listitem>
              <listitem>
                <para>
                  On some versions of Windows, paths were limited to
                  <constant>MAX_PATH</constant> characters.  To circumvent this,
                  Microsoft allowed paths to be <constant>32,767</constant>
                  characters long, under the conditions that the path is
                  expressed in Unicode (no Ansi version), and that the path is
                  prefixed with <filename>\\?\</filename>.  This convention is
                  applicable to any of the cases described above.
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            To summarize, what we've discussed so, let's put everything into a
            single table...
            <table>
              <title>DOS, Win32 and NT paths equivalences</title>
              <tgroup cols="3" align="left">
                <thead>
                  <row>
                    <entry>Type of path</entry>
                    <entry>Win32 example</entry>
                    <entry>NT equivalent</entry>
                    <entry>Rule to construct</entry>
                  </row>
                </thead>
                <tbody>
                  <row>
                    <entry>Full path</entry>
                    <entry><filename>c:\foo\bar.txt</filename></entry>
                    <entry><filename>\Global??\C:\foo\bar.txt</filename></entry>
                    <entry>Simple concatenation</entry>
                  </row>
                  <row>
                    <entry>Absolute path</entry>
                    <entry><filename>\foo\bar.txt</filename></entry>
                    <entry><filename>\Global??\J:\foo\bar.txt</filename></entry>
                    <entry>
                      Simple concatenation using the drive of the default
                      directory (here J:)
                    </entry>
                  </row>
                  <row>
                    <entry>Relative path</entry>
                    <entry><filename>gee\bar.txt</filename></entry>
                    <entry>
                      <filename>
                        \Global??\J:\mydir\mysubdir\gee\bar.txt
                      </filename>
                    </entry>
                    <entry>
                      Simple concatenation using the default directory
                      (here <filename>J:\mydir\mysubdir</filename>)
                    </entry>
                  </row>
                  <row>
                    <entry>Drive relative path</entry>
                    <entry><filename>j:gee\bar.txt</filename></entry>
                    <entry>
                      <msgtext>
                        <para>
                          <itemizedlist>
                            <listitem>
                              <para>
                                On Windows 9x (and DOS),
                                <filename>J:\toto\gee\bar.txt</filename>.
                              </para>
                            </listitem>
                            <listitem>
                              <para>
                                On Windows NT,
                                <filename>J:\gee\bar.txt</filename>.
                              </para>
                            </listitem>
                            <listitem>
                              <para>
                                On Windows NT,
                                <filename>J:\tata\titi\bar.txt</filename>.
                              </para>
                            </listitem>
                          </itemizedlist>
                        </para>
                      </msgtext>
                    </entry>
                    <entry>
                    <msgtext>
                        <para>
                          <itemizedlist>
                            <listitem>
                              <para>
                                On Windows NT (and DOS),
                                <filename>\toto</filename> is the default
                                directory on drive <filename>J:</filename>.
                              </para>
                            </listitem>
                            <listitem>
                              <para>
                                On Windows NT, if <envar>=J:</envar> isn't set.
                              </para>
                            </listitem>
                            <listitem>
                              <para>
                                On Windows NT, if <envar>=J:</envar> is set to
                                <filename>J:\tata\titi</filename>.
                              </para>
                            </listitem>
                          </itemizedlist>
                        </para>
                      </msgtext>
                    </entry>
                  </row>
                  <row>
                    <entry>UNC (Uniform Naming Convention) path</entry>
                    <entry><filename>\\host\share\foo\bar.txt</filename></entry>
                    <entry>
                      <filename>\Global??\UNC\host\share\foo\bar.txt</filename>
                    </entry>
                    <entry>
                      Simple concatenation.
                    </entry>
                  </row>
                  <row>
                    <entry>Device path</entry>
                    <entry><filename>\\.\device</filename></entry>
                    <entry><filename>\Global??\device</filename></entry>
                    <entry>Simple concatenation</entry>
                  </row>
                  <row>
                    <entry>Long paths</entry>
                    <entry><filename>\\?\...</filename></entry>
                    <entry></entry>
                    <entry>
                      With this prefix, paths can take up to
                      <constant>32,767</constant> characters, instead of
                      <constant>MAX_PATH</constant> for all the others).  Once
                      the prefix stripped, to be handled like one of the
                      previous ones, just providing internal buffers large
                      enough).
                    </entry>
                  </row>
                </tbody>
              </tgroup>
            </table>
          </para>
        </sect3>
      </sect2>
      
      <sect2>
        <title>Wine implementation</title>
        <para>
          We'll mainly cover in this section the way Wine opens a file (in the
          Unix sense) when given a Windows file name.  This will include mapping
          the Windows path onto a Unix path (including the devices case),
          handling the access rights, the sharing attribute if any...  
        </para>
        <sect3>
          <title>Mapping a Windows path into an absolute Windows path</title>
          <para>
            First of all, we described in previous section the way to convert
            any path in an absolute path.  Wine implements all the previous algorithms
            in order to achieve this.  Note also, that this transformation is
            done with information local to the process (default directory,
            environment variables...).  We'll assume in the rest of this section
            that all paths have now been transformed into absolute from.
          </para>
        </sect3>
        <sect3>
          <title>Mapping a Windows (absolute) path onto a Unix path</title>
          <para>
            When Wine is requested to map a path name (in DOS form, with a drive
            letter, e.g. <filename>c:\foo\bar\myfile.txt</filename>), Wine
            converts this into the following Unix path
            <filename>$(WINEPREFIX)/dosdevices/c:/foo/bar/myfile.txt</filename>.
            The Wine configuration process is responsible for setting
            <filename>$(WINEPREFIX)/dosdevices/c:</filename> to be a symbolic
            link pointing to the directory in Unix hierarchy the user wants to
            expose as the <filename>C:</filename> drive in the DOS forest of
            drives.  
          </para>
          <para>
            This scheme allows:
            <itemizedlist>
              <listitem>
                <para>
                  a very simple algorithm to map a DOS path name into a Unix one
                  (no need of Wine server calls) 
                </para>
              </listitem>
              <listitem>
                <para>
                  a very configurable implementation: it's very easy to change a
                  drive mapping 
                </para>
              </listitem>
              <listitem>
                <para>
                  a rather readable configuration: no need of sophisticated
                  tools to read a drive mapping, a <command>ls -l
                   $(WINEPREFIX)/dosdevices</command> 
                  says it all.
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            This scheme is also used to implement UNC path names.  For example,
            Wine maps <filename>\\host\share\foo\bar\MyRemoteFile.txt</filename>
            into 
            <filename>$(WINEPREFIX)/dosdevices/unc/host/share/foo/bar/MyRemoteFile.txt</filename>.
            It's then up to the user to decide where
            <filename>$(WINEPREFIX)/dosdevices/unc/host/share</filename> shall
            point to (or be).  For example, it can either be a symbolic link to a
            directory inside the local machine (just for emulation purpose), or
            a symbolic link to the mount point of a remote disk (done through
            Samba or NFS), or even the real mount point.  Wine will not do any
            checking here, nor will help in actually mounting the remote drive.
          </para>
          <para>
            We've seen how Wine maps a drive letter or a UNC path onto the Unix
            hierarchy, we now have to look on a the filename is searched within
            this hierarchy.  The main issue is about case sensivity.  Here's a
            reminder of the various properties for the file systems in the
            field.
            <table>
              <title>File systems' properties</title>
              <tgroup cols="4" align="left">
                <thead>
                  <row>
                    <entry>FS Name</entry>
                    <entry>Length of elements</entry>
                    <entry>Case sensitivity (on disk)</entry>
                    <entry>Case sensitivity for lookup</entry>
                  </row>
                </thead>
                <tbody>
                  <row>
                    <entry>FAT, FAT16 or FAT32</entry>
                    <entry>Short name (8+3)</entry>
                    <entry>Names are always stored in upper-case</entry> 
                    <entry>Case insensitive</entry>
                  </row>
                  <row>
                    <entry>VFAT</entry>
                    <entry>Short name (8+3) + alias on long name</entry> 
                    <entry>
                      Short names are always stored in upper-case.  Long names
                      are stored with case preservation.
                    </entry>
                    <entry>Case insensitive</entry>
                  </row>
                  <row>
                    <entry>NTFS</entry>
                    <entry>Long name + alias on short name (8+3).</entry>
                    <entry>
                      Long names are stored with case preservation.  Short names
                      are always stored in upper-case.
                    </entry>
                    <entry>Case insentivite</entry>
                  </row>
                  <row>
                    <entry>Linux FS (ext2fs, ext3fs, reiserfs...)</entry>
                    <entry>Long name</entry>
                    <entry>Case preserving</entry>
                    <entry>Case sensitive</entry>
                  </row>
                </tbody>
              </tgroup>
            </table>
          </para>
          <para>
            <note>
              <title>Case sensitivity vs. preservation</title>
              <para>
                When we say that most systems in NT are case insensitive, this
                has to be understood for looking up for a file, where the
                matches are made in a case insensitive mode.  This is different
                from VFAT or NTFS "case preservation" mechanism, which stores
                the file names as they are given when creating the file, while
                doing case insensitive matches.
              </para>
            </note>
            Since most file systems used in NT are case insensitive and since
            most Unix file systems are case sensitive, Wine undergo a case
            insensitive search when it has found the Unix path is has to look
            for.  This means, for example, that for opening the
            <filename>$(WINEPREFIX)/dosdevices/c:/foo/bar/myfile.txt</filename>,
            Wine will recursively open all directories in the path, and check,
            in this order, for the existence of the directory entry in the form
            given in the file name (ie. case sensitive), and if it's not found,
            in a case insensitive form.  This allows to also pass, in most Win32
            file API also a Unix path (instead of a DOS or NT path), but we'll
            come back to this later.  This also means that the algorithm
            described doesn't correctly handle the case of two files in the same
            directory, which names only differ on the case of the letters.  This
            means, that if, in the same directory, two files (which names match
            in a case sensitive comparison), Wine will pick-up the right one if
            the filename given matches on of the name (in a case sensitive way),
            but will pickup one of the two (without defining the one it's going
            to pickup) if the filename given matches none of the two names in a
            case sensitive way (but in a case insensitive way).  For example, if
            the two filenames are <filename>my_neat_file.txt</filename> and
            <filename>My_Neat_File.txt</filename>, Wine's behavior when opening
            <filename>MY_neat_FILE.txt</filename> is undefined.
          </para>
          <para>
            As Windows, at the early days, didn't support the notion of symbolic
            links on directories, lots of applications (and some old native
            DLLs) are not ready for this feature.  Mainly, they imply that the
            directory structure is a tree, which has lots of consequences on
            navigating in the forest of directories (ie: there cannot be two
            ways for going from directory to another, there cannot be
            cycles...).  In order to prevent some bad behavior for such
            applications, Wine sets up an option.  By default, symbolic links on
            directories are not followed by Wine.  There's an options to follow
            them (see the Wine User Guide), but this could be harmful.
          </para>
          <para>
            Wine considers that Unix file names <emphasis>are</emphasis> long
            filename.  This seems a reasonable approach; this is also the
            approach followed by most of the Unix OSes while mounting Windows
            partitions (with filesystems like FAT, FAT32 or NTFS).  Therefore,
            Wine tries to support short names the best it can.  Basically, they
            are two options:
            <itemizedlist>
              <listitem>
                <para>
                  The filesystem on which the inspected directory lies in a real
                  Windows FS (like FAT, or FAT32, or NTFS) and the OS has
                  support to access the short filename (for example, Linux does
                  this on FAT, FAT32 or VFAT).  In this case, Wine makes full use
                  of this information and really mimics the Windows behavior:
                  the short filename used for any file is the same than on
                  Windows.
                </para>
              </listitem>
              <listitem>
                <para>
                  If conditions listed above are not met (either, FS has no
                  physical short name support, or OS doesn't provide the access
                  access to the short name), Wine decides and computes on its
                  own the short filename for a given long filename.  We cannot
                  ensure that the generated short name is the same than on
                  Windows (because the algorithm on Windows takes into account
                  the order of creation of files, which cannot be implemented in
                  Wine: Wine would have to cache the short names of every
                  directory it uses!).  The short name is made up of part of the
                  long name (first characters) and the rest with a hashed
                  value.  This has several advantages:
                  <itemizedlist>
                    <listitem>
                      <para>
                        The algorithm is rather simple and low cost.
                      </para>
                    </listitem>
                    <listitem>
                      <para>
                        The algorithm is stateless (doesn't depend of the other
                        files in the directory).
                      </para>
                    </listitem>
                  </itemizedlist>
                  But, it also has the drawbacks (of the advantages):
                  <itemizedlist>
                    <listitem>
                      <para>
                        The algorithm isn't the same as on Windows, which means
                        a program cannot use short names generated on
                        Windows.  This could happen when copying an existing
                        installed program from Windows (for example, on a dual
                        boot machine).
                      </para>
                    </listitem>
                    <listitem>
                      <para>
                        Two long file names can end up with the same short name
                        (Windows handles the collision in this case, while Wine
                        doesn't).  We rely on our hash algorithm to lower at most
                        this possibility (even if it exists).
                      </para>
                    </listitem>
                  </itemizedlist>
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            Wine also allows in most file API to give as a parameter a full Unix
            path name.  This is handy when running a Wine (or Winelib) program
            from the command line, and one doesn't need to convert the path into
            the Windows form.  However, Wine checks that the Unix path given can
            be accessed from one of the defined drives, insuring that only part
            of the Unix <filename>/</filename> hierarchy can be accessed.
          </para>
          <para>
            As a side note, as Unix doesn't widely provide a Unicode interface
            to the filenames, and that Windows implements filenames as Unicode
            strings (even on the physical layer with NTFS, the FATs variant are
            ANSI), we need to properly map between the two.  At startup, Wine
            defines what's called the Unix Code Page, that's is the code page
            the Unix kernel uses as a reference for the strings.  Then Wine uses
            this code page for all the mappings it has to do between a Unicode
            path (on the Windows side) and a Ansi path to be used in a Unix path
            API.  Note, that this will work as long as a disk isn't mounted with
            a different code page than the one the kernel uses as a default.
          </para>
          <para>
            We describe below how Windows devices are mapped to Unix devices.  
            Before that, let's finish the pure file round-up with some basic
            operations.
          </para>
        </sect3>
        <sect3>
          <title>Access rights and file attributes</title>
          <para>
            Now that we have looked how Wine converts a Windows pathname into a
            Unix one, we need to cover the various meta-data attached to a file
            or a directory.
          </para>
          <para>
            In Windows, access rights are simplistic: a file can be read-only or
            read-write.  Wine sets the read-only flag if the file doesn't have
            the Unix user-write flag set.  As a matter of fact, there's no way
            Wine can return that a file cannot be read (that doesn't exist under
            Windows).  The file will be seen, but trying to open it will return
            an error.  The Unix exec-flag is never reported.  Wine doesn't use
            this information to allow/forbid running a new process (as Unix does
            with the exec-flag).  Last but not least: hidden files.  This exists
            on Windows but not really on Unix! To be exact, in Windows, the
            hidden flag is a metadata associated to any file or directoy; in
            Unix, it's a convention based on the syntax of the file name
            (whether it starts with a '.' or not).  Wine implements two behaviors
            (chosen by configuration).  This impacts file names and directory
            names starting by a '.'.  In first mode
            (<option>ShowDotFile</option> is <constant>FALSE</constant>), every 
            file or directory starting by '.' is returned with the hidden flag
            turned on.  This is the natural behavior on Unix (for
            <command>ls</command> or even file explorer).  In the second mode
            (<option>ShowDotFile</option> is <constant>TRUE</constant>), Wine
            never sets the hidden flag, hence every file will be seen.
          </para>
          <para>
            Last but not least, before opening a file, Windows makes use of
            sharing attributes in order to check whether the file can be opened;
            for example, a process, being the first in the system to open a
            given file, could forbid, while it maintains the file opened, that
            another process opens it for write access, whereas open for read
            access would be granted.  This is fully supported in Wine by moving
            all those checks in the Wine server for a global view on the system.
            Note also that what's moved in the Wine server is the check, when
            the file is opened, to implement the Windows sharing semantics.
            Further operation on the file (like reading and writing) will not
            require heavy support from the server.
          </para>
          <para>
            The other good reason for putting the code for actually opening a
            file in the server is that an opened files in Windows is managed
            through a handle, and handles can only be created in Wine server!
          </para>
          <para>
            Just a note about attributes on directories: while we can easily map
            the meaning of Windows' <constant>FILE_ATTRIBUTE_READONLY</constant>
            on a file, we cannot do it for a directory.  Windows' semantic (when
            this flag is set) means do not delete the directory, while the
            <constant>w</constant> attribute in Unix means don't write nor
            delete it.  Therefore, Wine uses an asymetric mapping here: if the
            directory (in Unix) isn't writable, then Wine reports the
            <constant>FILE_ATTRIBUTE_READONLY</constant> attribute; on the other
            way around, when asked to set a directory with
            <constant>FILE_ATTRIBUTE_READONLY</constant> attribute, Wine simply
            does nothing.
          </para>
        </sect3>
        <sect3>
          <title>Operations on file</title>
          <sect4>
            <title>Reading and writing</title>
            <para>
              Reading and writing are the basic operations on files.  Wine of
              course implements this, and bases the implementation on client
              side calls to Unix equivalents (like <function>read()</function>
              or <function>write()</function>).  Note, that the Wine server is
              involved in any read or write operation, as Wine needs to
              transform the Windows-handle to the file into a Unix file
              descriptor it can pass to any Unix file function.
            </para>
          </sect4>
          <sect4>
            <title>Getting a Unix fd</title>
            <para>
              This is major operation in any file related operation.  Basically,
              each file opened (at the Windows level), is first opened in the
              Wine server, where the fd is stored.  Then, Wine (on client side)
              uses <function>recvmsg()</function> to pass the fd from the wine
              server process to the client process.  Since this operation could
              be lengthy, Wine implement some kind of cache mechanism to send it
              only once, but getting a fd from a handle on a file (or any other
              Unix object which can be manipulated through a file descriptor)
              still requires a round trip to the Wine server.
            </para>
          </sect4>
          <sect4>
            <title>Locking</title>
            <para>
              Windows provides file locking capabilities.  When a lock is set
              (and a lock can be set on any contiguous range in a file), it
              controls how other processes in the system will have access to the
              range in the file.  Since locking range on a file are defined on a
              system wide manner, its implementation resides in
              <command>wineserver</command>.  It tries to make use Unix file
              locking (if the underlying OS and the mounted disk where the file
              sits support this feature) with <function>fcntl()</function> and
              the <constant>F_SETLK</constant> command.  If this isn't
              supported, then <command>wineserver</command> just pretends it
              works. 
            </para>
          </sect4>
          <sect4>
            <title>I/O control</title>
            <para>
              There's no need (so far) to implement support (for files and
              directories) for <function>DeviceIoControl()</function>, even if
              this is supported by Windows, but for very specific needs
              (like compression management, or file system related information).
              This isn't the case for devices (including disks), but we'll cover
              this in the hereafter section related to devices.
            </para>
          </sect4>
          <sect4>
            <title>Buffering</title>
            <para>
              Wine doesn't do any buffering on file accesses but rely on the
              underlying Unix kernel for that (when possible). This scheme is
              needed because it's easier to implement multiple accesses on the
              same file at the kernel level, rather than at Wine levels.  Doing
              lots of small reads on the same file can turn into a performance
              hog, because each read operation needs a round trip to the server
              in order to get a file descriptor (see above).
            </para>
          </sect4>
          <sect4>
            <title>Overlapped I/O</title>
            <para>
              Windows introduced the notion of overlapped I/O.  Basically, it
              just means that an I/O operation (think read / write to start
              with) will not wait until it's completed, but rather return to the
              caller as soon as possible, and let the caller handle the wait
              operation and determine when the data is ready (for a read
              operation) or has been sent (for a write operation).  Note that the
              overlapped operation is linked to a specific thread.
            </para>
            <para>
              There are several interests to this: a server can handle several
              clients without requiring multi-threading techniques; you can
              handle an event driven model more easily (ie how to kill properly
              a server while waiting in the lengthy <function>read()</function>
              operation).
            </para>
            <para>
              Note that Microsoft's support for this feature evolved along the
              various versions of Windows.  For example, Windows 95 or 98 only
              supports overlapped I/O for serial and parallel ports, while NT
              supports also files, disks, sockets, pipes, or mailslots.
            </para>
            <para>
              Wine implements overlapped I/O operations.  This is mainly done by
              queueing in the server a request that will be triggered when
              something the current state changes (like data available for a
              read operation).  This readiness is signaled to the calling
              processing by queueing a specific APC, which will be called within
              the next waiting operation the thread will have.  This specific
              APC will then do the hard work of the I/O operation.  This scheme
              allows to put in place a wait mechanism, to attach a routine to be
              called (on the thread context) when the state changes, and to be
              done is a rather transparent manner (embedded any the generic wait
              operation).  However, it isn't 100% perfect.  As the heavy
              operations are done in the context of the calling threads, if
              those operations are lengthy, there will be an impact on the
              calling thread, especially its latency.  In order to provide an
              effective support for this overlapped I/O operations, we would
              need to rely on Unix kernel features (AIO is a good example).
            </para>
          </sect4>
        </sect3>
        <sect3>
          <title>Devices & volume management</title>
          <para>
            We've covered so far the ways file names are mapped into Unix
            paths.  There's still need to cover it for devices.  As a regular
            file, devices are manipulated in Windows with both read / write
            operations, but also control mechanisms (speed or parity of a serial
            line; volume name of a hard disk...).  Since, this is also supported
            in Linux, there's also a need to open (in a Unix sense) a device
            when given a Windows device name.  This section applies to DOS device
            names, which are seen in NT as nicknames to other devices.
          </para>
          <para>
            Firstly, Wine implements the Win32 to NT mapping as described above,
            hence every device path (in NT sense) is of the following form:
            <filename>/??/devicename</filename> (or
            <filename>/DosDevices/devicename</filename>).  As Windows device
            names are case insensitive, Wine also converts them to lower case
            before any operation.  Then, the first operation Wine tries is to
            check whether
            <filename>$(WINEPREFIX)/dosdevices/devicename</filename> exists.  If
            so, it's used as the final Unix path for the device.  The
            configuration process is in charge of creating for example, a
            symbolic link between
            <filename>$(WINEPREFIX)/dosdevices/PhysicalDrive0</filename> and
            <filename>/dev/hda0</filename>.  If such a link cannot be found, and
            the device name looks like a DOS disk name (like
            <filename>C:</filename>), Wine first tries to get the Unix device
            from the path <filename>$(WINEPREFIX)/dosdevices/c:</filename>
            (i.e. the device which is mounted on the target of the symbol link);
            if this doesn't give a Unix device, Wine tries whether
            <filename>$(WINEPREFIX)/dosdevices/c::</filename> exists.  If so,
            it's assumed to be a link to the actual Unix device.  For example,
            for a CD Rom, <filename>$(WINEPREFIX)/dosdevices/e::</filename>
            would be a symbolic link to <filename>/dev/cdrom</filename>.  If
            this doesn't exist (we're still handling the a device name of the
            <filename>C:</filename> form), Wine tries to get the Unix device
            from the system information (<filename>/etc/mtab</filename> and
            <filename>/etc/fstab</filename> on Linux).  We cannot apply this
            method in all the cases, because we have no insurance that the
            directory can actually be found.  One could have, for example, a CD
            Rom which he/she want only to use as audio CD player (ie never
            mounted), thus not having any information of the device itself.  If
            all of this doesn't work either, some basic operations are checked:
            if the devicename is <filename>NUL</filename>, then
            <filename>/dev/null</filename> is returned.  If the device name is a
            default serial name (<filename>COM1</filename> up to
            <filename>COM9</filename>) (resp.  printer name
            <filename>LPT1</filename> up to <filename>LPT9</filename>), then
            Wine tries to open the Nth serial (resp. printer) in the system.
            Otherwise, some basic old DOS name support is done
            <filename>AUX</filename> is transformed into
            <filename>COM1</filename> and <filename>PRN</filename> into
            <filename>LPT1</filename>), and the whole process is retried with
            those new names.
          </para>
          <para>
            To sum up:
            <table>
              <title>
                Mapping of Windows device names into Unix device names
              </title>
              <tgroup cols="3" align="left">
                <thead>
                  <row>
                    <entry>Windows device name</entry>
                    <entry>NT device name</entry>
                    <entry>Mapping to Unix device name</entry>
                  </row>
                </thead>
                <tbody>
                  <row>
                    <entry><filename>&lt;any_path&gt;AUX</filename></entry>
                    <entry<filename>>\Global??\AUX</filename></entry>
                    <entry>
                      Treated as an alias to <filename>COM1</filename>
                    </entry>
                  </row>
                  <row>
                    <entry><filename>&lt;any_path&gt;PRN</filename></entry>
                    <entry><filename>\Global??\PRN</filename></entry>
                    <entry>Treated as an alias to <filename>LPT1</filename></entry>
                  </row>
                  <row>
                    <entry><filename>&lt;any_path&gt;COM1</filename></entry>
                    <entry><filename>\Global??\COM1</filename></entry>
                    <entry>
                      <filename>$(WINEPREFIX)/dosdevices/com1</filename>
                      (if the symbol link exists) or the Nth serial
                      line in the system (on Linux,
                      <filename>/dev/ttyS0</filename>).
                    </entry>
                  </row>
                  <row>
                    <entry><filename>&lt;any_path&gt;LPT1</filename></entry>
                    <entry><filename>\Global??\LPT1</filename></entry>
                    <entry>
                      <filename>$(WINEPREFIX)/dosdevices/lpt1</filename>
                      (if the symbol link exists) or the Nth printer
                      in the system (on Linux,
                      <filename>/dev/lp0</filename>).
                    </entry>
                  </row>
                  <row>
                    <entry><filename>&lt;any_path&gt;NUL</filename></entry>
                    <entry><filename>\Global??\NUL</filename></entry>
                    <entry><filename>/dev/null</filename></entry>
                  </row>
                  <row>
                    <entry><filename>\\.\E:</filename></entry>
                    <entry><filename>\Global??\E:</filename></entry>
                    <entry>
                      <filename>$(WINEPREFIX)/dosdevices/e::</filename> (if the
                      symbolic link exists) or guessing the device from
                      <filename>/etc/mtab</filename> or
                      <filename>/etc/fstab</filename>.
                    </entry>
                  </row>
                  <row>
                    <entry><filename>\\.\&lt;device_name&gt;</filename></entry>
                    <entry>
                      <filename>\Global??\&lt;device_name&gt;</filename>
                    </entry>
                    <entry>
                      <filename>$(WINEPREFIX)/dosdevices/&lt;device_name&gt;</filename>
                      (if the symbol link exists).
                    </entry>
                  </row>
                </tbody>
              </tgroup>
            </table>
          </para>
          <para>
            Now that we know which Unix device to open for a given Windows
            device, let's cover the operation on it.  Those operations can either
            be read / write, io control (and even others).
          </para>
          <para>
            Read and write operations are supported on Real disks & CDROM
            devices, under several conditions:
            <itemizedlist>
              <listitem>
                <para>
                  Foremost, as the <function>ReadFile()</function> and
                  <function>WriteFile()</function> calls are mapped onto the
                  Unix <function>read()</function> and
                  <function>write()</function> calls, the user (from the Unix
                  perspective of the one running the Wine executable) must have
                  read (resp.  write) access to the device.  It wouldn't be wise
                  to let a user write directly to a hard disk!!!
                </para>
              </listitem>
              <listitem>
                <para>
                  Blocks' size for read and write but be of the size of a
                  physical block (generally 512 for a hard disk, depends on the
                  type of CD used), and offsets must also be a multiple of the
                  block size.
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            Wine also reads (if the first condition above about access rights is
            met) the volume information from a hard disk or a CD ROM to be
            displayed to a user.
          </para>
          <!--
          <para>
            Handling of old DOS devices (<filename>COMx</filename>,
            <filename>LPTx</filename>, <filename>NUL</filename>...)
          </para>
          -->
          <para>
            Wine also recognizes VxD as devices.  But those VxD must be the
            Wine builtin ones (Wine will never allow to load native VxD).  Those
            are configured with symbolic links in the
            <filename>$(WINEPREFIX)/dosdevices/</filename> directory, and point
            to the actual builtin DLL.  This DLL exports a single entry point,
            that Wine will use when a call to
            <function>DeviceIoControl</function> is made, with a handle opened
            to this VxD.  This allows to provide some kind of compatibility for
            old Win9x apps, still talking directly to VxD.  This is no longer
            supported on Windows NT, newest programs are less likely to make use
            of this feature, so we don't expect lots of development in this
            area, even though the framework is there and working.  Note also that
            Wine doesn't provide support for native VxDs (as a game, report how
            many times this information is written in the documentation;  as an
            advanced exercise, find how many more occurrences we need in order to
            stop questions whether it's possible or not).
          </para>
        </sect3>
      </sect2>
    </sect1>
    <sect1 id="ntdll">
      <title><filename>NTDLL</filename> module</title>
      <para>
        <filename>NTDLL</filename> provides most of the services you'd expect
        from a kernel. In lots of cases, <filename>KERNEL32</filename> APIs are
        just wrappers to <filename>NTDLL</filename> APIs. There are however,
        some difference in the APIs (the <filename>NTDLL</filename> ones have
        quite often a bit wider semantics than their
        <filename>KERNEL32</filename> counterparts). All the detailed functions
        we've described since the beginning of this chapter are in fact
        implemented in <filename>NTDLL</filename>, plus a great numbers of
        others we haven's written about yet.
      </para>
    </sect1>

    <sect1>
      <title><filename>KERNEL32</filename> Module</title>
      
      <para>
        As already explained, <filename>KERNEL32</filename> maps quite a few of
        its APIs to <filename>NTDLL</filename>. There are however a couple of
        things which are handled directly in
        <filename>KERNEL32</filename>. Let's cover a few of them...
      </para>
      <sect2 id="consoles">
        <title>Console</title>
        <sect3>
          <title>NT implementation</title>
          <para>
            Windows implements console solely in the Win32 subsystem.  Under NT,
            the real implementation uses a dedicated subsystem
            <filename>csrss.exe</filename> Client/Server Run-time SubSystem)
            which is in charge, amont other things, of animating the consoles.
            Animating includes for example handling several processes on the
            same console (write operations must be atomic, but also a character
            keyed on the console must be read by a single process), or sending
            some information back to the processes (changing the size or
            attributes of the console, closing the console).  Windows NT uses a
            dedicated (RPC based) protocol between each process being attached
            to a console and the <command>csrss.exe</command> subsystem, which
            is in charge of the UI of every console in the system.
          </para>
        </sect3>
        <sect3>
          <title>Wine implementation</title>
          <para>
            Wine tries to integrate as much as possible into the Unix consoles,
            but the overall situation isn't perfect yet.  Basically, Wine
            implements three kinds of consoles:
            <itemizedlist>
              <listitem>
                <para>
                  the first one is a direct mapping of the Unix console into the
                  Windows environment.  From the windows program point of view,
                  it won't run in a Windows console, but it will see its
                  standard input and output streams redirected to files; thoses
                  files are hooked into the Unix console's output and input
                  streams respectively.  This is handy for running programs from
                  a Unix command line (and use the result of the program as it
                  was a Unix programs), but it lacks all the semantics of the
                  Windows consoles.
                </para>
              </listitem>
              <listitem>
                <para>
                  the second and third ones are closer to the NT scheme, albeit
                  different from what NT does.  The <command>wineserver</command>
                  plays the role of the <filename>csrss.exe</filename> subsystem
                  (all requests are sent to it), and are then dispatched to a
                  dedicated wine process, called (surprise!)
                  <command>wineconsole</command> which manages the UI of the
                  console.  There is a running instance of
                  <command>wineconsole</command> for every console in the
                  system.  Two  flavors of this scheme are actually implemented:
                  they vary on the backend for the
                  <command>wineconsole</command>.  The first one,  dubbed
                  <constant>user</constant>, creates a real GUI window
                  (hence the USER name) and renders the console in this window.
                  The second one uses the <filename>(n)curses</filename> library
                  to take full control of an existing Unix console; of course,
                  interaction with other Unix programs will not be as smooth as
                  the first solution.
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            The following table describes the main implementation differences
            between the three approaches.
            <table>
              <title>Function consoles implementation comparison</title>
              <tgroup cols="4" align="left">
                <thead>
                  <row>
                    <entry>Function</entry>
                    <entry>Bare streams</entry>
                    <entry>
                      <command>Wineconsole</command> &amp; user backend
                    </entry>
                    <entry>
                      <command>Wineconsole</command> &amp; curses backend
                    </entry>
                  </row>
                </thead>
                <tbody>
                  <row>
                    <entry>
                      Console as a Win32 Object (and associated handles)
                    </entry>
                    <entry>
                      No specific Win32 object is used in this case.  The
                      handles manipulated for the standard Win32 streams are in
                      fact "bare handles" to their corresponding Unix streams.
                      The mode manipulation functions
                      (<function>GetConsoleMode()</function> /
                      <function>SetConsoleMode()</function>) are not supported.
                    </entry>
                    <entry>
                      Implemented in server, and a specific Winelib program
                      (<command>wineconsole</command>) is in charge of the
                      rendering and user input.  The mode manipulation functions
                      behave as expected.
                    </entry>
                    <entry>
                      Implemented in server, and a specific Winelib program
                      (<command>wineconsole</command>) is in charge of the
                      rendering and user input.  The mode manipulation functions
                      behave as expected.
                    </entry>
                  </row>
                  <row>
                    <entry>
                      Inheritance (including handling in
                      <function>CreateProcess()</function> of
                      <constant>CREATE_DETACHED</constant>,
                      <constant>CREATE_NEW_CONSOLE</constant> flags).
                    </entry>
                    <entry>
                      Not supported.  Every process child of a process will
                      inherit the Unix streams, so will also inherit the Win32
                      standard streams.
                    </entry>
                    <entry>
                      Fully supported (each new console creation will be handled
                      by the creation of a new <filename>USER32</filename> window)
                    </entry>
                    <entry>
                      Fully supported, except for the creation of a new console,
                      which will be rendered on the same Unix terminal as the
                      previous one, leading to unpredictable results.
                    </entry>
                  </row>
                  <row>
                    <entry>
                      <function>ReadFile()</function> / 
                      <function>WriteFile()</function> operations
                    </entry> 
                    <entry>Fully supported</entry>
                    <entry>Fully supported</entry>
                    <entry>Fully supported</entry>
                  </row>
                  <row>
                    <entry>
                      Screen-buffer manipulation (creation, deletion, resizing...)
                    </entry>
                    <entry>Not supported</entry>
                    <entry>Fully supported</entry>
                    <entry>
                      Partly supported (this won't work too well as we don't
                      control (so far) the size of underlying Unix terminal
                    </entry>
                  </row>
                  <row>
                    <entry>
                      APIs for reading/writing screen-buffer content, cursor position
                    </entry>
                    <entry>Not supported</entry>
                    <entry>Fully supported</entry>
                    <entry>Fully supported</entry>
                  </row>
                  <row>
                    <entry>APIs for manipulating the rendering window size</entry>
                    <entry>Not supported</entry>
                    <entry>Fully supported</entry>
                    <entry>
                      Partly supported (this won't work too well as we don't
                      control (so far) the size of underlying Unix terminal
                    </entry>
                  </row>
                  <row>
                    <entry>
                      Signaling (in particular, Ctrl-C handling)
                    </entry>
                    <entry>
                      Nothing is done, which means that Ctrl-C will generate (as
                      usual) a <constant>SIGINT</constant> which will terminate
                      the program.
                    </entry>
                    <entry>
                      Partly supported (Ctrl-C behaves as expected, however the
                      other Win32 CUI signaling isn't properly implemented).
                    </entry>
                    <entry>
                      Partly supported (Ctrl-C behaves as expected, however the
                      other Win32 CUI signaling isn't properly implemented).
                    </entry>
                  </row>
                </tbody>
              </tgroup>
            </table>
          </para>
          <para>
            The Win32 objects behind a console can be created in several
            occasions:
            <itemizedlist>
              <listitem>
                <para>
                  When the program is started from
                  <command>wineconsole</command>, a new console object is
                  created and will be used (inherited) by the process launched
                  from <command>wineconsole</command>.
                </para>
              </listitem>
              <listitem>
                <para>
                  When a program, which isn't attached to a console, calls
                  <function>AllocConsole()</function>, Wine then launches
                  <command>wineconsole</command>, and attaches the current
                  program to this console.  In this mode, the
                  <filename>USER32</filename> mode is always selected as Wine
                  cannot tell the current state of the Unix console.
                </para>
              </listitem>
            </itemizedlist>
          </para>
          <para>
            Please also note, that starting a child process with the
            <constant>CREATE_NEW_CONSOLE</constant> flag, will end-up calling
            <function>AllocConsole()</function> in the child process, hence
            creating a <command>wineconsole</command> with the
            <filename>USER32</filename> backend.
          </para>
          <para>
            Another interesting point to note is that Windows implements handles
            to console objects (input and screen buffers) only in the
            <filename>KERNEL32</filename> DLL, and those are not sent nor seen
            from the <filename>NTDLL</filename> level, albeit, for example,
            console are waitable on input.  How is this possible?  Well, Windows
            NT is a bit tricky here.  Regular handles have an interesting
            property: their integral value is always a multiple of four (they
            are likely to be offsets from the beginning of a table).  Console
            handles, on the other hand, are not multiple of four, but have the
            two lower bit set (being a multiple of four means having the two
            lower bits reset).  When <filename>KERNEL32</filename> sees a handle
            with the two lower bits set, it then knows it's a console handle and
            takes appropriate decisions.  For example, in the various
            <function>kernel32!WaitFor*()</function> functions, it transforms
            any console handle (input and <emphasis>output</emphasis> -
            strangely enough handles to console's screen buffers are waitable)
            into a dedicated wait event for the targetted console.  There's an
            (undocumented) <filename>KERNEL32</filename> function
            <function>GetConsoleInputWaitHandle()</function> which returns the
            handle to this event in case you need it.  Another interesting
            handling of those console's handles is in
            <function>ReadFile()</function>
            (resp. <function>WriteFile()</function>), which behavior, for
            console's handles, is transferred to
            <function>ReadConsole()</function> (resp.
            <function>WriteConsole()</function>).  Note that's always the ANSI
            version of
            <function>ReadConsole()</function> /
            <function>WriteConsole()</function>
            which is called, hence using the default console's code page.  There
            are some other spots affected, but you can look in
            <filename>dlls/kernel</filename> to find them all.  All of this is
            implemented in Wine.
          </para>
          <para>
            Wine also implements the same layout of the registry for storing the
            preferences of the console as Windows does.  Those settings can
            either be defined globally, or on a per process name basis.
            <command>wineconsole</command> provides the choice to the user to
            pick you which registry part (global, current running program) it
            wishes to modify the settings for.
            <table>
              <title>Console registry settings</title>
              <tgroup cols="3" align="left">
                <thead>
                  <row>
                    <entry>Name</entry>
                    <entry>Default value</entry>
                    <entry>Purpose</entry>
                  </row>
                </thead>
                <tbody>
                  <row>
                    <entry>CursorSize</entry>
                    <entry>25</entry>
                    <entry>
                      Percentage of cell height to which the cursor extents
                    </entry>
                  </row>
                  <row>
                    <entry>CursorVisible</entry>
                    <entry>1</entry>
                    <entry>Whether the cursor is visible or not</entry>
                  </row>
                  <row>
                    <entry>EditionMode</entry>
                    <entry>0</entry>
                    <entry>
                      The way the edition takes place in the console: 0 is
                      insertion mode, 1 is overwrite mode.
                    </entry>
                  </row>
                  <row>
                    <entry>ExitOnDie</entry>
                    <entry>1</entry>
                    <entry>
                      Whether the console should close itself when last running
                      program attached to it dies
                    </entry>
                  </row>
                  <row>
                    <entry>FaceName</entry>
                    <entry>No default</entry>
                    <entry>
                      Name of the font to be used for display.  When none is
                      given, <command>wineconsole</command> tries its best to
                      pick up a decent font
                    </entry>
                  </row>
                  <row>
                    <entry>FontSize</entry>
                    <entry>0x0C08</entry>
                    <entry>
                      The high word in the font's cell height, and the low word
                      is the font cell's width.  The default value is 12 pixels
                      in height and 8 pixels in width.
                    </entry>
                  </row>
                  <row>
                    <entry>FontWeight</entry>
                    <entry>0</entry>
                    <entry>
                      Weigth of the font.  If none is given (or 0)
                      <command>wineconsole</command> picks up a decent font size
                    </entry>
                  </row>
                  <row>
                    <entry>HistoryBufferSize</entry>
                    <entry>50</entry>
                    <entry>
                      Number of entries in history buffer (not actually used)
                    </entry>
                  </row>
                  <row>
                    <entry>HistoryNoDup</entry>
                    <entry>0</entry>
                    <entry>
                      Whether the history should store twice the same entry
                    </entry>
                  </row>
                  <row>
                    <entry>MenuMask</entry>
                    <entry>0</entry>
                    <entry>
                      This mask only exists for Wine console handling.  It
                      allows to know which combination of extra keys are need to
                      open the configuration window on right click.  The mask
                      can include <constant>MK_CONTROL</constant> or
                      <constant>MK_SHIFT</constant> bits.  This can be needed
                      when programs actually need the right click to be passed
                      to them instead of being intercepted by
                      <command>wineconsole</command>.
                    </entry>
                  </row>
                  <row>
                    <entry>QuickEdit</entry>
                    <entry>0</entry>
                    <entry>
                      If null, mouse events are sent to the application.  If non
                      null, mouse events are used to select text on the window.
                      This setting must really be set on a application per
                      application basis, because it deals with the fact the CUI
                      application will use or not the mouse events.
                    </entry>
                  </row>
                  <row>
                    <entry>ScreenBufferSize</entry>
                    <entry>0x1950</entry>
                    <entry>
                      The high word is the number of font cells in the height of
                      the screen buffer, while the low word is the number of
                      font cells in the width of the screen buffer.
                    </entry>
                  </row>
                  <row>
                    <entry>ScreenColors</entry>
                    <entry>0x000F</entry>
                    <entry>
                      Default color attribute for the screen buffer (low char is
                      the foreground color, and high char is the background
                      color)
                    </entry>
                  </row>
                  <row>
                    <entry>WindowSize</entry>
                    <entry>0x1950</entry>
                    <entry>
                      The high word is the number of font cells in the height of
                      the window, while the low word is the number of font cells
                      in the width of the window.  This window is the visible
                      part of the screen buffer: this implies that a screen
                      buffer must always be bigger than its window, and that the
                      screen buffer can be scrolled so that every cell of the
                      screen buffer can be seen in the window.
                    </entry>
                  </row>
                </tbody>
              </tgroup>
            </table>
          </para>
        </sect3>
      </sect2>
      <sect2>
        <title>Win16 processes support</title>
        <sect3>
          <title>Starting a NE (Win16) process</title>
          <para>
            Wine is also able to run 16 bit processes, but this feature is only
            supported on Intel IA-32 architectures.
          </para>
          <para>
            When Wine is requested to run a NE (Win 16 process), it will in fact
            hand over the execution of it to a specific executable
            <command>winevdm</command>. VDM stands for Virtual DOS
            Machine. This <command>winevdm</command> is a Winelib application,
            but will in fact set up the correct 16 bit environment to run the
            executable. We will get back later on in details to what this means.
          </para>
          <para>
            Any new 16 bit process created by this executable (or its children)
            will run into the same <command>winevdm</command> instance. Among
            one instance, several functionalities will be provided to those 16
            bit processes, including the cooperative multitasking, sharing the
            same address space, managing the selectors for the 16 bit segments
            needed for code, data and stack.
          </para>
          <para>
            Note that several <command>winevdm</command> instances can run in
            the same Wine session, but the functionalities described above are
            only shared among a given instance, not among all the
            instances. <command>winevdm</command> is built as Winelib
            application, and hence has access to any facility a 32 bit
            application has.
          </para>
          <para>
            Each Win16 application is implemented in
            <command>winevdm</command> as a Win32
            thread. <command>winevdm</command> then implements its own
            scheduling facilities (in fact, the code for this feature is in the
            <filename>krnl386.exe</filename> DLL). Since the required Win16
            scheduling is non pre-emptive, this doesn't require any underlying
            OS kernel support.
          </para>
        </sect3>
        <sect3>
          <title>SysLevels</title>
          <para>
            SysLevels are an undocumented Windows-internal thread-safety system
            dedicated to 16 bit applications (or 32 bit applications that call -
            directly or indirectly - 16 bit code). They are basically critical
            sections which must be taken in a particular order. The mechanism is
            generic but there are always three syslevels:
            <itemizedlist>
              <listitem>
                <para>level 1 is the Win16 mutex,</para>
              </listitem>
              <listitem>
                <para>level 2 is the <filename>USER</filename> mutex,</para>
              </listitem>
              <listitem>
                <para>level 3 is the <filename>GDI</filename> mutex.</para>
              </listitem>
            </itemizedlist>
          </para>

          <para>
            When entering a syslevel, the code (in
            <filename>dlls/kernel/syslevel.c</filename>) will check that a
            higher syslevel is not already held and produce an error if so. This
            is because it's not legal to enter level 2 while holding level 3 -
            first, you must leave level 3.
          </para>

          <para>
            Throughout the code you may see calls to
            <function>_ConfirmSysLevel()</function> and
            <function>_CheckNotSysLevel()</function>. These functions are
            essentially assertions about the syslevel states and can be used to
            check that the rules have not been accidentally violated. In
            particular, <function>_CheckNotSysLevel()</function> will break
            probably into the debugger) if the check fails. If this happens the
            solution is to get a backtrace and find out, by reading the source
            of the wine functions called along the way, how Wine got into the
            invalid state.
          </para>
        </sect3>
        <sect3>
          <title>Memory management</title>
          <para>
            Every Win16 address is expressed in the form of selector:offset. The
            selector is an entry in the LDT, but a 16 bit entry, limiting each
            offset to 64 KB. Hence, the maximum available memory to a Win16
            process is 512 MB. Note, that the processor runs in protected mode,
            but using 16 bit selectors.
          </para>
          <para>
            Windows, for a 16 bit process, defines a few selectors to access the
            "real" memory (the one provided) by DOS. Basically, Wine also
            provides this area of memory.
          </para>
        </sect3>
      </sect2>
      <sect2>
        <title>DOS processes support</title>
        <para>
          The behaviour we just described also applies to DOS executables, which
          are handled the same way by <command>winevdm</command>. This is only
          supported on Intel IA-32 architectures.
        </para>
        <para>
          Wine implements also most of the DOS support in a Wine specific DLL
          (<filename>winedos</filename>). This DLL is called under certain
          conditions, like:
          <itemizedlist>
            <listitem>
              <para>
                In <command>winevdm</command>, when trying to launch a DOS
                application (<filename>.EXE</filename> or
                <filename>.COM</filename>, <filename>.PIF</filename>).
              </para>
            </listitem>
            <listitem>
              <para>
                In <filename>kernel32</filename>, when an attempt is made in the
                binary code to call some DOS or BIOS interrupts (like Int 21h
                for example).
              </para>
            </listitem>
          </itemizedlist>
        </para>
        <para>
          When <command>winevdm</command> runs a DOS program, this one runs in
          real mode (in fact in V86 mode from the IA-32 point of view).
        </para>
        <para>
          Wine also supports part of the DPMI (DOS Protected Mode Interface).
        </para>
        <para>
          Wine, when running a DOS programs, needs to map the 1 MB of virtual
          memory to the real memory (as seen by the DOS program). When this is
          not possible (like when someone else is already using this area), the
          DOS support is not possible. Not also that by doing so, access to
          linear address 0 is enabled (as it's also real mode address 0 which is
          valid). Hence, NULL pointer derefence faults are no longer catched.
        </para>
      </sect2>
    </sect1>
    
  </chapter>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
End:
-->