It makes no sense to have a left pointing arrow as the desktop icon.

[wine/testsucceed.git] / documentation / ole.sgml

  <chapter id="ole">
    <title>COM in Wine</title>

    <sect1 id="com-writing">
      <title>Writing COM Components for Wine</title>

      <para>
        This section describes how to create your own natively
        compiled COM components.
      </para>

      <sect2>
        <title>Macros to define a COM interface</title>

        <para>
          The goal of the following set of definitions is to provide a
          way to use the same header file definitions to provide both
          a C interface and a C++ object oriented interface to COM
          interfaces. The type of interface is selected automatically
          depending on the language but it is always possible to get
          the C interface in C++ by defining CINTERFACE.
        </para>
        <para>
          It is based on the following assumptions:
        </para>
        <itemizedlist>
          <listitem>
            <para>
              all COM interfaces derive from IUnknown, this should not
              be a problem.
            </para>
          </listitem>
          <listitem>
            <para>
              the header file only defines the interface, the actual
              fields are defined separately in the C file implementing
              the interface.
            </para>
          </listitem>
        </itemizedlist>
        <para>
          The natural approach to this problem would be to make sure
          we get a C++ class and virtual methods in C++ and a
          structure with a table of pointer to functions in C.
          Unfortunately the layout of the virtual table is compiler
          specific, the layout of g++ virtual tables is not the same
          as that of an egcs virtual table which is not the same as
          that generated by Visual C++. There are work arounds to make
          the virtual tables compatible via padding but unfortunately
          the one which is imposed to the Wine emulator by the Windows
          binaries, i.e. the Visual C++ one, is the most compact of
          all.
        </para>
        <para>
          So the solution I finally adopted does not use virtual
          tables. Instead I use in-line non virtual methods that
          dereference the method pointer themselves and perform the
          call.
        </para>
        <para>
          Let's take Direct3D as an example:
        </para>
        <programlisting>#define ICOM_INTERFACE IDirect3D
#define IDirect3D_METHODS \
    ICOM_METHOD1(HRESULT,Initialize,    REFIID,) \
    ICOM_METHOD2(HRESULT,EnumDevices,   LPD3DENUMDEVICESCALLBACK,, LPVOID,) \
    ICOM_METHOD2(HRESULT,CreateLight,   LPDIRECT3DLIGHT*,, IUnknown*,) \
    ICOM_METHOD2(HRESULT,CreateMaterial,LPDIRECT3DMATERIAL*,, IUnknown*,) \
    ICOM_METHOD2(HRESULT,CreateViewport,LPDIRECT3DVIEWPORT*,, IUnknown*,) \
    ICOM_METHOD2(HRESULT,FindDevice,    LPD3DFINDDEVICESEARCH,, LPD3DFINDDEVICERESULT,)
#define IDirect3D_IMETHODS \
    IUnknown_IMETHODS \
    IDirect3D_METHODS
ICOM_DEFINE(IDirect3D,IUnknown)
#undef ICOM_INTERFACE

#ifdef ICOM_CINTERFACE
// *** IUnknown methods *** //
#define IDirect3D_QueryInterface(p,a,b) ICOM_CALL2(QueryInterface,p,a,b)
#define IDirect3D_AddRef(p)             ICOM_CALL (AddRef,p)
#define IDirect3D_Release(p)            ICOM_CALL (Release,p)
// *** IDirect3D methods *** //
#define IDirect3D_Initialize(p,a)       ICOM_CALL1(Initialize,p,a)
#define IDirect3D_EnumDevices(p,a,b)    ICOM_CALL2(EnumDevice,p,a,b)
#define IDirect3D_CreateLight(p,a,b)    ICOM_CALL2(CreateLight,p,a,b)
#define IDirect3D_CreateMaterial(p,a,b) ICOM_CALL2(CreateMaterial,p,a,b)
#define IDirect3D_CreateViewport(p,a,b) ICOM_CALL2(CreateViewport,p,a,b)
#define IDirect3D_FindDevice(p,a,b)     ICOM_CALL2(FindDevice,p,a,b)
#endif</programlisting>
        <para>
          Comments:
        </para>
        <para>
          The ICOM_INTERFACE macro is used in the ICOM_METHOD macros
          to define the type of the 'this' pointer. Defining this
          macro here saves us the trouble of having to repeat the
          interface name everywhere. Note however that because of the
          way macros work, a macro like ICOM_METHOD1 cannot use
          'ICOM_INTERFACE##_VTABLE' because this would give
          'ICOM_INTERFACE_VTABLE' and not 'IDirect3D_VTABLE'.
        </para>
        <para>
          ICOM_METHODS defines the methods specific to this
          interface. It is then aggregated with the inherited methods
          to form ICOM_IMETHODS.
        </para>
        <para>
          ICOM_IMETHODS defines the list of methods that are
          inheritable from this interface. It must be written manually
          (rather than using a macro to generate the equivalent code)
          to avoid macro recursion (which compilers don't like).
        </para>
        <para>
          The ICOM_DEFINE finally declares all the structures
          necessary for the interface. We have to explicitly use the
          interface name for macro expansion reasons again.  Inherited
          methods are inherited in C by using the IDirect3D_METHODS
          macro and the parent's Xxx_IMETHODS macro. In C++ we need
          only use the IDirect3D_METHODS since method inheritance is
          taken care of by the language.
        </para>
        <para>
          In C++ the ICOM_METHOD macros generate a function prototype
          and a call to a function pointer method. This means using
          once 't1 p1, t2 p2, ...' and once 'p1, p2' without the
          types. The only way I found to handle this is to have one
          ICOM_METHOD macro per number of parameters and to have it
          take only the type information (with const if necessary) as
          parameters.  The 'undef ICOM_INTERFACE' is here to remind
          you that using ICOM_INTERFACE in the following macros will
          not work. This time it's because the ICOM_CALL macro
          expansion is done only once the 'IDirect3D_Xxx' macro is
          expanded. And by that time ICOM_INTERFACE will be long gone
          anyway.
        </para>
        <para>
          You may have noticed the double commas after each parameter
          type. This allows you to put the name of that parameter
          which I think is good for documentation. It is not required
          and since I did not know what to put there for this example
          (I could only find doc about IDirect3D2), I left them blank.
        </para>
        <para>
          Finally the set of 'IDirect3D_Xxx' macros is a standard set
          of macros defined to ease access to the interface methods in
          C. Unfortunately I don't see any way to avoid having to
          duplicate the inherited method definitions there. This time
          I could have used a trick to use only one macro whatever the
          number of parameters but I preferred to have it work the same
          way as above.
        </para>
        <para>
          You probably have noticed that we don't define the fields we
          need to actually implement this interface: reference count,
          pointer to other resources and miscellaneous fields. That's
          because these interfaces are just that: interfaces. They may
          be implemented more than once, in different contexts and
          sometimes not even in Wine. Thus it would not make sense to
          impose that the interface contains some specific fields.
        </para>
      </sect2>

      <sect2>
        <title>Bindings in C</title>

        <para>
          In C this gives:
        </para>
        <programlisting>typedef struct IDirect3DVtbl IDirect3DVtbl;
struct IDirect3D {
    IDirect3DVtbl* lpVtbl;
};
struct IDirect3DVtbl {
    HRESULT (*fnQueryInterface)(IDirect3D* me, REFIID riid, LPVOID* ppvObj);
    ULONG (*fnAddRef)(IDirect3D* me);
    ULONG (*fnRelease)(IDirect3D* me);
    HRESULT (*fnInitialize)(IDirect3D* me, REFIID a);
    HRESULT (*fnEnumDevices)(IDirect3D* me, LPD3DENUMDEVICESCALLBACK a, LPVOID b);
    HRESULT (*fnCreateLight)(IDirect3D* me, LPDIRECT3DLIGHT* a, IUnknown* b);
    HRESULT (*fnCreateMaterial)(IDirect3D* me, LPDIRECT3DMATERIAL* a, IUnknown* b);
    HRESULT (*fnCreateViewport)(IDirect3D* me, LPDIRECT3DVIEWPORT* a, IUnknown* b);
    HRESULT (*fnFindDevice)(IDirect3D* me, LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b);
}; 

#ifdef ICOM_CINTERFACE
// *** IUnknown methods *** //
#define IDirect3D_QueryInterface(p,a,b) (p)->lpVtbl->fnQueryInterface(p,a,b)
#define IDirect3D_AddRef(p)             (p)->lpVtbl->fnAddRef(p)
#define IDirect3D_Release(p)            (p)->lpVtbl->fnRelease(p)
// *** IDirect3D methods *** //
#define IDirect3D_Initialize(p,a)       (p)->lpVtbl->fnInitialize(p,a)
#define IDirect3D_EnumDevices(p,a,b)    (p)->lpVtbl->fnEnumDevice(p,a,b)
#define IDirect3D_CreateLight(p,a,b)    (p)->lpVtbl->fnCreateLight(p,a,b)
#define IDirect3D_CreateMaterial(p,a,b) (p)->lpVtbl->fnCreateMaterial(p,a,b)
#define IDirect3D_CreateViewport(p,a,b) (p)->lpVtbl->fnCreateViewport(p,a,b)
#define IDirect3D_FindDevice(p,a,b)     (p)->lpVtbl->fnFindDevice(p,a,b)
#endif</programlisting>
        <para>
          Comments:
        </para>
        <para>
          IDirect3D only contains a pointer to the IDirect3D
          virtual/jump table. This is the only thing the user needs to
          know to use the interface. Of course the structure we will
          define to implement this interface will have more fields but
          the first one will match this pointer.
        </para>
        <para>
          The code generated by ICOM_DEFINE defines both the structure
          representing the interface and the structure for the jump
          table. ICOM_DEFINE uses the parent's Xxx_IMETHODS macro to
          automatically repeat the prototypes of all the inherited
          methods and then uses IDirect3D_METHODS to define the
          IDirect3D methods.
        </para>
        <para>
          Each method is declared as a pointer to function field in
          the jump table. The implementation will fill this jump table
          with appropriate values, probably using a static variable,
          and initialize the lpVtbl field to point to this variable.
        </para>
        <para>
          The IDirect3D_Xxx macros then just dereference the lpVtbl
          pointer and use the function pointer corresponding to the
          macro name. This emulates the behavior of a virtual table
          and should be just as fast.
        </para>
        <para>
          This C code should be quite compatible with the Windows
          headers both for code that uses COM interfaces and for code
          implementing a COM interface.
        </para>
      </sect2>

      <sect2>
        <title>Bindings in C++</title>
        <para>
          And in C++ (with gcc's g++):
        </para>
        <programlisting>typedef struct IDirect3D: public IUnknown {
    private: HRESULT (*fnInitialize)(IDirect3D* me, REFIID a);
    public: inline HRESULT Initialize(REFIID a) { return ((IDirect3D*)t.lpVtbl)->fnInitialize(this,a); };
    private: HRESULT (*fnEnumDevices)(IDirect3D* me, LPD3DENUMDEVICESCALLBACK a, LPVOID b);
    public: inline HRESULT EnumDevices(LPD3DENUMDEVICESCALLBACK a, LPVOID b)
        { return ((IDirect3D*)t.lpVtbl)->fnEnumDevices(this,a,b); };
    private: HRESULT (*fnCreateLight)(IDirect3D* me, LPDIRECT3DLIGHT* a, IUnknown* b);
    public: inline HRESULT CreateLight(LPDIRECT3DLIGHT* a, IUnknown* b)
        { return ((IDirect3D*)t.lpVtbl)->fnCreateLight(this,a,b); };
    private: HRESULT (*fnCreateMaterial)(IDirect3D* me, LPDIRECT3DMATERIAL* a, IUnknown* b);
    public: inline HRESULT CreateMaterial(LPDIRECT3DMATERIAL* a, IUnknown* b)
        { return ((IDirect3D*)t.lpVtbl)->fnCreateMaterial(this,a,b); };
    private: HRESULT (*fnCreateViewport)(IDirect3D* me, LPDIRECT3DVIEWPORT* a, IUnknown* b);
    public: inline HRESULT CreateViewport(LPDIRECT3DVIEWPORT* a, IUnknown* b)
        { return ((IDirect3D*)t.lpVtbl)->fnCreateViewport(this,a,b); };
    private:  HRESULT (*fnFindDevice)(IDirect3D* me, LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b);
    public: inline HRESULT FindDevice(LPD3DFINDDEVICESEARCH a, LPD3DFINDDEVICERESULT b)
        { return ((IDirect3D*)t.lpVtbl)->fnFindDevice(this,a,b); };
};</programlisting>
        <para>
          Comments:
        </para>
        <para>
          In C++ IDirect3D does double duty as both the virtual/jump
          table and as the interface definition. The reason for this
          is to avoid having to duplicate the method definitions: once
          to have the function pointers in the jump table and once to
          have the methods in the interface class. Here one macro can
          generate both. This means though that the first pointer,
          t.lpVtbl defined in IUnknown, must be interpreted as the
          jump table pointer if we interpret the structure as the
          interface class, and as the function pointer to the
          QueryInterface method, t.fnQueryInterface, if we interpret
          the structure as the jump table. Fortunately this gymnastic
          is entirely taken care of in the header of IUnknown.
        </para>
        <para>
          Of course in C++ we use inheritance so that we don't have to
          duplicate the method definitions.
        </para>
        <para>
          Since IDirect3D does double duty, each ICOM_METHOD macro
          defines both a function pointer and a non-virtual inline
          method which dereferences it and calls it. This way this
          method behaves just like a virtual method but does not
          create a true C++ virtual table which would break the
          structure layout. If you look at the implementation of these
          methods you'll notice that they would not work for void
          functions. We have to return something and fortunately this
          seems to be what all the COM methods do (otherwise we would
          need another set of macros).
        </para>
        <para>
          Note how the ICOM_METHOD generates both function prototypes
          mixing types and formal parameter names and the method
          invocation using only the formal parameter name. This is the
          reason why we need different macros to handle different
          numbers of parameters.
        </para>
        <para>
          Finally there is no IDirect3D_Xxx macro. These are not
          needed in C++ unless the CINTERFACE macro is defined in
          which case we would not be here.
        </para>
        <para>
          This C++ code works well for code that just uses COM
          interfaces. But it will not work with C++ code implement a
          COM interface. That's because such code assumes the
          interface methods are declared as virtual C++ methods which
          is not the case here.
        </para>
      </sect2>

      <sect2>
        <title>Implementing a COM interface.</title>

        <para>
          This continues the above example. This example assumes that
          the implementation is in C.
        </para>
        <programlisting>typedef struct _IDirect3D {
    void* lpVtbl;
    // ...
 } _IDirect3D;

static ICOM_VTABLE(IDirect3D) d3dvt;

// implement the IDirect3D methods here

int IDirect3D_fnQueryInterface(IDirect3D* me)
{
    ICOM_THIS(IDirect3D,me);
    // ...
}

// ...

static ICOM_VTABLE(IDirect3D) d3dvt = {
    ICOM_MSVTABLE_COMPAT_DummyRTTIVALUE
    IDirect3D_fnQueryInterface,
    IDirect3D_fnAdd,
    IDirect3D_fnAdd2,
    IDirect3D_fnInitialize,
    IDirect3D_fnSetWidth
};</programlisting>
        <para>
          Comments:
        </para>
        <para>
          We first define what the interface really contains. This is
          the _IDirect3D structure. The first field must of course be
          the virtual table pointer. Everything else is free.
        </para>
        <para>
          Then we predeclare our static virtual table variable, we
          will need its address in some methods to initialize the
          virtual table pointer of the returned interface objects.
        </para>
        <para>
          Then we implement the interface methods. To match what has
          been declared in the header file they must take a pointer to
          an IDirect3D structure and we must cast it to an _IDirect3D
          so that we can manipulate the fields. This is performed by
          the ICOM_THIS macro.
        </para>
        <para>
          Finally we initialize the virtual table.
        </para>
      </sect2>
    </sect1>
                                         
    <sect1 id="dcom-1">
      <title>A brief introduction to DCOM in Wine</title>

      <para>
        This section explains the basic principles behind DCOM remoting as used by InstallShield and others.
      </para>

      <sect2>
        <title>Basics</title>

        <para>
          The basic idea behind DCOM is to take a COM object and make it location
          transparent. That means you can use it from other threads, processes and
          machines without having to worry about the fact that you can't just
          dereference the interface vtable pointer to call methods on it.
        </para>

        <para>
          You might be wondering about putting threads next to processes and
          machines in that last paragraph. You can access thread safe objects from
          multiple threads without DCOM normally, right? Why would you need RPC
          magic to do that?
        </para>

        <para>
          The answer is of course that COM doesn't assume that objects actually
          are thread-safe. Most real-world objects aren't, in fact, for various
          reasons. What these reasons are isn't too important here, though; it's
          just important to realize that the problem of thread-unsafe objects is
          what COM tries hard to solve with its apartment model. There are also
          ways to tell COM that your object is truly thread-safe (namely the
          free-threaded marshaller). In general, no object is truly thread-safe if
          it could potentially use another not so thread-safe object, though, so
          the free-threaded marshaller is less used than you'd think.
        </para>
        
        <para>
          For now, suffice it to say that COM lets you "marshal" interfaces into
          other "apartments". An apartment (you may see it referred to as a
          context in modern versions of COM) can be thought of as a location, and
          contains objects. 
        </para>

        <para>
          Every thread in a program that uses COM exists in an apartment. If a
          thread wishes to use an object from another apartment, marshalling and
          the whole DCOM infrastructure gets involved to make that happen behind
          the scenes.
        </para>

        <para>
          So. Each COM object resides in an apartment, and each apartment
          resides in a process, and each process resides in a machine, and each
          machine resides in a network. Allowing those objects to be used
          from <emphasis>any</emphasis> of these different places is what DCOM
          is all about.
        </para>

        <para>
          The process of marshalling refers to taking a function call in an
          apartment and actually performing it in another apartment. Let's say you
          have two machines, A and B, and on machine B there is an object sitting
          in a DLL on the hard disk. You want to create an instance of that object
          (activate it) and use it as if you had compiled it into your own
          program. This is hard, because the remote object is expecting to be
          called by code in its own address space - it may do things like accept
          pointers to linked lists and even return other objects.
        </para>

        <para>
          Very basic marshalling is easy enough to understand. You take a method
          on a remote interface (that is a COM interface that is
          implemented on the remote computer), copy each of its
          parameters into a buffer, and
          send it to the remote computer. On the other end, the remote server
          reads each parameter from the buffer, calls the method, writes the
          result into another buffer and sends it back.
        </para>

        <para>
          The tricky part is exactly how to encode those parameters in the buffer,
          and how to convert standard stdcall/cdecl method calls to network
          packets and back again. This is the job of the RPCRT4.DLL file - or the
          Remote Procedure Call Runtime. 
        </para>

        <para>
          The backbone of DCOM is this RPC runtime, which is an implementation
          of <ulink
          url="http://www.opengroup.org/onlinepubs/009629399/toc.htm">DCE
          RPC</ulink>. DCE RPC is not naturally object oriented, so this
          protocol is extended with some new constructs and by assigning new
          meanings to some of the packet fields, to produce ORPC or Object
          RPC. You might see it called MS-RPC as well.
        </para>

        <para>
          RPC packets contain a buffer containing marshalled data in NDR format.
          NDR is short for "Network Data Representation" and is similar
          to the XDR
          format used in SunRPC (the closest native equivalent on Linux to DCE
          RPC). NDR/XDR are all based on the idea of graph serialization and were
          worked out during the 80s, meaning they are very powerful and can do
          things like marshal doubly linked lists and other rather tricky
          structures.
        </para>

        <para>
          In Wine, our DCOM implementation is <emphasis>not</emphasis>
          currently based on the
          RPC runtime, as while few programs use DCOM even fewer use
          RPC directly so it was developed some time after
          OLE32/OLEAUT32 were. Eventually this will have to be fixed,
          otherwise our DCOM will never be compatible with
          Microsoft's. Bear this in mind as you read through the code
          however.
        </para>
      </sect2>
      
      <sect2>
        <title>Proxies and Stubs</title>

        <para>
          Manually marshalling and unmarshalling each method call using the NDR
          APIs (NdrConformantArrayMarshall etc) is very tedious work, so the
          Platform SDK ships with a tool called "midl" which is an IDL compiler.
          IDL or the "Interface Definition Language" is a tool designed
          specifically for describing interfaces in a reasonably language neutral
          fashion, though in reality it bears a close resemblence to C++.
        </para>

        <para>
          By describing the functions you want to expose via RPC in IDL therefore,
          it becomes possible to pass this file to MIDL which spits out a huge
          amount of C source code. That code defines functions which have the same
          prototype as the functions described in your IDL but which internally
          take each argument, marshal it using Ndr, send the packet, and unmarshal
          the return.
        </para>
        
        <para>
          Because this code proxies the code from the client to the server, the
          functions are called proxies. Easy, right?
        </para>

        <para>
          Of course, in the RPC server process at the other end, you need some way
          to unmarshal the RPCs, so you have functions also generated by MIDL
          which are the inverse of the proxies; they accept an NDR buffer, extract
          the parameters, call the real function and then marshal the result back.
          They are called stubs, and stand in for the real calling code in the
          client process.
        </para>
        
        <para>
          The sort of marshalling/unmarshalling code that MIDL spits out can be
          seen in dlls/oleaut32/oaidl_p.c - it's not exactly what it would look
          like as that file contains DCOM proxies/stubs which are different, but
          you get the idea. Proxy functions take the arguments and feed them to
          the NDR marshallers (or picklers), invoke an NdrProxySendReceive and
          then convert the out parameters and return code. There's a ton of goop
          in there for dealing with buffer allocation, exceptions and so on - it's
          really ugly code. But, this is the basic concept behind DCE RPC.
        </para>
      </sect2>
      
      <sect2>
        <title>Interface Marshalling</title>

        <para>
          Standard NDR only knows about C style function calls - they
          can accept and even return structures, but it has no concept
          of COM interfaces.  Confusingly DCE RPC <emphasis>does</emphasis> have a
          concept of RPC interfaces which are just convenient ways to
          bundle function calls together into namespaces, but let's
          ignore that for now as it just muddies the water. The
          primary extension made by Microsoft to NDR then was the
          ability to take a COM interface pointer and marshal that
          into the NDR stream.
        </para>

        <para>
          The basic theory of proxies and stubs and IDL is still here, but it's
          been modified slightly. Whereas before you could define a bunch of
          functions in IDL, now a new "object" keyword has appeared. This tells
          MIDL that you're describing a COM interface, and as a result the
          proxies/stubs it generates are also COM objects.
        </para>

        <para>
          That's a very important distinction. When you make a call to a remote
          COM object you do it via a proxy object that COM has constructed on the
          fly. Likewise, a stub object on the remote end unpacks the RPC packet
          and makes the call. 
        </para>

        <para>
          Because this is object-oriented RPC, there are a few complications: for
          instance, a call that goes via the same proxies/stubs may end up at a
          different object instance, so the RPC runtime keeps track of "this" and
          "that" in the RPC packets.
        </para>

        <para>
          This leads naturally onto the question of how we got those proxy/stub
          objects in the first place, and where they came from. You can use the
          CoCreateInstanceEx API to activate COM objects on a remote machine, this
          works like CoCreateInstance API. Behind the scenes, a lot of stuff is
          involved to do this (like IRemoteActivation, IOXIDResolver and so on)
          but let's gloss over that for now.
        </para>

        <para>
          When DCOM creates an object on a remote machine, the DCOM runtime on
          that machine activates the object in the usual way (by looking it up in
          the registry etc) and then marshals the requested interface back to the
          client. Marshalling an interface takes a pointer, and produces a buffer
          containing all the information DCOM needs to construct a proxy object in
          the client, a stub object in the server and link the two together.
        </para>

        <para>
          The structure of a marshalled interface pointer is somewhat complex.
          Let's ignore that too. The important thing is how COM proxies/stubs are
          loaded.
        </para>
      </sect2>

      <sect2>
        <title>COM Proxy/Stub System</title>

        <para>
          COM proxies are objects that implement both the interfaces needing to be
          proxied and also IRpcProxyBuffer. Likewise, COM stubs implement
          IRpcStubBuffer and understand how to invoke the methods of the requested
          interface.
        </para>

        <para>
          You may be wondering what the word "buffer" is doing in those interface
          names. I'm not sure either, except that a running theme in DCOM is that
          interfaces which have nothing to do with buffers have the word Buffer
          appended to them, seemingly at random. Ignore it and <emphasis>don't let it
            confuse you</emphasis> :) This stuff is convoluted enough ...
        </para>

        <para>
          The IRpc[Proxy/Stub]Buffer interfaces are used to control the proxy/stub
          objects and are one of the many semi-public interfaces used in DCOM.
        </para>

        <para>
          DCOM is theoretically an internet RFC 
          <ulink url="http://www.grimes.demon.co.uk/DCOM/DCOMSpec.htm">[2]</ulink> and is
          specced out, but in reality the only implementation of it apart from
          ours is Microsoft's, and as a result there are lots of interfaces
          which <emphasis>can</emphasis> be used if you want to customize or
          control DCOM but in practice are badly documented or not documented at
          all, or exist mostly as interfaces between MIDL generated code and COM
          itself. Don't pay too much attention to the MSDN definitions of these
          interfaces and APIs.
        </para>

        <para>
          COM proxies and stubs are like any other normal COM object - they are
          registered in the registry, they can be loaded with CoCreateInstance and
          so on. They have to be in process (in DLLs) however. They aren't
          activated directly by COM however, instead the process goes something
          like this:
          
          <itemizedlist>
            <listitem> <para> COM receives a marshalled interface packet, and retrieves the IID of
                the marshalled interface from it </para> </listitem>


            <listitem> <para> COM looks in
              HKEY_CLASSES_ROOT/Interface/{whatever-iid}/ProxyStubClsId32
              to retrieve the CLSID of another COM object, which
              implements IPSFactoryBuffer. </para> </listitem>
            
            <listitem> <para> IPSFactoryBuffer has only two methods, CreateProxy and CreateStub. COM
                calls whichever is appropriate: CreateStub for the server, CreateProxy
                for the client. MIDL will normally provide an implementation of this
                object for you in the code it generates. </para></listitem>
          </itemizedlist>
          
        </para>

        <para>
          Once CreateProxy has been called, the resultant object is QueryInterfaced to
          IRpcProxyBuffer, which only has 1 method, IRpcProxyBuffer::Connect.
          This method only takes one parameter, the IRpcChannelBuffer object which
          encapsulates the "RPC Channel" between the client and server.
        </para>

        <para>
          On the server side, a similar process is performed - the PSFactoryBuffer
          is created, CreateStub is called, result is QId to IRpcStubBuffer, and
          IRpcStubBuffer::Connect is used to link it to the RPC channel.
        </para>

      </sect2>

      <sect2>
        <title>RPC Channels</title>

        <para>
          Remember the RPC runtime? Well, that's not just responsible for
          marshalling stuff, it also controls the connection and protocols between
          the client and server. We can ignore the details of this for now,
          suffice it to say that an RPC Channel is a COM object that implements
          IRpcChannelBuffer, and it's basically an abstraction of different RPC
          methods. For instance, in the case of inter-thread marshalling (not
          covered here) the RPC connection code isn't used, only the NDR
          marshallers are, so IRpcChannelBuffer in that case isn't actually
          implemented by RPCRT4 but rather just by the COM/OLE DLLS.
        </para>

        <para>
          On this topic, Ove Kaaven says: It depends on the Windows version, I
          think. Windows 95 and Windows NT 4 certainly had very different models
          when I looked. I'm pretty sure the Windows 98 version of RPCRT4 was
          able to dispatch messages directly to individual apartments. I'd be
          surprised if some similar functionality was not added to Windows
          2000. After all, if an object on machine A wanted to use an object on
          machine B in an apartment C, wouldn't it be most efficient if the RPC
          system knew about apartments and could dispatch the message directly
          to it? And if RPC does know how to efficiently dispatch to apartments,
          why should COM duplicate this functionality? There were, however, no
          unified way to tell RPC about them across Windows versions, so in that
          old patch of mine, I let the COM/OLE dlls do the apartment dispatch,
          but even then, the RPC runtime was always involved. After all, it
          could be quite tricky to tell whether the call is merely interthread,
          without involving the RPC runtime...
        </para>

        <para>
          RPC channels are constructed on the fly by DCOM as part of the
          marshalling process. So, when you make a call on a COM proxy, it goes
          like this:
        </para>

        <para>
          Your code -&gt; COM proxy object -&gt; RPC Channel -&gt; COM stub object -&gt; Their code
        </para>

      </sect2>

      <sect2>
        <title>How this actually works in Wine</title>

        <para>
          Right now, Wine does not use the NDR marshallers or RPC to implement its
          DCOM. When you marshal an interface in Wine, in the server process a
          _StubMgrThread thread is started. I haven't gone into the stub manager
          here. The important thing is that eventually a _StubReaderThread is
          started which accepts marshalled DCOM RPCs, and then passes them to
          IRpcStubBuffer::Invoke on the correct stub object which in turn
          demarshals the packet and performs the call. The threads started by our
          implementation of DCOM are never terminated, they just hang around until
          the process dies.
        </para>

        <para>
          Remember that I said our DCOM doesn't use RPC? Well, you might be
          thinking "but we use IRpcStubBuffer like we're supposed to ... isn't
          that provided by MIDL which generates code that uses the NDR APIs?". If
          so pat yourself on the back, you're still with me. Go get a cup of
          coffee.
        </para>

      </sect2>

      <sect2>
        <title>Typelib Marshaller</title>

        <para>
          In fact, the reason for the PSFactoryBuffer layer of indirection is
          because not all interfaces are marshalled using MIDL generated code.
          Why not? Well, to understand <emphasis>that</emphasis>
          you have to see that one of the
          driving forces behind OLE and by extension DCOM was the development of
          Visual Basic. Microsoft wanted VB developers to be first class citizens
          in the COM world, but things like writing IDL and compiling them with a
          C compiler into DLLs wasn't easy enough.
        </para>

        <para>
          So, type libraries were invented. Actually they were invented as part of
          a parallel line of COM development known as "OLE Automation", but let's
          not get into that here. Type libraries are basically binary IDL files,
          except that despite there being two type library formats neither of them
          can fully express everything expressable in IDL. Anyway, with a type
          library (which can be embedded as a resource into a DLL) you have
          another option beyond compiling MIDL output - you can set the
          ProxyStubClsId32 registry entry for your interfaces to the CLSID of the
          "type library marshaller" or "universal marshaller". Both terms are
          used, but in the Wine source it's called the typelib marshaller.
        </para>

        <para>
          The type library marshaller constructs proxy and stub objects on the
          fly. It does so by having generic marshalling glue which reads the
          information from the type libraries, and takes the parameters directly
          off the stack. The CreateProxy method actually builds a vtable out of
          blocks of assembly stitched together which pass control to _xCall, which
          then does the marshalling. You can see all this magic in
          dlls/oleaut32/tmarshal.c
        </para>

        <para>
          In the case of InstallShield, it actually comes with typelibs for all
          the interfaces it needs to marshal (fixme: is this right?), but they
          actually use a mix of MIDL and typelib marshalling. In order to cover up
          for the fact that we don't really use RPC they're all forced to go via
          the typelib marshaller - that's what the 1 || hack is for and what the
          "Registering non-automation type library!" warning is about (I think).
        </para>
      </sect2>

      <sect2>
        <title>Apartments</title>

        <para>
          Before a thread can use COM it must enter an apartment. Apartments are 
          an abstraction of a COM objects thread safety level. There are many types
          of apartment but the only two we care about right now are single threaded
          apartments (STAs) and the multi-threaded apartment (MTA).
        </para>

        <para>
          Any given process may contain at most one MTA and potentially many STAs.
          This is because all objects in MTAs never care where they are invoked from
          and hence can all be treated the same. Since objects in STAs do care, they
          cannot be treated the same.
        </para>

        <para>
          You enter an apartment by calling <function>CoInitializeEx()</function> and 
          passing the desired thread model in as a parameter. The default if you use 
          the deprecated <function>CoInitialize()</function> is a STA, and this is the 
          most common type of apartment used in COM.
        </para>

        <para>
          An object in the multi-threaded apartment may be accessed concurrently by
          multiple threads: eg, it's supposed to be entirely thread safe. It must also
          not care about thread-affinity, the object should react the same way no matter
          which thread is calling it.
        </para>

        <para>
          An object inside a STA does not have to be thread safe, and all calls upon it
          should come from the same thread - the thread that entered the apartment in
          the first place.
        </para>

        <para>
          The apartment system was originally designed to deal with the disparity between
          the Windows NT/C++ world in which threading was given a strong emphasis, and the
          Visual Basic world in which threading was barely supported and even if it had
          been fully supported most developers would not have used it. Visual Basic code
          is not truly multi-threaded, instead if you start a new thread you get an entirely
          new VM, with separate sets of global variables. Changes made in one thread do
          <emphasis>not</emphasis> reflect in another, which pretty much violates the
          expected semantics of multi-threading entirely but this is Visual Basic, so what
          did you expect? If you access a VB object concurrently from multiple threads,
          behind the scenes each VM runs in a STA and the calls are marshaled between the
          threads using DCOM.
        </para>

        <para>
          In the Windows 2000 release of COM, several new types of apartment were added, the
          most important of which are RTAs (the rental threaded apartment) in which concurrent
          access are serialised by COM using an apartment-wide lock but thread affinity is
          not guaranteed. 
        </para>
      </sect2>

      <sect2>
        <title>Structure of a marshaled interface pointer</title>

        <para>
          When an interface is marshaled using <function>CoMarshalInterface()</function>,
          the result is a serialized OBJREF structure. An OBJREF actually contains a union,
          but we'll be assuming the variant that embeds a STDOBJREF here which is what's
          used by the system provided standard marshaling. A STDOBJREF (standard object
          reference) consists of the magic signature 'MEOW', then some flags, then the IID
          of the marshaled interface. Quite what MEOW stands for is a mystery, but it's
          definitely not "Microsoft Extended Object Wire". Next comes the STDOBJREF flags,
          identified by their SORF_ prefix. Most of these are reserved, and their purpose
          (if any) is unknown, but a few are defined.
        </para>

        <para>
          After the SORF flags comes a count of the references represented by this marshaled
          interface. Typically this will be 5 in the case of a normal marshal, but may be 0
          for table-strong and  table-weak marshals (the difference between these is explained below).
          The reasoning is this: In the general case, we want to know exactly when an object
          is unmarshaled and released, so we can accurately control the lifetime of the stub
          object. This is what happens when cPublicRefs is zero. However, in many cases, we
          only want to unmarshal an object once. Therefore, if we strengthen the rules to say
          when marshaling that we will only unmarshal once, then we no longer have to know when
          it is unmarshaled. Therefore, we can give out an arbitrary number of references when
          marshaling and basically say "don't call me, except when you die."
        </para>

        <para>
          The most interesting part of a STDOBJREF is the OXID, OID, IPID triple. This triple
          identifies any given marshaled interface pointer in the network. OXIDs are apartment
          identifiers, and are supposed to be unique network-wide. How this is guaranteed is
          currently unknown: the original algorithm Windows used was something like the current
          UNIX time and a local counter.
        </para>

        <para>
          OXIDs are generated and registered with the OXID resolver by performing local RPCs
          to the RPC subsystem (rpcss.exe). In a fully security-patched Windows system they
          appear to be randomly generated. This registration is done using the
          <function>ILocalOxidResolver</function> interface, however the exact structure of
          this interface is currently unknown.
        </para>

        <para>
          OIDs are object identifiers, and identify a stub manager. The stub manager manages
          interface stubs. For each exported COM object there are multiple interfaces and
          therefore multiple interface stubs (<function>IRpcStubBuffer</function> implementations).
          OIDs are apartment scoped. Each ifstub is identified by an IPID, which identifies
          a marshaled interface pointer. IPIDs are apartment scoped.
        </para>

        <para>
          Unmarshaling one of these streams therefore means setting up a connection to the
          object exporter (the apartment holding the marshaled interface pointer) and being
          able to send RPCs to the right ifstub. Each apartment has its own RPC endpoint and
          calls can be routed to the correct interface pointer by embedding the IPID into the
          call using RpcBindingSetObject. IRemUnknown, discussed below, uses a reserved IPID.
          Please note that this is true only in the current implementation. The native version
          generates an IPID as per any other object and simply notifies the SCM of this IPID.
        </para>

        <para>
          Both standard and handler marshaled OBJREFs contains an OXID resolver endpoint which
          is an RPC string binding in a DUALSTRINGARRAY. This is necessary because an OXID
          alone is not enough to contact the host, as it doesn't contain any network address
          data. Instead, the combination of the remote OXID resolver RPC endpoint and the OXID
          itself are passed to the local OXID resolver. It then returns the apartment string binding.
        </para>

        <para>
          This step is an optimisation: technically the OBJREF itself could contain the string
          binding of the apartment endpoint and the OXID resolver could be bypassed, but by using
          this DCOM can optimise out a server round-trip by having the local OXID resolver cache
          the query results. The OXID resolver is a service in the RPC subsystem (rpcss.exe) which
          implements a raw (non object-oriented) RPC interface called <function>IOXIDResolver</function>.
          Despite the identical naming convention this is not a COM interface.
        </para>

        <para>
          Unmarshaling an interface pointer stream therefore consists of
          reading the OXID, OID and IPID from the STDOBJREF, then reading
          one or more RPC string bindings for the remote OXID resolver.
          Then <function>RpcBindingFromStringBinding</function> is used
          to convert this remote string binding into an RPC binding handle
          which can be passed to the local 
          <function>IOXIDResolver::ResolveOxid</function> implementation
          along with the OXID. The local OXID resolver consults its list
          of same-machine OXIDs, then its cache of remote OXIDs, and if
          not found does an RPC to the remote OXID resolver using the
          binding handle passed in earlier. The result of the query is
          stored for future reference in the cache, and finally the
          unmarshaling application gets back the apartment string binding,
          the IPID of that apartments <function>IRemUnknown</function>
          implementation, and a security hint (let's ignore this for now).
        </para>

        <para>
          Once the remote apartments string binding has been located the
          unmarshalling process constructs an RPC Channel Buffer
          implementation with the connection handle and the IPID of the
          needed interface, loads and constructs the 
          <function>IRpcProxyBuffer</function> implementation for that
          IID and connects it to the channel. Finally the proxy is passed
          back to the application. 
        </para>
      </sect2>

      <sect2>
        <title>Handling IUnknown</title>

        <para>
          There are some subtleties here with respect to IUnknown. IUnknown
          itself is never marshaled directly: instead a version of it
          optimised for network usage is used. IRemUnknown is similar in
          concept to IUnknown except that it allows you to add and release
          arbitrary numbers of references at once, and it also allows you to
          query for multiple interfaces at once.
        </para>

        <para>
          IRemUnknown is used for lifecycle management, and for marshaling
          new interfaces on an object back to the client. Its definition can
          be seen in dcom.idl - basically the IRemUnknown::RemQueryInterface
          method takes an IPID and a list of IIDs, then returns STDOBJREFs
          of each new marshaled interface pointer.
        </para>

        <para>
          There is one IRemUnknown implementation per apartment, not per
          stub manager as you might expect. This is OK because IPIDs are
          apartment not object scoped (In fact, according to the DCOM draft
          spec, they are machine-scoped, but this implies apartment-scoped).
        </para>
      </sect2>

      <sect2>
        <title>Table marshaling</title>

        <para>
          Normally once you have unmarshaled a marshaled interface pointer
          that stream is dead, you can't unmarshal it again. Sometimes this
          isn't what you want. In this case, table marshaling can be used. 
          There are two types: strong and weak. In table-strong marshaling,
          selected by a specific flag to <function>CoMarshalInterface()</function>,
          a stream can be unmarshaled as many times as you like. Even if
          all the proxies are released, the marshaled object reference is
          still valid. Effectively the stream itself holds a ref on the object.
          To release the object entirely so its server can shut down, you 
          must use <function>CoReleaseMarshalData()</function> on the stream.
        </para>

        <para>
          In table-weak marshaling the stream can be unmarshaled many times,
          however the stream does not hold a ref. If you unmarshal the
          stream twice, once those two proxies have been released remote
          object will also be released. Attempting to unmarshal the stream
          at this point will yield <function>CO_E_DISCONNECTED</function>.
        </para>
      </sect2>

      <sect2>
        <title>RPC dispatch</title>

        <para>
          Exactly how RPC dispatch occurs depends on whether the exported
          object is in a STA or the MTA. If it's in the MTA then all is
          simple: the RPC dispatch thread can temporarily enter the MTA,
          perform the remote call, and then leave it again. If it's in a
          STA things get more complex, because of the requirement that only
          one thread can ever access the object.
        </para>

        <para>
          Instead, when entering a STA a hidden window is created implicitly
          by COM, and the user must manually pump the message loop in order
          to service incoming RPCs. The RPC dispatch thread performs the
          context switch into the STA by sending a message to the apartments
          window, which then proceeds to invoke the remote call in the right
          thread.
        </para>

        <para>
          RPC dispatch threads are pooled by the RPC runtime. When an incoming
          RPC needs to be serviced, a thread is pulled from the pool and
          invokes the call. The main RPC thread then goes back to listening
          for new calls. It's quite likely for objects in the MTA to therefore
          be servicing more than one call at once.
        </para>
      </sect2>

      <sect2>
        <title>Message filtering and re-entrancy</title>

        <para>
          When an outgoing call is made from a STA, it's possible that the
          remote server will re-enter the client, for instance to perform a
          callback. Because of this potential re-entrancy, when waiting for
          the reply to an RPC made inside a STA, COM will pump the message loop.
          That's because while this thread is blocked, the incoming callback
          will be dispatched by a thread from the RPC dispatch pool, so it
          must be processing messages.
        </para>

        <para>
          While COM is pumping the message loop, all incoming messages from
          the operating system are filtered through one or more message filters.
          These filters are themselves COM objects which can choose to discard,
          hold or forward window messages. The default message filter drops all
          input messages and forwards the rest. This is so that if the user
          chooses a menu option which triggers an RPC, they then cannot choose
          that menu option *again* and restart the function from the beginning.
          That type of unexpected re-entrancy is extremely difficult to debug,
          so it's disallowed.
        </para>

        <para>
          Unfortunately other window messages are allowed through, meaning that
          it's possible your UI will be required to repaint itself during an
          outgoing RPC. This makes programming with STAs more complex than it
          may appear, as you must be prepared to run all kinds of code any time
          an outgoing call is made. In turn this breaks the idea that COM
          should abstract object location from the programmer, because an
          object that was originally free-threaded and is then run from a STA
          could trigger new and untested codepaths in a program.  
        </para>
      </sect2>

      <sect2>
        <title>Wrapup</title>

        <para>
          Theres are still a lot of topics that have not been covered:
        </para>
        
        <itemizedlist>
          <listitem><para> Format strings/MOPs</para></listitem>
          
          <listitem><para> IRemoteActivation</para></listitem>
          
          <listitem><para> Complex/simple pings, distributed garbage collection</para></listitem>

          <listitem><para> Marshalling IDispatch</para></listitem>
          
          <listitem><para> ICallFrame</para></listitem>
          
          <listitem><para> Interface pointer swizzling</para></listitem>
          
          <listitem><para> Runtime class object registration (CoRegisterClassObject), ROT</para></listitem>
          
          <listitem><para> Exactly how InstallShield uses DCOM</para></listitem>
        </itemizedlist>

      </sect2>

      <sect2>
        <title>Further Reading</title>

        <para>
          Most of these documents assume you have knowledge only contained in
          other documents. You may have to reread them a few times for it all to
          make sense. Don't feel you need to read these to understand DCOM, you
          don't, you only need to look at them if you're planning to help
          implement it.
        </para>

        <itemizedlist>
            <listitem><para>
              <ulink url="http://www-csag.ucsd.edu/individual/achien/cs491-f97/projects/dcom-writeup.ps">
                http://www-csag.ucsd.edu/individual/achien/cs491-f97/projects/dcom-writeup.ps</ulink>

            </para></listitem>

            <listitem><para>
              <ulink url="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_n2p_459u.asp">
                http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_n2p_459u.asp</ulink>

            </para></listitem>


            <listitem><para>
              <ulink url="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_q2z_5ygi.asp">
                http://msdn.microsoft.com/library/default.asp?url=/library/en-us/com/htm/cmi_q2z_5ygi.asp</ulink>
            </para></listitem>


            <listitem><para>
                <ulink url="http://www.microsoft.com/msj/0398/dcom.aspx">
                  http://www.microsoft.com/msj/0398/dcom.aspx</ulink>
            </para></listitem>

            <listitem><para>
                <ulink url="http://www.microsoft.com/ntserver/techresources/appserv/COM/DCOM/4_ConnectionMgmt.asp">
                  http://www.microsoft.com/ntserver/techresources/appserv/COM/DCOM/4_ConnectionMgmt.asp</ulink>
            </para></listitem>


            <listitem><para><ulink url="http://www.idevresource.com/com/library/articles/comonlinux.asp">
                  http://www.idevresource.com/com/library/articles/comonlinux.asp</ulink>

                (unfortunately part 2 of this article does not seem to exist anymore, if it was ever written)</para></listitem>
        </itemizedlist>
      </sect2>
    </sect1>
  </chapter>

<!-- Keep this comment at the end of the file
Local variables:
mode: sgml
sgml-parent-document:("wine-devel.sgml" "set" "book" "part" "chapter" "")
End:
-->