Mac/Demo/building.html

   1 <HTML>
   2 <HEAD>
   3 <TITLE>Building Mac Python from source</TITLE>
   4 </HEAD>
   5 <BODY>
   6 <H1>Building Mac Python from source</H1>
   7 <HR>
   8
   9 This document explains how to build MacPython from source. This is
  10 necessary if you want to write extension modules for 68K Python, and
  11 currently also probably the easiest way to build PPC extension
  12 modules. Building Python is not something to be undertaken lightly,
  13 the process is not very streamlined so you need a reasonable working
  14 knowledge of the CodeWarrior development environment, a good net
  15 connection and probably quite some time too. <p>
  16
  17 The information density in this file is high, so you should probably
  18 print it and read it at your leasure. Most things are explained only
  19 once (and probably in the wrong place:-). <p>
  20
  21 I am very interested in feedback on this document, contact me at <A
  22 HREF="mailto:jack@cwi.nl">&lt;jack@cwi.nl&gt;</A> or send your
  23 comments to the <A
  24 HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
  25 Interest Group</A>.
  26
  27 <H2>What you need.</H2>
  28
  29 The following things you definitely need:
  30
  31 <UL>
  32
  33 <LI> You need a MacPython source distribution, of course. You can
  34 obtain one from <A
  35 HREF="ftp://ftp.cwi.nl/pub/jack/python/mac">ftp://ftp.cwi.nl/pub/jack/python/mac</A>,
  36 and possibly also from the standard <A
  37 HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
  38 site</A>. Everything you need is also included in the standard Python
  39 source distribution, but the organization is different. Look in
  40 directory <code>Mac/mwerks/projects</code> for the project files and
  41 related stuff.
  42
  43 <LI> You need MetroWerks CodeWarrior. The current distribution has
  44 been built with version 9 of CodeWarrior. Ordering information is
  45 available on the <A HREF="http://www.metrowerks.com/">MetroWerks
  46 homepage</A>. You might still be able to build Python with MPW or
  47 Think/Symantec C but you are basically on your own.
  48
  49 <LI> You need GUSI, the Grand Unified Socket Interface, by Matthias
  50 Neeracher. The current distribution has been built with CWGUSI 1.7.2,
  51 obtainable from <A
  52 HREF="ftp://ftp.switch.ch/software/mac/src/mw_c">ftp://ftp.switch.ch/software/mac/src/mw_c</A>.
  53 It is possible to build a non-GUSI Python, see below.
  54
  55 </UL>
  56
  57 <A NAME="optional">The MacPython project files are configured to
  58 include a plethora of optional modules</A>, and these modules need a
  59 number extra packages. To use the project files as-is you have to
  60 download these packages too. PPC and CFM68K Python have all such modules as
  61 dynamically loaded modules, so if you don't need a certain package it
  62 suffices to just refrain from builing the extension module. For static 68K
  63 Python things are a bit more complicated: you have to edit the
  64 interpreter project file to remove the reference to the module (and
  65 the libraries it uses), and edit the <code>Mac:mwerks:mwerks_nonshared_config.h</code>
  66 file to remove the <code>USE_...</code> line.  Here are the locations for the various things
  67 you need:
  68
  69 <UL>
  70
  71 <LI> Tcl and Tk can be obtained from <A
  72 HREF="ftp://ftp.smli.com/pub/tcl/mac/">ftp://ftp.smli.com/pub/tcl/mac/</A>.
  73 The current distributions, Tcl 7.5p1 and Tk 4.1p1 need a bit of work,
  74 see the section on <A HREF="#tcltk">building Tcl/Tk Python</A>
  75 below. Get the "full source" distribution, which includes MoreFiles.
  76
  77 <LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
  78 HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
  79 was built using version 1.2, which you can obtain from <A
  80 HREF="ftp://rhino.harvard.edu/pub/dan/WASTE">&lt;ftp://rhino.harvard.edu/pub/dan/WASTE&gt;</A>
  81 and various other places.
  82
  83 <LI> JPEG library by the Independent JPEG Group. Python is still built
  84 using an archaic version of the library, version 4. It can be obtained
  85 from the <A HREF="ftp://ftp.cwi.nl/pub/jack/python/mac">
  86 ftp://ftp.cwi.nl/pub/jack/python/mac</A> directory, complete with CW8
  87 projects. If someone manages to build Python with the version 6
  88 library I would be grateful if they sent me the changes needed.  The
  89 most recent JPEG library can always be obtained from <A
  90 HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>.
  91
  92 <LI> The netpbm/pbmplus and libtiff libraries. The netpbm distribution
  93 (which includes libtiff) is generally available on Internet ftp
  94 servers. For Python pbmplus, an older incarnation of netpbm, is
  95 functionally identical to netpbm, since Python only uses the library
  96 and not the complete applications. A distribution with correct
  97 projects and library source only is available from, you guessed it, <A
  98 HREF="ftp://ftp.cwi.nl/pub/jack/python/mac">ftp://ftp.cwi.nl/pub/jack/python/mac</A>.
  99
 100 </UL>
 101
 102 <H2>Setting Up</H2>
 103
 104 Now that you have collected everything you should start with building
 105 the various parts. Everything is independent, with the single
 106 exception that Tcl and Tk depend on CWGUSI. If you don't want to fix
 107 access paths try to set things up as follows:
 108
 109 <PRE>
 110 Top-level-folder:
 111         CWGUSI
 112         imglibs
 113                 libjpeg
 114                 pbmplus
 115                         libtiff
 116         MoreFiles 1.4.2 (not needed by Python, only by tcl/tk)
 117         Python
 118         Tcl 7.5
 119         Tk 4.1
 120         Waste 1.2 distribution (if you want waste)
 121 </PRE>
 122
 123 Now build all the libraries. In <code>CWGUSI</code> you build the
 124 projects <code>GUSI.68K.µ</code>, <code>GUSI.CFM68K.µ</code>
 125 and <code>GUSI.PPC.µ</code>, in
 126 <code>MoreFiles</code>, <code>libjpeg</code>, <code>pbmplus</code>
 127 and<code>libtiff</code> you build all projects. Tcl/tk is a special
 128 case, see below. Of course, if you are only interested in one of
 129 static 68K, CFM68K or PPC you can skip building the other libraries.
 130
 131 <H2><A NAME="tcltk">Building Tcl/Tk</H2>
 132
 133 You need to make a minor organizational change to the Tcl/Tk
 134 distribution. The current instructions are for the
 135 <code>tcl7.5.1</code> and <code>tk4.1.1</code> distribution:
 136
 137 <UL>
 138
 139 <LI> Rename the <code>compat</code> folders to <code>(compat)</code>
 140 in both the Tcl and Tk folders.
 141
 142 <LI> In the Tcl folder, move <code>strncasecmp.c</code> and
 143 <code>tclErrno.h</code> from <code>(compat)</code> to the main Tcl
 144 folder.
 145
 146 <LI> Fix <code>dnr.c</code> as provided by MetroWerks by inserting
 147 <pre><code> #pragma ANSI_strict off </code></pre> at the
 148 beginning. The tcl library is built with strict ANSI on, and this file
 149 uses C++ style comments.
 150
 151 <LI> If you want to build <code>SimpleTcl</code> and
 152 <code>SimpleTk</code> you will probably have to remove the references
 153 to <code>libmoto</code> from the project.
 154
 155 <LI> You are <EM>strongly</EM> advised to add a line
 156 <pre><code>
 157 #define USE_TCLALLOC 1
 158 </code></pre>
 159 somewhere at the beginning of <code>MW_TclHeader.pch</code>.
 160 As distributed, tcl and tk assume that malloc calls always succeed and
 161 use the resulting pointer without checking for <code>NULL</code>
 162 values. Needless to say, this wreaks havoc on a Macintosh.
 163
 164 <LI> If you want to build for CFM68K you have to modify <code>TclMacNotify.c</code>
 165 because there is an error in the Apple Universal headers (sic!). Read the
 166 comments at the beginning of <code>Mac:Python:macglue.c</code> and copy the
 167 code to <code>TclMacNotify.c</code>. If you get linker errors on <code>GetEvQHdr</code>
 168 you have not done this correctly.
 169
 170 </UL>
 171
 172 Build first the MoreFiles library, then the Tcl library, then
 173 SimpleTcl (test it by typing <code>ls -l</code> in the window you get)
 174 then the Tk library, then SimpleTk (which can again be tested with
 175 <code>ls -l</code>). If this all worked you are all set to try
 176 building Python.
 177
 178 <H2>Building Waste</H2>
 179
 180 You do not need to build the Waste libraries, as Python includes the
 181 source modules themselves. You have to make one modification,
 182 though. In file <code>ICCFMGlue.c</code> in folder <code>Minimal IC
 183 APIs</code>, add the following lines:
 184
 185 <blockquote><pre><code>
 186 #include <Gestalt.h>
 187 #include <Errors.h>
 188 </code></pre></blockquote>
 189
 190 <H2>The organization of the Python source tree</H2>
 191
 192 Time for a short break, while we have a look at the organization of
 193 the Python source tree.  At the top level, we find the following
 194 folders:
 195
 196 <DL>
 197 <DT> build.mac68k.stand
 198 <DD> This is where you will build static 68K interpreters.
 199
 200 <DT> build.mac68k.shared
 201 <DD> This is where you build the CFM68K shared library, interpreter
 202 and applet framework.
 203
 204 <DT> build.macppc.shared
 205 <DD> This is where you build the PPC shared library, interpreter and
 206 applet framework. You can also build the fat applet framework here.
 207
 208 <DT> build.macppc.stand
 209 <DD> This is where you build a nonshared PPC interpreter (optional).
 210
 211 <DT> Demo
 212 <DD> Demo programs that are not Mac-specific. Some of these may not
 213 work, the file <code>README-Mac</code> has some details.
 214
 215 <DT> Extensions
 216 <DD> Extensions to the interpreter that are not Mac-specific. Contains
 217 only the <code>img</code> extension in this distribution. Extensions
 218 are <em>not</em> built here, as they are on Unix, but incorporated in
 219 the core interpreter or built as plugin modules.
 220
 221 <DT> Grammar
 222 <DD> The Python grammar. Included for reference only, you cannot build
 223 the parser on a Mac.
 224
 225 <DT> Include
 226 <DD> Machine-independent header files.
 227
 228 <DT> Modules
 229 <DD> Machine-independent optional modules. Not all of these will work
 230 on the Mac.
 231
 232 <DT> Objects
 233 <DD> Machine-independent code for various objects. Most of these are
 234 not really optional: the interpreter will not function without them.
 235
 236 <DT> Parser
 237 <DD> The Python parser (machine-independent).
 238
 239 <DT> PlugIns
 240 <DD> This is where you build the PPC and CFM68K dynamically-loaded plugin modules.
 241
 242 <DT> Python
 243 <DD> The core interpreter. Most files are machine-independent, some
 244 are unix-specific and not used on the Mac.
 245
 246 <DT> Tools
 247 <DD> Tools for python developers. Contains <code>modulator</code>
 248 which builds skeleton C extension modules and <code>bgen</code> which
 249 generates complete interface modules from information in C header
 250 files. There are some readme files, but more documentation is sorely
 251 needed.
 252
 253 </DL>
 254
 255 All the mac-specific stuff lives in the <code>Mac</code> folder:
 256 <DL>
 257
 258 <DT> Compat
 259 <DD> Unix-compatability routines. Some of these are not used anymore,
 260 since CWGUSI provides a rather complete emulation, but you may need
 261 these if you are trying to build a non-GUSI python.
 262
 263 <DT> Demo
 264 <DD> Mac-specific demo programs, some of them annotated.
 265
 266 <DT> Include
 267 <DD> Mac-specific but compiler-independent include files.
 268
 269 <DT> Lib
 270 <DD> Mac-specific standard modules. The <code>toolbox</code> folder
 271 contains modules specifically needed with various MacOS toolbox
 272 interface modules.
 273
 274 <DT> Modules
 275 <DD> Mac-specific builtin modules. Theoretically these are all
 276 optional, but some are rather essential (like
 277 <code>macmodule</code>). A lot of these modules are generated with
 278 <code>bgen</code>, in which case the bgen input files are included so
 279 you can attempt to regenerate them or extend them.
 280
 281 <DT> MPW
 282 <DD> MPW-specific files. These have not been used or kept up-to-date
 283 for a long time, so use at your own risk.
 284
 285 <DT> mwerks
 286 <DD> Mwerks-specific sources and headers. Contains glue code for
 287 Pythons shared-library architecture, a replacement for
 288 <code>malloc</code> and a directory with various projects for building
 289 variations on the Python interpreter. The <code>mwerks_*.h</code>
 290 files here are the option-setting files for the various interpreters
 291 and such, comparable to the unix command-line <code>-D</code> options
 292 to the compiler. Each project uses the correct option file as its
 293 "prefix file" in the "C/C++ language" settings. Disabling optional
 294 modules (for the 68K interpreter), building non-GUSI interpreters and
 295 various other things are accomplished by modifying these files (and
 296 possibly changing the list of files included in the project window, of
 297 course).
 298
 299 <DT> Python
 300 <DD> Mac-specific parts of the core interpreter.
 301
 302 <DT> Resources
 303 <DD> Resource files needed to build the interpreter.
 304
 305 <DT> Scripts
 306 <DD> A collection of various mac-specific Python scripts. Some are
 307 essential, some are useful but few are documented, so you will have to
 308 use your imagination to work them out.
 309
 310 <DT> Unsupported
 311 <DD> Modules that are not supported any longer but may still work with a little effort.
 312 </DL>
 313
 314 <H2>Building the 68K interpreter</H2>
 315
 316 If you have all the optional libraries mentioned <A
 317 HREF="#optional">above</A> loaded buildin Python for 68K macs is a
 318 breeze: open the project in the folder <code>build.mac68k.stand</code>
 319 and build it.  Do <em>not</em> run it yet, this will possibly result
 320 in a garbled preferences file. <p>
 321
 322 First remove the <code>Python preferences</code> file from your
 323 preference folder, only if you had an older version of Python
 324 installed.  (this is also what you do if you did not heed the last
 325 sentence of the preceeding paragraph). Next, move the interpreter to
 326 the main Python folder (up one level) and run it there. This will
 327 create a correct initial preferences file. You are now all set, and
 328 your tree should be completely compatible with a binary-only
 329 distribution. Read the release notes
 330 (<code>Relnotes-somethingorother</code>) and
 331 <code>ReadMeOrSuffer</code> in the <code>Mac</code> folder.
 332
 333 <H2>Building the CFM68K interpreter</H2>
 334
 335 Building the CFM68K interpreter is as almost exactly the same as building
 336 the PPC interpreter, with the exception that you should read "CFM68K"
 337 for "PPC" every time. Continue reading with the next section.
 338
 339 <H2>Building the PPC interpreter</H2>
 340
 341 First you build the interpreter, core library and applet skeleton in
 342 folder <code>build.macppc.stand</code>.  The order to build things is
 343 the following:
 344
 345 <DL>
 346 <DT> MWRuntimeStaticLib
 347 <DD> A modified version of the MetroWerks runtime library that is
 348 suitable for Pythons' shared library architecture. The sources all
 349 come from the MW distribution.
 350
 351 <DT> PythonCorePPC
 352 <DD> The shared library that contains the bulk of the interpreter and
 353 its resources. It is a good idea to immedeately put an alias to this
 354 shared library in the <code>Extensions</code> folder of your system
 355 folder.  Do exactly that: put an <em>alias</em> there, copying or
 356 moving the file will cause you grief later.
 357
 358 <DT> PythonPPC
 359 <DD> The interpreter. This is basically a routine to call out to the
 360 shared library. Because of the organization of GUSI it also contains
 361 the Gusi settings resource (together with a ResEdit template, so you
 362 can change the gusi settings should you feel like doing so).  Do
 363 <em>not</em> run it yet, this will possibly result in a garbled
 364 preferences file. <p>
 365
 366 <DT> PythonApplet
 367 <DD> The applet skeleton application. Very similar to
 368 <code>PythonPPC</code>, but it calls to a different entrypoint in the
 369 core library. The <code>mkapplet</code> script will copy this complete
 370 file, and add a <code>'PYC '</code> with the module to generate an
 371 applet. <p>
 372
 373 </DL>
 374
 375 After creating the alias to <code>PythonCore</code> you should move
 376 <code>PythonPPC</code> to the main Python folder. Next you remove any
 377 old <code>Python Preferences</code> file from the
 378 <code>Preferences</code> folder (if you had python installed on your
 379 system before) and run the interpreter once to create the correct
 380 preferences file. You should also make an alias to
 381 <code>PythonApplet</code> in the main Python folder. (again: making an
 382 alias is preferrable to copying or moving the file, since this will
 383 cause the correct file to be used if you ever rebuild
 384 PythonApplet). <p>
 385
 386 Next, you have to build the extension modules in the
 387 <code>PlugIns</code> folder. Open each project with <code>.ppc</code> in the
 388 name and build it. After all
 389 the dynamically loaded modules are built you have to create a number
 390 of aliases: some modules live together in a single dynamic
 391 library. Run the <code>MkPluginAliases.py</code> script from
 392 <code>Mac:scripts</code> to create the aliases. <p>
 393
 394 Finally, you must build the standard applets:
 395 <code>EditPythonPrefs</code>, <code>mkapplet</code>, etc. This is
 396 easiest done with the <code>fullbuild</code> script from
 397 <code>Mac:scripts</code>. Answer <em>no</em> to all questions except
 398 when it asks whether to build the applets. <p>
 399
 400 <BLOCKQUOTE>
 401 Actually, the <code>fullbuild</code> script can be used to build
 402 everything, but you need a fully-functional interpreter before you can
 403 use it (and one that isn't rebuilt in the process: you cannot rebuild
 404 a running program). You could copy the 68K interpreter to a different
 405 place and use that to run fullbuild, or use the standalone PPC python
 406 for this. I tend to keep a standalone interpreter in a safe place for
 407 this use only.
 408 </BLOCKQUOTE>
 409
 410 You are all set now, and should read the release notes and
 411 <code>ReadMeOrSuffer</code> file from the <code>Mac</code> folder.
 412
 413 <H2>Rebuilding <code>.exp</code> files for PPC and CFM68K</H2>
 414
 415 Occasionally it may be necessary to rebuild your PythonCore <code>.exp</code>
 416 file, a file that controls which symbols are exported by your PythonCore
 417 shared library. Rebuild it if you get unexpected undefined symbols when you
 418 are building a plugin module. <p>
 419
 420 Rebuilding the .exp file is done by first removing the file and removing the
 421 reference to it in the project (in the "config" section). Next, build PythonCore.
 422 This will create a new .exp file. Edit this file to remove the references to
 423 the symbols <code>__initialize</code>, <code>__terminate</code>, <code>setjmp</code>,
 424 <code>longjmp</code> and <code>__ptmf_null</code>. Next, add the .exp file to the project
 425 again and rebuild PythonCore. <p>
 426
 427 This rather convoluted procedure is needed to ensure that plugin modules don't
 428 accidentally link with those entrypoints from PythonCore, which will not work because
 429 those routines have to be in the same code fragment as they are used from.
 430
 431 <H2>Odds and ends</H2>
 432
 433 Some remarks that I could not fit in elsewhere:
 434
 435 <UL>
 436
 437 <LI> It may be possible to use the <code>PythonCore</code> shared
 438 library to embed Python in another program, if your program can live
 439 with using GUSI for I/O. Use PythonCore in stead of your C library
 440 (or, at the very least, link it before the normal C library). Let me
 441 know whether this works.
 442
 443 <LI> It is possible to build PPC extension modules without building a
 444 complete Python. Take the binary distribution, add folders
 445 <code>Include</code>, <code>Mac:Include</code> and
 446 <code>Mac:mwerks</code> from the source distribution and you should be
 447 all set. A template for a dynamic module can be found in
 448 <code>xx.ppc.µ</code> or <code>xx.CFM68K.µ</code>.
 449
 450 <LI> The Python shared library architecture is a variant of the architecture
 451 described as "application with shared libraries and dropins" in the MetroWerks
 452 "Targeting MacOS" documentation. The Python Application and applet-template use
 453 the <code>AppRuntime.Lib</code> runtime library (with properly set CFM
 454 initialization and termination routines). PythonCore uses <code>ShlibRuntime.Lib</code>
 455 and <code>MWRuntimeStaticLib.Lib</code>, which is almost identical to the MW
 456 standard <code>MWRuntimeLib</code>, but not dynamically loaded. This library contains
 457 the part of the runtime that can (or must) be shared between all modules in the program.
 458 It is linked statically into PythonCore (and exported to the applications and plugins)
 459 so we do not have to distribute yet another shared library. Plugin modules use
 460 <code>ShlibRuntime.Lib</code> and obtain the rest from PythonCore. PythonCore uses a
 461 non-standard initialization entry point, <code>__initialize_with_resources</code>, to
 462 be able to obtain resources from the library file lateron. Plugins can do the same or
 463 use the standard <code>__initialize</code> entry point.
 464
 465
 466 <UL>
 467 </BODY>
 468 </HTML>