Mac/Demo/building.html

   1 <HTML>
   2 <HEAD>
   3 <TITLE>Building Mac Python from source</TITLE>
   4 </HEAD>
   5 <BODY>
   6 <H1>Building Mac Python from source</H1>
   7 <HR>
   8
   9 <B>Note</B>: This document is still for Python 2.0. It is wildly incorrect.
  10 <p>
  11
  12 This document explains how to build MacPython from source. This is
  13 necessary if you want to make modifications to the Python core. Building
  14 Python is not something to be undertaken lightly, you need a reasonable
  15 working knowledge of the CodeWarrior development environment, a good net
  16 connection and probably quite some time too. <p>
  17
  18 The information density in this file is high, so you should probably
  19 print it and read it at your leasure. Most things are explained only
  20 once (and probably in the wrong place:-). <p>
  21
  22 <blockquote>
  23 First a warning: this information may become outdated if a new CodeWarrior is
  24 released after MacPython. The
  25 <a href="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</a> will
  26 hopefully have updated instructions in that case.
  27 </blockquote>
  28
  29 I am very interested in feedback on this document, send your
  30 comments to the <A
  31 HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
  32 Interest Group</A>.
  33
  34 <H2>What you need.</H2>
  35
  36 The following things you definitely need:
  37
  38 <UL>
  39
  40 <LI> You need a MacPython source distribution, of course. You can
  41 obtain one from <A HREF="ftp://ftp.cwi.nl/pub/jack/python/mac">
  42 ftp://ftp.cwi.nl/pub/jack/python/mac</A> or from the companion webpage
  43 at <A HREF="http://www.cwi.nl/~jack/macpython.html">
  44 http://www.cwi.nl/~jack/macpython.html</A> (which has up-to-date links
  45 to the other packages needed too) and possibly also from the standard
  46 <A HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
  47 site</A>. <BR>
  48
  49 A better alternative is to check the sources straight out of the CVS
  50 repository, see below. Most of the packages mentioned here are also
  51 available through CVS. Check the section on <a href="#cvs">CVS
  52 repository use</a> below.
  53
  54 <LI> You need MetroWerks CodeWarrior. The current distribution has
  55 been built with CodeWarrior Pro 5.2. Ordering information is
  56 available on the <A HREF="http://www.metrowerks.com/">MetroWerks
  57 homepage</A>. Building Python with MPW or Think/Symantec C is
  58 probably impossible without major surgery.
  59
  60 <LI> You need GUSI version 2, the Grand Unified Socket Interface, by
  61 Matthias Neeracher. The original GUSI is obtainable from <A
  62 HREF="ftp://gusi.sourceforge.net/pub/gusi/">
  63 ftp://gusi.sourceforge.net/pub/gusi/</A>. At
  64 the moment Python is built with a slightly modified version of GUSI
  65 2.1.1, so it may be better to check the <A
  66 HREF="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</A>
  67 for a GUSI that is most easily used for building Python.
  68 <br>
  69
  70 The modified GUSI is also in the MacPython cvs source repository, in the
  71 directory <code>lib-src/GUSI2</code>. As GUSI is moving to sourceforge
  72 this info may be outdated by the time you read it so better check the
  73 MacPython homepage, probably.
  74 </UL>
  75
  76 <A NAME="optional">The MacPython project files are configured to
  77 include a plethora of optional modules</A>, and these modules need a
  78 number of extra packages. To use the project files as-is you have to
  79 download these packages too. Python has all such modules as
  80 dynamically loaded modules, so if you don't need a certain package it
  81 suffices to just refrain from builing the extension module.
  82 Here are the locations for the various things
  83 you need:
  84
  85 <UL>
  86
  87 <LI> Tcl and Tk are in a sad state on the Mac, the standard source distributions
  88 simply don't compile, so I have created a distribution especially for use
  89 with MacPython.
  90 See the section on <A HREF="#tcltk">building Tcl/Tk Python</A>
  91 below.
  92
  93 <LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
  94 HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
  95 was built using version 1.3, which you can obtain from <A
  96 HREF="http://www.boingo.com/waste">&lt;http://www.boingo.com/waste&gt;</A>
  97 and various other places.
  98
  99 <LI> Gdbm library for the Mac. Available from Jack's Mac software page at
 100 <A HREF="http://www.cwi.nl/~jack/macsoftware.html">
 101 http://www.cwi.nl/~jack/macsoftware.html</A> and <A HREF="ftp://ftp.cwi.nl/pub/jack/mac">
 102 ftp://ftp.cwi.nl/pub/jack/mac</A>. Also in the MacPython cvs repository at
 103 <code>lib-src/gdbm</code>.
 104
 105 <LI> JPEG library by the Independent JPEG Group. A version including
 106 Mac projects can be found at Jack's page mentioned above.
 107 The most recent JPEG library can always be obtained from <A
 108 HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>. Again,
 109 also in the MacPython cvs repository at <code>lib-src/jpeg</code>.
 110
 111 <LI> The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution
 112 (which includes libtiff) is generally available on Internet ftp
 113 servers. For Python pbmplus, an older incarnation of netpbm, is
 114 functionally identical to netpbm, since Python only uses the library
 115 and not the complete applications. A distribution with correct
 116 projects and library source only is available from, you guessed it, Jack's Mac software
 117 page mentioned above. And, guessed it again, in the MacPython cvs repository
 118 at <code>lib-src/netpbm</code>, etc. The only gotcha is that libtiff lives in
 119 <code>lib-src/netpbm/libtiff</code>, for historical reasons.
 120
 121 </UL>
 122
 123 <H2>Setting Up</H2>
 124
 125 Now that you have collected everything you should start with building
 126 the various parts.  If you don't want to fix
 127 access paths try to set things up as follows:
 128
 129 <PRE>
 130 Top-level-folder:
 131         GUSI2
 132         imglibs
 133                 jpeg
 134                 netpbm
 135                         libtiff
 136                 zlib
 137                 png
 138         gdbm
 139         Python
 140                 Modules
 141                 ...
 142                 Mac
 143                         Modules
 144                         Build
 145                         ...
 146         Tcl/Tk Folder
 147                 tcl8.0
 148                 tk8.0
 149                 MoreFiles 1.4.3
 150         Waste 1.3 distribution (if you want waste)
 151 </PRE>
 152
 153 If your setup of the libraries is exactly the same as mine (which is
 154 not very likely, unless you happen to work from the same CVS
 155 repository) you can use the project <code>buildlibs.prj</code> in the
 156 <code>:Mac:Build</code> folder to build all needed libraries in one
 157 fell swoop, otherwise you will have to build the libraries one by
 158 one. <p>
 159
 160 First build GUSI. If you didn't get the python-specific GUSI you have to
 161 massage some of the Python code (missing routines) and you'll miss the
 162 functionality of "delay console".
 163  <p>
 164
 165 Next, in
 166 <code>libjpeg</code>, <code>pbmplus</code>,
 167 <code>zlib</code>, <code>libpng</code>, <code>gdbm</code>,
 168 and<code>libtiff</code> you build all projects. Usually the projects are in "mac"
 169 subfolders, sometimes they are in the main folder. Tcl/tk is a special
 170 case, see below.
 171
 172 <H2><A NAME="tcltk">Building Tcl/Tk</H2>
 173
 174 The Tcl/Tk 8.3.0 source distribution does not work on the Mac. I have created
 175 an archive of the sources that I used to build _tkinter for MacPython,
 176 you can obtain this from <a
 177 href="ftp://ftp.cwi.nl/pub/jack/python/mac/tcltk830src-for-python.sit">
 178 ftp://ftp.cwi.nl/pub/jack/python/mac/tcltk830src-for-python.sit</a>. Only the
 179 libraries needed to build _tkinter for PPC have been fixed. <P>
 180
 181 Note that if you use a different release of Tcl and Tk than the ones
 182 I have used you may have to adapt the Python <code>tkresources.rsrc</code> file.
 183 This is easiest done by building <code>SimpleTk</code> and copying the TEXT, ICON
 184 and CRSR resources from it to <code>tkresources.rsrc</code>. This allows
 185 the <code>_tkinter</code> module to work without an installed Tk/Tcl on your
 186 machine. <P>
 187
 188 Also note that the <code>_tkinter.ppc.slb</code> that is normally distributed
 189 in the <code>PlugIns</code> folder is the one from the Imaging extension,
 190 which has some extra features needed by PIL (and which features should not
 191 hinder normal operation).
 192
 193 </UL>
 194
 195 Build first the Tcl library, then
 196 SimpleTcl (test it by typing <code>ls -l</code> in the window you get)
 197 then the Tk library, then SimpleTk (which can again be tested with
 198 <code>ls -l</code>). If this all worked you are all set to try
 199 building Python.
 200
 201 <H2>Building Waste</H2>
 202
 203 You do not need to build the Waste libraries, as Python includes the
 204 source modules themselves.
 205
 206 <H2>The organization of the Python source tree</H2>
 207
 208 Time for a short break, while we have a look at the organization of
 209 the Python source tree.  At the top level, we find the following
 210 folders:
 211
 212 <DL>
 213 <DT> Demo
 214 <DD> Demo programs that are not Mac-specific. Some of these may not
 215 work.
 216
 217 <DT> Extensions
 218 <DD> Extensions to the interpreter that are not Mac-specific. Contains
 219 the <code>img</code>, <code>Imaging</code> and <code>Numerical</code> extensions
 220 in this distribution.
 221
 222 <DT> Grammar
 223 <DD> The Python grammar. Included for reference only, you cannot build
 224 the parser on a Mac.
 225
 226 <DT> Include
 227 <DD> Machine-independent header files.
 228
 229 <DT> Modules
 230 <DD> Machine-independent optional modules. Not all of these will work
 231 on the Mac.
 232
 233 <DT> Objects
 234 <DD> Machine-independent code for various object types. Most of these are
 235 not really optional: the interpreter will not function without them.
 236
 237 <DT> Parser
 238 <DD> The Python parser (machine-independent).
 239
 240 <DT> Python
 241 <DD> The core interpreter. Most files are machine-independent, some
 242 are unix-specific and not used on the Mac.
 243
 244 <DT> Tools
 245 <DD> Tools for python developers. Contains <code>modulator</code> which
 246 builds skeleton C extension modules, <code>bgen</code> which generates
 247 complete interface modules from information in C header files and
 248 <code>freeze</code> which is used to turn Python scripts into real
 249 applications (used by MacFreeze and BuildApplication) There are some
 250 readme files, but more documentation is sorely needed.
 251
 252 </DL>
 253
 254 All the mac-specific stuff lives in the <code>Mac</code> folder:
 255 <DL>
 256 <DT> Build
 257 <DD> This is where the project files live and where you build the
 258 libraries, shared libraries, executables and plugin modules. All the
 259 resulting binaries, except for intermedeate results, are deposited in
 260 the toplevel folder or the Mac:PlugIns folder (for plugin modules).
 261
 262 <DT> Compat
 263 <DD> Unix-compatability routines. Most of these are not used anymore,
 264 since GUSI provides a rather complete emulation, but you may need
 265 these if you are trying to build a non-GUSI python.
 266
 267 <DT> Demo
 268 <DD> Mac-specific demo programs, some of them annotated.
 269
 270 <DT> Include
 271 <DD> Mac-specific but compiler-independent include files.
 272
 273 <DT> Lib
 274 <DD> Mac-specific standard modules. The <code>toolbox</code> folder
 275 contains modules specifically needed with various MacOS toolbox
 276 interface modules.
 277
 278 <DT> Modules
 279 <DD> Mac-specific builtin modules. Theoretically these are all
 280 optional, but some are rather essential (like
 281 <code>macmodule</code>). A lot of these modules are generated with
 282 <code>bgen</code>, in which case the bgen input files are included so
 283 you can attempt to regenerate them or extend them.
 284
 285 <DT> MPW
 286 <DD> MPW-specific files. These have not been used or kept up-to-date
 287 for a long time, so use at your own risk.
 288
 289 <DT> mwerks
 290 <DD> Mwerks-specific sources and headers. Contains glue code for
 291 Pythons shared-library architecture, a replacement for
 292 <code>malloc</code> and a directory with various projects for building
 293 variations on the Python interpreter. The <code>mwerks_*.h</code>
 294 files here are the option-setting files for the various interpreters
 295 and such, comparable to the unix command-line <code>-D</code> options
 296 to the compiler. Each project uses the correct option file as its
 297 "prefix file" in the "C/C++ language" settings. Disabling optional
 298 modules (for the 68K interpreter), building non-GUSI interpreters and
 299 various other things are accomplished by modifying these files (and
 300 possibly changing the list of files included in the project window, of
 301 course).
 302
 303 <DT> PlugIns
 304 <DD> This is where the PPC and CFM68K dynamically-loaded plugin modules live.
 305
 306 <DT> Python
 307 <DD> Mac-specific parts of the core interpreter.
 308
 309 <DT> Resources
 310 <DD> Resource files needed to build the interpreter.
 311
 312 <DT> Scripts
 313 <DD> A collection of various mac-specific Python scripts. Some are
 314 essential, some are useful but few are documented, so you will have to
 315 use your imagination to work them out.
 316
 317 <DT> Tools
 318 <DD> A collection of tools, usually bigger than those in the scripts
 319 folder. The important ones here are the IDE and macfreeze. The IDE is built
 320 with the buildIDE.py script, which puts the resulting applet in the toplevel
 321 folder. Macfreeze is usually invoked through the BuildApplication script,
 322 but for more control over the freezing process you can run the main script here.
 323
 324
 325 <DT> Unsupported
 326 <DD> Modules that are not supported any longer but may still work with a little effort.
 327 </DL>
 328
 329 <H2>Building the 68K interpreter</H2>
 330
 331 68K Python is no longer supported, and the projects are not included in the
 332 source distirbution anymore. If you really want to build Python for the 68K
 333 your best bet is to check the sources out of the CVS repository. The latest
 334 projects (in :Mac:build:) that support 68K development are tagged as such,
 335 and are dated around August 2000. If you plan on doing this announce it on
 336 the SIG, please. <p>
 337
 338 <H2>Building the PPC interpreter</H2>
 339
 340 First you optionally build the external libraries with buildlibs.prj. Next,
 341 the projects for
 342 interpreter, core library and applet skeleton  are all linked together, so
 343 building the fat target in <code>PythonEngine.prj</code>
 344 will result in everything being built. The
 345 resulting applications and fat shared library are deposited in the main
 346 Python folder. Finally, you build all the plugins with the plugins.prj project.
 347
 348 For completeness sake here is a breakdown of the projects:
 349
 350 <DL>
 351
 352 <DT> PythonCore (with subproject PythonCorePPC)
 353 <DD> The shared library that contains the bulk of the interpreter and
 354 its resources. It is a good idea to immedeately put an alias to this
 355 shared library in the <code>Extensions</code> folder of your system
 356 folder.  Do exactly that: put an <em>alias</em> there, copying or
 357 moving the file will cause you grief later if you rebuild the library and
 358 forget to copy it to the extensions folder again. The InstallPython applet
 359 will also do this, along with creating the plugin aliases. <br>
 360 Note that the subproject looks a bit silly nowadays (with no more CFM68K
 361 support) but you will have to live with that for this release.
 362
 363 <DT> PythonInterpeter
 364 <DD> The interpreter. This is basically a routine to call out to the
 365 shared library. Unlike in previous releases the same program is used for
 366 creating applets (for which formerly PythonApplet was used). <p>
 367
 368 <DT> Plugin projects
 369 <DD> Each plugin module has a separate project. The <code>Plugins.prj</code>
 370 project tries to build them all, but is known to be flakey. See <code>fullbuild</code>
 371 below for a better way to build everything.
 372 </DL>
 373
 374 After creating the alias to <code>PythonCore</code> you remove any old
 375 <code>Python 2.0b1 Preferences</code> file from the <code>Preferences</code> folder
 376 (if you had python installed on your system before) and run the interpreter once
 377 to create the correct preferences file.  <p>
 378
 379 Next, you have to build the extension modules.
 380 The <code>PlugIns.ppc</code> project has all the
 381 other projects as subprojects and builds everything (but see the gotcha above).
 382 <p>
 383
 384 Finally, you must build the standard applets:
 385 <code>EditPythonPrefs</code>, <code>BuildApplet</code>, etc. This is
 386 easiest done with the <code>fullbuild</code> script from
 387 <code>Mac:scripts</code>. <p>
 388
 389 <BLOCKQUOTE>
 390 Actually, the <code>fullbuild</code> script can be used to build
 391 everything, but you need a fully-functional interpreter before you can
 392 use it (and one that isn't rebuilt in the process: you cannot rebuild
 393 a running program). You could copy the interpreter to a different
 394 place and use that to run fullbuild. The <code>PythonStandSmall.prj</code>
 395 project builds an interpreter that is suited to this, and it can also come
 396 in handy if you need to debug things (which is easier in a static program). <p>
 397
 398 </BLOCKQUOTE>
 399
 400 You are all set now, and should read the release notes and
 401 <code>ReadMe</code> file from the <code>Mac</code> folder.
 402
 403 <H2>Rebuilding <code>.exp</code> files for PPC and CFM68K</H2>
 404
 405 Occasionally it may be necessary to rebuild your PythonCore <code>.exp</code>
 406 file, a file that controls which symbols are exported by your PythonCore
 407 shared library. Rebuild it if you get unexpected undefined symbols when you
 408 are building a plugin module. <p>
 409
 410 Rebuilding the .exp file is done by first removing the file and removing the
 411 reference to it in the project (in the "config" section). Next, build PythonCore.
 412 This will create a new .exp file. Edit this file to remove the references to
 413 the symbols <code>__initialize</code>, <code>__terminate</code>, <code>setjmp</code>,
 414 <code>longjmp</code>, <code>vec_longjmp</code>, <code>main</code> and (for PPC) <code>__ptmf_null</code> or (for
 415 CFM68K) <code>__start</code> and <code>dummy_init_routine</code>.
 416 Next, add the .exp file to the project
 417 again and rebuild PythonCore. <p>
 418
 419 This rather convoluted procedure is needed to ensure that plugin modules don't
 420 accidentally link with those entrypoints from PythonCore, which will not work because
 421 those routines have to be in the same code fragment as they are used from.
 422
 423 <H2><a name="cvs">Using the CVS source archive</a></H2>
 424
 425 It is possible (and probably best) to access the Python sources through remote CVS. The
 426 advantage of this is that you get the very latest sources, so any bug
 427 fixed or new features will be immedeately available. This is also the
 428 disadvantage, of course: as this is the same tree as is used for
 429 development it may sometimes be a little less stable. <p>
 430
 431 The CVS client of choice is Alexandre Parenteau's MacCVS. It can be
 432 obtained through the <a href="http://www.wincvs.org">WinCVS
 433 homepage</a>. MacCVS uses Internet Config to set file types correctly
 434 based on the filename extension. In the maccvs preferences you should
 435 also set (in the "binary files" section) "use mac encoding:
 436 applesingle" and (in the "text files" section) "use ISO latin 1
 437 conversion". <p>
 438
 439 It is also a good idea to disable Quicktime Exchange in the Quicktime
 440 control panel. Quicktime Exchange will magically map some extensions to
 441 filetypes, and this can seriously hinder you if, for instance, <code>.bmp</code>
 442 is not a Windows bitmap file. <p>
 443
 444 The machine-independent Python sources are checked out from the main
 445 Python CVS archive on sourceforge.net, see the <a
 446 href="http://www.python.org/download/cvs.html">Source access via
 447 CVS</a> page for details. When you check the sources out you will get
 448 something like <code>Python:dist:src</code>, and under that the
 449 <code>Modules</code>, <code>Lib</code>, etc hierarchy. The
 450 <code>src</code> folder should be renamed to <code>Python</code>, and
 451 is what this document refers to as the "toplevel Python folder". <P>
 452
 453 Next, <em>in a separate folder</em>, you check out the
 454 mac-specific sources. You then move the <code>Mac</code> folder from this
 455 checkout (the only folder with anything in it) to the Python source folder.
 456 Note that the checking out in a separate folder and moving is necessary,
 457 due to the way cvs works.
 458
 459 The CVS path to use for the mac stuff can be found
 460 at the <a href="http://www.cwi.nl/~jack/macpython.html">MacPython
 461 homepage</a>. Finally, you check out the external libraries needed in
 462 the parent of the toplevel Python folder. The CVS path for these libraries is
 463 also mentioned at the MacPython homepage. <p>
 464
 465 Neither of the pages mentioned above contains the passwords for the
 466 CVS sites, for obvious reasons, but they do contain instructions on
 467 how to obtain the passwords. <p>
 468
 469 You should end up with a folder structure as described at the top of this
 470 document. <p>
 471
 472 Note that while the Mac folder is now a subfolder of your toplevel Python
 473 folder this does not mean that they "act as one" as far as CVS is concerned.
 474 To update all your sources you have to do a "cvs update" in the toplevel
 475 Python folder and another one in the Mac folder. This is again a cvs problem:
 476 it cannot deal with subpackages coming from different repositories. <p>
 477
 478 <H2>Odds and ends</H2>
 479
 480 Some remarks that I could not fit in elsewhere:
 481
 482 <UL>
 483
 484 <LI> It may be possible to use the <code>PythonCore</code> shared
 485 library to embed Python in another program, if your program can live
 486 with using GUSI for I/O. Use PythonCore in stead of your MSL C library
 487 (or, at the very least, link it before the normal C library).
 488
 489 <LI> It is possible to build PPC extension modules without building a
 490 complete Python. The binary distribution installer can optionally install
 491 all the needed folders (the develop option). A template for a dynamic module can be found in
 492 <code>xx.prj</code>.
 493
 494 <LI> The Python shared library architecture is a variant of the architecture
 495 described as "application with shared libraries and dropins" in the MetroWerks
 496 "Targeting MacOS" documentation. The Python Application and applet-template use
 497 the <code>MSL AppRuntime.Lib</code> runtime library (with properly set CFM
 498 initialization and termination routines). PythonCore uses <code>MSL Runtime.Lib</code>,
 499 which is really intended for standalone programs but which we fool into working by
 500 providing a dummy main program.
 501 It is linked statically into PythonCore (and exported to the applications and plugins)
 502 so we do not have to distribute yet another shared library. Plugin modules use
 503 <code>MSL ShlibRuntime.Lib</code> (not the dropin runtime: modules are never unloaded)
 504 and obtain the rest from PythonCore. PythonCore uses a
 505 non-standard initialization entry point, <code>__initialize_with_resources</code>, to
 506 be able to obtain resources from the library file later on. Plugins can do the same
 507 (_tkinter does) or use the standard <code>__initialize</code> entry point.
 508
 509
 510 </UL>
 511 </BODY>
 512 </HTML>