Mac/Demo/building.html

   1 <HTML>
   2 <HEAD>
   3 <TITLE>Building Mac Python from source</TITLE>
   4 </HEAD>
   5 <BODY>
   6 <H1>Building Mac Python from source</H1>
   7 <HR>
   8
   9 This document explains how to build MacPython from source. This is
  10 necessary if you want to make modifications to the Python core. Building
  11 Python is not something to be undertaken lightly, you need a reasonable
  12 working knowledge of the CodeWarrior development environment, a good net
  13 connection and probably quite some time too. <p>
  14
  15 The information density in this file is high, so you should probably
  16 print it and read it at your leasure. Most things are explained only
  17 once (and probably in the wrong place:-). <p>
  18
  19 <blockquote>
  20 First a warning: this information may become outdated if a new CodeWarrior is
  21 released after MacPython. The
  22 <a href="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</a> will
  23 hopefully have updated instructions in that case.
  24 </blockquote>
  25
  26 I am very interested in feedback on this document, send your
  27 comments to the <A
  28 HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
  29 Interest Group</A>.
  30
  31 <H2>What you need.</H2>
  32
  33 The following things you definitely need:
  34
  35 <UL>
  36
  37 <LI> You need a MacPython source distribution, of course. You can
  38 obtain one via <A HREF="http://www.cwi.nl/~jack/macpython.html">
  39 http://www.cwi.nl/~jack/macpython.html</A>  (which has up-to-date links
  40 to the other packages needed too) and possibly also from the standard
  41 <A HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
  42 site</A>. <BR>
  43
  44 A better alternative is to check the sources straight out of the CVS
  45 repository, see below. Most of the packages mentioned here are also
  46 available through CVS. Check the section on <a href="#cvs">CVS
  47 repository use</a> below.
  48
  49 <LI> You need MetroWerks CodeWarrior. The current distribution has
  50 been built with CodeWarrior Pro 6.1. Ordering information is
  51 available on the <A HREF="http://www.metrowerks.com/">MetroWerks
  52 homepage</A>. Building Python with MPW, Think/Symantec C or the OSX
  53 developer tools is impossible without major surgery.
  54
  55 <LI> You need GUSI version 2, the Grand Unified Socket Interface, by
  56 Matthias Neeracher. The original GUSI is obtainable from <A
  57 HREF="ftp://gusi.sourceforge.net/pub/gusi/">
  58 ftp://gusi.sourceforge.net/pub/gusi/</A>. At
  59 the moment Python is built with a slightly modified version of GUSI
  60 2.1.1, so it may be better to check the <A
  61 HREF="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</A>
  62 for a GUSI that is most easily used for building Python.
  63
  64 </UL>
  65
  66 <A NAME="optional">The MacPython project files are configured to
  67 include a plethora of optional modules</A>, and these modules need a
  68 number of extra packages. To use the project files as-is you have to
  69 download these packages too. Python has all such modules as
  70 dynamically loaded modules, so if you don't need a certain package it
  71 suffices to just refrain from builing the extension module.
  72 Here are the locations for the various things
  73 you need:
  74
  75 <UL>
  76
  77 <LI> Tcl and Tk are in a sad state on the Mac, the standard source distributions
  78 simply don't compile, so I have created a distribution especially for use
  79 with MacPython.
  80 See the section on <A HREF="#tcltk">building Tcl/Tk Python</A>
  81 below.
  82
  83 <LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
  84 HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
  85 was built using version 2.0, which is included in the CodeWarrior
  86 package. You can also obtain it from <A
  87 HREF="http://www.merzwaren.com/waste">&lt;http://www.merzwaren.com/waste&gt;</A>
  88 and various other places.
  89
  90 <LI> Gdbm library for the Mac. Available from Jack's Mac software page at
  91 <A HREF="http://www.cwi.nl/~jack/macsoftware.html">
  92 http://www.cwi.nl/~jack/macsoftware.html</A> and <A HREF="ftp://ftp.cwi.nl/pub/jack/mac">
  93 ftp://ftp.cwi.nl/pub/jack/mac</A>.
  94
  95 <LI> JPEG library by the Independent JPEG Group. A version including
  96 Mac projects can be found at Jack's page mentioned above.
  97 The most recent JPEG library can always be obtained from <A
  98 HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>.
  99
 100 <LI> The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution
 101 (which includes libtiff) is generally available on Internet ftp
 102 servers. For Python pbmplus, an older incarnation of netpbm, is
 103 functionally identical to netpbm, since Python only uses the library
 104 and not the complete applications. A distribution with correct
 105 projects and library source only is available from, you guessed it, Jack's Mac software
 106 page mentioned above.
 107
 108 </UL>
 109
 110 <H2>Setting Up</H2>
 111
 112 Now that you have collected everything you should start with building
 113 the various parts.  If you don't want to fix
 114 access paths try to set things up as follows:
 115
 116 <PRE>
 117 Top-level-folder:
 118         GUSI2
 119         GUSI2Carbon
 120         imglibs
 121                 jpeg
 122                 netpbm
 123                         libtiff
 124                 zlib
 125                 png
 126         gdbm
 127         Python
 128                 Modules
 129                 ...
 130                 Mac
 131                         Modules
 132                         Build
 133                         ...
 134         Tcl/Tk Folder
 135                 tcl8.0
 136                 tk8.0
 137                 MoreFiles 1.4.3
 138 </PRE>
 139
 140 If your setup of the libraries is exactly the same as mine (which is
 141 not very likely, unless you happen to work from the same CVS
 142 repository) you can use the project <code>buildlibs.prj</code> in the
 143 <code>:Mac:Build</code> folder to build all needed libraries in one
 144 fell swoop, otherwise you will have to build the libraries one by
 145 one. <p>
 146
 147 First build GUSI, both the norla one and the Carbon variant.
 148  <p>
 149
 150 Next, in
 151 <code>libjpeg</code>, <code>pbmplus</code>,
 152 <code>zlib</code>, <code>libpng</code>, <code>gdbm</code>,
 153 and<code>libtiff</code> you build all projects. Usually the projects are in "mac"
 154 subfolders, sometimes they are in the main folder. Tcl/tk is a special
 155 case, see below.
 156
 157 <H2><A NAME="tcltk">Building Tcl/Tk</H2>
 158
 159 The Tcl/Tk 8.3.0 source distribution does not work on the Mac. I have created
 160 an archive of the sources that I used to build _tkinter for MacPython,
 161 you can obtain this from <a
 162 href="ftp://ftp.cwi.nl/pub/jack/python/mac/tcltk830src-for-python.sit">
 163 ftp://ftp.cwi.nl/pub/jack/python/mac/tcltk830src-for-python.sit</a>. Only the
 164 libraries needed to build _tkinter for PPC have been fixed. <P>
 165
 166 Note that if you use a different release of Tcl and Tk than the ones
 167 I have used you may have to adapt the Python <code>tkresources.rsrc</code> file.
 168 This is easiest done by building <code>SimpleTk</code> and copying the TEXT, ICON
 169 and CRSR resources from it to <code>tkresources.rsrc</code>. This allows
 170 the <code>_tkinter</code> module to work without an installed Tk/Tcl on your
 171 machine. <P>
 172
 173 Also note that the <code>_tkinter.ppc.slb</code> that is normally distributed
 174 in the <code>PlugIns</code> folder is the one from the Imaging extension,
 175 which has some extra features needed by PIL (and which features should not
 176 hinder normal operation).
 177
 178 </UL>
 179
 180 Build first the Tcl library, then
 181 SimpleTcl (test it by typing <code>ls -l</code> in the window you get)
 182 then the Tk library, then SimpleTk (which can again be tested with
 183 <code>ls -l</code>). If this all worked you are all set to try
 184 building Python.
 185
 186 <H2>Building Waste</H2>
 187
 188 You do not need to build the Waste libraries, as Python includes the
 189 source modules themselves.
 190
 191 <H2>The organization of the Python source tree</H2>
 192
 193 Time for a short break, while we have a look at the organization of
 194 the Python source tree.  At the top level, we find the following
 195 folders:
 196
 197 <DL>
 198 <DT> Demo
 199 <DD> Demo programs that are not Mac-specific. Some of these may not
 200 work.
 201
 202 <DT> Extensions
 203 <DD> Extensions to the interpreter that are not Mac-specific. Contains
 204 the <code>img</code>, <code>Imaging</code> and <code>Numerical</code> extensions
 205 in this distribution.
 206
 207 <DT> Grammar
 208 <DD> The Python grammar. Included for reference only, you cannot build
 209 the parser on a Mac.
 210
 211 <DT> Include
 212 <DD> Machine-independent header files.
 213
 214 <DT> Modules
 215 <DD> Machine-independent optional modules. Not all of these will work
 216 on the Mac.
 217
 218 <DT> Objects
 219 <DD> Machine-independent code for various object types. Most of these are
 220 not really optional: the interpreter will not function without them.
 221
 222 <DT> Parser
 223 <DD> The Python parser (machine-independent).
 224
 225 <DT> Python
 226 <DD> The core interpreter. Most files are machine-independent, some
 227 are unix-specific and not used on the Mac.
 228
 229 <DT> Tools
 230 <DD> Tools for python developers. Contains <code>modulator</code> which
 231 builds skeleton C extension modules, <code>bgen</code> which generates
 232 complete interface modules from information in C header files and
 233 <code>freeze</code> which is used to turn Python scripts into real
 234 applications (used by MacFreeze and BuildApplication) There are some
 235 readme files, but more documentation is sorely needed.
 236
 237 </DL>
 238
 239 All the mac-specific stuff lives in the <code>Mac</code> folder:
 240 <DL>
 241 <DT> Build
 242 <DD> This is where the project files live and where you build the
 243 libraries, shared libraries, executables and plugin modules. All the
 244 resulting binaries, except for intermedeate results, are deposited in
 245 the toplevel folder or the Mac:PlugIns folder (for plugin modules).
 246
 247 <DT> Compat
 248 <DD> Unix-compatability routines. Most of these are not used anymore,
 249 since GUSI provides a rather complete emulation, but you may need
 250 these if you are trying to build a non-GUSI python.
 251
 252 <DT> Demo
 253 <DD> Mac-specific demo programs, some of them annotated.
 254
 255 <DT> Include
 256 <DD> Mac-specific but compiler-independent include files.
 257
 258 <DT> Lib
 259 <DD> Mac-specific standard modules. The <code>toolbox</code> folder
 260 contains modules specifically needed with various MacOS toolbox
 261 interface modules.
 262
 263 <DT> Modules
 264 <DD> Mac-specific builtin modules. Theoretically these are all
 265 optional, but some are rather essential (like
 266 <code>macosmodule</code>). A lot of these modules are generated with
 267 <code>bgen</code>, in which case the bgen input files are included so
 268 you can attempt to regenerate them or extend them.
 269
 270 <DT> MPW
 271 <DD> MPW-specific files. These have not been used or kept up-to-date
 272 for a long time, so use at your own risk.
 273
 274 <DT> mwerks
 275 <DD> Mwerks-specific sources and headers. Contains glue code for
 276 Pythons shared-library architecture, a replacement for
 277 <code>malloc</code> and a directory with various projects for building
 278 variations on the Python interpreter. The <code>mwerks_*.h</code>
 279 files here are the option-setting files for the various interpreters
 280 and such, comparable to the unix command-line <code>-D</code> options
 281 to the compiler. Each project uses the correct option file as its
 282 "prefix file" in the "C/C++ language" settings. Disabling optional
 283 modules (for the 68K interpreter), building non-GUSI interpreters and
 284 various other things are accomplished by modifying these files (and
 285 possibly changing the list of files included in the project window, of
 286 course).
 287
 288 <DT> PlugIns
 289 <DD> This is where the Classic and Carbon dynamically-loaded plugin modules live.
 290
 291 <DT> Python
 292 <DD> Mac-specific parts of the core interpreter.
 293
 294 <DT> Resources
 295 <DD> Resource files needed to build the interpreter.
 296
 297 <DT> Scripts
 298 <DD> A collection of various mac-specific Python scripts. Some are
 299 essential, some are useful but few are documented, so you will have to
 300 use your imagination to work them out.
 301
 302 <DT> Tools
 303 <DD> A collection of tools, usually bigger than those in the scripts
 304 folder. The important ones here are the IDE and macfreeze. The IDE is built
 305 with the buildIDE.py script, which puts the resulting applet in the toplevel
 306 folder. Macfreeze is usually invoked through the BuildApplication script,
 307 but for more control over the freezing process you can run the main script here.
 308
 309
 310 <DT> Unsupported
 311 <DD> Modules that are not supported any longer but may still work with a little effort.
 312 </DL>
 313
 314 <H2>Building the PPC interpreter</H2>
 315 <em>This is different under 2.1. You are best off using the fullbuild.py
 316 script, see <a href="#fullbuild">below</a>. </em><p>
 317
 318 First you optionally build the external libraries with buildlibs.prj. Next,
 319 the projects for
 320 interpreter and core library are linked together, so
 321 building the PythonInterpreterClassic and/or PythonInterpreterCarbon target
 322 in <code>PythonInterpreter.prj</code>
 323 will result in everything being built. The result, however, is an "Application
 324 template", (filetype Atmp). If you don't use fullbuild you can manually
 325 turn either of these into an interpreter by copying it to PythonInterpreter
 326 and setting the filetype to APPL (with ResEdit or some such). <p>
 327
 328 Fullbuild does this for you, and the Atmp files is also how ConfigurePythonCarbon
 329 and ConfigurePythonClassic work their magic. <p>
 330
 331 For completeness sake here is a breakdown of the projects:
 332
 333 <DL>
 334
 335 <DT> PythonCore
 336 <DD> The shared library that contains the bulk of the interpreter and
 337 its resources. It has targets for PythonCore and PythonCoreCarbon.
 338 It is a good idea to immedeately put an alias to this
 339 shared library in the <code>Extensions</code> folder of your system
 340 folder.  Do exactly that: put an <em>alias</em> there, copying or
 341 moving the file will cause you grief later if you rebuild the library and
 342 forget to copy it to the extensions folder again. The ConfigurePythonXXX applets
 343 will also do this. <br>
 344
 345 <DT> PythonInterpeter
 346 <DD> The interpreter. This is basically a routine to call out to the
 347 shared library. Unlike in previous releases the same program is used for
 348 creating applets (for which formerly PythonApplet was used). <p>
 349
 350 <DT> Plugin projects
 351 <DD> Each plugin module has a separate project, and these can be rebuilt on
 352 the fly. Fullbuild (or actually it's little helper genpluginprojects) takes
 353 care of this.
 354 </DL>
 355
 356 After creating the alias to <code>PythonCore</code> you remove any old
 357 <code>Python XXXX Preferences</code> file from the <code>Preferences</code> folder
 358 (if you had python installed on your system before) and run the interpreter once
 359 to create the correct preferences file.  <p>
 360
 361 Next, you have to build the extension modules.
 362 If you don't use fullbuild simply open each project and build it.
 363 <p>
 364
 365 Finally, you must build the standard applets:
 366 <code>EditPythonPrefs</code>, <code>BuildApplet</code>, etc. For the N-th time:
 367 fullbuild does this for you, but you can also manually drag/drop them onto
 368 BuildApplet. <p>
 369
 370 <BLOCKQUOTE>
 371 <a name="fullbuild"></a>
 372 The <code>fullbuild</code> script can be used to build
 373 everything, but you need a fully-functional interpreter before you can
 374 use it (and one that isn't rebuilt in the process: you cannot rebuild
 375 a running program). You could copy the interpreter to a different
 376 place and use that to run fullbuild. The <code>PythonStandSmall.prj</code>
 377 project builds an interpreter that is suited to this, and it can also come
 378 in handy if you need to debug things (which is easier in a static program). <p>
 379
 380 </BLOCKQUOTE>
 381
 382 You are all set now, and should read the release notes and
 383 <code>ReadMe</code> file from the <code>Mac</code> folder.
 384
 385 <H2>Rebuilding <code>.exp</code> files</H2>
 386
 387 Occasionally it may be necessary to rebuild your PythonCore <code>.exp</code>
 388 file, a file that controls which symbols are exported by your PythonCore
 389 shared library. Rebuild it if you get unexpected undefined symbols when you
 390 are building a plugin module. <p>
 391
 392 Rebuilding the .exp file is done by first both removing the file and removing the
 393 reference to it in the project (in the "config" section). Next, build PythonCore or
 394 PythonCoreCarbon.
 395 This will create a new .exp file, with the name <code>PythonCore.mcp.exp</code>.
 396 Rename this file to either <code>PythonCore.exp</code> or <code>PythonCoreCarbon.exp</code>
 397 and add this file back to the project. Next, edit ot to remove the references to
 398 the symbols <code>__initialize</code>, <code>__terminate</code>, <code>setjmp</code>,
 399 <code>longjmp</code>, <code>vec_longjmp</code>, <code>main</code> and <code>__ptmf_null</code>.
 400 They are all close together about halfway the file.
 401
 402 Finally rebuild again. <p>
 403
 404 This rather convoluted procedure is needed to ensure that plugin modules don't
 405 accidentally link with those entrypoints from PythonCore, which will not work because
 406 those routines have to be in the same code fragment as they are used from.
 407
 408 <H2><a name="cvs">Using the CVS source archive</a></H2>
 409
 410 It is possible (and probably best) to access the Python sources through remote CVS. The
 411 advantage of this is that you get the very latest sources, so any bug
 412 fixed or new features will be immedeately available. This is also the
 413 disadvantage, of course: as this is the same tree as is used for
 414 development it may sometimes be a little less stable. <p>
 415
 416 The CVS client of choice is Alexandre Parenteau's MacCVS. It can be
 417 obtained through the <a href="http://www.wincvs.org">WinCVS
 418 homepage</a>. MacCVS uses Internet Config to set file types correctly
 419 based on the filename extension. In the maccvs preferences you should
 420 also set (in the "binary files" section) "use mac encoding:
 421 applesingle" and (in the "text files" section) "use ISO latin 1
 422 conversion". <p>
 423
 424 <blockquote>
 425 There is one group of people for whom MacCVS is not the best choice: people with
 426 checkin rights to the Python repository. You will have to use MacCVS Pro
 427 (completely unrelated) from www.maccvs.org, because it has working SSH support.
 428 </blockquote>
 429
 430 It is a good idea to disable Quicktime Exchange in the Quicktime
 431 control panel. Quicktime Exchange will magically map some extensions to
 432 filetypes, and this can seriously hinder you if, for instance, <code>.bmp</code>
 433 is not a Windows bitmap file. <p>
 434
 435 The Python sources are checked out from the main
 436 Python CVS archive on sourceforge.net, see the <a
 437 href="http://www.python.org/download/cvs.html">Source access via
 438 CVS</a> page for details. When you check the sources out you will get
 439 something like <code>Python:dist:src</code>, and under that the
 440 <code>Modules</code>, <code>Lib</code>, <code>Mac</code> etc hierarchy. The
 441 <code>src</code> folder can be renamed to <code>Python</code>, and
 442 is what this document refers to as the "toplevel Python folder". <P>
 443
 444 The CVS repository does not contain all the projects for the plugin modules,
 445 these are built with <code>fullbuild.py</code> normally. For this reason
 446 it is probably a good idea to first build <code>PythonStandSmall.prj</code>,
 447 which builds a fairly minimal interpreter, and then follow the
 448 <a href="#fullbuild">fullbuild instructions</a>.
 449
 450 <H2>Odds and ends</H2>
 451
 452 Some remarks that I could not fit in elsewhere:
 453
 454 <UL>
 455
 456 <LI> It may be possible to use the <code>PythonCore</code> shared
 457 library to embed Python in another program, if your program can live
 458 with using GUSI for I/O. Use PythonCore in stead of your MSL C library
 459 (or, at the very least, link it before the normal C library).
 460
 461 <LI> It is possible to build PPC extension modules without building a
 462 complete Python. The binary distribution installer can optionally install
 463 all the needed folders (the develop option). A template for a dynamic module can be found in
 464 <code>xx.prj</code>.
 465
 466 <LI> The Python shared library architecture is a variant of the architecture
 467 described as "application with shared libraries and dropins" in the MetroWerks
 468 "Targeting MacOS" documentation. The Python Application and applet-template use
 469 the <code>MSL AppRuntime.Lib</code> runtime library (with properly set CFM
 470 initialization and termination routines). PythonCore uses <code>MSL Runtime.Lib</code>,
 471 which is really intended for standalone programs but which we fool into working by
 472 providing a dummy main program.
 473 It is linked statically into PythonCore (and exported to the applications and plugins)
 474 so we do not have to distribute yet another shared library. Plugin modules use
 475 <code>MSL ShlibRuntime.Lib</code> (not the dropin runtime: modules are never unloaded)
 476 and obtain the rest from PythonCore. PythonCore uses a
 477 non-standard initialization entry point, <code>__initialize_with_resources</code>, to
 478 be able to obtain resources from the library file later on. Plugins can do the same
 479 (_tkinter does) or use the standard <code>__initialize</code> entry point.
 480
 481
 482 </UL>
 483 </BODY>
 484 </HTML>