Mac/Demo/building.html

   1 <HTML>
   2 <HEAD>
   3 <TITLE>Building Mac Python from source</TITLE>
   4 </HEAD>
   5 <BODY>
   6 <H1>Building Mac Python from source</H1>
   7 <HR>
   8
   9 This document explains how to build MacPython from source. This is
  10 necessary if you want to write extension modules for 68K Python, and
  11 currently also probably the easiest way to build PPC extension
  12 modules. Building Python is not something to be undertaken lightly,
  13 the process is not very streamlined so you need a reasonable working
  14 knowledge of the CodeWarrior development environment, a good net
  15 connection and probably quite some time too. <p>
  16
  17 The information density in this file is high, so you should probably
  18 print it and read it at your leasure. Most things are explained only
  19 once (and probably in the wrong place:-). <p>
  20
  21 I am very interested in feedback on this document, contact me at <A
  22 HREF="mailto:jack@cwi.nl">&lt;jack@cwi.nl&gt;</A> or send your
  23 comments to the <A
  24 HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
  25 Interest Group</A>.
  26
  27 <H2>What you need.</H2>
  28
  29 The following things you definitely need:
  30
  31 <UL>
  32
  33 <LI> You need a MacPython source distribution, of course. You can
  34 obtain one from <A
  35 HREF="ftp://ftp.cwi.nl/pub/jack/python/mac">ftp://ftp.cwi.nl/pub/jack/python/mac</A>
  36 or from the companion webpage at <A HREF="http://www.cwi.nl/~jack/macpython.html">
  37 http://www.cwi.nl/~jack/macpython.html</A> (which has up-to-date links to the other
  38 packages needed too)
  39 and possibly also from the standard <A
  40 HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
  41 site</A>. Everything you need is also included in the standard Python
  42 source distribution, but the organization is different. Look in
  43 directory <code>Mac/mwerks/projects</code> for the project files and
  44 related stuff.
  45
  46 <LI> You need MetroWerks CodeWarrior. The current distribution has
  47 been built with CodeWarrior Pro 2. Ordering information is
  48 available on the <A HREF="http://www.metrowerks.com/">MetroWerks
  49 homepage</A>. Building Python with MPW or Think/Symantec C is
  50 probably impossible without major surgery.
  51
  52 <LI> You need GUSI, the Grand Unified Socket Interface, by Matthias
  53 Neeracher. The original CWGUSI is
  54 obtainable from <A
  55 HREF="ftp://sunsite.cnlab-switch.ch/software/platform/macos/src">
  56 ftp://sunsite.cnlab-switch.ch/software/platform/macos/src</A>.
  57 At the moment Python is built with a slightly modified version of GUSI,
  58 these modifications are available in folder <code>Python:Mac:GUSI-mods</code>.
  59
  60 </UL>
  61
  62 <A NAME="optional">The MacPython project files are configured to
  63 include a plethora of optional modules</A>, and these modules need a
  64 number of extra packages. To use the project files as-is you have to
  65 download these packages too. PPC and CFM68K Python have all such modules as
  66 dynamically loaded modules, so if you don't need a certain package it
  67 suffices to just refrain from builing the extension module. For static 68K
  68 Python things are a bit more complicated: you have to edit the
  69 interpreter project file to remove the reference to the module (and
  70 the libraries it uses), and edit the <code>Mac:mwerks:mwerks_nonshared_config.h</code>
  71 file to remove the <code>USE_...</code> line.  Here are the locations for the various things
  72 you need:
  73
  74 <UL>
  75
  76 <LI> Tcl and Tk can be obtained from <A
  77 HREF="ftp://ftp.smli.com/pub/tcl/mac/">ftp://ftp.smli.com/pub/tcl/mac/</A>.
  78 The current distributions, Tcl 8.0p2 and Tk 8.0p2 need a bit of work,
  79 see the section on <A HREF="#tcltk">building Tcl/Tk Python</A>
  80 below. Get the "full source" distribution, which includes MoreFiles.
  81
  82 <LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
  83 HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
  84 was built using version 1.3, which you can obtain from <A
  85 HREF="http://www.boingo.com/waste">&lt;http://www.boingo.com/waste&gt;</A>
  86 and various other places.
  87
  88 <LI> Gdbm library for the Mac. Available from Jack's Mac software page at
  89 <A HREF="http://www.cwi.nl/~jack/macsoftware.html">
  90 http://www.cwi.nl/~jack/macsoftware.html</A> and <A HREF="ftp://ftp.cwi.nl/pub/jack/mac">
  91 ftp://ftp.cwi.nl/pub/jack/mac</A>.
  92
  93 <LI> JPEG library by the Independent JPEG Group. A version including
  94 Mac projects can be found at Jack's page mentioned above.
  95 The most recent JPEG library can always be obtained from <A
  96 HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>.
  97
  98 <LI> The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution
  99 (which includes libtiff) is generally available on Internet ftp
 100 servers. For Python pbmplus, an older incarnation of netpbm, is
 101 functionally identical to netpbm, since Python only uses the library
 102 and not the complete applications. A distribution with correct
 103 projects and library source only is available from, you guessed it, Jack's Mac software
 104 page mentioned above.
 105
 106 </UL>
 107
 108 <H2>Setting Up</H2>
 109
 110 Now that you have collected everything you should start with building
 111 the various parts.  If you don't want to fix
 112 access paths try to set things up as follows:
 113
 114 <PRE>
 115 Top-level-folder:
 116         CWGUSI
 117         imglibs
 118                 jpeg
 119                 netpbm
 120                         libtiff
 121                 zlib
 122                 png
 123         gdbm
 124         Python
 125         Tcl/Tk Folder
 126                 tcl8.0
 127                 tk8.0
 128                 MoreFiles 1.4.3
 129         Waste 1.3 distribution (if you want waste)
 130 </PRE>
 131
 132 If your setup of the libraries is exactly the same as mine (which is
 133 not very likely, unless you happen to work from the same CVS
 134 repository) you can use the project <code>buildlibs.prj</code> in the
 135 <code>build.mac</code> folder to build all needed libraries in one
 136 fell swoop, otherwise you will have to build the libraries one by
 137 one. <p>
 138
 139 First build GUSI. If you didn't get the python-specific GUSI you have to
 140 move the files from the "CWGUSI-mods" to the right
 141 place in the CWGUSI distribution folder. Build the MSL version for your
 142 platform (ppc, 68k, cfm68k). <p>
 143
 144 Next, in
 145 <code>MoreFiles</code>, <code>libjpeg</code>, <code>pbmplus</code>,
 146 <code>zlib</code>, <code>libpng</code>, <code>gdbm</code>,
 147 and<code>libtiff</code> you build all projects. Usually the projects are in "mac"
 148 subfolders, sometimes they are in the main folder. Tcl/tk is a special
 149 case, see below. Of course, if you are only interested in one of
 150 static 68K, CFM68K or PPC you can skip building the other libraries.
 151
 152 <H2><A NAME="tcltk">Building Tcl/Tk</H2>
 153
 154 You need to make some minor changes to the Tcl/Tk 8.0
 155 distribution. You should make the CW Pro projects (in the mac subfolders).
 156 <UL>
 157
 158 <LI> There are no cfm68k targets. You make these by copying the 68k targets,
 159 setting the "68k target" to "cfm68k library" and changing the output filename,
 160 and changing the prefix
 161 header filename in the C/C++ settings panel to "MW_???HeaderCFM68K".
 162
 163 <LI> I had to add Search.c (from MoreFiles) to the tcl library projects. I don't
 164 understand why this is, but it seemed to cure the problems I had.
 165
 166 <LI> Note that if you use a different release of Tcl and Tk than the ones
 167 I have used you may have to adapt the Python <code>tkresources.rsrc</code> file.
 168 This is easiest done by building <code>SimpleTk</code> and copying the TEXT, ICON
 169 and CRSR resources from it to <code>tkresources.rsrc</code>. This allows
 170 the <code>_tkinter</code> module to work without an installed Tk/Tcl on your
 171 machine.
 172
 173 </UL>
 174
 175 Build first the Tcl library, then
 176 SimpleTcl (test it by typing <code>ls -l</code> in the window you get)
 177 then the Tk library, then SimpleTk (which can again be tested with
 178 <code>ls -l</code>). If this all worked you are all set to try
 179 building Python.
 180
 181 <H2>Building Waste</H2>
 182
 183 You do not need to build the Waste libraries, as Python includes the
 184 source modules themselves.
 185
 186 <H2>The organization of the Python source tree</H2>
 187
 188 Time for a short break, while we have a look at the organization of
 189 the Python source tree.  At the top level, we find the following
 190 folders:
 191
 192 <DL>
 193 <DT> build.macstand
 194 <DD> This is where you build static 68K interpreter, and possibly (if
 195 you want them) static PPC and fat interpreters. There is also a project
 196 here that builds smaller versions of the static interpreter.
 197
 198 <DT> build.mac
 199 <DD> This is where you build the CFM68K, PPC and fat shared library,
 200 interpreter and applet framework. The fat targets are deposited in the
 201 main folder (one level up), the others, which are intermediate results
 202 really, are left here.
 203
 204 <DT> Demo
 205 <DD> Demo programs that are not Mac-specific. Some of these may not
 206 work, the file <code>README-Mac</code> has some details.
 207
 208 <DT> Extensions
 209 <DD> Extensions to the interpreter that are not Mac-specific. Contains
 210 only the <code>img</code> extension in this distribution. Extensions
 211 are not always built here, as they are on Unix, but sometimes incorporated in
 212 the core interpreter or built as plugin modules.
 213
 214 <DT> Grammar
 215 <DD> The Python grammar. Included for reference only, you cannot build
 216 the parser on a Mac.
 217
 218 <DT> Include
 219 <DD> Machine-independent header files.
 220
 221 <DT> Modules
 222 <DD> Machine-independent optional modules. Not all of these will work
 223 on the Mac.
 224
 225 <DT> Objects
 226 <DD> Machine-independent code for various objects. Most of these are
 227 not really optional: the interpreter will not function without them.
 228
 229 <DT> Parser
 230 <DD> The Python parser (machine-independent).
 231
 232 <DT> PlugIns
 233 <DD> This is where you build the PPC and CFM68K dynamically-loaded plugin modules.
 234
 235 <DT> Python
 236 <DD> The core interpreter. Most files are machine-independent, some
 237 are unix-specific and not used on the Mac.
 238
 239 <DT> Tools
 240 <DD> Tools for python developers. Contains <code>modulator</code>
 241 which builds skeleton C extension modules and <code>bgen</code> which
 242 generates complete interface modules from information in C header
 243 files. There are some readme files, but more documentation is sorely
 244 needed.
 245
 246 </DL>
 247
 248 All the mac-specific stuff lives in the <code>Mac</code> folder:
 249 <DL>
 250
 251 <DT> Compat
 252 <DD> Unix-compatability routines. Some of these are not used anymore,
 253 since CWGUSI provides a rather complete emulation, but you may need
 254 these if you are trying to build a non-GUSI python.
 255
 256 <DT> Demo
 257 <DD> Mac-specific demo programs, some of them annotated.
 258
 259 <DT> Include
 260 <DD> Mac-specific but compiler-independent include files.
 261
 262 <DT> Lib
 263 <DD> Mac-specific standard modules. The <code>toolbox</code> folder
 264 contains modules specifically needed with various MacOS toolbox
 265 interface modules.
 266
 267 <DT> Modules
 268 <DD> Mac-specific builtin modules. Theoretically these are all
 269 optional, but some are rather essential (like
 270 <code>macmodule</code>). A lot of these modules are generated with
 271 <code>bgen</code>, in which case the bgen input files are included so
 272 you can attempt to regenerate them or extend them.
 273
 274 <DT> MPW
 275 <DD> MPW-specific files. These have not been used or kept up-to-date
 276 for a long time, so use at your own risk.
 277
 278 <DT> mwerks
 279 <DD> Mwerks-specific sources and headers. Contains glue code for
 280 Pythons shared-library architecture, a replacement for
 281 <code>malloc</code> and a directory with various projects for building
 282 variations on the Python interpreter. The <code>mwerks_*.h</code>
 283 files here are the option-setting files for the various interpreters
 284 and such, comparable to the unix command-line <code>-D</code> options
 285 to the compiler. Each project uses the correct option file as its
 286 "prefix file" in the "C/C++ language" settings. Disabling optional
 287 modules (for the 68K interpreter), building non-GUSI interpreters and
 288 various other things are accomplished by modifying these files (and
 289 possibly changing the list of files included in the project window, of
 290 course).
 291
 292 <DT> Python
 293 <DD> Mac-specific parts of the core interpreter.
 294
 295 <DT> Resources
 296 <DD> Resource files needed to build the interpreter.
 297
 298 <DT> Scripts
 299 <DD> A collection of various mac-specific Python scripts. Some are
 300 essential, some are useful but few are documented, so you will have to
 301 use your imagination to work them out.
 302
 303 <DT> Unsupported
 304 <DD> Modules that are not supported any longer but may still work with a little effort.
 305 </DL>
 306
 307 <H2>Building the 68K interpreter</H2>
 308
 309 If you have all the optional libraries mentioned <A
 310 HREF="#optional">above</A> loaded buildin Python for 68K macs is a
 311 breeze: open the project in the folder <code>build.macstand</code> and
 312 build the 68K target.  Do <em>not</em> run it yet, this will possibly
 313 result in a garbled preferences file. <p>
 314
 315 First remove the <code>Python XXX preferences</code> file from your
 316 preference folder, only if you had an older version of Python
 317 installed.  (this is also what you do if you did not heed the last
 318 sentence of the preceeding paragraph). Next, move the interpreter to
 319 the main Python folder (up one level) and run it there. This will
 320 create a correct initial preferences file. You are now all set, and
 321 your tree should be completely compatible with a binary-only
 322 distribution. Read the release notes
 323 (<code>Relnotes-somethingorother</code>) and
 324 <code>ReadMe</code> in the <code>Mac</code> folder.
 325
 326 <H2>Building the PPC and CFM68K interpreter</H2>
 327
 328 First you build the interpreter, core library and applet skeleton in
 329 folder <code>build.mac</code>. The projects are all linked together, so
 330 building the fat targets in <code>Python.prj</code> and
 331 <code>PythonApplet.prj</code> will result in everything being built. The
 332 resulting applications and fat shared library are deposited in the main
 333 Python folder. For completeness sake here is a breakdown of the
 334 projects:
 335
 336 <DL>
 337
 338 <DT> PythonCore (with subprojects PythonCorePPC and PythonCoreCFM68K)
 339 <DD> The shared library that contains the bulk of the interpreter and
 340 its resources. It is a good idea to immedeately put an alias to this
 341 shared library in the <code>Extensions</code> folder of your system
 342 folder.  Do exactly that: put an <em>alias</em> there, copying or
 343 moving the file will cause you grief later if you rebuild the library and
 344 forget to copy it to the extensions folder again.
 345
 346 <DT> Python
 347 <DD> The interpreter. This is basically a routine to call out to the
 348 shared library.  <p>
 349
 350 <DT> PythonAppletPPC
 351 <DD> The applet skeleton application. Very similar to
 352 <code>PythonPPC</code>, but it calls to a different entrypoint in the
 353 core library. The <code>mkapplet</code> script will copy this complete
 354 file, and add a <code>'PYC '</code> with the module to generate an
 355 applet. <p>
 356
 357 </DL>
 358
 359 After creating the alias to <code>PythonCore</code> you remove any old
 360 <code>Python XXX Preferences</code> file from the <code>Preferences</code> folder
 361 (if you had python installed on your system before) and run the interpreter once
 362 to create the correct preferences file.  <p>
 363
 364 Next, you have to build the extension modules in the
 365 <code>PlugIns</code> folder. The <code>PlugIns.ppc</code> project has all the
 366 other projects as subprojects and builds everything. After all
 367 the dynamically loaded modules are built you have to create a number
 368 of aliases: some modules live together in a single dynamic
 369 library. Run the <code>ConfigurePython.py</code> script from
 370 <code>Mac:scripts</code> to create the aliases. <p>
 371
 372 Finally, you must build the standard applets:
 373 <code>EditPythonPrefs</code>, <code>BuildApplet</code>, etc. This is
 374 easiest done with the <code>fullbuild</code> script from
 375 <code>Mac:scripts</code>. <p>
 376
 377 <BLOCKQUOTE>
 378 Actually, the <code>fullbuild</code> script can be used to build
 379 everything, but you need a fully-functional interpreter before you can
 380 use it (and one that isn't rebuilt in the process: you cannot rebuild
 381 a running program). You could copy the 68K interpreter to a different
 382 place and use that to run fullbuild, or use the standalone PPC python
 383 for this. I tend to keep a standalone interpreter in a safe place for
 384 this use only. <p>
 385
 386 </BLOCKQUOTE>
 387
 388 You are all set now, and should read the release notes and
 389 <code>ReadMe</code> file from the <code>Mac</code> folder.
 390
 391 <H2>Rebuilding <code>.exp</code> files for PPC and CFM68K</H2>
 392
 393 Occasionally it may be necessary to rebuild your PythonCore <code>.exp</code>
 394 file, a file that controls which symbols are exported by your PythonCore
 395 shared library. Rebuild it if you get unexpected undefined symbols when you
 396 are building a plugin module. <p>
 397
 398 Rebuilding the .exp file is done by first removing the file and removing the
 399 reference to it in the project (in the "config" section). Next, build PythonCore.
 400 This will create a new .exp file. Edit this file to remove the references to
 401 the symbols <code>__initialize</code>, <code>__terminate</code>, <code>setjmp</code>,
 402 <code>longjmp</code>, <code>main</code> and (for PPC) <code>__ptmf_null</code> or (for
 403 CFM68K) <code>__start</code> and <code>dummy_init_routine</code>.
 404 Next, add the .exp file to the project
 405 again and rebuild PythonCore. <p>
 406
 407 This rather convoluted procedure is needed to ensure that plugin modules don't
 408 accidentally link with those entrypoints from PythonCore, which will not work because
 409 those routines have to be in the same code fragment as they are used from.
 410
 411 <H2>Odds and ends</H2>
 412
 413 Some remarks that I could not fit in elsewhere:
 414
 415 <UL>
 416
 417 <LI> It may be possible to use the <code>PythonCore</code> shared
 418 library to embed Python in another program, if your program can live
 419 with using GUSI for I/O. Use PythonCore in stead of your MSL C library
 420 (or, at the very least, link it before the normal C library). Let me
 421 know whether this works.
 422
 423 <LI> It is possible to build PPC extension modules without building a
 424 complete Python. Take the binary distribution, add folders
 425 <code>Include</code>, <code>Mac:Include</code> and
 426 <code>Mac:mwerks</code> from the source distribution and you should be
 427 all set. A template for a dynamic module can be found in
 428 <code>xx.prj</code>.
 429
 430 <LI> The Python shared library architecture is a variant of the architecture
 431 described as "application with shared libraries and dropins" in the MetroWerks
 432 "Targeting MacOS" documentation. The Python Application and applet-template use
 433 the <code>MSL AppRuntime.Lib</code> runtime library (with properly set CFM
 434 initialization and termination routines). PythonCore uses <code>MSL Runtime.Lib</code>,
 435 which is really intended for standalone programs but which we fool into working by
 436 providing a dummy main program.
 437 It is linked statically into PythonCore (and exported to the applications and plugins)
 438 so we do not have to distribute yet another shared library. Plugin modules use
 439 <code>MSL ShlibRuntime.Lib</code> (not the dropin runtime: modules are never unloaded)
 440 and obtain the rest from PythonCore. PythonCore uses a
 441 non-standard initialization entry point, <code>__initialize_with_resources</code>, to
 442 be able to obtain resources from the library file later on. Plugins can do the same
 443 (_tkinter does) or use the standard <code>__initialize</code> entry point.
 444
 445
 446 </UL>
 447 </BODY>
 448 </HTML>