Mac/Demo/building.html

   1 <HTML>
   2 <HEAD>
   3 <TITLE>Building MacPython-OS9 from source</TITLE>
   4 </HEAD>
   5 <BODY>
   6 <H1>Building MacPython-OS9 from source</H1>
   7 <HR>
   8
   9 This document explains how to build MacPython-OS9 from source. This is
  10 necessary if you want to make modifications to the Python core. Building
  11 Python is not something to be undertaken lightly, you need a reasonable
  12 working knowledge of the CodeWarrior development environment, a good net
  13 connection and probably quite some time too. <p>
  14
  15 Note that if you only want to build new extension modules you don't need to
  16 build Python from source, see the <a href="#extending">note on extending Python</a>.<p>
  17
  18 The information density in this file is high, so you should probably
  19 print it and read it at your leasure. Most things are explained only
  20 once (and probably in the wrong place:-). <p>
  21
  22 <blockquote>
  23 First a warning: this information may become outdated if a new CodeWarrior is
  24 released after MacPython. The
  25 <a href="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</a> will
  26 hopefully have updated instructions in that case. These instructions are for CW7,
  27 it is rumoured you may encounter some problems with newer versions of CodeWarrior.
  28 </blockquote>
  29
  30 I am interested in feedback on this document, send your
  31 comments to the <A
  32 HREF="http://www.python.org/sigs/pythonmac-sig/">Mac Python Special
  33 Interest Group</A>.
  34
  35 <H2>What you need.</H2>
  36
  37 The following things you definitely need:
  38
  39 <UL>
  40
  41 <LI> You need a MacPython source distribution, of course. You can
  42 obtain one via <A HREF="http://www.cwi.nl/~jack/macpython.html">
  43 http://www.cwi.nl/~jack/macpython.html</A>  (which has up-to-date links
  44 to the other packages needed too) and possibly also from the standard
  45 <A HREF="ftp://ftp.python.org/pub/python/mac">python.org ftp
  46 site</A>. <BR>
  47
  48 A better alternative is to check the sources straight out of the CVS
  49 repository, see below. Most of the packages mentioned here are also
  50 available through CVS. Check the section on <a href="#cvs">CVS
  51 repository use</a> below.
  52
  53 <LI> You need MetroWerks CodeWarrior. The current distribution has
  54 been built with CodeWarrior Pro 7.1. Ordering information is
  55 available on the <A HREF="http://www.metrowerks.com/">MetroWerks
  56 homepage</A>. Building Python with MPW, Think/Symantec C or the OSX
  57 developer tools is impossible without major surgery.
  58
  59 <LI> You need GUSI version 2, the Grand Unified Socket Interface, by
  60 Matthias Neeracher. The original GUSI is obtainable from <A
  61 HREF="ftp://gusi.sourceforge.net/pub/gusi/">
  62 ftp://gusi.sourceforge.net/pub/gusi/</A>. At
  63 the moment Python is built with a modified version of GUSI
  64 with Carbon adaptations, so it may be better to check the <A
  65 HREF="http://www.cwi.nl/~jack/macpython.html">MacPython homepage</A>
  66 for a GUSI that is most easily used for building Python.
  67
  68 </UL>
  69
  70 <A NAME="optional">The MacPython project files are configured to
  71 include a plethora of optional modules</A>, and these modules need a
  72 number of extra packages. To use the project files as-is you have to
  73 download these packages too. Python has all such modules as
  74 dynamically loaded modules, so if you don't need a certain package it
  75 suffices to just refrain from builing the extension module.
  76 Here are the locations for the various things
  77 you need:
  78
  79 <UL>
  80
  81 <LI> Waste, a TextEdit replacement written by Marco Piovanelli, <A
  82 HREF="mailto:piovanel@kagi.com">&lt;piovanel@kagi.com&gt;</A>.  Python
  83 was built using version 2.0, which is included in the CodeWarrior
  84 package. You can also obtain it from <A
  85 HREF="http://www.merzwaren.com/waste">&lt;http://www.merzwaren.com/waste&gt;</A>
  86 and various other places.
  87
  88 <LI> Gdbm library for the Mac. Available from Jack's Mac software page at
  89 <A HREF="http://www.cwi.nl/~jack/macsoftware.html">
  90 http://www.cwi.nl/~jack/macsoftware.html</A> and <A HREF="ftp://ftp.cwi.nl/pub/jack/mac">
  91 ftp://ftp.cwi.nl/pub/jack/mac</A>.
  92
  93 <LI> JPEG library by the Independent JPEG Group. A version including
  94 Mac projects can be found at Jack's page mentioned above.
  95 The most recent JPEG library can always be obtained from <A
  96 HREF="ftp://ftp.uu.net/graphics/jpeg/">ftp://ftp.uu.net/graphics/jpeg/</A>.
  97
  98 <LI> The netpbm/pbmplus, libtiff, zlib and png libraries. The netpbm distribution
  99 (which includes libtiff) is generally available on Internet ftp
 100 servers. For Python pbmplus, an older incarnation of netpbm, is
 101 functionally identical to netpbm, since Python only uses the library
 102 and not the complete applications. A distribution with correct
 103 projects and library source only is available from, you guessed it, Jack's Mac software
 104 page mentioned above.
 105
 106 </UL>
 107
 108 <H2>Setting Up</H2>
 109
 110 Now that you have collected everything you should start with building
 111 the various parts.  If you don't want to fix
 112 access paths try to set things up as follows:
 113
 114 <PRE>
 115 Top-level-folder:
 116         GUSI2
 117         imglibs
 118                 jpeg
 119                 netpbm
 120                         libtiff
 121                 zlib
 122                 png
 123         gdbm
 124         Python
 125                 Modules
 126                 ...
 127                 Mac
 128                         Modules
 129                         Build
 130                         ...
 131 </PRE>
 132
 133 If your setup of the libraries is exactly the same as mine (which is
 134 not very likely, unless you happen to work from the same CVS
 135 repository) you can use the project <code>buildlibs.prj</code> in the
 136 <code>:Mac:Build</code> folder to build all needed libraries in one
 137 fell swoop, otherwise you will have to build the libraries one by
 138 one. <p>
 139
 140 First build GUSI, the Carbon variant.
 141  <p>
 142
 143 Next, in
 144 <code>libjpeg</code>, <code>pbmplus</code>,
 145 <code>zlib</code>, <code>libpng</code>, <code>gdbm</code>,
 146 and<code>libtiff</code> you build all projects. Usually the projects are in "mac"
 147 subfolders, sometimes they are in the main folder. Tcl/tk is a special
 148 case, see below.
 149
 150 <H2>The organization of the Python source tree</H2>
 151
 152 Time for a short break, while we have a look at the organization of
 153 the Python source tree.  At the top level, we find the following
 154 folders:
 155
 156 <DL>
 157 <DT> Demo
 158 <DD> Demo programs that are not Mac-specific. Some of these may not
 159 work.
 160
 161 <DT> Extensions
 162 <DD> Extensions to the interpreter that are not Mac-specific. Contains
 163 the <code>img</code>, <code>Imaging</code> and <code>Numerical</code> extensions
 164 in this distribution.
 165
 166 <DT> Grammar
 167 <DD> The Python grammar. Included for reference only, you cannot build
 168 the parser on a Mac.
 169
 170 <DT> Include
 171 <DD> Machine-independent header files.
 172
 173 <DT> Modules
 174 <DD> Machine-independent optional modules. Not all of these will work
 175 on the Mac.
 176
 177 <DT> Lib
 178 <DD> Machine-independent modules in Python.
 179
 180 <DT> Lib:lib-dynload
 181 <DD> This is where the dynamically-loaded plugin modules live.
 182
 183 <DT> Lib:plat-mac
 184 <DD> This is where most of the Mac-specific modules live. The modules here
 185 are available both in MacPython-OS9 and MacPython-OSX.
 186
 187 <DT> Objects
 188 <DD> Machine-independent code for various object types. Most of these are
 189 not really optional: the interpreter will not function without them.
 190
 191 <DT> Parser
 192 <DD> The Python parser (machine-independent).
 193
 194 <DT> Python
 195 <DD> The core interpreter. Most files are machine-independent, some
 196 are unix-specific and not used on the Mac.
 197
 198 <DT> Tools
 199 <DD> Tools for python developers. Contains <code>modulator</code> which
 200 builds skeleton C extension modules, <code>bgen</code> which generates
 201 complete interface modules from information in C header files and
 202 <code>freeze</code> which is used to turn Python scripts into real
 203 applications (used by MacFreeze and BuildApplication) There are some
 204 readme files, but more documentation is sorely needed.
 205
 206 </DL>
 207
 208 The mac-specific stuff lives in the <code>Mac</code> folder:
 209 <DL>
 210 <DT> Build
 211 <DD> This is where the project files live and where you build the
 212 libraries, shared libraries, executables and plugin modules. All the
 213 resulting binaries, except for intermedeate results, are deposited in
 214 the toplevel folder or the :Lib:lib-dynload folder (for plugin modules).
 215
 216 <DT> Compat
 217 <DD> Unix-compatability routines. Most of these are not used anymore,
 218 since GUSI provides a rather complete emulation, but you may need
 219 these if you are trying to build a non-GUSI python.
 220
 221 <DT> Demo
 222 <DD> Mac-specific demo programs, some of them annotated.
 223
 224 <DT> Include
 225 <DD> Mac-specific but compiler-independent include files.
 226
 227 <DT> Lib
 228 <DD> MacPython-OS9 specific standard modules which are not shared with
 229 MacPython-OSX.
 230
 231 <DT> Modules
 232 <DD> Mac-specific builtin modules. Theoretically these are all
 233 optional, but some are rather essential (like
 234 <code>macosmodule</code>). A lot of these modules are generated with
 235 <code>bgen</code>, in which case the bgen input files are included so
 236 you can attempt to regenerate them or extend them.
 237
 238 <DT> MPW
 239 <DD> MPW-specific files. These have not been used or kept up-to-date
 240 for a long time, so use at your own risk.
 241
 242 <DT> mwerks
 243 <DD> Mwerks-specific sources and headers. Contains glue code for
 244 Pythons shared-library architecture, a replacement for
 245 <code>malloc</code> and a directory with various projects for building
 246 variations on the Python interpreter. The <code>mwerks_*.h</code>
 247 files here are the option-setting files for the various interpreters
 248 and such, comparable to the unix command-line <code>-D</code> options
 249 to the compiler. Each project uses the correct option file as its
 250 "prefix file" in the "C/C++ language" settings. Disabling optional
 251 modules (for the 68K interpreter), building non-GUSI interpreters and
 252 various other things are accomplished by modifying these files (and
 253 possibly changing the list of files included in the project window, of
 254 course).
 255
 256 <DT> OSX
 257 <DD> Specific to MacPython-OSX, not used by MacPython-OS9.
 258
 259 <DT> OSXResources
 260 <DD> Specific to MacPython-OSX, not used by MacPython-OS9.
 261
 262 <DT> Python
 263 <DD> Mac-specific parts of the core interpreter.
 264
 265 <DT> Resources
 266 <DD> Resource files needed to build the interpreter.
 267
 268 <DT> Scripts
 269 <DD> A collection of various mac-specific Python scripts. Some are
 270 essential, some are useful but few are documented, so you will have to
 271 use your imagination to work them out.
 272
 273 <DT> Tools
 274 <DD> A collection of tools, usually bigger than those in the scripts
 275 folder. The important ones here are the IDE and macfreeze. The IDE is built
 276 with the buildIDE.py script, which puts the resulting applet in the toplevel
 277 folder. Macfreeze is usually invoked through the BuildApplication script,
 278 but for more control over the freezing process you can run the main script here.
 279
 280
 281 <DT> Unsupported
 282 <DD> Modules that are not supported any longer but may still work with a little effort.
 283 </DL>
 284
 285 <H2>Building the PPC interpreter</H2>
 286
 287 First you optionally build the external libraries with buildlibs.prj. <p>
 288
 289 Then, the <code>fullbuild</code> script can be used to build
 290 everything, but you need a fully-functional interpreter before you can
 291 use it (and one that isn't rebuilt in the process: you cannot rebuild
 292 a running program). You could copy the interpreter to a different
 293 place and use that to run fullbuild. The <code>PythonStandSmall.prj</code>
 294 project builds an interpreter that is suited to this, and it can also come
 295 in handy if you need to debug things (which is easier in a static program). <p>
 296
 297 In case you want to build by hand, or in case the <code>fullbuild</code>
 298 script does not work, here is a breakdown of the various projects. <p>
 299
 300 The projects for interpreter and core library are linked together, so
 301 building the PythonInterpreter target
 302 in <code>PythonInterpreter.prj</code>
 303 will result in the whole core being built, but not the extension modules.  <p>
 304
 305 You will get about 100 warnings on "missing prototype" for the various module init
 306 routines, ignore these. You will also get numerous warnings on functions from GUSI which
 307 override functions from MSL, ignore these too. <p>
 308
 309 Here is a breakdown of the projects:
 310
 311 <DL>
 312
 313 <DT> PythonCore
 314 <DD> The shared library that contains the bulk of the interpreter and
 315 its resources.
 316 It is a good idea to immedeately put an alias to this
 317 shared library in the <code>Extensions</code> folder of your system
 318 folder.  Do exactly that: put an <em>alias</em> there, copying or
 319 moving the file will cause you grief later if you rebuild the library and
 320 forget to copy it to the extensions folder again. The ConfigurePythonXXX applets
 321 will also do this. <br>
 322
 323 <DT> PythonInterpeter
 324 <DD> The interpreter. This is basically a routine to call out to the
 325 shared library. <p>
 326
 327 <DT> Plugin projects
 328 <DD> Each plugin module has a separate project, and these can be rebuilt on
 329 the fly. Fullbuild (or actually it's little helper genpluginprojects) takes
 330 care of this.
 331 </DL>
 332
 333 After creating the alias to <code>PythonCore</code> you remove any old
 334 <code>Python XXXX Preferences</code> file from the <code>Preferences</code> folder
 335 (if you had python installed on your system before) and run the interpreter once
 336 to create the correct preferences file.  <p>
 337
 338 Next, you have to build the extension modules.
 339 If you don't use fullbuild simply open each project and build it.
 340 <p>
 341
 342 Finally, you must build the standard applets:
 343 <code>EditPythonPrefs</code>, <code>BuildApplet</code>, etc. For the N-th time:
 344 fullbuild does this for you, but you can also manually drag/drop them onto
 345 BuildApplet. <p>
 346
 347 You are all set now, and should read the release notes and
 348 <code>ReadMe</code> file from the <code>Mac</code> folder.
 349
 350 Rebuilding .exp files is no longer needed since CodeWarrior 7.
 351
 352 <H2><a name="cvs">Using the CVS source archive</a></H2>
 353
 354 It is possible (and probably best) to access the Python sources through remote CVS. The
 355 advantage of this is that you get the very latest sources, so any bug
 356 fixed or new features will be immedeately available. This is also the
 357 disadvantage, of course: as this is the same tree as is used for
 358 development it may sometimes be a little less stable. <p>
 359
 360 The CVS client of choice is Alexandre Parenteau's MacCVS. It can be
 361 obtained through the <a href="http://www.wincvs.org">WinCVS
 362 homepage</a>. MacCVS uses Internet Config to set file types correctly
 363 based on the filename extension. In the maccvs preferences you should
 364 also set (in the "binary files" section) "use mac encoding:
 365 applesingle" and (in the "text files" section) "use ISO latin 1
 366 conversion". <p>
 367
 368 It is a good idea to disable Quicktime Exchange in the Quicktime control
 369 panel if you are on OS9 or before. Quicktime Exchange will magically map
 370 some extensions to filetypes, and this can seriously hinder you if, for
 371 instance, <code>.bmp</code> is not a Windows bitmap file. <p>
 372
 373 The Python sources are checked out from the main
 374 Python CVS archive on sourceforge.net, see the <a
 375 href="http://www.python.org/download/cvs.html">Source access via
 376 CVS</a> page for details. When you check the sources out you will get
 377 something like <code>Python:dist:src</code>, and under that the
 378 <code>Modules</code>, <code>Lib</code>, <code>Mac</code> etc hierarchy. The
 379 <code>src</code> folder can be renamed to <code>Python</code>, and
 380 is what this document refers to as the "toplevel Python folder". <P>
 381
 382 The CVS repository does not contain all the projects for the plugin modules,
 383 these are built with <code>fullbuild.py</code> normally. For this reason
 384 it is probably a good idea to first build <code>PythonStandSmall.prj</code>,
 385 which builds a fairly minimal interpreter, and then follow the
 386 fullbuild instructions</a>.
 387
 388 <H2>Odds and ends</H2>
 389
 390 Some remarks that I could not fit in elsewhere:
 391
 392 <UL>
 393
 394 <LI> It may be possible to use the <code>PythonCore</code> shared
 395 library to embed Python in another program, if your program can live
 396 with using GUSI for I/O. Use PythonCore in stead of your MSL C library
 397 (or, at the very least, link it before the normal C library). Ask for help
 398 on PythonMac-SIG if you have problems with this.
 399
 400 <LI> <a name="extending"></a>It is possible to build PPC extension
 401 modules without building a complete Python. The binary distribution
 402 installer can optionally install all the needed folders (the develop
 403 option). A template for a dynamic module can be found in
 404 <code>xx.prj</code>.
 405
 406 <LI> The Python shared library architecture is a variant of the architecture
 407 described as "application with shared libraries and dropins" in the MetroWerks
 408 "Targeting MacOS" documentation. The Python Application and applet-template use
 409 the <code>MSL AppRuntime.Lib</code> runtime library (with properly set CFM
 410 initialization and termination routines). PythonCore uses <code>MSL Runtime.Lib</code>,
 411 which is really intended for standalone programs but which we fool into working by
 412 providing a dummy main program.
 413 It is linked statically into PythonCore (and exported to the applications and plugins)
 414 so we do not have to distribute yet another shared library. Plugin modules use
 415 <code>MSL ShlibRuntime.Lib</code> (not the dropin runtime: modules are never unloaded)
 416 and obtain the rest from PythonCore. PythonCore uses a
 417 non-standard initialization entry point, <code>__initialize_with_resources</code>, to
 418 be able to obtain resources from the library file later on. Plugins can do the same
 419 (_tkinter does) or use the standard <code>__initialize</code> entry point.
 420
 421
 422 </UL>
 423 </BODY>
 424 </HTML>