Patrick Welche <prlw1@cam.ac.uk>

[netbsd-mini2440.git] / distrib / syspkg / notes / PROPOSAL

Proposal: NetBSD System Installation Packages
=============================================

CONTENTS
--------
0. Introduction
1. System Packages
  1.1 Package Format
  1.2 Package Granularity
    1.2.1 Root/User/Share separation
2. Package Sets
  2.1 Set format
3. Creation of Packages and Sets
4. Modifications to the NetBSD installation process
A. Working Plan

------------------------------------------------------------------------

0. Introduction

  The current NetBSD installation process involves the downloading
  of binary `sets', which the user can choose among at install time.
  A set is a tarred, gzipped set of files, to be untarred relative
  to '/'.  No facility exists to choose convenient subsets of the files
  in a set to be installed, or to remove a set which has been installed.

  The current granularity of sets is very large, being divided into:

        base    -- general system binaries
        comp    -- compilers and related tools
        etc     -- system configuration files
        games   -- games and other amusements
        man     -- system manual pages
        misc    -- items not falling into other categories
        secr    -- items not exportable under US law
        text    -- text processing tools
        xbase   -- general X11R6 binaries
        xcomp   -- X11R6 development items
        xfont   -- X11R6 fonts
        xserver -- X11R6 servers for various video hardware

  Users who wish to install part of a set need to either install
  the full set and then determine which files they need to remove,
  or abandon the normal install process, and figure out which files
  to unpack by hand.  Similarly, if a set is later determined to
  be unnecessary, the only way to remove it is to figure out which
  files on the system belonged to that set, and remove them by hand.

  When it comes time to upgrade a system which has been installed this
  way, the usual procedure is to unpack a new version of each installed
  set over the previous version.  When a file is moved, renamed, or
  removed in a newer version of a set, the old version often remains on
  the system for some time.  In at least one recent instance (the move
  of /sbin/mountd to /usr/sbin/mountd) this has resulted in much
  confusion, and large amounts of traffic on the relevant mailing lists.

  The remainder of this document describes a proposed method of handling
  these and other problems with the current install set system by
  moving to the use of fine-grained `system packages', based on the
  currently existing package system for third-party software, and
  allowing users to choose among either `package sets' at the same
  granularity as our current install sets, or individual `packages'
  at a much finer level of granularity.  In either case, the new system
  would also greatly simplify upgrading or removal of such packages
  and sets at a later time, and would allow tracking of dependencies
  between the various sets and packages distributed as part of NetBSD.

  First, the format of system packages in the proposed system is
  discussed, followed by the format of package sets, which will serve
  as a replacement for the current install sets.  The creation of
  packages in an automated fashion from a NetBSD source tree is
  discussed as is the effect of this system on the NetBSD installation
  process.  An appendix discusses my work plan to implement this new
  system.

  It is hoped that this document will serve as a basis for discussion
  of what is involved in changing NetBSD to use system packages for
  system installation and upgrades, and that after several iterations
  of discussion and revision, it will serve as a plan for the actual
  implementation of this system.

------------------------------------------------------------------------

1. System Packages

  System packages will be the basic building blocks of a NetBSD system.
  At install time, the user will choose which system packages to install,
  subject to dependencies between packages.  After system install,
  users will be able to install additional packages or remove installed
  packages.  When it comes time to upgrade the system, packages can
  be removed and reinstalled in a reliable fashion.  All of this
  functionality is already available for third-party software via the
  use of the software package system in /usr/pkgsrc.  This proposal
  extends that functionality to the NetBSD system itself.

1.1 Package Format

  System packages will be identical in format to the binary packages
  used by the current third-party package system.  This will allow the
  same tools to be used for working with system packages as are
  currently used for working with third-party packages.  This will also
  also allow the system to benefit from the fact that the workings of
  the current package system are well understood.

1.2 Package Granularity

  System packages will be at the granularity of groups of related tools
  and their support files.  Thus, `Kerberos', `UUCP', `Text formatting'
  and `amd' might each be packages which depended on nothing but a few
  base packages, while `C Development' and `Fortran development' might
  be separate packages which each depended upon `Binutils' and `Base
  EGCS utilities' packages.  Packages sets, described below, would add
  the ability to choose entire broad categories of software to install,
  like todays install sets, while maintaining the ability to remove
  individual packages later.

1.2.1 Root/User/Share separation

  In order to support a variety of system configurations, it is crucial
  that the new package system support the possibility of some part of
  a system residing on a server and possibly being shared between
  multiple machines on a network.  A machine which has some filesystems
  local and some shared must, at the very least, be able to add and
  remove packages from local filesystems, and should be able to
  determine what packages have been added or removed from the volumes
  mounted over the network.

  The most common shared configurations are to have a system share
  /usr/share from the network, and have all other filesystems local,
  or to share the entirety of /usr from the network, and maintain
  local root and /var hierarchies, possibly as a single filesystem.
  Other commonly shared hierarchies include /usr/X11R6 and /usr/pkg.

  Two steps are necessary to support this type of sharing: the system
  must be able to check separate repositories for packages installed
  on different filesystems, and packages must be designed so as to
  allow a client to install only those parts of the system which reside
  on local filesystems.
  
  The first of these is addressed by a set of patches described by
  Alistair Crooks in a post to the netbsd-current mailing list on
  Friday, September 18, 1998.  These patches, which have not yet been
  committed cause third-party software packages installed in /usr/pkg
  to be registered in /usr/pkg/etc/pkg, and packages installed in
  /usr/X11R6 to be registered in /usr/X11R6/etc/pkg.  This could be
  extended easily to allow sharing of system package installations by
  having the new system X11R6 packages also use /usr/X11R6/etc/pkg
  for package registration, to have system packages installed in /usr
  use /usr/etc/pkg for package registration, and to have system
  packages installed in / and /var use /etc/pkg for package
  registration.  This would allow all of the types of filesystem
  sharing described above, without introducing too much complication
  into the package system.

  The second step, that of insuring that a client can choose to install
  only the parts of the system which reside on local volumes can be
  most easily addressed by careful consideration of package contents.
  A look through the contents of the current install sets suggests
  that relatively few packages will in fact need to install in more
  than one of /, /usr, /usr/share and /usr/X11R6.  Were such packages
  split into separate components, based on filesystem boundaries,
  users would easily be able to install only the parts which are local
  in their particular configuration.

------------------------------------------------------------------------

2. Package Sets

  In moving to fine-grained system packages, it is important that
  beginning users still be able to select broad categories of software
  to install at once.  The introduction of `package sets', analogous
  in granularity, but not mechanism, to the current binary install sets
  addresses this concern, while maintaining the ability of more advanced
  users to choose among individual packages at install time, and
  maintaining the ability to remove, upgrade, or add individual
  packages at a later time.

  These package sets will maintain the same layout as the current
  install sets, so that a user who chooses the same sets as he would
  have chosen now will see the same results.  In the new system,
  however, these sets will be made up of binary packages, and installing
  a set will simply result in the installation of the constituent
  packages.

2.1 Set format

  A set will be a tar archive containing the packages which make up the
  set plus a contents file.  At the least, the index file will contain
  the name of each included package, plus a one line description of each
  package's contents.  Installation utilities will offer the option of
  installing the whole set, or choosing among individual packages,
  based on the descriptions in the contents file.  It is expected that the
  contents file itself will be automatically generated from the one-line
  descriptions provided in each package's pkg/COMMENT file.

  When a set is installed, the contents file will be recorded in a
  manner similar to the registration of package information in the
  current third-party package system.  This will allow users to remove
  an entire set at a later date, without needing to know what individual
  packages came from that set.

------------------------------------------------------------------------

3. Creation of Packages and Sets

  Under the current distribution-building system, the Makefile in
  /usr/src/etc creates binary install sets from an installed system,
  based on the set lists in /usr/src/distrib/sets/lists.  In the new
  system, a new directory hierarchy, /usr/src/distrib/pkg, will
  contain Makefiles and data files relevant to the creation of
  system packages and package sets.

  The directory /usr/src/distrib/pkg/sets will contain a directory
  for each package set, and each of these directories will contain
  a directory for each package in that set.  The Makefile in
  /usr/src/distrib/pkg/sets will recurse into these set directories
  to build each set.  The individual set Makefiles will recurse into
  each package directory to build the individual packages, and will
  then create a set file from the constituent packages and from the
  contents file, which will be automatically generated from the
  package directories.

  The package directories will resemble the package directories for
  third-party software packages in /usr/pkgsrc, except that they will
  probably rely on the files making up the package already being
  present in ${DESTDIR}, rather than building them directly.  This
  assumption is already present in the current distribution package
  Makefile code, and is probably reasonable to keep.

------------------------------------------------------------------------

4. Modifications to the NetBSD installation process

  Once the NetBSD system is available as system packages and package
  sets, it will be possible to modify the various installation tools
  to use these sets to install the system.  It is expected that
  installation tools will default to allow users to choose among
  package sets at install time, but allow an `advanced mode' in which
  packages could be selected and deselected on an individual basis.

  This will require that the various package tools (at least pkg_add)
  be present on install media to be used with system packages.
  Modifications to sysinst and other install tools are beyond the
  current scope of this proposal, but will be necessary to take
  advantage of the new capabilities provided by this system.

------------------------------------------------------------------------

A. Working Plan

  My current plan for implementing system packages and package sets
  for NetBSD consists of four steps.  All of these steps should be
  taken in the CVS source tree (segregated into src/distrib/pkg, of
  course), and hopefully will involve other contributors in addition
  to myself:

    1.) Hammer this proposal into a more detailed specification
    
        I am submitting this proposal now in the hopes that it
        will spark discussion which will lead to a refinement
        of the planned system package system.  Once some sort
        of consensus is reached on the relevant mailing lists,
        I will begin work in earnest on implementing this.

    2.) Create the /usr/src/distrib/pkg hierarchy, and a template
        package

        The first step in actually implementing this system will
        be to create either an actual or mocked-up system package
        which can be used as a template for creation of the
        remaining system packages.

    3.) Create system packages

        I expect that this step will involve most of the actual
        work in implementing the new system.  Packages will have
        to be created for each functional group of binaries
        currently shipped with NetBSD.  A lot of discussion and
        design will have to go into the decisions as to how
        many packages should make up each set and what files
        belong in which packages.

    4.) Create Package Sets

        Once all system packages exist, it will be necessary to
        put together some code to automatically generate set
        contents files and to create sets from each directory
        of packages in /usr/src/distrib/pkg/sets.

  Once these steps are complete, NetBSD will have system packages,
  and it will be possible to begin looking at modifying the NetBSD
  install process to use them.  It is important to note that none
  of these changes will require modifying the current installation
  set building code in any way, so the use of the current system
  can continue unhindered while the new system is being implemented.

------------------------------------------------------------------------
$Id: PROPOSAL,v 1.2 2004/01/17 05:30:01 lukem Exp $