Updated hint

[linux_from_scratch_hints.git] / crablfs.txt

AUTHOR: Zhou Peng(Chowroc) <chowroc dot z at gmail dot com>

DATE: 2006-12-05

LICENSE: GNU Free Documentation License Version 1.2

SYNOPSIS: crablfs -- User Based Package Management System

DESCRIPTION:
        Every package and its files belong to a system normal user, and
a batch script to rebuild or migrate your personalized LFS/BLFS
system by install all your packages all at once.

ATTACHMENTS:
    http://cheeseshop.python.org/pypi?:action=display&name=crablfs
    https://sourceforge.net/project/showfiles.php?group_id=180695

PREREQUISITES:
    python-2.4
    pexpect-2.1(optional)

HINT:

(1) Introduction:
crablfs: User Based Package Management System which is written in
python. Every package and its files belong to a system normal user,
and a batch script to rebuild or migrate your personalized LFS/BLFS
system by install all your packages automatically all at once. The
goal of "crablfs" is to exert more control on packages that compiled
from source, and reduce the costs of system migration and backup.

The basic theory is: every package in the system has a corresponding
user it belongs to, the installation actions(copy, mkdir, etc) will
run by this user, thus all the files a package generated have the
mode of this user. At the same time, all those users belongs to a
'install' group, and the key directories in the system such as
/usr/local, /usr/local/bin, etc have g+s,o+t mode, and owned to the
'install' group, to make different packages create their own files
and directories in those directories.

There are many benefits to use this method. which can be looked up
from the "LFS hints":

More Control and Package Management using Package Users (v1.2)
http://www.linuxfromscratch.org/hints/downloads/files/more_control_and_pkg_man.txt

Based on this theory, the package manager "userpack" is a command
line interface like rpm and paco, for finishing some necessary
operations. It can install, remove, query a package, or list all
packages of current system, or find the package owner of a specified
file. the package name($pkgname-$version) will add to the list file
(/usr/src/packages.list for default).

Every package user has its own HOME dir, which is default to
/usr/src/$pkgname. It will stores compressed source archives, all
patches and supplementary files for installation, and installation
profiles which contains the information of the package, such as
user/group, source archive, patches, create time and those commands
to build this package.

By default, "userpack" use interactive mode, which will start a
shell like command line, in which you type "cmd $command" to execute
the corresponding command, and those commands will been recorded to
"installation profile"(described before). This profile can be used
to execute installation commands automatically in noninteractive
mode.

The script "crablfs" is a batch script write in python, it is another
ALFS instance in other words, which make use of the automated ability
of "userpack". It can use the list file(/usr/src/packages.list)
mentioned before, to operate the compressed source archives and
installation profiles you appointed automatically. And you can also
specified a directory that has the same structure as $homepre
(/usr/src) to build a ALFS system automatically. This means that you
can migrate from an old system to new ones more easily -- and rebuild
your personalized system more quickly.

The essential differences among Linux distributions is the
differences of the package management ways. As a distribution, LFS
actually does not has a mature package manager itself, this of course
roots to the flexibility of LFS, but for daily use, a mature package
management system is necessary. It is recommended to use User Based
Management in the BLFS document, but the LFS hints only give the basic
 principles which is very creative, but the shell scripts is not
enough to form a self-consistency system. So I write one regardless of
unveil my superficialness, a hope that my crude remarks may draw forth
by abler people.

Of course crablfs is not only used for LFS, it aims to those packages
that build from source code, so it can used on other distributions, too.

It's now available from:
https://sourceforge.net/project/showfiles.php?group_id=180695
http://cheeseshop.python.org/pypi?:action=display&name=crablfs

(2) install crablfs
To make sure that crablfs also under the control of package
management, to install by the following way to build the tools chain:
# export PYTHONPATH=/opt/lib/python2.4/site-packages
// sys.path will not be influenced by PYTHONPATH

*** Now the pexpect package is optional ***
# tar xfz pexpect-2.1.tar.gz
# cd pexpect-2.1
# python setup.py install --prefix=/opt
# cd ..

# tar xfz crablfs-0.1.tar.gz
# cd crablfs-0.1
# vi userpack.dirs
*** define all key directories ***
*** these dirs will belong to 'install' group.and have g+s,o+t mode ***
/usr/local/
/usr/local/bin/
/usr/local/doc/
/usr/local/etc/
/usr/local/include/
/usr/local/info/
/usr/local/lib/
/usr/local/libexec/
/usr/local/man/
/usr/local/man/man1/
/usr/local/man/man2/
/usr/local/man/man3/
/usr/local/man/man4/
/usr/local/man/man5/
/usr/local/man/man6/
/usr/local/man/man7/
/usr/local/man/man8/
/usr/local/sbin/
/usr/local/share/
/usr/local/share/doc/
/usr/local/share/info/
/usr/local/share/locale/
/usr/local/share/man/
/usr/local/share/man/man1/
/usr/local/share/man/man2/
/usr/local/share/man/man3/
/usr/local/share/man/man4/
/usr/local/share/man/man5/
/usr/local/share/man/man6/
/usr/local/share/man/man7/
/usr/local/share/man/man8/
/usr/local/share/misc/
/usr/local/share/terminfo/
/usr/local/share/zoneinfo/
/usr/lib/python2.4/site-packages/
/usr/lib64/python2.4/site-packages/
// Adjust it for you own needs
# python setup.py install --prefix=/opt
# cd ..

# /opt/bin/userpack init
# /opt/bin/userpack install -f /tmp/crablfs-0.1.tar.gz crablfs-0.1
crablfs> cmd tar xfz crablfs-0.1.tar.gz
crablfs> cmd cd crablfs-0.1
crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
crablfs> cmd cd ..
crablfs> cmd rm -rf crablfs-0.1
crablfs> commit

*** Now the pexpect package is optional ***
# userpack install -f /tmp/pexpect-2.1.tar.gz pexpect-2.1
crablfs> cmd tar xfz pexpect-2.1.tar.gz
crablfs> cmd cd pexpect-2.1
crablfs> cmd python setup.py install
crablfs> cmd cd ..
crablfs> cmd rm -rf pexpect-2.1
crablfs> commit

# unset PYTHONPATH

(3) package management:
Manager is 'userpack'.

a. install a package:
# userpack install $pkgname-$version
for example:
# userpack install rxvt-2.7.10
you can classify:
# userpack install meida.mplayer-1.0pre8
because the system support '.' for user name.

The short way is:
# upm i $pkgname-$version

******
NOTICE:
* It is not true for LFS system,the shadow package that LFS use
only support [-a-z0-9_] chars, so I wrote a shadow module to access
/etc/passwd, /etc/group, /etc/shadow, /etc/gshadow directly.

*. So far the classification function have not been tested strictly.
******

Under the conditions above, the default action will taken, that is:
the compressed source archive and patches are considered to have
been in the dir of HOME(/usr/src/$pkgname), other will you should
specify it with argument:
# userpack install -f /mnt/file/packages/rxvt-2.7.10.tar.gz rxvt-2.7.10
to make those archive and patches to be copied to HOME.

This command will start an interactive command line, it is simple and
only several command. In fact it is like a shell, you can type any
'shell' command to install the package, only need add 'cmd' before the
command to make the 'userpack' records the commands. These commands
will be stored into HOME/.config(installation profile).
crablfs> cmd tar xfz rxvt-2.7.10.tar.gz
crablfs> cmd cd rxvt-2.7.10
crablfs> cmd ./configure
crablfs> cmd make
crablfs> cmd make install
crablfs> cmd cd ..
crablfs> cmd rm -rf rxvt-2.7.10
crablfs> commit

You should known that the current user is the package user, not root;
and current working directory is the user's HOME, so there may be
ownership problem(What is the purpose of User Based Package
Management System).

When you type commands, you can use 'list' to see what you have typed:
crablfs> list
0, tar xfz rxvt-2.7.10.tar.gz
1, cd rxvt-2.7.10
2, ./configure
3, make
4, make install

Use 'rollback' to clean up these, and 'del N' to delete specified
command.

If you want to execute a command but not want it to be recorded into
the 'list', just 'do':
crablfs> do ls

You must 'commit/cmt' at last, thus all the commands will be recorded
to the .config, the $pkgname-$version couple will been add to list
file(/usr/src/packages.list for default), to indicate that the package
has been installed. If there is a command error makes the installation
failed, and the fault can't been clean up by yourself, just type
'quit' and the installation terminated, but the archive and patches
will still be copied to HOME.

If there is any other supplementary files:
# userpack install -f /mnt/file/packages/MPlayer-1.0per8.tar.bz2 \
    -p /mnt/file/packages/all-20060611.tar.bz2 \
    -p /mnt/file/packages/Blue-1.6.tar.bz2 mplayer-1.0pre8
// all is mplayer's codecs.Blue is GUI Skin.

If you have the installation profile before, you can use -a|--auto
to install automatically:
# userpack install -a rxvt-2.7.10
This means the HOME/.config exists, otherwise use -c|--profile:
# userpack install -ac /mnt/packages/confiles/rxvt-2.7.10 rxvt-2.7.10

If you have burned /usr/src to DCD-R's sources dir, you can also
execute the automatically installation below:
# userpack install -as /mnt/dvdrom/sources rxvt-2.7.10

Moreover, the sources could also on the net such as ftp or ssh:
ssh://localhost/sources/mlterm-2.9.3.tar.gz
scp://localhost/sources/mlterm-2.9.3.tar.gz
the 2 above is same,
ftp://localhost/pub/sources/mlterm-2.9.3.tar.gz
for example:
# userpack install \
    -f ftp://192.168.0.1/pub/sources/mlterm-2.9.3.tar.gz mlterm-2.9.3

******
If the package name(mlterm-2.9.3) is not given, it can be parsed
from the archive name(mlterm-2.9.3.tar.gz) automatically by 'userpack'.
******

If a archive has problem and makes installation failed, the archive
still has been copied to HOME, and the next time 'userpack' will check
whether the archive exists in HOME, if true, copy will not occur. To
change this, use -C|--copy-force:
# userpack install -Caf libesmtp-1.0.3r1.tar.bz2

About the user/group ID, 'userpack' can define its own range,
default from 1000 to 20000, U can define in
/usr/lib/python2.4/site-packages/crablfs/shadow.py:
UID_MIN        1000
UID_MAX        20000
GID_MIN        1000
GID_MAX        20000
ADM_MIN        950
ADM_MAX        1000

This mean the normal user ids range is (1000, 19999), admin group ids
is (950, 999). So 'install' group will be 950 by default.

Besides the 'install', other admin groups will be used as 'application
groups'. For example, 'glib2', 'atk', 'gtk', 'gtk2' all belongs to
'GTK' group for some limited directories sharing, such as
/usr/share/gtk-doc/
/usr/share/gtk-doc/html
thus no modification of userpack.dirs is necessary.

Correspondingly, add a '-G' option to support multiple groups, for
example, 'xfce4' belogs to 'xfce4', 'install' and 'GTK' group.
(default is 'xfce4' and 'install').

Now I can do this to support 'applications group' and 'multiple groups':
# upm i -G GTK glib2-2.6.4
crablfs> ...
......
crablfs> chgrp GTK /usr/share/gtk-doc/{,html}
crablfs> chmod g+w,o+t /usr/share/gtk-doc/{,html}
......

# upm i -G GTK atk-1.9.1
......

These all can be read from the installation profiles and be executed
automatically.

b. remove:
# userpack remove rxvt-2.7.10
or
# upm r rxvt-2.7.10

c. list all packages under the control:
# userpack packs
or
# upm a
rxvt-2.7.10
mplayer-1.0pre8
crablfs-0.1
pexpect-2.1
mlterm-2.9.3

d. query a file's package owner:
# userpack owner /usr/local/bin/mlterm
or
# upm o /usr/local/bin/mlterm
mlterm-2.9.3

e. list all files a package contains:
# userpack files pexpect-2.1
or
# upm l pexpect-2.1
/usr/lib/python2.4/site-packages/fdpexpect.py
/usr/lib/python2.4/site-packages/pxssh.py
/usr/lib/python2.4/site-packages/pexpect.py
/usr/lib/python2.4/site-packages/pxssh.pyc
/usr/lib/python2.4/site-packages/pexpect.pyc
/usr/lib/python2.4/site-packages/fdpexpect.pyc

This will read $HOME/.files to print all files the package have. This
file is generated by the Package Manager 'userpack' as soon as the
installation is finished, but you can force 'userpack' to regenerate
it by run:
# upm L pexpect-2.1
This command will also print all the files of the package.

(4) About the pkgname & version rule:
This is the argument to
'userpack', the form is '$pkgname-$version', the python re pattern
is:
'^(?P<pkgname>(\w+-{0,1})+)(-(?P<version>(\d[\w]*[-\.])*\d[\w]*))'

That is '$pkgname' part is 'part1[-part2][-part3][...]', every part
could contains '[-a-zA-Z0-9_]', no rules for start char, but it is
recommended to use alphabet; when meet '-[0-9][a-zA-Z0-9_]\.'
pattern('-' follows a string start with digit, and follows a '.'),
this will be considered as start part of $version, $version can also
contains other '-', and when meet a string not start with digit,
$version ends. chars follows are ignored.

(5) ALFS
It is mentioned before that all commands will be recorded to
HOME/.config, and 'userpack' can also use this commands to run
automatically, so there could be a batch script to use packages.list
to install those packages all at once. Based on this, we can get an
instance of ALFS.

The target of the instance is extended to tranditional nALFS and
jhalfs, for more personalized environment, for example: The first
time, I build LFS/BLFS step by step, and the packages installed,
related with their own's information & operations are recorded to
the profiles, the next time, I can use those profiles to build the
software envrionment I need with a 'turn key' way, exactly as those
before. This means I can migrate many times, and others can also use
my profiles to build their own personalized system. The end target
also contains a platform independent operation.

This batch script is 'crablfs'.

******
So far, it is only tested under the condition that the LFS base
system has been built, to use 'crablfs' to build a bunch of BLFS
packages, so it is the future to make the LFS base system under the
control of package management system.
******

At first, adjust the tool chain:
# cd /blfs-sources
# tar xfj python-2.4.1.tar.bz2
# cd python-2.4.1
# patch -Np1 -i ../python-2.4.1-gdbm-1.patch
# ./configure --prefix=/opt --enable-shared
# make && make install
# cd ..
# rm -rf python-2.4.1
// U'd better delete it, otherwise may be problems

# export PYTHONPATH=/opt/lib/python2.4/site-packages

*** Now pexpect is optional ***
# tar xfz pexpect-2.1.tar.gz
# cd pexpect-2.1
# /opt/bin/python setup.py install --prefix=/opt
# cd ..
# rm -rf pexpect-2.1

# tar xfz crablfs-0.1.tar.gz
# cd crablfs-0.1
# cp userpack.dirs.blfs userpack.dirs
// Adjust for you own need and system
// Whenever U meet pemission problem, just adjust /etc/userpack.dirs
//  and run 'userpack init'
# /opt/bin/python setup.py install --prefix=/opt

# /opt/bin/userpack init
# /opt/bin/userpack install -f python-2.4.1.tar.bz2 -p
# python-2.4.1-gdbm-1.patch python-2.4.1
crablfs> cmd tar xfj python-2.4.1.tar.bz2
crablfs> cmd cd python-2.4.1
crablfs> cmd ./configure --prefix=/usr --enable-shared
crablfs> cmd make
crablfs> cmd make install
crablfs> cmd cd ..
crablfs> cmd rm -rf python-2.4.1
crablfs> commit

# /opt/bin/userpack install -f crablfs-0.1.tar.gz crablfs-0.1
crablfs> cmd tar xfz crablfs-0.1.tar.gz
crablfs> cmd cd crablfs-0.1
crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
crablfs> cmd cd ..
crablfs> cmd rm -rf crablfs-0.1
crablfs> commit

*** Now the pexpect is optional ***
# userpack install -f pexpect-2.1.tar.gz pexpect-2.1
crablfs> cmd tar xfz pexpect-2.1.tar.gz
crablfs> cmd cd pexpect-2.1
crablfs> cmd python setup.py install
crablfs> cmd cd ..
crablfs> cmd rm -rf pexpect-2.1
crablfs> commit

# unset PYTHONPATH
# userpack packs
python-2.4.1
crablfs-0.1
pexpect-2.1

******
Now there is a script 'scripts/setup-blfs' to do these all!
******

And use the same way, I installed some packages:
# userpack packs
# cat /usr/src/packages.list
python-2.4.1
crablfs-0.1.1
pexpect-2.1
net-tools-1.60
openssl-0.9.7g
cracklib-2.8.3
Linux-PAM-0.80
iptables-1.3.3
gnupg-1.4.1
pcre-6.1
libxml-1.8.17
libxml2-2.6.20
libxslt-1.1.14
gdbm-1.8.3
pkg-config-0.19
glib-1.2.10
glib2-2.6.4
expat-1.95.8
libesmtp-1.0.3r1
lzo-2.01
libusb-0.1.10a
libjpeg-6b
libpng-1.2.8
which-2.16
unzip-5.52
zip-2.32
pciutils-2.1.11
cpio-2.6
libpcap-0.9.3
ppp-2.4.3
rp-pppoe-3.8
libsoup-2.2.3
subversion-1.1.4
wget-1.9.1
bind-9.3.1
gc-6.8
w3m-0.5.1
gpm-1.20.1
dhcpcd-1.3.22
freetype-2.1.10
fontconfig-2.3.2
Xorg-6.9.0
915resolution-0.5.2
gtk-1.2.10
pango-1.8.1
atk-1.9.1
gtk2-2.6.7
startup-notification-0.8
hicolor-icon-theme-0.8
xfce-4.2.2
mlterm-2.9.3

The X Window is OK and xfce is usable, so I try to *migrate*:
# mv /usr/src/packages.list /blfs-sources/
# copy-profiles /usr/src
// This will generate a profiles dir in current dir,
// which contains all the profiles from $homepre(/usr/src)
# mv profiles /blfs-sources
# crablfs -t alfs \
    -C /blfs-sources/profiles/ \
    -F /blfs-sources/ \
    /blfs-sources/packages.list

If you have installed such a BLFS system and burned /usr/src all to
the DVD-R, you can do the following for another system to install all
package once a time:
# crablfs -t crablfs[/default] \
    -s /mnt/dvdrom/sources/ \
    /mnt/dvdrom/sources/packages.list

You may want to adjust 'packages.list' for your own needs, if you want
, you can also edit the profiles.

crablfs will record current successfully installed package name to
/var/log/crablfs/.mark. Because of the packages dependencies
problems, if one package failed, the 'crablfs' process terminated.
After you clean up the problem, you can continue from the
breakpoint.

So if .mark point to the last item of packages.list, 'crablfs' will
do nothing. Adjust it for you need.

Moreover, 'crablfs' will check packages.list, illegal package name
will terminate 'crablfs' and do nothing; if the name in .mark not in
packages.list, you will get a "x not in list" error report, you can
adjust corresponding, or just delete the .mark file.

So far, there are few permission problems, and all can avoid by
changing the installtion arguments. The wrappers for chmod, chgrp,
chown, install may be needed, but so far I have not see the
neccessary. To dicide this for further using and testing.

(6) The next plan
* Add upgrade part
* Add gettext internationlized support
* Whole infomations show of a package
* PyUnit
* Add code for cmdline to read .config history commands
* Make cmdline can execute 'root' commands, as 'sudo', by IPC?
* A command editor?
* File lock of packages.list
* Change .config to the form of '$pkgname-$version'
* Enganced platform independent support:
    Add var and env var to cmdline for different $ARCH & $LOCALE
* Support multi installation sources
* Enganced classification mechanism
* Intergrated checking
* Enganced configuration getting & setting: Plain Tree & XML
* Consider package dependencies problems
* Better documents: internal, design doc and manual
* Make 'userpack init' can adjust those dir items deleted from 'userpack.dirs'

******
*** LIMITATIONS ***
When want to install 'glib2-2.6.4', but type in error with
'glib-2.6.4':
# upm i -a -c profiles/glib2 glib-2.6.4
This will mistakenly remove all files of glib-1.2.10!!! Because when a
package installation failed, the Manager will remove all the files
that generated by 'install', 'cp', 'mv' commands during the process.

So far, this limitation roots in the limitation of package version
control, this also makes 'upgrade' not easy.
******

ACKNOWLEDGEMENTS:
    Matthias S. Benkmann built the original principle

CHANGELOG:
2006-12-05:
    For crablfs-0.1.1beta.

2006-10-25:
    crablfs-0.1 has been distributed at 2006-10-20, so I write this hint.