TopGit - A different patch queue manager
DESCRIPTION
-----------
TopGit aims to make handling of large amount of interdependent topic
branches easier. In fact, it is designed especially for the case
when you maintain a queue of third-party patches on top of another
(perhaps Git-controlled) project and want to easily organize, maintain
and submit them - TopGit achieves that by keeping a separate topic
branch for each patch and providing few tools to maintain the branches.
RATIONALE
---------
Why not use something like StGIT or Guilt or rebase -i for maintaining
your patch queue? The advantage of these tools is their simplicity;
they work with patch _series_ and defer to the reflog facility for
version control of patches (reordering of patches is not
version-controlled at all). But there are several disadvantages -
for one, these tools (especially StGIT) do not actually fit well
with plain Git at all: it is basically impossible to take advantage
of the index effectively when using StGIT. But more importantly,
these tools horribly fail in the face of distributed environment.
TopGit has been designed around three main tenets:
(i) TopGit is as thin layer on top of Git as possible.
You still maintain your index and commit using Git, TopGit will
only automate few indispensable tasks.
(ii) TopGit is anxious about _keeping_ your history. It will
never rewrite your history and all metadata is also tracked by Git,
smoothly and non-obnoxiously. It is good to have a _single_ point
when the history is cleaned up, and that is at the point of inclusion
in the upstream project; locally, you can see how your patch has evolved
and easily return to older versions.
(iii) TopGit is specifically designed to work in distributed
environment. You can have several instances of TopGit-aware repositories
and smoothly keep them all up-to-date and transfer your changes between
them.
As mentioned above, the main intended use-case for TopGit is tracking
third-party patches, where each patch is effectively a single topic
branch. In order to flexibly accommodate even complex scenarios when
you track many patches where many are independent but some depend
on others, TopGit ignores the ancient Quilt heritage of patch series
and instead allows the patches to freely form graphs (DAGs just like
Git history itself, only "one level higher"). For now, you have
to manually specify which patches does the current one depend
on, but TopGit might help you with that in the future in a darcs-like
fashion.
A glossary plug: The union (i.e. merge) of patch dependencies is
called a _base_ of the patch (topic branch).
Of course, TopGit is perhaps not the right tool for you:
(i) TopGit is not complicated, but StGIT et al. are somewhat
simpler, conceptually. If you just want to make a linear purely-local
patch queue, deferring to StGIT instead might make more sense.
(ii) When using TopGit, your history can get a little hairy
over time, especially with all the merges rippling through. ;-)
SYNOPSIS
--------
## Create and evolve a topic branch
$ tg create t/gitweb/pathinfo-action
tg: Automatically marking dependency on master
tg: Creating t/gitweb/pathinfo-action base from master...
$ ..hack..
$ git commit
$ ..fix a mistake..
$ git commit
## Create another topic branch on top of the former one
$ tg create t/gitweb/nifty-links
tg: Automatically marking dependency on t/gitweb/pathinfo-action
tg: Creating t/gitweb/nifty-links base from t/gitweb/pathinfo-action...
$ ..hack..
$ git commit
## Create another topic branch on top of master and submit
## the resulting patch upstream
$ tg create t/revlist/author-fixed master
tg: Creating t/revlist/author-fixed base from master...
$ ..hack..
$ git commit
$ tg patch -m
tg: Sent t/revlist/author-fixed
From: pasky@suse.cz
To: git@vger.kernel.org
Cc: gitster@pobox.com
Subject: [PATCH] Fix broken revlist --author when --fixed-string
## Create another topic branch depending on two others non-trivially
$ tg create t/whatever t/revlist/author-fixed t/gitweb/nifty-links
tg: Creating t/whatever base from t/revlist/author-fixed...
tg: Merging t/whatever base with t/gitweb/nifty-links...
Merge failed!
tg: Please commit merge resolution and call: tg create
tg: It is also safe to abort this operation using `git reset --hard`
tg: but please remember you are on the base branch now;
tg: you will want to switch to a different branch.
$ ..resolve..
$ git commit
$ tg create
tg: Resuming t/whatever setup...
$ ..hack..
$ git commit
## Update a single topic branch and propagate the changes to
## a different one
$ git checkout t/gitweb/nifty-links
$ ..hack..
$ git commit
$ git checkout t/whatever
$ tg info
Topic Branch: t/whatever (1 commit)
Subject: [PATCH] Whatever patch
Base: 3f47ebc1
Depends: t/revlist/author-fixed t/gitweb/nifty-links
Needs update from:
t/gitweb/nifty-links (1 commit)
$ tg update
tg: Updating base with t/gitweb/nifty-links changes...
Merge failed!
tg: Please commit merge resolution and call `tg update` again.
tg: It is also safe to abort this operation using `git reset --hard`,
tg: but please remember you are on the base branch now;
tg: you will want to switch to a different branch.
$ ..resolve..
$ git commit
$ tg update
tg: Updating t/whatever against new base...
Merge failed!
tg: Please resolve the merge and commit. No need to do anything else.
tg: You can abort this operation using `git reset --hard` now
tg: and retry this merge later using `tg update`.
$ ..resolve..
$ git commit
## Update a single topic branch and propagate the changes
## further through the dependency chain
$ git checkout t/gitweb/pathinfo-action
$ ..hack..
$ git commit
$ git checkout t/whatever
$ tg info
Topic Branch: t/whatever (1/2 commits)
Subject: [PATCH] Whatever patch
Base: 0ab2c9b3
Depends: t/revlist/author-fixed t/gitweb/nifty-links
Needs update from:
t/gitweb/pathinfo-action (<= t/gitweb/nifty-links) (1 commit)
$ tg update
tg: Recursing to t/gitweb/nifty-links...
[t/gitweb/nifty-links] tg: Updating base with t/gitweb/pathinfo-action changes...
Merge failed!
[t/gitweb/nifty-links] tg: Please commit merge resolution and call `tg update` again.
[t/gitweb/nifty-links] tg: It is also safe to abort this operation using `git reset --hard`,
[t/gitweb/nifty-links] tg: but please remember you are on the base branch now;
[t/gitweb/nifty-links] tg: you will want to switch to a different branch.
[t/gitweb/nifty-links] tg: You are in a subshell. If you abort the merge,
[t/gitweb/nifty-links] tg: use `exit` to abort the recursive update altogether.
[t/gitweb/nifty-links] $ ..resolve..
[t/gitweb/nifty-links] $ git commit
[t/gitweb/nifty-links] $ tg update
[t/gitweb/nifty-links] tg: Updating t/gitweb/nifty-links against new base...
Merge failed!
[t/gitweb/nifty-links] tg: Please resolve the merge and commit.
[t/gitweb/nifty-links] tg: You can abort this operation using `git reset --hard`.
[t/gitweb/nifty-links] tg: You are in a subshell. After you either commit or abort
[t/gitweb/nifty-links] tg: your merge, use `exit` to proceed with the recursive update.
[t/gitweb/nifty-links] $ ..resolve..
[t/gitweb/nifty-links] $ git commit
[t/gitweb/nifty-links] $ exit
tg: Updating base with t/gitweb/nifty-links changes...
tg: Updating t/whatever against new base...
## Clone a TopGit-controlled repository
$ git clone URL repo
$ cd repo
$ tg remote --populate origin
...
$ git fetch
$ tg update
## Add a TopGit remote to a repository and push to it
$ git remote add foo URL
$ tg remote foo
$ git push foo
## Update from a non-default TopGit remote
$ git fetch foo
$ tg -r foo summary
$ tg -r foo update
USAGE
-----
The 'tg' tool of TopGit has several subcommands:
tg help
~~~~~~~
Our sophisticated integrated help facility. Doesn't do
a whole lot for now.
tg create
~~~~~~~~~
Create a new TopGit-controlled topic branch of a given name
(required argument) and switch to it. If no dependencies
are specified (by extra arguments passed after the first one),
the current branch is assumed to be the only dependency.
After `tg create`, you should insert the patch description
to the '.topmsg' file, which will already contain some
prefilled bits. You can set topgit.to, topgit.cc and topgit.bcc
configuration variables in order to have `tg create`
add these headers with given default values to '.topmsg'.
The main task of `tg create` is to set up the topic branch
base from the dependencies. This may fail due to merge conflicts.
In that case, after you commit the conflicts resolution,
you should call `tg create` again (without any arguments);
it will detect that you are on a topic branch base ref and
resume the topic branch creation operation.
In an alternative use case, if '-r BRANCH' is given instead
of dependency list, the topic branch is created based on
the given remote branch.
tg delete
~~~~~~~~~
Remove a TopGit-controlled topic branch of given name
(required argument). Normally, this command will remove
only empty branch (base == head); use '-f' to remove
non-empty branch.
The '-f' option is also useful to force removal of a branch's base, if
you used 'git branch -D B' to remove the branch B, and then certain
TopGit commands complain, because the base of branch B is still there.
Currently, this command will _NOT_ remove the branch from
the dependency list in other branches. You need to take
care of this _manually_. This is even more complicated
in combination with '-f', in that case you need to manually
unmerge the removed branch's changes from the branches
depending on it.
TODO: '-a' to delete all empty branches, depfix, revert
tg depend
~~~~~~~~~
Change dependencies of a TopGit-controlled topic branch.
This should have several subcommands, but only 'add' is
supported right now.
The 'add' subcommand takes an argument of a topic branch
to be added, adds it to '.topdeps', performs a commit and
then updates your topic branch accordingly. If you want to
do other things related to the dependency addition, like
adjusting '.topmsg', prepare them in the index before
calling 'tg depend add'.
TODO: Subcommand for removing dependencies, obviously
tg info
~~~~~~~
Show a summary information about the current or specified
topic branch.
tg patch
~~~~~~~~
Generate a patch from the current or specified topic branch.
This means that the diff between the topic branch base and
head (latest commit) is shown, appended to the description
found in the .topmsg file.
The patch is by default simply dumped to stdout. In the future,
tg patch will be able to automatically send the patches by mail
or save them to files. (TODO)
Options:
-i base patch generation on index instead of branch
-w base patch generation on working tree instead of branch
tg mail
~~~~~~~
Send a patch from the current or specified topic branch as
email.
Takes the patch given on the command line and emails it out.
Destination addresses such as To, Cc and Bcc are taken from the
patch header.
Since it actually boils down to `git send-email` please refer to
its documentation for details on how to setup email for git.
You can pass arbitrary options to this command through the
'-s' parameter, but you must double-quote everything.
The '-r' parameter with msgid can be used to generate in-reply-to
and reference headers to an earlier mail.
Note: be careful when using this command. It easily sends out several
mails. You might want to run
git config sendemail.confirm always
to let `git send-email` ask for confirmation before sending any mail.
TODO: 'tg mail patchfile' to mail an already exported patch
TODO: mailing patch series
TODO: specifying additional options and addresses on command
line
tg remote
~~~~~~~~~
Register given remote as TopGit-controlled. This will create
the namespace for the remote branch bases and teach 'git fetch'
and 'git push' to operate on them. (Do NOT use 'git push --all'
for your pushes - plain 'git push' will do the right thing.)
It takes a optional remote name argument, and optional
'--populate' switch - use that for your origin-style remote,
it will seed the local topic branch system based on the
remote topic branches. '--populate' will also make 'tg remote'
automatically fetch the remote and 'tg update' to look at
branches of this remote for updates by default.
tg summary
~~~~~~~~~~
Show overview of all TopGit-tracked topic branches and their
up-to-date status ('>' marks the current topic branch,
'0' marks that it introduces no own changes,
'l'/'r' marks that it is local-only or has remote mate,
'L'/'R' marks that it is ahead/out-of-date wrt. its remote mate,
'D' marks that it is out-of-date wrt. its dependencies,
'!' marks that it has missing dependencies (even recursively),
'B' marks that it is out-of-date wrt. its base).
This can take long time to accurately determine all the relevant
information about each branch; you can pass '-t' to get just
terse list of topic branch names quickly. Alternately, you can
pass '--graphviz' to get a dot-suitable output to draw a dependency
graph between the topic branches.
You can also use the --sort option to sort the branches using
a topological sort. This is especially useful if each
TopGit-tracked topic branch depends on a single parent branch,
since it will then print the branches in the dependency
order. In more complex scenarios, a text graph view would be
much more useful, but that is not yet implemented.
The --deps option outputs dependency informations between
branches in a machine-readable format. Feed this to "tsort"
to get the output from --sort.
TODO: Speed up by an order of magnitude
TODO: Text graph view
tg export
~~~~~~~~~
Export a tidied-up history of the current topic branch
and its dependencies, suitable for feeding upstream.
Each topic branch corresponds to a single commit or patch
in the cleaned up history (corresponding basically exactly
to `tg patch` output for the topic branch).
The command has three possible outputs now - either a Git branch with
the collapsed history, a Git branch with a linearized history, or a
quilt series in new directory.
In case of producing collapsed history in new branch,
you can use this collapsed structure either for providing
a pull source for upstream, or further linearization e.g.
for creation of a quilt series using git log:
git log --pretty=email -p --topo-order origin..exported
To better understand the function of `tg export`,
consider this dependency structure of topic branches:
origin/master - t/foo/blue - t/foo/red - master
`- t/bar/good <,----------'
`- t/baz ------------'
(Where each of the branches may have hefty history.) Then
master$ tg export for-linus
will create this commit structure on branch for-linus:
origin/master - t/foo/blue -. merge - t/foo/red -.. merge - master
`- t/bar/good <,-------------------'/
`- t/baz ---------------------'
In case of using the linearize mode:
master$ tg export --linearize for-linus
you get a linear history respecting the dependencies of your patches in
a new branch for-linus. The result should be more or less the same as
using quilt mode and reimporting it into a Git branch. (More or less
because the topologic order can usually be extended in more than one
way into a complete ordering and the two methods may choose different
one's.) The result might be more appropriate for merging upstream as
it contains fewer merges.
Note that you might get conflicts during linearization because the
patches are reordered to get a linear history.
In case of the quilt mode,
master$ tg export --quilt for-linus
would create this directory for-linus:
for-linus/t/foo/blue.diff
for-linus/t/foo/red.diff
for-linus/t/bar/good.diff
for-linus/t/baz.diff
for-linus/series:
t/foo/blue.diff -p1
t/bar/good.diff -p1
t/foo/red.diff -p1
t/baz.diff -p1
The command works on the current topic branch
and can be called either without a parameter
(in that case, '--collapse' is assumed)
and with one mandatory argument: the name of the branch
where the exported result shall be stored.
The branch will be silently overwritten if it exists already!
Use git reflog to recover in case of mistake.
Alternatively, call it with the '--quilt' parameter
and an argument specifying the directory
where the quilt series should be saved.
With '--quilt', you can also pass '-b' parameter followed by
a comma-separated explicit list of branches to export. This
mode of operation is currently not supported with collapse.
In '--quilt' mode the patches are named like the originating topgit
branch. So usually they end up in subdirectories of the output
directory. With option '--flatten' the names are mangled such that
they end up directly in the output dir (i.e. slashed are substituted by
underscores). With '--numbered' (which implies '--flatten') the patch
names get a number as prefix to allow getting the order without
consulting the series file, which eases sending out the patches.
Usage: tg export ([(--collapse | --linearize)] BRANCH | --quilt DIR)
TODO: Make stripping of non-essential headers configurable
TODO: Make stripping of [PATCH] and other prefixes configurable
TODO: --mbox option for other mode of operation
TODO: -a option to export all branches
TODO: For quilt exporting, export the linearized history created in a
temporary branch---this would allow producing conflict-less
series
tg import
~~~~~~~~~
Import commits within the given revision range into TopGit,
creating one topic branch per commit, the dependencies forming
a linear sequence starting on your current branch (or a branch
specified by the '-d' parameter).
The branch names are auto-guessed from the commit messages
and prefixed by t/ by default; use '-p PREFIX' to specify
an alternative prefix (even an empty one).
Alternatively, you can use the '-s NAME' parameter to specify
the name of target branch; the command will then take one more
argument describing a single commit to import.
tg update
~~~~~~~~~
Update the current or specified topic branch wrt. changes in the
branches it depends on and remote branches.
This is performed in two phases - first,
changes within the dependencies are merged to the base,
then the base is merged into the topic branch.
The output will guide you in case of conflicts.
After the update the current branch is the specified one.
In case your dependencies are not up-to-date, tg update
will first recurse into them and update these.
If a remote branch update brings dependencies on branches
not yet instantiated locally, you can either bring in all
the new branches from the remote using 'tg remote --populate'
or only pick out the missing ones using 'tg create -r'
('tg summary' will point out branches with incomplete
dependencies by showing an '!' near to them).
TODO: tg update -a for updating all topic branches
tg push
~~~~~~~
pushes a TopGit-controlled topic branch to a remote
repository. By default the remote gets all dependencies
(both tgish and non-tgish) and bases pushed to.
tg base
~~~~~~~
Prints the base commit of the current topic branch. Silently
exits with exit code 1 if you are not working on a TopGit
branch.
tg log
~~~~~~
Prints the git log of the named topgit branch.
Note: if you have shared the TopGit branch, this command
only lists the commits that were made in the current
repository, so you will not see work done by your
collaborators.
TODO: tg rename
IMPLEMENTATION
--------------
TopGit stores all the topic branches in the regular refs/heads/
namespace, (we recommend to mark them with the 't/' prefix).
Except that, TopGit also maintains a set of auxiliary refs in
refs/top-*. Currently, only refs/top-bases/ is used, containing
the current _base_ of the given topic branch - this is basically
a merge of all the branches the topic branch depends on; it is
updated during `tg update` and then merged to the topic branch,
and it is the base of a patch generated from the topic branch by
`tg patch`.
All the metadata is tracked within the source tree and history
of the topic branch itself, in .top* files; these files are kept
isolated within the topic branches during TopGit-controlled merges
and are of course omitted during `tg patch`. The state of these
files in base commits is undefined; look at them only in the topic
branches themselves. Currently, two files are defined:
.topmsg: Contains the description of the topic branch
in a mail-like format, plus the author information,
whatever Cc headers you choose or the post-three-dashes message.
When mailing out your patch, basically only few extra headers
mail headers are inserted and the patch itself is appended.
Thus, as your patches evolve, you can record nuances like whether
the particular patch should have To-list/Cc-maintainer or vice
versa and similar nuances, if your project is into that.
From is prefilled from your current GIT_AUTHOR_IDENT, other headers
can be prefilled from various optional topgit.* config options.
.topdeps: Contains the one-per-line list of branches
your patch depends on, pre-seeded with `tg create`. (Continuously
updated) merge of these branches will be the "base" of your topic
branch. DO NOT EDIT THIS FILE MANUALLY!!! If you do so, you need
to know exactly what are you doing, since this file must stay in
sync with the Git history information, otherwise very bad things
will happen.
TopGit also automagically installs a bunch of custom commit-related
hooks that will verify if you are committing the .top* files in sane
state. It will add the hooks to separate files within the hooks/
subdirectory and merely insert calls of them to the appropriate hooks
and make them executable (but make sure the original hooks code
is not called if the hook was not executable beforehand).
Another automagically installed piece is .git/info/attributes specifier
for an 'ours' merge strategy for the files .topmsg and .topdeps, and
the (intuitive) 'ours' merge strategy definition in .git/config.
REMOTE HANDLING
---------------
There are three issues with accessing topic branches in remote repositories:
(i) Fetching/pushing accurate picture of the remote topic branch setup
(ii) Referring to remote topic branches from your local repository
(iii) Developing some of the remote topic branches locally
(ii) and (iii) are fairly interconnected problems, while (i) is largely
independent. The issue is to accurately reflect the current state of the
quickly changing topic branches set - this can be easily done
with the current facilities like 'git remote prune' and 'git push --mirror' -
and to properly upload also the bases of the topic branches.
For this, we need to modify the fetch/push refspecs to also include
the refs/top-bases/ ref namespace; we shall provide a special 'tg remote'
command to set up an existing remote for TopGit usage.
About (ii) and (iii), there are two somewhat contradicting design
considerations:
(a) Hacking on multiple independent TopGit remotes in a single
repository
(b) Having a self-contained topic system in local refs space
To us, (a) does not appear to be very convincing, while (b) is quite desirable
for 'git-log topic' etc. working, 'git push' automatically creating
self-contained topic system in the remote repository, and increased conceptual
simplicity.
Thus, we choose to instantiate all the topic branches of given remote locally;
this is performed by 'tg remote --populate'.
'tg update' will also check if a branch can be updated from its corresponding
remote branch. The logic is somewhat involved if we should DTRT.
First, we update the base, handling the remote branch as if it was the first
dependency; thus, conflict resolutions made in the remote branch will be
carried over to our local base automagically. Then, the base is merged into
remote branch and the result is merged to local branch - again, to carry over
remote conflict resolutions. In the future, this order might be adjustable
per-update in case local changes are diverging more than the remote ones.
All commands by default refer to the remote that 'tg remote --populate'
was called on the last time ('topgit.remote' configuration variable). You can
manually run any command with a different base remote by passing '-r REMOTE'
_before_ the subcommand name.
POINTERS
--------
The following references are useful to understand the development of topgit and
its subcommands.
tg depend:
http://lists-archives.org/git/688698-add-list-and-rm-sub-commands-to-tg-depend.html
