Reimplement Language Modelling weights

[xapian.git] / xapian-core / HACKING

Instructions for hacking on Xapian
==================================

.. contents:: Table of contents

.. tip::

   Before reading the contents of this file, you should look at
   `the Xapian developer guide`_. If you want, you can `grab
   it from github`_ and build it locally.

   The bulk of the previous contents of this file now live in the developer
   guide, and it is likely that the rest will follow in due course.

   .. _the Xapian developer guide: https://xapian-developer-guide.readthedocs.io/
   .. _grab it from github: https://github.com/xapian/xapian-developer-guide


Snapshots
=========

If you want to try unreleased Xapian code, you can fetch it from our git
repository.  For convenience, we also provide bootstrapped tarballs (much like
the sourcecode download for any release version) which get built every 20
minutes if there have been any changes checked in.  These tarballs need to
pass "make distcheck" to be automatically uploaded, so using them will help
to assure that you don't pick a "bad" version.  The snapshots are available
from the "Bleeding Edge" page of the Xapian website.

Building from git
=================

When building from a git checkout, we *strongly* recommend that you use
the ``bootstrap`` script in the top level directory to set up the tree ready
for building.  This script will check which directories you have checked out,
so you can bootstrap a partial tree.  You can also ``touch .nobootstrap`` in
a subdirectory to tell bootstrap to ignore it.

You will need the following tools installed to build from git:

* GNU m4 >= 4.6 (for autoconf)
* perl >= 5.6 (for automake; also for various maintainer scripts)
* python >= 3.3 (for generating the Python bindings)
* GNU make (or another make which support VPATH for explicit rules)
* GNU bison (for building SWIG, used for generating the bindings)
* Tcl (to generate unicode/unicode-data.cc)

For a recent version of Debian or Ubuntu, this command should ensure you have
all the necessary tools and libraries::

    apt-get install build-essential m4 perl python3 zlib1g-dev uuid-dev wget bison tcl

If you want to build Omega, you'll also need::

    apt-get install libpcre2-dev libmagic-dev

On Fedora, the uuid library can be installed by doing::

    yum install libuuid-devel

On macOS, if you're using macports you'll want the following:

  * file (magic.h in configure)

If you're using homebrew you'll want the following::

    brew install libmagic pcre2

If you're doing much development work, you'll probably also want the following
tools installed:

* valgrind for better testsuite error finding
* ccache for faster rebuilds
* eatmydata for faster testsuite runs

The repository does not contain any automatically generated files
(such as configure, Makefile.in, Snowball-generated stemmers, Lemon-generated
parsers, SWIG-generated code, etc) because experience shows it's best to keep
these out of version control.  To avoid requiring you to install the correct
versions of the tools required, we either include the source to these tools in
the repo directly (in the case of Snowball and Lemon), or the bootstrap script
will download them as tarballs (autoconf, automake, libtool) or
from git (SWIG), build them, and install them within the source tree.

To download source tarballs, bootstrap will use wget, curl or lwp-request if
installed.  If not, it will give an error telling you the URL to download from
by hand and where to copy the file to.

Bootstrap will then run autoreconf on each of the checked-out subdirectories,
and generate a top-level configure script.  This configure script allows you to
configure xapian-core and any other modules you've checked out with single
simple command, such that the other modules link against the uninstalled
xapian-core (which is very handy for development work and a bit fiddly to set
up by hand).  It automatically passes --enable-maintainer-mode to the
subprojects so that the autotools will be rerun if configure.ac, Makefile.am,
etc are modified.

The bootstrap script doesn't care what the current directory is.  The top-level
configure script generated by it supports building in a separate directory to
the sources: simply create the directory you want to build in, and then run the
configure script from inside that directory.  For example, to build in a
directory called "build" (starting in the top level source directory)::

  ./bootstrap
  mkdir build
  cd build
  ../configure

When running bootstrap, if you need to add any extra macro directories to the
path searched by aclocal (which is part of automake), you can do this by
specifying these in the ACLOCAL_FLAGS environment variable, e.g.::

  ACLOCAL_FLAGS=-I/extra/macro/directory ./bootstrap

If you wish to prevent bootstrap from downloading and building the autotools
pass the --without-autotools option.  You can force it to delete the downloaded
and installed versions by passing --clean.

If you are tracking development in git, there will sometimes be changes
to the build system sources which require regeneration of the generated
makefiles and associated machinery.  We aim to make the build system
automatically regenerate the necessary files, but in the event that a build
fails after an update, it may be worth re-running the bootstrap script to
regenerate the build system from scratch, before looking for the cause of the
error elsewhere.

Tools required to build documentation
-------------------------------------

If you want to be able to build distribution tarballs (with "make dist") then
you'll also need some further tools.  If you don't want to have to install all
these tools, then pass --disable-documentation to configure to disable these
rules (the default state of this follows the setting of
--enable-maintainer-mode, so in a non-maintainer-mode tree, you can pass
--enable-documentation to enable these rules).  Without the documentation,
"make dist" will fail (to prevent accidentally distributing tarballs without
documentation), but you can configure and build.

The documentation tools are:

* doxygen (v1.8.8 is used for 1.3.x snapshots and releases; 1.7.6.1 fails to
  process trunk after PL2Weight was added).
* dot (part of Graphviz.  Doxygen's DOT_MULTI_TARGETS option apparently needs
  ">1.8.10")
* help2man
* rst2html or rst2html.py (in python3-docutils on Debian/Ubuntu)
* pngcrush (optional - used to reduce the size of PNG files in the HTML
  apidocs)
* sphinx-doc (in python3-sphinx on Debian/Ubuntu, or as sphinx via pip install)

For a recent version of Debian or Ubuntu, this command should install all the
required documentation tools::

    apt-get install doxygen graphviz help2man python-docutils pngcrush python-sphinx python3-sphinx

Documentation builds on macOS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On macOS, if you're using homebrew, you'll want the following::

    brew install doxygen help2man graphviz pngcrush

(Ensure you're up to date with brew, as earlier packaging of graphviz
didn't properly install dot.)

You also need sphinx and docutils, which are python packages; you can
install them via pip::

    pip3 install sphinx docutils

You may find it easier to use homebrew to install python first, so
these packages are separate from the system python::

    brew install python

On macOS, if you're using macports you'll want the following:

  * texlive (pdflatex during build)
  * texlive-basic (for makeindex in configure)
  * texlive-latex-extra (latex style)

The homebrew texlive package only supports 32 bit systems, so even if you're
using homebrew, you'll probably want to install MacTeX from
https://www.tug.org/mactex/ instead.

As of 1.3.2, we no longer build PDF versions of the API docs by default, but
you can build them yourself with::

    make -C docs apidoc.pdf

Autotools versions
------------------

* autoconf 2.69 is used to generate snapshots and releases.

  autoconf 2.64 is a hard minimum requirement.

  autoconf 2.60 is required for docdir support and AC_TYPE_SSIZE_T.

  autoconf 2.62 generates faster configure scripts and warns about unrecognised
  options passed to configure.

  autoconf 2.63 fixes a regression in AC_C_BIGENDIAN introduced in 2.62
  (Omega uses this macro).

  autoconf 2.64 generates smaller configure scripts by using shell functions.

* automake 1.16.1 is used to generate snapshots and releases.

  automake 1.13 is a hard minimum requirement, needed for
  `AC_CONFIG_MACRO_DIRS`.

* libtool 2.4.6 is used to generate snapshots and releases.

  libtool 2.2.8 is the current hard minimum requirement.

  libtool 2.2 is required for us to be able to override link_all_deplibs_CXX
  and sys_lib_dlsearch_path_spec in configure.  It also fixes some
  long-standing issues and is significantly faster.

Please tell us if you find that newer versions of any of these tools work or
fail to work.

There is a good GNU autotools tutorial at
<https://www.lrde.epita.fr/~adl/autotools.html>.

Building from git on Windows with MSVC
--------------------------------------

Building using MSVC is now supported by the autotools build system.  You need
to install a set of Unix-like tools first - we recommended MSYS2:
https://www.msys2.org/

For details of how to specify MSVC to ``configure`` see the "INSTALL" document.

When building from git, by default you'll need some extra tools to generate
Unicode tables (Tcl) and build documentation (doxygen, help2man, sphinx-doc).
We don't currently have detailed advice on how to do this (if you can provide
some then please send a patch).

You can avoid needing Tcl by copying ``xapian-core/unicode/unicode-data.cc``
from another platform or a release which uses the same Unicode version.  You
can avoid needing most of the documentation tools by running configure with
the ``--disable-documentation`` option.


Submitting Patches:
===================

If you have a patch to fix a problem in Xapian, or to add a new feature,
please send it to us for inclusion.  Any major changes should be discussed
on the xapian-devel mailing list first:
<https://xapian.org/lists>

Also, please read the following section on licensing of patches before
submitting a patch.

If you're working from a tarball, you can unpack a second clean copy of the
files and compare the two versions with "diff -pruN" (-p reports the function
name for each chunk, -r acts recursively, -u does a unified diff, and -N shows
new files in the diff).  Alternatively "ptardiff" (which comes with perl, at
least on Debian and Ubuntu) can diff against the original tarball, unpacking
it on the fly.

Please try to follow the style of the existing code.

We will do our best to give credit where credit is due - if we have used
patches from you, or received helpful reports or advice, we will add your name
to the AUTHORS file (unless you specifically request us not to).  If you see we
have forgotten to do this, please draw it to our attention so that we can
address the omission.


Tips for Submitting a Good Patch
================================

Make sure the tests are right
-----------------------------

 * If you're adding a feature, also add feature tests for it.  These both
   ensure that the feature isn't broken to start with and detect if later
   changes stop it working as intended.

 * If you've fixed a bug, make sure there's a regression test which
   fails on the existing code and succeeds after your changes.

 * If you're adding a new testcase to demonstrate an existing bug, and not
   checking a fix in at the same time, mark the testcase as a known failure (by
   calling ``XFAIL("explanatory message")`` at the start of your testcase (if
   necessary this can be conditional on backend or other factors - the backend
   case has explicit support via ``XFAIL_FOR_BACKEND("backend", "message")``).

   This will mean that this testcase failing will be reported as "XFAIL" which
   won't cause the test run to fail.  If such a testcase in fact passes, that
   gets reported as "XPASS" and *will* cause the test run to fail.  A testcase
   should not be flagged as "XFAIL" for a long time, but it can be useful to be
   able to add such testcases during development.  It also allows a patch
   series which fixes a bug to first demonstrate the bug via a new testcase
   marked as "XFAIL", then fix the bug and remove the "XFAIL" - this makes it
   clear that the regression test actually failed before the fix.

   Note that failures which are due to valgrind errors or leaked fds are not
   affected by this macro - such errors are inherently not suitable for "XFAIL"
   as they go away when the testsuite is run without valgrind or on a platform
   where our fd leak detector code isn't supported.

 * Make sure all existing tests continue to pass.

If you don't know how to write tests using the Xapian test rig, then
ask.  It's reasonably simple once you've done it once.  There is a brief
introduction to the Xapian test system in ``docs/tests.html``.


Update trac
-----------

If there's a related trac ticket, update it (if the issue is completely
addressed by the changes you've made, then close it).

Update the release notes for the most recent release with a copy of the
patch.  If the commit from git applies cleanly, you can just link to
it.  If it fails to apply, please attach an adjusted patch which does.
If there are conflicts in test cases which aren't easy to resolve, it is
acceptable to just drop those changes from the patch if we can still be
confident that the issue is actually fixed by the patch.


Commit message
--------------

If there's a trac ticket or other reference for the bug, mention it in the
commit message - it's a great help to future developers trying to work out
why a change was made.

.. vim: syntax=rst