Merge remote-tracking branch 'flapflap/de-network_configuration'

[tails-test.git] / wiki / src / contribute / release_process / test.mdwn

[[!meta title="Manual test suite"]]

[[!toc levels=1]]

Some [[test results]] that might be useful to keep are saved.

<div class="caution">
Read this document from the branch used to prepare the release.
</div>

# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

        git diff --find-renames <old tag>..<ISO commit>

e.g. `git diff 0.17..06fa1ab80d55c9f29274b7459bd198edb1a8d53d`

## Result

Compare the list of bundled packages and versions with the one shipped last
time. `.packages` are usually attached to the email announcing the ISO is ready.

        /usr/bin/diff -u \
            wiki/src/torrents/files/tails-i386-0.16.packages \
            tails-i386-0.17.packages \
            | wdiff --diff-input  --terminal

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

## Image size

Check the image size has not changed much since the last release.

In a directory with many Tails ISO images:

    find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5

# Automated test suite

Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. It's design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

## Running the automated test suite

See [[test/setup]] and [[test/usage]].

## Automated test suite migration progress

The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.

# Tor Browser

## Security and fingerprinting

* Run the [tests the TBB folks
  use](https://trac.torproject.org/projects/tor/wiki/doc/build/BuildSignoff#TestPagestoUse).
* Compare the fingerprint of Tails and the latest TBB using at least
  <https://panopticlick.eff.org/>
  - The exposed User-Agent should match the latest TBB's one.
  - Update the [[fingerprint section|support/known_issues#fingerprint]] of the
    known issues page if needed.
* WebRTC should be disabled:
  - In `about:config` check that `media.peerconnection.enabled` is set to
    `false`.
  - <http://mozilla.github.io/webrtc-landing/>, especially the `getUserMedia`
    test. It's expected that the audio test works if you agree to share a
    microphone with the remote website; anything else should fail.
  - <http://net.ipcalf.com/> should display
    `ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1`
* One should be able to switch identities from the web browser.
* Running `getTorBrowserUserAgent` should produce the User-Agent set by the
  installed version of Torbutton, and used in the Tor Browser.

## Functionality

* Browsing (by IP) a HTTP or HTTPS server on the LAN should be possible.
* Browsing (by IP) a FTP server on the LAN should be possible.

# Pidgin

(automate: [[!tails_ticket 7820]])

* Check that you can initiate an OTR conversation.
* Check that XMPP is working with a new test profile.
  For example using Riseup:
  - Username: username
  - Domain: riseup.net
  - Connect server: 4cjw6cwpeaeppfqz.onion
  - Then try to create and connect to a new room:
    - Room: testing
    - Server: conference.riseup.net
    - Handle: username
* Check that Pidgin doesn't leak too much information when replying to
  CTCP requests:
  * Start Tails, launch Pidgin, and join #tails.
  * Also join #tails from a client that supports CTCP commands
    properly, e.g. Konversation.
  * Try to send `/ctcp <Tails_account_nick> COMMAND` from the other client to Pidgin:
    - You should get no answer apart for the PING and VERSION commands
      ([[!tails_ticket 5823]]).
    - List of `/ctcp` commands, see [this page](http://www.wikkedwire.com/irccommands):
      - PING
      - VERSION
      - FINGER
      - USERINFO
      - CLIENTINFO
      - TIME

# Tor

(automate: [[!tails_ticket 7821]])

* The version of Tor should be the latest stable one, which is the highest version number
  before alpha releases on <http://deb.torproject.org/torproject.org/pool/main/t/tor/>.
* Check that the firewall-level Tor enforcement is effective:
  - check output of `iptables -L -n -v`
  - check output of `iptables -t nat -L -n -v`
  - try connecting to the Internet after unsetting `$http_proxy` and
    `$HTTP_PROXY` using a piece of software that does not obey the
    GNOME proxy settings, *and* is not explicitly torified in Tails:
    
                unset http_proxy ; unset HTTP_PROXY
                wget --no-proxy http://monip.org/
    
    ... should only give you "Connection refused" error message.
* Check that IPv6 traffic is blocked:
  - check output of `ip6tables -L -n`
  - at a place with working IPv6: try connecting to a known-working
    IPv6-enabled server on its IPv6 address over TCP and icmp6.
* After DHCP has been set up, `/etc/resolv.conf` must read `nameserver 127.0.0.1`.
* Before DHCP has been set up, `/etc/resolv.conf` must read `nameserver 127.0.0.1`.
* [[doc/first_steps/startup_options/bridge_mode]] should work:
  1. Set up an administrator password.
  1. Enable network configuration in Tails Greeter.
  1. Configure a few bridges in Tor Launcher:
     
                bridge 198.252.153.59:9001
                obfs2 198.252.153.59:16492
                obfs3 198.252.153.59:16493
     
  1. Use the Internet.
  1. Check that the only outgoing direct network connections go to the
     configured bridges:
     
                sudo watch "netstat -taupen | grep ESTABLISHED"

* Verify that all destinations reached from an intensive Tails session
  are tor routers or
  authorities:
  1. Boot Tails without the network in.
  1. Set up an administration password.
  1. Start dumping your whole session's network activity with `sudo
     tcpdump -n -i any -w dump` (or better, do the dump on another machine,
     or on the host OS if Tails is running in a VM).
  1. Plug the network.
  1. Wait for Tor to be functional.
  1. Save `/var/lib/tor/cached-microdesc-consensus` out of the VM (it's needed
     to analyze the network dump later on).
  1. Do *a lot* of network stuff (why not run do this while doing all
     the other tests **but** I2P and the unsafe browser, which would
     show many false positives?)
  1. Then check all destinations, e.g. by using tshark and the script below:

         # set DUMP to the output of tcpdump above
         DUMP=dump
         # set CONSENSUS to Tor's consensus from the Tails session
         CONSENSUS=cached-microdesc-consensus
         NODES=$(mktemp)
         awk '/^r / { print $6 }' ${CONSENSUS} > ${NODES}
         # Note that these default directory authorities may change! To be
         # sure, check in Tor's source, src/or/config.c:~900
         DIR_AUTHS="
         128.31.0.39
         86.59.21.38
         194.109.206.212
         82.94.251.203
         76.73.17.194
         212.112.245.170
         193.23.244.244
         208.83.223.34
         171.25.193.9
         154.35.32.5
         "
         tshark -r ${DUMP} -T fields -e ip.dst | sort | uniq | \
         while read x; do
             ip_expr=$(echo ${x} | sed -e "s@\.@\\\.@g")
             if echo ${DIR_AUTHS} | grep -qe "${ip_expr}"; then
                 continue
             fi
             if ! grep -qe "^${ip_expr}$" ${NODES}; then
                 echo "${x} is bad"
             fi
         done
         rm ${NODES}

     Note that this script will produce some false positives, like your
     gateway, broadcasts, etc.

## Stream isolation

See our [[stream isolation design
page|contribute/design/stream_isolation]] for details such as port
numbers, that are not duplicated here to avoid desynchronization.

Assumptions for the following tests: first, Tor stream isolation
features properly do their work; second, our `torrc` sets the right
`SocksPort` options to implement what we want.

**Note**: the following commands would advantageously be replaced with
the appropriate tcpdump or tshark filters.

* Make sure Claws Mail use its dedicated `SocksPort` when connecting
  to IMAP / POP3 / SMTP servers:

      sudo watch -n 0.1 'netstat -taupen | grep claws'

* Make sure these use the `SocksPort` dedicated for Tails-specific applications:
  - htpdate — as root, run:

        service htpdate stop \
           && rm -f /var/run/htpdate/{done,success} \
           && service htpdate start

    ... with the following command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep curl'

  - security check — run `tails-security-check` with the following
    command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep perl'

  - incremental upgrades — run `tails-upgrade-frontend-wrapper` with
    the following command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep perl'

* Make sure the Tor Browser uses its dedicated `SocksPort`: quit the Tor Browser
  then start it with the following command running in another
  terminal:

      sudo watch -n 0.1 'netstat -taupen | grep firefox'

* Make sure other applications use the default system-wide
  `SocksPort`:
  - Polipo — run:

        wget https://tails.boum.org/

    ... with the following command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep polipo'

  - Gobby 0.5 — start Gobby 0.5 from the *Applications* menu and
    connect to a server (for example `gobby.debian.org`), with the following command running in
    another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep gobby'

  - SSH — run (no need to authenticate the server or to login):

        ssh lizard.tails.boum.org

    ... with the following command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep -E "connect-proxy|ssh"'

  - whois — run:

        whois example.com

    ... with the following command running in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep whois'

* Make sure a random application run using `torify` and `torsocks`
  uses the default system-wide `SocksPort`. Run:

      torify /usr/bin/gobby-0.5

    ... and connect to a server (for example `gobby.debian.org`), with the following command running
    in another terminal:

        sudo watch -n 0.1 'netstat -taupen | grep gobby'

    Then do the same test for:

      torsocks /usr/bin/gobby-0.5

# Use of untrusted partitions

(automate: [[!tails_ticket 7822]])

* Is any local hard-disk swap partition used as swap?
  boot on a (possibly virtual) machine that has a cleartext swap
  partition not managed by LVM. To verify that a local GTP partition is swap,
  check its type code with `sgdisk -p`, Linux swap is code 8200.

  This swap partition must not be used by Tails. Run `cat /proc/swaps`.

* Is a persistence volume on a local hard-disk partition used?
  (Hint: setup a libvirt USB disk with GPT and a partition labeled
  `TailsData`, set the `removable` flag on it, check that
  tails-greeter proposes to enable persistence. Then remove the
  `removable` flag, and check that tails-greeter does not propose to
  enable persistence anymore.)

# Claws

* Check mail over IMAP using:
  - a "clearnet" IMAP server.
  - a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion
    with SSL).
* Send an email using:
  - a "clearnet" SMTP server.
  - a hidden service SMTP server (see above).
* Check that the profile works and is torified:
  1. Send an email using Claws and a non-anonymizing SMTP relay (a
     SMTP relay that writes the IP address of the client it is
     relaying email for in the Received header).
  1. Then check that email's headers once received, especially the
     `Received:` one.
* Also check that the EHLO/HELO SMTP message is not leaking anything
  at the application level:
  1. Start Claws using the panel icon.
  2. Disable SSL/TLS for SMTP in Claws (so take precautions for not
     leaking your password in plaintext by either changing it
     temporarily or using a disposable account).
  3. Run `sudo tcpdump -n -i lo -w dump` to capture the packets before
     Tor encrypts it, then close tcpdump
  4. Check the dump for the HELO/EHLO message and
     verify that it only contains `localhost`: `tcpdump -A -r dump`
  5. Check the `Received:` and `Message-Id` fields in the received
     message: it must not leak the hostname, nor the local IP.

# WhisperBack

* I should be able to send a bug report with WhisperBack.
* When we receive this bug report on the tails-bugs mailing-list,
  Schleuder tells us that it was sent encrypted.

# Time

(automate: [[!tails_ticket 5836]])

1. Boot Tails without a network cable connected.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 down`.)
2. Set an administration password.
3. set the time to an obviously wrong one:

           date --set="Mon, 01 Mar 2000 15:45:34 - 0800"

4. Connect the network cable.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 up`)

=> the date should be corrected and Tor/Vidalia should start
correctly.

# Erase memory on shutdown

- `memlockd` must be running
- `udev-watchdog` must monitor the right device when booted off USB (automate: [[!tails_ticket 5560]])
- `udev-watchdog` must monitor the right device when booted off DVD (automate: [[!tails_ticket 5560]])
- After booting from DVD, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).
  (automate: [[!tails_ticket 5472]])
- After booting from USB, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).

# Root access control

* Check you can login as root with `su` neither with the `amnesia` password nor
  with the `live` one.
* Check that the `$TAILS_USER_PASSWORD` variable, if still existing in the system
  environment after the boot has finished, does not contain the clear text
  password.

# Virtualization support

* Test that Tails starts and the browser launches in VirtualBox.

# I2P

Make sure that I2P is up-to-date, at least if the
[changelogs](https://geti2p.net/en/blog/) mention that
security critical bugs were fixed.

Start I2P by appending `i2p` to the kernel command line.

* Check that I2P starts when a network interface is up:
  - Within 30 seconds you should get the "I2P router console is ready"
    pop-up
  - Start the I2P Browser via "Applications -> Internet -> I2P Browser":
    * You get the "Starting I2P Browser..." pop-up.
    * The router console (<http://127.0.0.1:7657>) opens successfully
      upon success.
    * On exiting I2P Browser, check that its chroot gets properly torn
      down on exit (there should be nothing mounted inside
      `/var/lib/i2p-browser`).
  - After a few minutes you should get the "I2P is ready" pop-up
  - Go to <http://127.0.0.1:7657/i2ptunnelmgr> in the I2P Browser:
    * You should get "Network: Hidden" in the "General" section.
    * The numbers in the "Peers" section of the sidebar should be
      non-zero.
    * Check that you can reach some eepsites within Iceweasel, like
      <http://i2p-projekt.i2p> and <http://forum.i2p>.
  - Check that you can connect to the I2P IRC server through Pidgin
    and the preconfigured IRC account on 127.0.0.1.
* Check I2P failure modes:
  - Router console failure:
    * Boot without network so I2P doesn't start automatically.
    * Block the router console port: `nc -l -p 7657 -t 127.0.0.1`
    * Plug the network
    * You should get the "I2P failed to start" pop-up, and I2P should
      not be running (check with `service i2p status`)
  - Bootstrap failure:
    * Detach the network immediately after getting the "I2P router
      console is ready" pop-up
    * Wait for up to six minutes
    * You should get the "I2P is not ready" pop-up
    * The I2P router console should still be accessible on
      <http://127.0.0.1:7657>

# Git

* clone a repository over `git://`

  git clone git://git.tails.boum.org/htp

* clone a repository over `https://`

  git clone https://git-tails.immerda.ch/htp

* clone a repository over SSH

# SSH

* Connecting over SSH to a server on the Internet should work (and
  appear in Vidalia's connections list).
* Connecting (by IP) over SSH to a server on the LAN should work.
* Connecting to a sftp server on the Internet using GNOME's "Connect
  to a server" should work.

# APT

     grep -r deb.tails.boum.org /etc/apt/sources.list*

* Make sure the Tails repository suite in matching the release tag (for example
  the release version number) is in APT sources.
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.

<a id="incremental-upgrades"></a>

# Incremental upgrades

* List the versions from which an upgrade paths to this one is described.
  In the `stable` or `testing` branch:

      git grep -l "  version: '\?0.23'\?" wiki/src/upgrade/

* For each description file, open it and verify if it allows incremental upgrade
  or only full upgrade.

* For each previous version from which an upgrade paths is described, install it
  and try to upgrade:
  * For every incremental upgrade path: make sure the resulting updated
    system "works fine" (boots and pretends to be the correct version).
  * For upgrade paths that only propose a full upgrade: make sure the
    user is told to do a manual upgrade.

  If:
  
  * the update-description files have been published on the
    *alpha* channel already (see <https://tails.boum.org/upgrade/v1/Tails/>)
  * and the IUK has been published already (see
    <https://archive.torproject.org/amnesia.boum.org/tails/alpha/>
    and <https://archive.torproject.org/amnesia.boum.org/tails/stable/>):

  then:
  
        echo 'TAILS_CHANNEL="alpha"' | sudo tee --append /etc/os-release && \
        tails-upgrade-frontend-wrapper

  Else, use a local test setup:

  * A web server on the LAN.
  * A copy of `wiki/src/upgrade` from the `stable` or `testing` branch,
    for example in `/var/www/tails/upgrade/v1/Tails/0.14~rc2/i386/stable/updates.yml`
  * A copy of the `iuk` directory of our HTTP mirrors,
    for example in `/var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk`.

    To synchronize your local copy:

        torsocks rsync -rt --progress --delete rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/

  * Patch `/etc/hosts` in Tails to point to your web server:

        echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts

  * Patch sudo configuration to allow passing arbitrary arguments to
    `tails-upgrade-frontend`:

        sudo sed -i \
            -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
            /etc/sudoers.d/zzz_upgrade

  * Call the upgrader must be called, from inside the system to upgrade,
    with every needed option to use the local web server rather than the
    online one, for example:

        DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
        tails-upgrade-frontend-wrapper --override-baseurl \
        http://192.168.1.4/tails

# Windows Camouflage

Enable Windows camouflage via the Tails Greeter checkbox and:

* Tails OpenPGP Applet's context menu should look readable
* The Tor Browser should use a Internet Explorer theme
* The Unsafe Browser has no scary red theme

# Unsafe Web Browser

(automate: [[!tails_ticket 7823]])

* On start, if no DNS server was configured in NetworkManager
  (e.g. if there's no network connection), there must be an error.
* Once started, check that:
  - the Tor Browser instance runs as the `clearnet` user.
  - it has no proxy configured.
  - no extensions are installed.
  - there are no bookmarks except the default Firefox ones.
* On exit, check that:
  - make sure that its chroot gets properly teared down on exit (there
    should be nothing mounted inside `/var/lib/unsafe-browser`).

# Real (non-VM) hardware

`[can't-automate]`

* Boot on bare-metal on USB.
* Boot on bare-metal on DVD.
* Measure boot time (from syslinux menu the GNOME dektop ready - quickly press
  enter in the greeter), then on some reference bare metal hardware, and
  compare with previous version. The new one should not be significantly
  slower to start.

# Documentation

* The "Tails documentation" desktop launcher should open the
  [[getting started]] page (automate: [[!tails_ticket 8788]]):
  - in English
  - in one language to which the website is translated
  - in one language to which the website is not translated (=> English)
* Browse around in the documentation shipped in the image. Internal
  links should be fine.

# Internationalization

Boot and check basic functionality is working for every supported
language. You *really* have to reboot between each language.

* The chosen keyboard layout must be applied.
* The virtual keyboard must work and be auto-configured to use the same keyboard
  layout as the X session.
* The Startpage search engine must be localized for the languages we ship a
  search plugin for:

      find /usr/local/lib/tor-browser/distribution/searchplugins/locale -iname startpage-*.xml

* The Wikipedia search engine must be localized for all languages.

## Spellchecking

* Check that every supported language is listed in the list of languages for
  spell checking.
  - Visit <https://translate.google.com/>.
  - Right-click and choose "Check spelling".
  - Right-click and check the list of available languages.
* For a few languages, check the spell checking:
  - Type something in the textarea.
  - Right-click and select a language.
  - Verify that the spelling suggestion are from that language.
* Once [[!tails_ticket 5962]] is fixed, the browser spelling dictionary must be
  localized (for languages that are supported by our branding extension).

# Misc

* Check that Tails Greeter's "more options" screen displays properly
  on a display with 600 px height, preferably in a language that's
  more verbose than English (e.g. French).
* Check that all seems well during init (mostly that all services
  start without errors), and that `/var/log/syslog` seems OK.
* MAT should be able to clean a PDF file, such as:
  <http://examples.itextpdf.com/results/part3/chapter12/pdf_metadata.pdf>
* The Tails signing key shipped should be up-to-date (that is, neither it nor
  one its subkeys must have expired, or be about to expire any time soon).
  - `gpg --list-keys --with-colons 1202821CBE2CD9C1`
* The "Report an error" desktop launcher should open the [[support]]
  page, both in English and in one language to which the website is
  translated (automate: [[!tails_ticket 6904]]).
* One should be able to refresh the GnuPG keyring in Seahorse (with
  the workaround documented in comment 4 on [[!tails_ticket 7051]],
  until that ticket is fixed for real).