TODO epan/dissectors/asn1/kerberos/packet-kerberos-template.c new GSS flags

[wireshark-sm.git] / doc / wsug_src / wsug_mate.adoc

[#ChMate]

== MATE

[#ChMateIntroduction]

=== Introduction

MATE: Meta Analysis and Tracing Engine

What is MATE? Well, to keep it very short, with MATE you can create user
configurable extension(s) of the display filter engine.

MATE's goal is to enable users to filter frames based on information extracted
from related frames or information on how frames relate to each other. MATE
was written to help troubleshooting gateways and other systems where a "use"
involves more protocols. However, MATE can be used as well to analyze other
issues regarding an interaction between packets like response times,
incompleteness of transactions, presence/absence of certain attributes in a
group of Protocol Data Units (PDUs) and more.

MATE is a Wireshark plugin that allows the user to specify how different
frames are related to each other. To do so, MATE extracts data from the frames'
tree and then, using that information, tries to group the frames based on how
MATE is configured. Once the PDUs are related, MATE will create a "protocol"
tree with fields the user can filter with. The fields will be almost the same
for all the related frames, so one can filter a complete session spanning
several frames containing more protocols based on an attribute appearing in
some related frame. Other than that MATE allows to filter frames based on
response times, number of PDUs in a group and a lot more.

So far MATE has been used to:

* Filter all packets of a call using various protocols knowing just the
calling number. (MATE's original goal)
* Filter all packets of all calls using various protocols based on the release
cause of one of its "segments".
* Extrapolate slow transactions from very "dense" captures. (finding requests
that timeout)
* Find incomplete transactions (no responses)
* Follow requests through more gateways/proxies.
* more...

[#ChMateGettingStarted]

=== Getting Started

These are the steps to try out MATE:

* Run Wireshark and check if the plugin is installed (MATE should
appear in Help->About Wireshark:Plugins)
* Get a configuration file e.g., <<File_tcp_mate,tcp.mate>> (see {wireshark-wiki-url}Mate/Examples[Mate/Examples]
for more) and place it somewhere on your harddisk.
* Go to Edit->Preferences...->Protocols->MATE and set the Configuration Filename to the file
you want to use and restart Wireshark.
* Load a corresponding capture file (e.g.,
{wireshark-wiki-url}uploads/27707187aeb30df68e70c8fb9d614981/http.cap[http.cap]) and see if MATE
has added some new display filter fields, something like: `mate tcp_pdu:1->tcp_ses:1`
or, at prompt: `path_to/wireshark -o "mate.config: tcp.mate" -r http.cap`.

If everything went well, your packet details might look something like this:

.Packet Details - MATE TCP Session (tcp.mate)
image::images/ws-mate-tcp-output.png[]

[#ChMateManual]

=== MATE Overview

==== Introduction

MATE creates a filterable tree based on information contained in frames that
share some relationship with information obtained from other frames. The way
these relationships are made is described in a configuration file. The
configuration file tells MATE what makes a PDU and how to relate it to other
PDUs.

MATE analyzes each frame to extract relevant information from the "protocol"
tree of that frame. The extracted information is contained in MATE PDUs;
these contain a list of relevant attributes taken from the tree. From now on, I
will use the term "PDU" to refer to the objects created by MATE containing the
relevant information extracted from the frame; I'll use "frame" to refer to the
"raw" information extracted by the various dissectors that pre-analyzed the frame.

For every PDU, MATE checks if it belongs to an existing "Group of PDUs" (GOP).
If it does, it assigns the PDU to that GOP and moves any new relevant attributes
to the GOP's attribute list. How and when do PDUs belong to GOPs is described
in the configuration file as well.

Every time a GOP is assigned a new PDU, MATE will check if it matches the
conditions to make it belong to a "Group of Groups" (GOG). Naturally the
conditions that make a GOP belong to a GOG are taken from the configuration
file as well.

Once MATE is done analyzing the frame it will be able to create a "protocol"
tree for each frame based on the PDUs, the GOPs they belong to and naturally any
GOGs the former belongs to.

How to tell MATE what to extract, how to group it and then how to relate those
groups is made using AVPs and AVPLs.

Information in MATE is contained in Attribute Value Pairs (AVPs). AVPs are made
of two strings: the name and the value. AVPs are used in the configuration and
there they have an operator as well. There are various ways AVPs can be matched
against each other using those operators.

AVPs are grouped into AVP Lists (AVPLs). PDUs, GOPs and GOGs have an AVPL each.
Their AVPLs will be matched in various ways against others coming from the
configuration file.

MATE will be instructed how to extract AVPs from frames in order to create a PDU
with an AVPL. It will be instructed as well, how to match that AVPL against the
AVPLs of other similar PDUs in order to relate them. In MATE the relationship
between PDUs is a GOP, it has an AVPL as well. MATE will be configured with other
AVPLs to operate against the GOP's AVPL to relate GOPs together into GOGs.

A good understanding on how AVPs and AVPLs work is fundamental to understand how
MATE works.

===== About MATE

MATE was originally written by Luis Ontanon, a Telecommunications systems
troubleshooter, as a way to save time filtering out the packets of a single call
from huge capture files using just the calling number. Later he used the time he
had saved to make it flexible enough to work with protocols other than the ones
he was directly involved with.

[#AVP]
==== Attribute Value Pairs (AVP)

Information used by MATE to relate different frames is contained in Attribute
Value Pairs (AVPs). AVPs are made of two strings - the name and the value. When
AVPs are used in the configuration, an operator is defined as well. There are
various ways AVPs can be matched against each other using those operators.

----
  avp_name="avp's value"
  another_name= "1234 is the value"
----

The name is a string used to refer to a "type" of an AVP. Two AVPs won't match
unless their names are identical.


The name must start with a lowercase letter (a-z) and can contain only alphanumeric characters
(a-zA-Z0-9) and the special characters "_", "-", and ".". The name ends with an operator.

You should not use uppercase characters in names, or names that start with “.” or
“_”. Capitalized names are reserved for configuration parameters (we'll call them
keywords); nothing forbids you from using capitalized strings for other things as
well but it probably would be confusing. I'll avoid using capitalized words for
anything but the keywords in this document, the reference manual, the examples
and the base library. Names that start with a “.” would be very confusing as well
because in the old grammar, AVPL transforms use names starting with a “.” to
indicate they belong to the replacement AVPL.

The value is a string that is either set in the configuration (for configuration
AVPs) or by Wireshark while extracting interesting fields from a frame's tree.
The values extracted from fields use the same representation as they do in filter
strings except that no quotes are used.

*The value will be dealt with as a string even if it is a number.* If there are
any spaces in the value, the value must be between quotes "".
Values that are also keywords such as True and False should also be wrapped
in quotes ("True", "False").

----
   ip_addr=10.10.10.11
   tcp_port=1234
   binary_data=01:23:45:67:89:ab:cd:ef
   parameter12=0x23aa
   parameter_with_spaces="this value has spaces"
----

The way two AVPs with the same name might match is described by the operator.
Remember two AVPs won't match unless their names are identical. In MATE, match
operations are always made between the AVPs extracted from frames (called data
AVPs) and the configuration's AVPs.

Currently defined MATE AVP match operators are:

* <<Equal,Equal>> _=_ will match if the string given completely matches the data
AVP's value string
* <<NotEqual,Not Equal>> _!_ will match only if the given value string is not equal to
the data AVP's value string
* <<OneOf,One Of>> _{}_ will match if one of the possible strings listed is equal to
the data AVP's value string
* <<StartsWith,Starts With>> _^_ will match if the string given matches the first
characters of the data AVP's value string
* <<EndsWith,Ends With>> _$_ will match if the string given matches the last characters
of the data AVP's value string
* <<Contains,Contains>> _~_ will match if the string given matches any substring of the
data AVP's value string
* <<LowerThan,Lower Than>> _<_ will match if the data AVP's value string is semantically
lower than the string given
* <<HigherThan,Higher Than>> _>_ will match if the data AVP's value string is semantically
higher than the string given
* <<Exists,Exists>> _?_ (the ? can be omitted) will match as far as a data AVP of the
given name exists

==== AVP lists (AVPL)

An AVPL is a set of diverse AVPs that can be matched against other AVPLs. Every
PDU, GOP and GOG has an AVPL that contains the information regarding it. The
rules that MATE uses to group PDUs and GOPs are AVPL operations.

*There will never be two identical AVPs in a given AVPL.* However, we can have
more than one AVP with the same name in an AVPL as long as their values are
different.

Some AVPL examples:
----
  ( addr=10.20.30.40, addr=192.168.0.1, tcp_port=21, tcp_port=32534, user_cmd=PORT,
        data_port=12344, data_addr=192.168.0.1 )
  ( addr=10.20.30.40, addr=192.168.0.1, channel_id=22:23, message_type=Setup,
        calling_number=1244556673 )
  ( addr=10.20.30.40, addr=192.168.0.1, ses_id=01:23:45:67:89:ab:cd:ef )
  ( user_id=pippo, calling_number=1244556673, assigned_ip=10.23.22.123 )
----

In MATE there are two types of AVPLs:

* data AVPLs that contain information extracted from frames.
* configuration AVPLs that come from the configuration and are used to tell MATE how
to relate items based on their data AVPLs.

Data AVPLs can be operated against configuration AVPLs in various ways:

* <<Loose,Loose Match>>: Will match if at least one of the AVPs of each AVPL
match. If it matches it will return an AVPL containing all AVPs from the data
AVPL that did match the configuration AVPs.
* <<Every,"Every" Match>>: Will match if none of the AVPs of the configuration AVPL
fails to match a present AVP in the data AVPL, even if not all of the
configuration AVPs have a match. If it matches it will return an AVPL containing
all AVPs from the data AVPL that did match one AVP in the configuration AVPL.
* <<Strict,Strict Match>>: Will match if and only if every one of the configuration
AVPs have at least one match in the data AVPL. If it matches it will return
an AVPL containing the AVPs from the data AVPL that matched.
* There's also a <<Merge,Merge>> operation that is to be performed between AVPLs
where all the AVPs that don't exist in the data AVPL but exist in the configuration
will be added to the data AVPL.
* Other than that, there are <<Transform,Transforms>> - a combination
of a match AVPL and an AVPL to merge.

=== MATE Frame Analysis

MATE's analysis of a frame is performed in three phases:

* In the first phase, MATE attempts to extract a MATE PDU from the frame's
protocol tree. MATE will create a PDU if MATE's config has a _Pdu_ declaration
whose _Proto_ is contained in the frame.

* In the second phase, if a PDU has been extracted from the frame, MATE will try
to group it to other PDUs into a GOP (Group of PDUs) by matching the key
criteria given by a _Gop_ declaration. If there is no GOP yet with the key
criteria for the PDU, MATE will try to create a new GOP for it if it matches the
_Start_ criteria given in the _Gop_ declaration.

* In the third phase, if there's a GOP for the PDU, MATE will try to group this
GOP with other GOPs into a GOG (Group of Groups) using the criteria given by the
_Member_ criteria of a _Gog_ declaration.

.MATE Analysis (PDU->GOP->GOG) flowchart
image::images/ws-mate-analysis.png[]

The extraction and matching logic comes from MATE's configuration; MATE's
configuration file is specified by the _mate.config_ preference. By default it is
an empty string which means: do not configure MATE.

The config file tells MATE what to look for in frames; How to make PDUs out of
it; How will PDUs be related to other similar PDUs into GOPs; And how GOPs
relate into GOGs.

The MATE configuration file is a list of declarations. There are 4 types of
declarations: _Transform_, _Pdu_, _Gop_, and _Gog_. A _Transform_ block must be
before any of the other block declarations that may use it.

==== Create PDUs (Phase 1)

MATE will look in the tree of every frame to see if there is useful data to
extract, and if there is, it will create one or more PDU objects containing the
useful information.

The first part of MATE's analysis is the "PDU extraction".

===== PDU data extraction

MATE will make a PDU for each different proto field of _Proto_ type present in the
frame. MATE will fetch from the field's tree those fields that are defined in
the <<Pdu>> declaration whose initial offset in the frame is within the
boundaries of the current _Proto_ and those of the given _Transport_ and _Payload_
statements.

----
Pdu dns_pdu Proto dns Transport ip {
    Extract addr From ip.addr;
    Extract dns_id From dns.id;
    Extract dns_resp From dns.flags.response;
};
----

.Wireshark window - fields for PDU extraction
image::images/ws-mate-dns_pane.png[]

Once MATE has found a _Proto_ field for which to create a PDU from the frame it
will move backwards in the frame looking for the respective _Transport_ fields.
After that it will create AVPs named as each of those given in the rest of the
AVPL for every instance of the fields declared as its values.

.Frame fields mapped to PDU attributes
image::images/ws-mate-dns_pdu.png[]

Sometimes we need information from more than one _Transport_ protocol. In that
case MATE will check the frame looking backwards to look for the various
_Transport_ protocols in the given stack. MATE will choose only the closest
transport boundary per "protocol" in the frame.

This way we'll have all PDUs for every _Proto_ that appears in a frame match its
relative transports.

----
Pdu isup_pdu Proto isup Transport mtp3/ip {
    Extract addr From ip.addr;

    Extract m3pc From mtp3.dpc;
    Extract m3pc From mtp3.opc;

    Extract cic From isup.cic;
    Extract isup_msg From isup.message_type;
};
----

.Frame containing multiple PDUs
image::images/ws-mate-isup_over_mtp3_over_ip.png[]

This allows to assign the right _Transport_ to the PDU avoiding duplicate
transport protocol entries (in case of tunneled ip over ip for example).

----
Pdu ftp_pdu Proto ftp Transport tcp/ip {
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract ftp_cmd From ftp.command;
};
----

.Frame with encapsulated (tunneled) fields
image::images/ws-mate-ftp_over_gre.png[]

Other than the mandatory _Transport_ there is also an optional _Payload_
statement, which works pretty much as _Transport_ but refers to elements after
the _Proto_'s range. It is useful in those cases where the payload protocol
might not appear in a PDU but nevertheless the PDU belongs to the same category.

----
Pdu mmse_over_http_pdu Proto http Transport tcp/ip {

    Payload mmse;

    Extract addr From ip.addr;
    Extract port From tcp.port;

    Extract content From http.content_type;
    Extract host From http.host;
    Extract http_rq From http.request;
    Extract method From http.request.method;
    Extract resp From http.response.code;

    Extract msg_type From mmse.message_type;
    Extract notify_status From mmse.status;
    Extract send_status From mmse.response_status;
    Extract trx From mmse.transaction_id;
};
----

.Extract from Payload fields
image::images/ws-mate-mmse_over_http.png[]

===== Conditions on which to create PDUs

There might be cases in which we won't want MATE to create a PDU unless some of
its extracted attributes meet or do not meet some criteria. For that we use the
_Criteria_ statements of the _Pdu_ declarations.

----
Pdu isup_pdu Proto isup Transport mtp3/ip {
    ...

    // MATE will create isup_pdu PDUs only when there is not a point code '1234'
    Criteria Reject Strict (m3pc=1234);
};

Pdu ftp_pdu Proto ftp Transport tcp/ip {
    ...

    // MATE will create ftp_pdu PDUs only when they go to port 21 of our ftp_server
    Criteria Accept Strict (addr=10.10.10.10, port=21);
};
----

The _Criteria_ statement is given an action (_Accept_ or _Reject_), a match type
(_Strict_, _Loose_ or _Every_) and an AVPL against which to match the currently
extracted one.

===== Transforming the attributes of a PDU

Once the fields have been extracted into the PDU's AVPL, MATE will apply any
declared _Transform_ to it. The way transforms are applied and how they work
is described later on. However, it's useful to know that once the AVPL for the
PDU is created, it may be transformed before being analyzed. That way we can
massage the data to simplify the analysis.

===== MATE's PDU tree

Every successfully created PDU will add a MATE tree to the frame dissection. If
the PDU is not related to any GOP, the tree for the PDU will contain just the
PDU's info. If it is assigned to a GOP, the tree will also contain the GOP items,
and the same applies for the GOG level.

----
mate dns_pdu:1
    dns_pdu: 1
        dns_pdu time: 3.750000
        dns_pdu Attributes
            dns_resp: False
            dns_id: 0x8cac
            addr: 10.194.4.11
            addr: 10.194.24.35
----

The PDU's tree contains some filterable fields

* _mate.dns_pdu_ will contain the number of the "dns_pdu" PDU
* _mate.dns_pdu.RelativeTime_ will contain the time passed since the beginning
of the capture in seconds
* the tree will contain the various attributes of the PDU as well, these will
all be strings (to be used in filters as "10.0.0.1", not as 10.0.0.1)
** mate.dns_pdu.dns_resp
** mate.dns_pdu.dns_id
** mate.dns_pdu.addr

==== Grouping PDUs together (GOP) (Phase 2)

Once MATE has created the PDUs it passes to the PDU analysis phase. During the
PDU analysis phase MATE will try to group PDUs of the same type into 'Groups of
PDUs' (aka *GOP*++s++) and copy some AVPs from the PDU's AVPL to the GOP's AVPL.

.Grouping PDUs (GOP) flowchart
image::images/ws-mate-pdu_analysis.png[]

===== What can belong to a GOP

Given a PDU, the first thing MATE will do is to check if there is any GOP
declaration in the configuration for the given PDU type. If so, it will use its
_Match_ AVPL to match it against the PDU's AVPL; if they don't match, the
analysis phase is done. If there is a match, the AVPL is the GOP's candidate key
which will be used to search the index of GOPs for the GOP to which to assign
the current PDU. If there is no such GOP and this PDU does not match the
_Start_ criteria of a _Gop_ declaration for the PDU type, the PDU will remain
unassigned and only the analysis phase will be done.

----
Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {...};
Gop dns_req On dns_pdu Match (addr, addr, dns_id) {...};
Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {...};
----

===== Start of a GOP

If there was a match, the candidate key will be used to search the index of GOPs
to see if there is already a GOP matching the GOP's key the same way. If there
is such a match in the GOPs collection, and the PDU doesn't match the _Start_
AVPL for its type, the PDU will be assigned to the matching GOP. If it is a
_Start_ match, MATE will check whether or not that GOP has been already
stopped. If the GOP has been stopped, a new GOP will be created and will replace
the old one in the index of GOPs.

----
Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
};

Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
    Start (dns_resp="True");
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1);
};
----

If no _Start_ is given for a GOP, a PDU whose AVPL matches an existing GOP's
key will act as the start of a GOP.

===== What goes into the GOP's AVPL

Once we know a GOP exists and the PDU has been assigned to it, MATE will copy
into the GOP's AVPL all the attributes matching the key plus any AVPs of the
PDU's AVPL matching the _Extra_ AVPL.

----
Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
    Extra (pasv_prt, pasv_addr);
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1);
    Extra (calling, called);
};
----

===== End of a GOP

Once the PDU has been assigned to the GOP, MATE will check whether or not the
PDU matches the _Stop_, if it happens, MATE will mark the GOP as stopped. Even
after stopped, a GOP may get assigned new PDUs matching its key, unless such
PDU matches _Start_. If it does, MATE will instead create a new GOP starting
with that PDU.

----
Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
    Stop (ftp_cmd=QUIT); // The response to the QUIT command will be assigned to the same GOP
    Extra (pasv_prt, pasv_addr);
};

Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
    Start (dns_resp="False");
    Stop (dns_resp="True");
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1); // IAM
    Stop (isup_msg=16); // RLC
    Extra (calling, called);
};
----

If no _Stop_ criterium is stated for a given GOP, the GOP will be stopped as
soon as it is created. However, as with any other GOP, PDUs matching the GOP's
key will still be assigned to the GOP unless they match a _Start_ condition,
in which case a new GOP using the same key will be created. To group multiple
PDUs that match the _Start_, add a bogus _Stop_ such as
----
Gop frame_ses On frame_pdu Match (frame_time) {
    Start (frame_time);
    Stop (frame_time="FOO");
};
----

===== GOP's tree

For every frame containing a PDU that belongs to a GOP, MATE will create a tree
for that GOP.

The example below represents the tree created by the _dns_pdu_ and _dns_req_
examples.

----
...
MATE dns_pdu:6->dns_req:1
    dns_pdu: 6
        dns_pdu time: 2.103063
        dns_pdu time since beginning of Gop: 2.103063
        dns_pdu Attributes
            dns_resp: True
            dns_id: 0x8cac
            addr: 10.194.4.11
            addr: 10.194.24.35
    dns_req: 1
        GOP Key:  addr=10.194.4.11; addr=10.194.24.35; dns_id=0x8cac;
        dns_req Attributes
            dns_id: 0x8cac
            addr: 10.194.4.11
            addr: 10.194.24.35
        dns_req Times
            dns_req start time: 0.000000
            dns_req hold time: 2.103063
            dns_req duration: 2.103063
        dns_req number of PDUs: 2
            Start PDU: in frame 1
            Stop PDU: in frame 6 (2.103063 : 2.103063)

----

Other than the PDU's tree, this one contains information regarding the
relationship between the PDUs that belong to the GOP. That way we have:

* mate.dns_req which contains the id of this dns_req GOP. This will be present
in frames that belong to dns_req GOPs.
* mate.dns_req.dns_id and mate.dns_req.addr which represent the values of the
attributes copied into the GOP.
* the timers of the GOP
** mate.dns_req.StartTime time (in seconds) passed since beginning of capture
until GOP's start.
** mate.dns_req.Time time passed between the start PDU and the stop PDU assigned
to this GOP (only created if a Stop criterion has been declared for the GOP and
a matching PDU has arrived).
** mate.dns_req.Duration time passed between the start PDU and the last PDU
assigned to this GOP.
* mate.dns_req.NumOfPdus the number of PDUs that belong to this GOP
** mate.dns_req.Pdu a filterable list of frame numbers of the PDUs of this GOP

===== GOP's timers

Note that there are two "timers" for a GOP:

* *Time*, which is defined only for GOPs that have been Stopped, and gives the
time passed between the _Start_ and the _Stop_ PDUs.
* *Duration*, which is defined for every GOP regardless of its state, and give
the time passed between its _Start_ PDU and the last PDU that was assigned to
that GOP.

So:

* we can filter for PDUs that belong to GOPs that have been Stopped with
*mate.xxx.Time*
* we can filter for PDUs that belong to unstopped GOPs with *mate.xxx &&
!mate.xxx.Time*
* we can filter for PDUs that belong to stopped GOPs using *mate.xxx.Duration*
* we can filter for PDUs that belong to GOPs that have taken more (or less) time
that 0.5s to complete with *mate.xxx.Time > 0.5* (you can try these also as
color filters to find out when response times start to grow)

==== Grouping GOPs together (GOG) (Phase 3)

When GOPs are created, or whenever their AVPL changes, GOPs are (re)analyzed to
check if they match an existent group of groups (GOG) or can create a new one.
The GOP analysis is divided into two phases. In the first phase, the still
unassigned GOP is checked to verify whether it belongs to an already existing
GOG or may create a new one. The second phase eventually checks the GOG and
registers its keys in the index of GOGs.

.Grouping GOPs (GOG) flowchart
image::images/ws-mate-gop_analysis.png[]

There are several reasons for the author to believe that this feature needs to
be reimplemented, so probably there will be deep changes in the way this is done
in the near future. This section of the documentation reflects the version of
MATE as of Wireshark 0.10.9; in future releases this will change.

===== Declaring a Group Of Groups (GOG)

The first thing we have to do configuring a GOG is to tell MATE that it exists.

----
Gog http_use {
   ...
};
----

===== Telling MATE what could be a GOG member

Then we have to tell MATE what to look for a match in the candidate GOPs.

----
Gog http_use {
    Member http_ses (host);
    Member dns_req (host);
};
----

===== Getting interesting data into the GOG

Most often, also other attributes than those used for matching would be
interesting. In order to copy from GOP to GOG other interesting attributes, we
might use _Extra_ like we do for GOPs.

----
Gog http_use {
    ...
    Extra (cookie);
};
----

===== GOG's tree

----
mate http_pdu:4->http_req:2->http_use:1
    http_pdu: 4
        http_pdu time: 1.309847
        http_pdu time since beginning of Gop: 0.218930
        http_req: 2
            ... (the gop's tree for http_req: 2) ..
        http_use: 1
            http_use Attributes
                host: www.example.com
            http_use Times
                http_use start time: 0.000000
                http_use duration: 1.309847
            number of GOPs: 3
                dns_req: 1
                    ... (the gop's tree for dns_req: 1) ..
                http_req: 1
                    ... (the gop's tree for http_req: 1) ..
                http_req of current frame: 2
----

We can filter on:

* the timers of the GOG
** *mate.http_use.StartTime* time (in seconds) passed since beginning of capture until GOG's start.
** *mate.http_use.Duration* time elapsed between the first frame of a GOG and the last one assigned to it.
* the attributes passed to the GOG
** *mate.http_use.host*
* *mate.http_use.NumOfGops* the number of GOPs that belong to this GOG
* *mate.http_use.GopStart* the start frame of a GOP
* *mate.http_use.GopStop* the stop frame of a GOP

==== Adjust data (AVPL Transforms)

A Transform is a sequence of Match rules optionally completed with modification
of the match result by an additional AVPL. Such modification may be an Insert
(merge) or a Replace. Transforms can be used as helpers to manipulate an item's
AVPL before it is processed further. They come to be very helpful in several
cases.

===== Syntax

AVPL Transforms are declared in the following way:

----
Transform name {
    Match [Strict|Every|Loose] match_avpl [Insert|Replace] modify_avpl;
    ...
};
----

The *name* is the handle to the AVPL transform. It is used to refer to the
transform when invoking it later.

The _Match_ declarations instruct MATE what and how to match against the data
AVPL and how to modify the data AVPL if the match succeeds. They will be
executed in the order they appear in the config file whenever they are invoked.

The optional match type qualifier (_Strict_, _Every_, or _Loose_) is used
to choose the <<Match,Match type>>; _Strict_ is the default value which
may be omitted.

The optional modification mode qualifier instructs MATE how the modify AVPL
should be used:

* the default value _Insert_ (which may be omitted) causes the _modify_avpl_
to be *merged* to the existing data AVPL,
* _Replace_ causes all the matching AVPs from the data AVPL to be
*replaced* by the _modify_avpl_.

The _modify_avpl_ may be an empty one; this comes useful in some cases for
both _Insert_ and _Replace_ modification modes.

----
Transform rm_client_from_http_resp1 {
    Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted
    Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests
};

Transform rm_client_from_http_resp2 {
    Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing
};
----

Examples:

----
Transform insert_name_and {
    Match Strict (host=10.10.10.10, port=2345) Insert (name=JohnDoe);
};
----

adds name=JohnDoe to the data AVPL if it contains host=10.10.10.10 *and*
port=2345

----
Transform insert_name_or {
    Match Loose (host=10.10.10.10, port=2345) Insert (name=JohnDoe);
};
----

adds name=JohnDoe to the data AVPL if it contains host=10.10.10.10 *or*
port=2345

----
Transform replace_ip_address {
    Match (host=10.10.10.10) Replace (host=192.168.10.10);
};
----

replaces the original host=10.10.10.10 by host=192.168.10.10

----
Transform add_ip_address {
    Match (host=10.10.10.10) (host=192.168.10.10);
};
----

adds (inserts) host=192.168.10.10 to the AVPL, keeping the original
host=10.10.10.10 in it too

----
 Transform replace_may_be_surprising {
    Match Loose (a=aaaa, b=bbbb) Replace (c=cccc, d=dddd);
 };
----

gives the following results:

* (a=aaaa, b=eeee) gets transformed to (b=eeee, c=cccc, d=dddd) because a=aaaa
did match so it got replaced while b=eeee did not match so it has been left
intact,
* (a=aaaa, b=bbbb) gets transformed to (c=cccc, d=dddd) because both a=aaaa and
b=bbbb did match.

===== Usage

Once declared, Transforms can be added to the declarations of PDUs, GOPs or
GOGs. This is done by adding the _Transform name_list_ statement to the
declaration:

----
Pdu my_proto_pdu Proto my_proto Transport ip {
    Extract addr From ip.addr;
    ...
    Transform my_pdu_transform[, other_pdu_transform[, yet_another_pdu_transform]];
};
----

* In case of PDU, the list of transforms is applied against the PDU's AVPL
after its creation.
* In case of GOP and GOG, the list of transforms is applied against their
respective AVPLs when they are created and every time they change.

===== Operation

.Applying Transform flowchart
image::images/ws-mate-transform.png[]

* A list of previously declared Transforms may be given to every Item (_Pdu_, _Gop_,
or _Gog_), using the _Transform_ statement.
* Every time the AVPL of an item changes, it will be operated against *all* the
Transforms on the list given to that item. The Transforms on the list are
applied left to right.
* Inside each of the Transforms, the item's AVPL will be operated against the
Transform's Match clauses starting from the topmost one, until all have been
tried or until one of them succeeds.

MATE's Transforms can be used for many different things, like:

===== Multiple Start/Stop conditions for a GOP

Using _Transforms_ we can add more than one start or stop condition to a GOP.

----
Transform start_cond {
    Match (attr1=aaa,attr2=bbb) (msg_type=start);
    Match (attr3=www,attr2=bbb) (msg_type=start);
    Match (attr5^a) (msg_type=stop);
    Match (attr6$z) (msg_type=stop);
};

Pdu pdu ... {
    ...
    Transform start_cond;
}

Gop gop ... {
    Start (msg_type=start);
    Stop (msg_type=stop);
    ...
}
----

===== Marking GOPs and GOGs to filter them easily

----
Transform marks {
    Match (addr=10.10.10.10, user=john) (john_at_host);
    Match (addr=10.10.10.10, user=tom) (tom_at_host);
}

...

Gop my_gop ... {
    ...
    Transform marks;
}
----

After that we can use a display filter *mate.my_gop.john_at_host* or
*mate.my_gop.tom_at_host*

===== Adding (Insert) direction knowledge to MATE

----
Transform direction_as_text {
    Match (src=192.168.0.2, dst=192.168.0.3) Insert (direction=from_2_to_3);
    Match (src=192.168.0.3, dst=192.168.0.2) Insert (direction=from_3_to_2);
};

Pdu my_pdu Proto my_proto Transport tcp/ip {
    Extract src From ip.src;
    Extract dst From ip.dst;
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract start From tcp.flags.syn;
    Extract stop From tcp.flags.fin;
    Extract stop From tcp.flags.rst;
    Transform direction_as_text;
}

Gop my_gop On my_pdu Match (addr,addr,port,port) {
    ...
    Extra (direction);
}
----

The original example (below) would delete _src_ and _dst_ then add _direction_.
----
Transform direction_as_text {
    Match (src=192.168.0.2, dst=192.168.0.3) Replace (direction=from_2_to_3);
    Match (src=192.168.0.3, dst=192.168.0.2) Replace (direction=from_3_to_2);
};
----

===== NAT

NAT can create problems when tracing, but we can easily work around it by
Transforming the NATed IP address and the Ethernet address of the router into
the non-NAT address:

----
Transform denat {
    Match (addr=192.168.0.5, ether=01:02:03:04:05:06) Replace (addr=123.45.67.89);
    Match (addr=192.168.0.6, ether=01:02:03:04:05:06) Replace (addr=123.45.67.90);
    Match (addr=192.168.0.7, ether=01:02:03:04:05:06) Replace (addr=123.45.67.91);
}

Pdu my_pdu Proto my_proto transport tcp/ip/eth {
    Extract ether From eth.addr;
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Transform denat;
}
----

[#ChMateConfigurationTutorial]

=== MATE's configuration tutorial

We'll show a MATE configuration that first creates GOPs for every DNS and HTTP
request, then it ties the GOPs together in a GOG based on the host. Finally,
we'll separate into different GOGs request coming from different users.

With this MATE configuration loaded we can:

* use *mate.http_use.Duration > 5.5* to filter frames based on the time it takes
to load a complete page from the DNS request to resolve its name until the last
image gets loaded.
* use *mate.http_use.client == "10.10.10.20" && mate.http_use.host == "www.example.com"*
to isolate DNS and HTTP packets related to a visit of a certain user.
* use *mate.http_req.Duration > 1.5* to filter all the packets of HTTP requests
that take more than 1.5 seconds to complete.

The complete config file is available on the Wireshark Wiki:
{wireshark-wiki-url}Mate/Tutorial

Note: This example uses _dns.qry.name_ which is defined since Wireshark
version 0.10.9. Supposing you have a MATE plugin already installed you can test
it with the current Wireshark version.

==== A GOP for DNS requests

First we'll tell MATE how to create a GOP for each DNS request/response.

MATE needs to know what makes a DNS PDU. We describe it using a _Pdu_
declaration:

----
Pdu dns_pdu Proto dns Transport ip {
    Extract addr From ip.addr;
    Extract dns_id From dns.id;
    Extract dns_resp From dns.flags.response;
};
----

Using _Proto dns_ we tell MATE to create PDUs every time it finds _dns_. Using
_Transport ip_ we inform MATE that some of the fields we are interested are
in the _ip_ part of the frame. Finally, we tell MATE to import _ip.addr_ as
_addr_, _dns.id_ as _dns_id_ and _dns.flags.response_ as _dns_resp_.

Once we've told MATE how to extract _dns_pdus_ we'll tell it how to match
requests and responses and group them into a GOP. For this we'll use a _Gop_
declaration to define the GOP, and then, _Start_ and _Stop_ statements to
tell it when the GOP starts and ends.

----
Gop dns_req On dns_pdu Match (addr,addr,dns_id) {
    Start (dns_resp="False");
    Stop (dns_resp="True");
};
----

Using the *Gop* declaration we tell MATE that the *Name* of the GOP is _dns_req_,
that _dns_pdus_s can become members of the GOP, and what is the key used to match
the PDUs to the GOP.

The key for this GOP is _"addr, addr, dns_id"_. That means that in order to
belong to the same GOP, _dns_pdus_ have to have both addresses and the
_request id_ identical. We then instruct MATE that a _dns_req_ starts whenever
a _dns_pdu_ matches _"dns_resp=++"++False++"++"_ and that it stops when another _dns_pdu_
matches _"dns_resp=++"++True++"++"_.

At this point, if we open a capture file using this configuration, we are able
to use a display filter *mate.dns_req.Time > 1* to see only the packets of
DNS requests that take more than one second to complete.

We can use a display filter *mate.dns_req && ! mate.dns_req.Time* to find
requests for which no response was given. *mate.xxx.Time* is set only for GOPs
that have being stopped.

==== A GOP for HTTP requests

This other example creates a GOP for every HTTP request.

----
Pdu http_pdu Proto http Transport tcp/ip {
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract http_rq From http.request.method;
    Extract http_rs From http.response;
    DiscardPduData true;
};

Gop http_req On http_pdu Match (addr, addr, port, port) {
    Start (http_rq);
    Stop (http_rs);
};
----

So, if we open a capture using this configuration

* filtering with *mate.http_req.Time > 1* will give all the requests where the
response header takes more than one second to come
* filtering with *mate.http_req.Duration > 1.5* will show those request that
take more than 1.5 seconds to complete.

You have to know that *mate.xxx.Time* gives the time in seconds between the PDU
matching the GOP *Start* clause and the PDU matching the GOP *Stop* clause (yes, you can create
timers using this!). On the other hand, *mate.xxx.Duration* gives you the time
passed between the GOP *Start* and the last PDU assigned to that GOP regardless
whether it is a *Stop* or not. After the GOP *Stop*, PDUs matching the GOP's Key will
still be assigned to the same GOP as far as they don't match the GOP *Start*, in
which case a new GOP with the same key will be created.

==== Getting DNS and HTTP together into a GOG

We'll tie together to a single GOG all the HTTP packets belonging to requests
and responses to a certain host and the DNS request and response used to resolve
its domain name using the _Pdu_ and _Gop_ definitions of the previous examples

To be able to group DNS and HTTP requests together, we need to import into the
PDUs and GOPs some part of information that both those protocols share. Once the
PDUs and GOPs have been defined, we can use _Extract_ (for PDUs) and
_Extract_ (for GOPs) statements to tell MATE what other protocol fields are to
be added to PDU's and GOP's AVPLs. We add the following statements to the
appropriate declarations:

----
    Extract host From http.host; // to Pdu http_pdu as the last Extract in the list
    Extra (host); // to Gop http_req after the Stop

    Extract host From dns.qry.name; // to Pdu dns_pdu as the last Extract in the list
    Extra (host); // to Gop dns_req after the Stop
----

Here we've told MATE to import _http.host_ into _http_pdu_ and _dns.qry.name_
into _dns_pdu_ as _host_. We also have to tell MATE to copy the _host_
attribute from the PDUs to the GOPs - we do this using _Extra_.

Once we have all the data we need in PDUs and GOPs, we tell MATE what makes
different GOPs belong to a certain GOG.

----
Gog http_use {
    Member http_req (host);
    Member dns_req (host);
    Expiration 0.75;
};
----

Using the _Gog_ declaration, we tell MATE to define a GOG type named
_http_use_ whose expiration is 0.75 seconds after all the GOPs that belong to it
had been stopped. After that time, an eventual new GOP with the same key match
will create a new GOG instead of been added to the previous GOG.

Using the _Member_ statements, we tell MATE that *http_req*s with the same
*host* belong to the same GOG, same thing for *dns_req*s.

So far we have instructed MATE to group every packet related to sessions towards
a certain host. At this point if we open a capture file and:

* a display filter *mate.http_use.Duration > 5* will show only those requests
that have taken more than 5 seconds to complete starting from the DNS request
and ending with the last packet of the HTTP responses.

* a display filter *mate.http_use.host == "www.w3c.org"* will show all the
packets (both DNS and HTTP) related to the requests directed to www.w3c.org

==== Separating requests from multiple users

"Houston: we've had a problem here."

This configuration works fine if used for captures taken at the client's side
but deeper in the network we'd got a real mess. Requests from many users get
mixed together into _http_uses_. GOGs are created and stopped almost randomly
(depending on the timing in which GOPs start and stop). How do we get requests
from individual users separated from each other?

MATE has a tool that can be used to resolve this kind of grouping issues. This
tool are the _Transforms_. Once defined, they can be applied against PDUs,
GOPs and GOGs and they might replace or insert more attributes based on what's
there. We'll use them to create an attribute named *client*, using which we'll
separate different requests.

For DNS we need the ip.src of the request moved into the GOP only from the DNS
request.

So we first tell MATE to import ip.src as client:

----
    Extract client From ip.src;
----

Next, we tell MATE to replace ( *dns_resp="True", client* ) with just *dns_resp="True"* in
the PDU. That way, we'll keep the attribute *client* only in the DNS request
PDUs (i.e., packets coming from the client).To do so, we have to add a
_Transform_ declaration (in this case, with just one clause) before the _Pdu_
declaration which uses it:

----
Transform rm_client_from_dns_resp {
    Match (dns_resp="True", client) Replace (dns_resp="True");
};
----

Next, we invoke the transform by adding the following line after the _Extract_
list of the dns_pdu PDU:

----
    Transform rm_client_from_dns_resp;
----

HTTP is a little trickier. We have to remove the attribute carrying ip.src from
both the response and the "continuations" of the response, but as there is
nothing to filter on for the continuations, we have to add a fake attribute
first. And then we have to remove *client* when the fake attribute appears.
This is possible due to the fact that the _Match_ clauses in the _Transform_
are executed one by one until one of them succeeds. First, we declare another
two _Transforms_:

----
Transform rm_client_from_http_resp1 {
    Match (http_rq); //first match wins so the request won't get the not_rq attribute inserted
    Match Every (addr) Insert (not_rq); //this line won't be evaluated if the first one matched so not_rq won't be inserted to requests
};

Transform rm_client_from_http_resp2 {
    Match (not_rq, client) Replace (); //replace "client and not_rq" with nothing (will happen only in the response and eventual parts of it)
};
----

Next, we add another _Extract_ statement to the _http_pdu_ declaration, and
apply both _Transforms_ declared above in a proper order:

----
    Extract client From ip.src;
    Transform rm_client_from_http_resp1, rm_client_from_http_resp2;
----

In MATE, all the _Transform_s listed for an item will be evaluated, while
inside a single _Transform_, the evaluation will stop at the first successful
_Match_ clause. That's why we first just match _http_rq_ to get out of the
first sequence before adding the _not_rq_ attribute. Then we apply the second
_Transform_ which removes both _not_rq_ and _client_ if both are there. Yes,
_Transform_s are cumbersome, but they are very useful.

Once we got all what we need in the PDUs, we have to tell MATE to copy the
attribute _client_ from the PDUs to the respective GOPs, by adding client to
_Extra_ lists of both _Gop_ declarations:

----
    Extra (host, client);
----

On top of that, we need to modify the old declarations of GOP key to new ones
that include both _client_ and _host_. So we change the _Gog_ *Member*
declarations the following way:

----
    Member http_req (host, client);
    Member dns_req (host, client);
----

Now we got it, every "usage" gets its own GOG.

[#ChMateConfigurationExamples]

=== MATE configuration examples

The following is a collection of various configuration examples for MATE. Many
of them are useless because the "conversations" facility does a better job.
Anyway they are meant to help users understanding how to configure MATE.

[#File_tcp_mate]
==== TCP session (tcp.mate)

The following example creates a GOP out of every TCP session.

----
Transform add_tcp_stop {
    Match (tcp_flags_reset="True") Insert (tcp_stop="True");
    Match (tcp_flags_fin="True") Insert (tcp_stop="True");
};

Pdu tcp_pdu Proto tcp Transport ip {
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract tcp_start From tcp.flags.syn;
    Extract tcp_flags_reset From tcp.flags.reset;
    Extract tcp_flags_fin From tcp.flags.fin;
    Transform add_tcp_stop;
};

Gop tcp_ses On tcp_pdu Match (addr, addr, port, port) {
    Start (tcp_start="True");
    Stop (tcp_stop="True");
};

Done;
----

This probably would do fine in 99.9% of the cases but 10.0.0.1:20->10.0.0.2:22 and 10.0.0.1:22->10.0.0.2:20 would both fall into the same gop if they happen to overlap in time.

* filtering with *mate.tcp_ses.Time > 1* will give all the sessions that last more than one second
* filtering with *mate.tcp_ses.NumOfPdus < 5* will show all tcp sessions that have less than 5 packets.
* filtering with *mate.tcp_ses.Id == 3* will show all the packets for the third tcp session MATE has found

==== a GOG for a complete FTP session

This configuration allows to tie a complete passive FTP session (including the
data transfer) in a single GOG.

----
Pdu ftp_pdu Proto ftp Transport tcp/ip {
    Extract ftp_addr From ip.addr;
    Extract ftp_port From tcp.port;
    Extract ftp_resp From ftp.response.code;
    Extract ftp_req From ftp.request.command;
    Extract server_addr From ftp.passive.ip;
    Extract server_port From ftp.passive.port;

    LastPdu true;
};

Pdu ftp_data_pdu Proto ftp-data Transport tcp/ip{
    Extract server_addr From ip.src;
    Extract server_port From tcp.srcport;

};

Gop ftp_data On ftp_data_pdu Match (server_addr, server_port) {
    Start (server_addr);
};

Gop ftp_ctl On ftp_pdu Match (ftp_addr, ftp_addr, ftp_port, ftp_port) {
    Start (ftp_resp=220);
    Stop (ftp_resp=221);
    Extra (server_addr, server_port);
};

Gog ftp_ses {
    Member ftp_ctl (ftp_addr, ftp_addr, ftp_port, ftp_port);
    Member ftp_data (server_addr, server_port);
};

Done;
----

Note: not having anything to distinguish between ftp-data packets makes this
config to create one GOP for every ftp-data packet instead of each transfer.
Pre-started GOPs would avoid this.

==== using RADIUS to filter SMTP traffic of a specific user

Spying on people, in addition to being immoral, is illegal in many countries.
This is an example meant to explain how to do it not an invitation to do so.
It's up to the police to do this kind of job when there is a good reason to do
so.

----
Pdu radius_pdu On radius Transport udp/ip {
    Extract addr From ip.addr;
    Extract port From udp.port;
    Extract radius_id From radius.id;
    Extract radius_code From radius.code;
    Extract user_ip From radius.framed_addr;
    Extract username From radius.username;
}

Gop radius_req On radius_pdu (radius_id, addr, addr, port, port) {
    Start (radius_code {1|4|7} );
    Stop (radius_code {2|3|5|8|9} );
    Extra (user_ip, username);
}

// we define the smtp traffic we want to filter
Pdu user_smtp Proto smtp Transport tcp/ip {
    Extract user_ip From ip.addr;
    Extract smtp_port From tcp.port;
    Extract tcp_start From tcp.flags.syn;
    Extract tcp_stop From tcp.flags.reset;
}

Gop user_smtp_ses On user_smtp (user_ip, user_ip, smtp_port!25) {
    Start (tcp_start=1);
    Stop (tcp_stop=1);
}

// with the following group of groups we'll group together the radius and the smtp
// we set a long expiration to avoid the session expire on long pauses.
Gog user_mail {
    Expiration 1800;
    Member radius_req (user_ip);
    Member user_smtp_ses (user_ip);
    Extra (username);
}

Done;
----

Filtering the capture file with *mate.user_mail.username == "theuser"* will
filter the RADIUS packets and SMTP traffic for _"theuser"_.

==== H323 Calls

This configuration will create a GOG out of every call.

----
Pdu q931 Proto q931 Transport ip {
    Extract addr From ip.addr;
    Extract call_ref From q931.call_ref;
    Extract q931_msg From q931.message_type;
    Extract calling From q931.calling_party_number.digits;
    Extract called From q931.called_party_number.digits;
    Extract guid From h225.guid;
    Extract q931_cause From q931.cause_value;
};

Gop q931_leg On q931 Match (addr, addr, call_ref) {
    Start (q931_msg=5);
    Stop (q931_msg=90);
    Extra (calling, called, guid, q931_cause);
};

Pdu ras Proto h225.RasMessage Transport ip {
    Extract addr From ip.addr;
    Extract ras_sn From h225.requestSeqNum;
    Extract ras_msg From h225.RasMessage;
    Extract guid From h225.guid;
};

Gop ras_req On ras Match (addr, addr, ras_sn) {
    Start (ras_msg {0|3|6|9|12|15|18|21|26|30} );
    Stop (ras_msg {1|2|4|5|7|8|10|11|13|14|16|17|19|20|22|24|27|28|29|31});
    Extra (guid);
};

Gog call {
    Member ras_req (guid);
    Member q931_leg (guid);
    Extra (called,calling,q931_cause);
};

Done;
----

with this we can:

* filter all signalling for a specific caller: *mate.call.caller == "123456789"*
* filter all signalling for calls with a specific release cause: *mate.call.q931_cause == 31*
* filter all signalling for very short calls: *mate.q931_leg.Time < 5*

==== MMS

With this example, all the components of an MMS send or receive will be tied
into a single GOG. Note that this example uses the _Payload_ clause because
MMS delivery uses MMSE over either HTTP or WSP. As it is not possible to relate
the retrieve request to a response by the means of MMSE only (the request is
just an HTTP GET without any MMSE), a GOP is made of HTTP PDUs but MMSE data
need to be extracted from the bodies.

----
## WARNING: this example has been blindly translated from the "old" MATE syntax
## and it has been verified that Wireshark accepts it. However, it has not been
## tested against any capture file due to lack of the latter.

Transform rm_client_from_http_resp1 {
    Match (http_rq);
    Match Every (addr) Insert (not_rq);
};

Transform rm_client_from_http_resp2 {
    Match (not_rq,ue) Replace ();
};

Pdu mmse_over_http_pdu Proto http Transport tcp/ip {
    Payload mmse;
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract http_rq From http.request;
    Extract content From http.content_type;
    Extract resp From http.response.code;
    Extract method From http.request.method;
    Extract host From http.host;
    Extract content From http.content_type;
    Extract trx From mmse.transaction_id;
    Extract msg_type From mmse.message_type;
    Extract notify_status From mmse.status;
    Extract send_status From mmse.response_status;
    Transform rm_client_from_http_resp1, rm_client_from_http_resp2;
};

Gop mmse_over_http On mmse_over_http_pdu Match (addr, addr, port, port) {
    Start (http_rq);
    Stop (http_rs);
    Extra (host, ue, resp, notify_status, send_status, trx);
};

Transform mms_start {
    Match Loose() Insert (mms_start);
};

Pdu mmse_over_wsp_pdu Proto wsp Transport ip {
    Payload mmse;
    Extract trx From mmse.transaction_id;
    Extract msg_type From mmse.message_type;
    Extract notify_status From mmse.status;
    Extract send_status From mmse.response_status;
    Transform mms_start;
};

Gop mmse_over_wsp On mmse_over_wsp_pdu Match (trx) {
    Start (mms_start);
    Stop (never);
    Extra (ue, notify_status, send_status);
};

Gog mms {
    Member mmse_over_http (trx);
    Member mmse_over_wsp (trx);
    Extra (ue, notify_status, send_status, resp, host, trx);
    Expiration 60.0;
};
----

[#ChMateConfigurationLibrary]

=== MATE's configuration library

The MATE library (will) contains GOP definitions for several protocols. Library
protocols are included in your MATE config using: +_Action=Include;
Lib=proto_name;_+.

For Every protocol with a library entry, we'll find defined what from the PDU is
needed to create a GOP for that protocol, eventually any criteria and the very
essential GOP definition (i.e., __Gop__, _Start_ and _Stop_).

[NOTE]
====
It seems that this code is written in the old syntax of MATE. So far it has not
been transcribed into the new format. It may still form the basis to recreate
these in the new format.
====

==== General use protocols

===== TCP

It will create a GOP for every TCP session. If it is used it should be the last
one in the list. And every other proto on top of TCP should be declared with
_LastPdu=TRUE;_ so that a TCP PDU is not created where another pdu type exists.

----
Transform add_tcp_stop {
    Match (tcp_flags_reset="True") Insert (tcp_stop="True");
    Match (tcp_flags_fin="True") Insert (tcp_stop="True");
};

Pdu tcp_pdu Proto tcp Transport ip {
    Extract addr From ip.addr;
    Extract port From tcp.port;
    Extract tcp_start From tcp.flags.syn;
    Extract tcp_flags_reset From tcp.flags.reset;
    Extract tcp_flags_fin From tcp.flags.fin;
    Transform add_tcp_stop;
};

Gop tcp_ses On tcp_pdu Match (addr, addr, port, port) {
    Start (tcp_start="True");
    Stop (tcp_stop="True");
};

Done;
----

===== DNS

will create a GOP containing every request and its response (eventually
retransmissions too).

----
Action=PduDef; Name=dns_pdu; Proto=dns; Transport=udp/ip; addr=ip.addr; port=udp.port; dns_id=dns.id; dns_rsp=dns.flags.response;

Action=GopDef; Name=dns_req; On=dns_pdu; addr; addr; port!53; dns_id;
Action=GopStart; For=dns_req; dns_rsp=0;
Action=GopStop; For=dns_req; dns_rsp=1;
----

===== RADIUS

A GOP for every transaction.

----
Action=PduDef; Name=radius_pdu; Proto=radius; Transport=udp/ip; addr=ip.addr; port=udp.port; radius_id=radius.id; radius_code=radius.code;

Action=GopDef; Name=radius_req; On=radius_pdu; radius_id; addr; addr; port; port;
Action=GopStart; For=radius_req; radius_code|1|4|7;
Action=GopStop; For=radius_req; radius_code|2|3|5|8|9;
----

===== RTSP

----
Action=PduDef; Name=rtsp_pdu; Proto=rtsp; Transport=tcp/ip; addr=ip.addr; port=tcp.port; rtsp_method=rtsp.method;
Action=PduExtra; For=rtsp_pdu; rtsp_ses=rtsp.session; rtsp_url=rtsp.url;

Action=GopDef; Name=rtsp_ses; On=rtsp_pdu; addr; addr; port; port;
Action=GopStart; For=rtsp_ses; rtsp_method=DESCRIBE;
Action=GopStop; For=rtsp_ses; rtsp_method=TEARDOWN;
Action=GopExtra; For=rtsp_ses; rtsp_ses; rtsp_url;
----

==== VoIP/Telephony

Most protocol definitions here will create one GOP for every Call Leg unless
stated.

===== ISUP

----
Action=PduDef; Name=isup_pdu; Proto=isup; Transport=mtp3; mtp3pc=mtp3.dpc; mtp3pc=mtp3.opc; cic=isup.cic; isup_msg=isup.message_type;

Action=GopDef; Name=isup_leg; On=isup_pdu; ShowPduTree=TRUE; mtp3pc; mtp3pc; cic;
Action=GopStart; For=isup_leg; isup_msg=1;
Action=GopStop; For=isup_leg; isup_msg=16;
----

===== Q931

----
Action=PduDef; Name=q931_pdu; Proto=q931; Stop=TRUE; Transport=tcp/ip; addr=ip.addr; call_ref=q931.call_ref; q931_msg=q931.message_type;

Action=GopDef; Name=q931_leg; On=q931_pdu; addr; addr; call_ref;
Action=GopStart; For=q931_leg; q931_msg=5;
Action=GopStop; For=q931_leg; q931_msg=90;
----

===== H225 RAS

----
Action=PduDef; Name=ras_pdu; Proto=h225.RasMessage; Transport=udp/ip; addr=ip.addr; ras_sn=h225.RequestSeqNum; ras_msg=h225.RasMessage;
Action=PduExtra; For=ras_pdu; guid=h225.guid;

Action=GopDef; Name=ras_leg; On=ras_pdu; addr; addr; ras_sn;
Action=GopStart; For=ras_leg; ras_msg|0|3|6|9|12|15|18|21|26|30;
Action=GopStop; For=ras_leg; ras_msg|1|2|4|5|7|8|10|11|13|14|16|17|19|20|22|24|27|28|29|31;
Action=GopExtra; For=ras_leg; guid;
----

===== SIP

----
Action=PduDef; Proto=sip_pdu; Transport=tcp/ip; addr=ip.addr; port=tcp.port; sip_method=sip.Method; sip_callid=sip.Call-ID; calling=sdp.owner.username;

Action=GopDef; Name=sip_leg; On=sip_pdu; addr; addr; port; port;
Action=GopStart; For=sip; sip_method=INVITE;
Action=GopStop; For=sip; sip_method=BYE;
----

===== MEGACO

Will create a GOP out of every transaction.

To "tie" them to your call's GoG use: _Action=GogKey; Name=your_call; On=mgc_tr;
addr!mgc_addr; megaco_ctx;_

----
Action=PduDef; Name=mgc_pdu; Proto=megaco; Transport=ip; addr=ip.addr; megaco_ctx=megaco.context; megaco_trx=megaco.transid; megaco_msg=megaco.transaction; term=megaco.termid;

Action=GopDef; Name=mgc_tr; On=mgc_pdu; addr; addr; megaco_trx;
Action=GopStart; For=mgc_tr; megaco_msg|Request|Notify;
Action=GopStop; For=mgc_tr; megaco_msg=Reply;
Action=GopExtra; For=mgc_tr; term^DS1; megaco_ctx!Choose one;
----

[#ChMateReferenceManual]

=== MATE's reference manual

==== Attribute Value Pairs (AVP)

MATE uses AVPs for almost everything: to keep the data it has extracted from the
frames' trees as well as to keep the elements of the configuration.

These "pairs" (actually tuples) are made of a name, a value and, in case of
configuration AVPs, an operator. Names and values are strings. AVPs with
operators other than '=' are used only in the configuration and are used for
matching AVPs of PDUs, GOPs and GOGs in the analysis phase.

===== Name

The name is a string used to refer to a type of AVP. Two attributes won't
match unless their names are identical. Capitalized names are reserved for
keywords (you can use them for your elements if you want but I think it's not
the case). MATE attribute names can be used in Wireshark's display filters the
same way like names of protocol fields provided by dissectors, but they are not
just references to (or aliases of) protocol fields.

===== Value

The value is a string. It is either set in the configuration (for configuration
AVPs) or by MATE while extracting interesting fields from a dissection tree
and/or manipulating them later. The values extracted from fields use the same
representation as they do in filter strings.

==== AVP Operators (=,!,{},^,$,~,<,>,?)

Currently only match operators are defined (there are plans to (re)add transform
attributes but some internal issues have to be solved before that). The match
operations are always performed between two operands: the value of an AVP stated
in the configuration and the value of an AVP (or several AVPs with the same name)
extracted from packet data (called "data AVPs"). It is not possible to match
data AVPs to each other.

The defined match operators are:

* <<Equal,Equal>> _=_ test for equality, that is: either the value strings are identical
or the match will fail.
* <<NotEqual,Not Equal>> _!_ will match only if the value strings aren't equal.
* <<OneOf,One Of>> _{}_ will match if one of the value strings listed is equal to the
data AVP's string. Items inside the list's curly braces are
separated with the | character.
* <<StartsWith,Starts With>> _^_ will match if the configuration value string matches the
first characters of the data AVP's value string.
* <<EndsWith,Ends With>> _$_ will match if the configuration value string matches the
last characters of the data AVP's value string.
* <<Contains,Contains>> _~_ will match if the configuration value string matches a
substring of the characters of the data AVP's value string.
* <<LowerThan,Lower Than>> _<_ will match if the data AVP's value string is semantically
lower than the configuration value string.
* <<HigherThan,Higher Than>> _>_ will match if the data AVP's value string is semantically
higher than the configuration value string.
* <<Exists,Exists>> _?_ (can be omitted) will match if the AVP name matches, regardless
what the value string is.

[#Equal]
===== Equal AVP Operator (=)

This operator tests whether the values of the operator and the operand AVP are
equal.

Example::
attrib=aaa *matches* attrib=aaa +
attrib=aaa *does not match* attrib=bbb

[#NotEqual]
===== Not equal AVP operator (!)

This operator matches if the value strings of two AVPs are not equal.

Example::
attrib=aaa matches attrib!bbb +
attrib=aaa does not match attrib!aaa

[#OneOf]
===== "One of" AVP operator ({})

The "one of" operator matches if the data AVP value is equal to one of the
values listed in the "one of" AVP.

Example::
attrib=1 matches attrib{1|2|3} +
attrib=2 matches attrib{1|2|3} +
attrib=4 does not match attrib{1|2|3}

[#StartsWith]
===== "Starts with" AVP operator (^)

The "starts with" operator matches if the first characters of the data AVP
value are identical to the configuration AVP value.

Example::
attrib=abcd matches attrib^abc +
attrib=abc matches attrib^abc +
attrib=ab does not match attrib^abc +
attrib=abcd does not match attrib^bcd +
attrib=abc does not match attrib^abcd +

[#EndsWith]
===== "Ends with" operator ($)

The ends with operator will match if the last bytes of the data AVP value are
equal to the configuration AVP value.

Example::
attrib=wxyz matches attrib$xyz +
attrib=yz does not match attrib$xyz +
attrib=abc...wxyz does not match attrib$abc

[#Contains]
===== Contains operator (~)

The "contains" operator will match if the data AVP value contains a string
identical to the configuration AVP value.

Example::
attrib=abcde matches attrib~bcd +
attrib=abcde matches attrib~abc +
attrib=abcde matches attrib~cde +
attrib=abcde does not match attrib~xyz

[#LowerThan]
===== "Lower than" operator (<)

The "lower than" operator will match if the data AVP value is semantically lower
than the configuration AVP value.

Example::
attrib=abc matches attrib<bcd +
attrib=1 matches attrib<2 +
but beware: attrib=10 does not match attrib<9 +
attrib=bcd does not match attrib<abc +
attrib=bcd does not match attrib<bcd +

BUGS

It should check whether the values are numbers and compare them numerically

[#HigherThan]
===== "Higher than" operator (>)

The "higher than" operator will match if the data AVP value is semantically
higher than the configuration AVP value.

Examples

attrib=bcd matches attrib>abc +
attrib=3 matches attrib>2 +
but beware: attrib=9 does not match attrib>10 +
attrib=abc does not match attrib>bcd +
attrib=abc does not match attrib>abc +

BUGS

It should check whether the values are numbers and compare them numerically

[#Exists]
===== Exists operator (?)

The exists operator will always match as far as the two operands have the same
name.

Examples

attrib=abc matches attrib? +
attrib=abc matches attrib (this is just an alternative notation of the previous example) +
obviously attrib=abc does not match other_attrib? +

==== Attribute Value Pair List (AVPL)
PDUs, GOPs and GOGs use an AVPL to contain the tracing information. An AVPL is
an unsorted set of <<AVP,AVPs>> that can be matched against other AVPLs.

[#Match]
==== Operations between AVPLs (Match)

There are three types of match operations that can be performed between AVPLs.
The PDU's/GOP's/GOG's AVPL will be always one of the operands; the AVPL operator
(match type) and the second operand AVPL will always come from the
<<Config,configuration>>.
Note that a diverse AVP match operator may be specified for each AVP in the
configuration AVPL.

An AVPL match operation returns a result AVPL. In <<Transform,Transform>>s, the
result AVPL may be replaced by another AVPL. The replacement means that the
existing data AVPs are dropped and the replacement AVPL from the
<<Config,configuration>> is <<Merge,Merged>> to the data AVPL of the
PDU/GOP/GOG.

* <<Loose,Loose Match>>: Will match if at least one of the AVPs of the two
operand AVPLs match. If it matches, it returns a result AVPL containing all AVPs
from the data AVPL that did match the configuration's AVPs.
* <<Every,"Every" Match>>: Will match if none of the AVPs of the configuration
AVPL fails to match an AVP in the data AVPL, even if not all of the
configuration AVPs have a match. If it matches, it returns a result AVPL
containing all AVPs from the data AVPL that did match an AVP in the
configuration AVPL.
* <<Strict,Strict Match>>: Will match if and only if each of the AVPs in the
configuration AVPL has at least one match in the data AVPL. If it matches, it
returns a result AVPL containing those AVPs from the data AVPL that matched.

[#Loose]
===== Loose Match

A loose match between AVPLs succeeds if at least one of the data AVPs matches at
least one of the configuration AVPs. Its result AVPL contains all the data AVPs
that matched.

Loose matches are used in Extra operations against the <<Pdu,PDU>>'s AVPL to
merge the result into <<Gop,GOP>>'s AVPL, and against <<Gop,GOP>>'s AVPL to
merge the result into <<Gog,GOG>>'s AVPL. They may also be used in
<<Criteria,Criteria>> and <<Transform,Transform>>s.

[NOTE]
====
As of current (2.0.1), Loose Match does not work as described here, see
https://gitlab.com/wireshark/wireshark/issues/12184[issue 12184]. Only use
in Transforms and Criteria is effectively affected by the bug.
====

Loose Match Examples

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Loose (attr_a?, attr_c?) ==> (attr_a=aaa, attr_c=xxx)

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Loose (attr_a?, attr_c=ccc) ==> (attr_a=aaa)

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Loose (attr_a=xxx; attr_c=ccc) ==> No Match!

[#Every]
===== Every Match

An "every" match between AVPLs succeeds if none of the configuration's AVPs that
have a counterpart in the data AVPL fails to match. Its result AVPL contains all
the data AVPs that matched.

These may only be used in <<Criteria,Criteria>> and <<Transform,Transform>>s.

[NOTE]
====
As of current (2.0.1), Loose Match does not work as described here, see
https://gitlab.com/wireshark/wireshark/-/issues/12184[issue 12184].
====

"Every" Match Examples

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Every (attr_a?, attr_c?) ==> (attr_a=aaa, attr_c=xxx)

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Every (attr_a?, attr_c?, attr_d=ddd) ==> (attr_a=aaa, attr_c=xxx)

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Every (attr_a?, attr_c=ccc) ==> No Match!

(attr_a=aaa; attr_b=bbb; attr_c=xxx) Match Every (attr_a=xxx, attr_c=ccc) ==> No Match!

[#Strict]
===== Strict Match

A Strict match between AVPLs succeeds if and only if every AVP in the
configuration AVPL has at least one counterpart in the data AVPL and none of the
AVP matches fails. The result AVPL contains all the data AVPs that matched.

These are used between GOP keys (key AVPLs) and PDU AVPLs. They may also be used
in <<Criteria,Criteria>> and <<Transform,Transform>>s.

Examples

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Strict (attr_a?, attr_c=xxx) ==> (attr_a=aaa, attr_c=xxx)

(attr_a=aaa, attr_b=bbb, attr_c=xxx, attr_c=yyy) Match Strict (attr_a?, attr_c?) ==> (attr_a=aaa, attr_c=xxx, attr_c=yyy)

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Strict (attr_a?, attr_c=ccc) ==> No Match!

(attr_a=aaa, attr_b=bbb, attr_c=xxx) Match Strict (attr_a?, attr_c?, attr_d?) ==> No Match!

[#Merge]
==== AVPL Merge

An AVPL may be merged into another one. That would add to the latter every AVP
from the former that does not already exist there.

This operation is done

* between the result of a key match and the GOP's or GOG's AVPL,
* between the result of an Extra match and the GOP's or GOG's AVPL,
* between the result of a <<Transform,Transform>> match and PDU's/GOP's AVPL. If
the operation specified by the Match clause is Replace, the result AVPL of the
match is removed from the item's AVPL before the modify_avpl is merged into it.

Examples

(attr_a=aaa, attr_b=bbb) "merge" (attr_a=aaa, attr_c=xxx) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx)

Can't have multiple "attr_a" with same value "aaa"

(attr_a=aaa, attr_b=bbb) "merge" (attr_a=aaa, attr_a=xxx) former becomes (attr_a=aaa, attr_a=xxx, attr_b=bbb)

Multiple "attr_a" with different values "aaa" and "xxx"

(attr_a=aaa, attr_b=bbb) "merge" (attr_c=xxx, attr_d=ddd) former becomes (attr_a=aaa, attr_b=bbb, attr_c=xxx, attr_d=ddd)

All AVP names are unique so resulting AVPL contains all AVPs from both AVPLs

[#Config]
=== Configuration Reference (mate.config)
[#Pdu]
==== PDU declaration block

The following configuration AVPLs deal with PDU creation and data extraction.

===== _Pdu_ declaration block header

In each frame of the capture, MATE will look for source _proto_name_'s PDUs in
the order in which the declarations appear in its configuration and will create
PDUs of every type it can from that frame, unless specifically instructed that
some PDU type is the last one to be looked for in the frame. If told so for a
given type, MATE will extract all PDUs of that type and the previously declared
types it finds in the frame but not those declared later.

The complete declaration of a _Pdu_ looks as below; the mandatory order of the
diverse clauses is as shown.

----
Pdu name Proto proto_name Transport {proto1[/proto2/proto3[/...]|mate}; {
    Payload proto; //optional, no default value
    Extract attribute From proto.field ; //may occur multiple times, at least once
    Transform transform1[, transform2[, ...]]; //optional
    Criteria {Accept|Reject} {Strict|Every|Loose} match_avpl; //optional
    DropUnassigned {TRUE|FALSE}; //optional, default=FALSE
    DiscardPduData {TRUE|FALSE}; //optional, default=FALSE
    LastPdu {TRUE|FALSE}; //optional, default=FALSE
};
----

====== Pdu name

The _name_ is a mandatory attribute of a _Pdu_ declaration. It is chosen
arbitrarily, except that each _name_ may only be used once in MATE's
configuration, regardless the class of an item it is used for. The _name_ is
used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_
is also used as part of the filterable fields' names related to this type of PDU
which MATE creates.

However, several _Pdu_ declarations may share the same _name_. In such case, all
of them are created from each source PDU matching their _Proto_, _Transport_,
and _Payload_ clauses, while the bodies of their declarations may be totally
different from each other. Together with the _Accept_ (or _Reject_) clauses,
this feature is useful when it is necessary to build the PDU's AVPL from
different sets of source fields depending on contents (or mere presence) of
other source fields.

====== Proto and Transport clauses

Every instance of the protocol _proto_name_ PDU in a frame will generate one
PDU with the AVPs extracted from fields that are in the _proto_name_'s range
and/or the ranges of underlying protocols specified by the _Transport_ list.
It is a mandatory attribute of a _Pdu_ declaration. The _proto_name_ is the name
of the protocol as used in Wireshark display filter.

The PDU's _Proto_, and its _Transport_ list of protocols separated by / tell
MATE which fields of a frame can get into the PDU's AVPL. In order that MATE
would extract an attribute from a frame's protocol tree, the area representing
the field in the hex display of the frame must be within the area of either the
_Proto_ or its relative _Transport_++s++. _Transport_++s++ are chosen moving backwards
from the protocol area, in the order they are given.

_Proto http Transport tcp/ip_ does what you'd expect it to - it selects the
nearest tcp range that precedes the current http range, and the nearest ip range
that precedes that tcp range. If there is another ip range before the nearest
one (e.g., in case of IP tunneling), that one is not going to be selected.
_Transport_ tcp/ip/ip that "logically" should select the encapsulating IP header
too doesn't work so far.

Once we've selected the _Proto_ and _Transport_ ranges, MATE will fetch those
protocol fields belonging to them whose extraction is declared using the
_Extract_ clauses for the PDU type. The _Transport_ list is also mandatory,
if you actually don't want to use any transport protocol, use _Transport mate_.
(This didn't work until 0.10.9).

===== Payload clause

Other than the PDU's _Proto_ and its _Transport_ protocols, there is also a
_Payload_ attribute to tell MATE from which ranges of _Proto_'s payload to
extract fields of a frame into the PDU. In order to extract an attribute from a
frame's tree the highlighted area of the field in the hex display must be within
the area of the _Proto_'s relative payload(s). _Payload_++s++ are chosen moving
forward from the protocol area, in the order they are given.
_Proto http Transport tcp/ip Payload mmse_ will select the first mmse range
after the current http range. Once we've selected the _Payload_ ranges, MATE
will fetch those protocol fields belonging to them whose extraction is declared
using the _Extract_ clauses for the PDU type.

===== Extract clause

Each _Extract_ clause tells MATE which protocol field value to extract as an AVP
value and what string to use as the AVP name. The protocol fields are referred
to using the names used in Wireshark display filters. If there is more than one
such protocol field in the frame, each instance that fulfills the criteria
stated above is extracted into its own AVP. The AVP names may be chosen
arbitrarily, but to be able to match values originally coming from different
PDUs (e.g., hostname from DNS query and a hostname from HTTP GET request) later
in the analysis, identical AVP names must be assigned to them and the dissectors
must provide the field values in identical format (which is not always the case).

===== Transform clause

The _Transform_ clause specifies a list of previously declared _Transform_++s++ to
be performed on the PDU's AVPL after all protocol fields have been extracted to
it. The list is always executed completely, left to right. On the contrary, the
list of Match clauses inside each individual _Transform_ is executed only until
the first match succeeds.

[#Criteria]
===== Criteria clause

This clause tells MATE whether to use the PDU for analysis. It specifies a match
AVPL, an AVPL <<Match,Match type>> (_Strict_, _Every_, or _Loose_) and the action to be
performed (_Accept_ or _Reject_) if the match succeeds. Once every attribute has
been extracted and eventual transform list has been executed, and if the
_Criteria_ clause is present, the PDU's AVPL is matched against the match AVPL;
if the match succeeds, the action specified is executed, i.e., the PDU is
accepted or rejected. The default behaviors used if the respective keywords are
omitted are _Strict_ and _Accept_. Accordingly, if the clause is omitted, all
PDUs are accepted.

===== DropUnassigned clause

If set to _TRUE_, MATE will destroy the PDU if it cannot assign it to a GOP.
If set to _FALSE_ (the default if not given), MATE will keep them.

===== DiscardPduData clause

If set to _TRUE_, MATE will delete the PDU's AVPL once it has analyzed it and
eventually extracted some AVPs from it into the GOP's AVPL. This is useful to
save memory (of which MATE uses a lot). If set to _FALSE_ (the default if not
given), MATE will keep the PDU attributes.

===== LastPdu clause

If set to _FALSE_ (the default if not given), MATE will continue to look for
PDUs of other types in the frame. If set to _TRUE_, it will not try to create
PDUs of other types from the current frame, yet it will continue to try for the
current type.

[#Gop]
==== GOP declaration block

===== _Gop_ declaration block header

Declares a Gop type and its candidate key.

----
Gop name On pduname Match key {
    Start match_avpl; // optional
    Stop match_avpl; // optional
    Extra match_avpl; // optional
    Transform transform_list; // optional
    Expiration time; // optional
    IdleTimeout time; // optional
    Lifetime time; // optional
    DropUnassigned [TRUE|FALSE]; //optional
    ShowTree [NoTree|PduTree|FrameTree|BasicTree]; //optional
    ShowTimes [TRUE|FALSE]; //optional, default TRUE
};
----

====== Gop name

The _name_ is a mandatory attribute of a _Gop_ declaration. It is chosen
arbitrarily, except that each _name_ may only be used once in MATE's
configuration, regardless the class of an item it is used for. The _name_ is
used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_
is also used as part of the filterable fields' names related to this type of
GOP which MATE creates.

====== On clause

The _name_ of PDUs which this type of GOP is supposed to be grouping. It is
mandatory.

====== Match clause

Defines what AVPs form up the _key_ part of the GOP's AVPL (the GOP's _key_ AVPL
or simply the GOP's _key_). All PDUs matching the _key_ AVPL of an active GOP
are assigned to that GOP; a PDU which contains the AVPs whose attribute names
are listed in the GOP's _key_ AVPL, but they do not strictly match any active
GOP's _key_ AVPL, will create a new GOP (unless a _Start_ clause is given).
When a GOP is created, the elements of its key AVPL are copied from the creating
PDU.

===== Start clause

If given, it tells MATE what match_avpl must a PDU's AVPL match, in addition to
matching the GOP's _key_, in order to start a GOP. If not given, any PDU whose
AVPL matches the GOP's _key_ AVPL will act as a start for a GOP. The PDU's AVPs
matching the match_avpl are not automatically copied into the GOP's AVPL.

===== Stop clause

If given, it tells MATE what match_avpl must a PDU's AVPL match, in addition to
matching the GOP's _key_, in order to stop a GOP. If omitted, the GOP is
"auto-stopped" - that is, the GOP is marked as stopped as soon as it is created.
The PDU's AVPs matching the match_avpl are not automatically copied into the
GOP's AVPL.

===== Extra clause

If given, tells MATE which AVPs from the PDU's AVPL are to be copied into the
GOP's AVPL in addition to the GOP's key.

===== Transform clause

The _Transform_ clause specifies a list of previously declared _Transform_++s++ to
be performed on the GOP's AVPL after the AVPs from each new PDU, specified by
the _key_ AVPL and the _Extra_ clause's match_avpl, have been merged into it.
The list is always executed completely, left to right. On the contrary, the list
of _Match_ clauses inside each individual _Transform_ is executed only until
the first match succeeds.

===== Expiration clause

A (floating) number of seconds after a GOP is _Stop_ ped during which further
PDUs matching the _Stop_ ped GOP's key but not the _Start_ condition will still
be assigned to that GOP. The default value of zero has an actual meaning of
infinity, as it disables this timer, so all PDUs matching the _Stop_ ped GOP's
key will be assigned to that GOP unless they match the _Start_ condition.

===== IdleTimeout clause

A (floating) number of seconds elapsed from the last PDU assigned to the GOP
after which the GOP will be considered released. The default value of zero has
an actual meaning of infinity, as it disables this timer, so the GOP won't be
released even if no PDUs arrive - unless the _Lifetime_ timer expires.

===== Lifetime clause

A (floating) of seconds after the GOP _Start_ after which the GOP will be
considered released regardless anything else. The default value of zero has an
actual meaning of infinity.

===== DropUnassigned clause

Whether or not a GOP that has not being assigned to any GOG should be discarded.
If _TRUE_, the GOP is discarded right after creation. If _FALSE_, the default,
the unassigned GOP is kept. Setting it to _TRUE_ helps save memory and speed up
filtering.

===== TreeMode clause

Controls the display of PDUs subtree of the GOP:

* _NoTree_: completely suppresses showing the tree
* _PduTree_: the tree is shown and shows the PDUs by PDU Id
* _FrameTree_: the tree is shown and shows the PDUs by the frame number in which
they are
* _BasicTree_: needs investigation

===== ShowTimes clause

Whether or not to show the times subtree of the GOP. If _TRUE_, the default,
the subtree with the timers is added to the GOP's tree. If _FALSE_, the subtree
is suppressed.

[#Gog]
==== GOG declaration block

===== _Gog_ declaration block header

Declares a Gog type and its candidate key.

----
Gog name {
    Member gopname (key); // mandatory, at least one
    Extra match_avpl; // optional
    Transform transform_list; // optional
    Expiration time; // optional, default 2.0
    GopTree [NoTree|PduTree|FrameTree|BasicTree]; // optional
    ShowTimes [TRUE|FALSE]; // optional, default TRUE
};
----

====== Gog name

The _name_ is a mandatory attribute of a _Gog_ declaration. It is chosen
arbitrarily, except that each _name_ may only be used once in MATE's
configuration, regardless the class of an item it is used for. The _name_ is
used to distinguish between different types of PDUs, GOPs, and GOGs. The _name_
is also used as part of the filterable fields' names related to this type of
GOG which MATE creates.

===== Member clause

Defines the _key_ AVPL for the GOG individually for each GOP type _gopname_.
All _gopname_ type GOPs whose _key_ AVPL matches the corresponding _key_ AVPL
of an active GOG are assigned to that GOG; a GOP which contains the AVPs whose
attribute names are listed in the GOG's corresponding _key_ AVPL, but they do
not strictly match any active GOG's _key_ AVPL, will create a new GOG. When a
GOG is created, the elements of its _key_ AVPL are copied from the creating GOP.

Although the _key_ AVPLs are specified separately for each of the Member
_gopname_++s++, in most cases they are identical, as the very purpose of a GOG is
to group together GOPs made of PDUs of different types.

===== Extra clause

If given, tells MATE which AVPs from any of the GOP's AVPL are to be copied
into the GOG's AVPL in addition to the GOG's key.

===== Expiration clause

A (floating) number of seconds after all the GOPs assigned to a GOG have been
released during which new GOPs matching any of the session keys should still be
assigned to the existing GOG instead of creating a new one. Its value can range
from 0.0 to infinite. Defaults to 2.0 seconds.

===== Transform clause

The _Transform_ clause specifies a list of previously declared _Transform_++s++ to
be performed on the GOG's AVPL after the AVPs from each new GOP, specified by
the _key_ AVPL and the _Extra_ clause's match_avpl, have been merged into it.
The list is always executed completely, left to right. On the contrary, the list
of _Match_ clauses inside each individual _Transform_ is executed only until
the first match succeeds.

===== TreeMode clause

Controls the display of GOPs subtree of the GOG:

* _NoTree_: completely suppresses showing the tree
* _BasicTree_: needs investigation
* _FullTree_: needs investigation

===== ShowTimes clause

Whether or not to show the times subtree of the GOG. If _TRUE_, the default,
the subtree with the timers is added to the GOG's tree. If _FALSE_, the subtree
is suppressed.

[#Transform]
==== Transform declaration block

A Transform is a sequence of Match rules optionally followed by an instruction
how to modify the match result using an additional AVPL. Such modification may
be an Insert (merge) or a Replace. The syntax is as follows:

----
Transform name {
    Match [Strict|Every|Loose] match_avpl [[Insert|Replace] modify_avpl] ; // may occur multiple times, at least once
};
----

For examples of Transforms, check the <<ChMateManual,Manual>> page.

TODO: migrate the examples here?

The list of Match rules inside a Transform is processed top to bottom;
the processing ends as soon as either a Match rule succeeds or all have been
tried in vain.

Transforms can be used as helpers to manipulate an item's AVPL before the item
is processed further. An item declaration may contain a Transform clause
indicating a list of previously declared Transforms. Regardless whether the
individual transforms succeed or fail, the list is always executed completely
and in the order given, i.e., left to right.

In MATE configuration file, a Transform must be declared before declaring any
item which uses it.

==== Settings configuration AVPL

[NOTE]
====
The *Settings* parameters have been moved to other configuration parameters
or deprecated. Leave for now until rest of document is updated for current syntax.
====

The *Settings* config element is used to pass to MATE various operational
parameters. the possible parameters are

===== GogExpiration

How long in seconds after all the GOPs assigned to a GOG have been released new
GOPs matching any of the session keys should create a new GOG instead of being
assigned to the previous one. Its value can range from 0.0 to infinite.
Defaults to 2.0 seconds.

===== DiscardPduData

Whether or not the AVPL of every PDU should be deleted after it was being
processed (saves memory). It can be either _TRUE_ or _FALSE_. Defaults to _TRUE_.
Setting it to _FALSE_ can save you from a headache if your config does not work.

===== DiscardUnassignedPdu

Whether PDUs should be deleted if they are not assigned to any GOP. It can be
either _TRUE_ or _FALSE_. Defaults to _FALSE_. Set it to _TRUE_ to save memory
if unassigned PDUs are useless.

===== DiscardUnassignedGop

Whether GOPs should be deleted if they are not assigned to any session. It can
be either _TRUE_ or _FALSE_. Defaults to _FALSE_. Setting it to _TRUE_ saves
memory.

===== ShowPduTree

===== ShowGopTimes

==== Debugging Stuff

The following settings are used to debug MATE and its configuration. All levels
are integers ranging from 0 (print only errors) to 9 (flood me with junk),
defaulting to 0.

===== Debug declaration block header

----
Debug {
    Filename "path/name"; //optional, no default value
    Level [0-9]; //optional, generic debug level
    Pdu Level [0-9]; //optional, specific debug level for Pdu handling
    Gop Level [0-9]; //optional, specific debug level for Gop handling
    Gog Level [0-9]; //optional, specific debug level for Gog handling
};
----

====== Filename clause

The  {{{path/name}}} is a full path to the file to which debug output is to be
written. Non-existent file will be created, existing file will be overwritten
at each opening of a capture file. If the statement is missing, debug messages
are written to console, which means they are invisible on Windows.

====== Level clause

Sets the level of debugging for generic debug messages. It is an integer
ranging from 0 (print only errors) to 9 (flood me with junk).

====== Pdu Level clause

Sets the level of debugging for messages regarding PDU creation. It is an
integer ranging from 0 (print only errors) to 9 (flood me with junk).

====== Gop Level clause

Sets the level of debugging for messages regarding PDU analysis (that is how do
they fit into ?GOPs). It is an integer ranging from 0 (print only errors) to 9
(flood me with junk).

====== Gog Level clause

Sets the level of debugging for messages regarding GOP analysis (that is how do
they fit into ?GOGs). It is an integer ranging from 0 (print only errors) to 9
(flood me with junk).

===== Settings Example
----
Action=Settings; SessionExpiration=3.5; DiscardPduData=FALSE;
----

==== Action=Include

Will include a file to the configuration.

----
Action=Include; {Filename=filename;|Lib=libname;}
----

===== Filename

The filename of the file to include. If it does not begin with '/' it will look
for the file in the current path.

===== Lib

The name of the lib config to include. will look for libname.mate in
wiresharks_dir/matelib.

===== Include Example
----
Action=Include; Filename=rtsp.mate;
----

This will include the file called "rtsp.mate" into the current config.