BIG documentation clear out and clean up. Still more work to do, but this is a big...

[freeems-vanilla.git] / docs / migrate / latex / SerialProtocolCoreSpecification.tex

\documentclass[12pt,a4paper,titlepage]{article}
\include{FreeEMS}

\begin {document}

\begin{titlepage} 
\begin{center}

\includegraphics[width=1\textwidth]{./logos/freeems}\\[1cm] \vfill

\HRule \\[0.8cm]
{ \huge \bfseries Serial Protocol -- Core}\\[0.4cm]
\HRule \vfill

\Large \emph{Author:}\\
\textsc{Fred Cooke} \vfill

{\large Version 0.7.0 --- Saturday 1\textsuperscript{st} January, 2011}

\include{FreeEMStoc}




\section {Foreword}

The purpose of the protocol is to provide reliable two-way serial communication
with FreeEMS and associated devices over multiple physical protocols. This will
commonly be used for live tuning, configuration changes, firmware updates and
data acquisition of various types.

This document covers the basic data transfer protocol, which all
implementations should adhere to. It describes the methods through which
various functions are achieved, right down to the hardware, at all but the
application level.

Each implementation will provide documentation on the device-specific
functionality offered and the payload data formats used. Not all devices
will support all functions, either because they have limited functionality
(a display device for example), or because the software versions may differ.
Where logging/debug facilities are available, the payload type of unrecognised
packets should be noted before discarding them.




\section {Main Attributes}
\begin{itemize}
\item Compact binary data format to maximize speed and bandwidth.
\item Data is packetized with an escaping scheme.
\item Each packet consists of a header, payload and checksum.
\item Compulsory integrated low-cost checksum.
\item Compulsory acknowledgment of received packets.
\item Optional sequence numbers
\item Big endian multi byte number format
\item Compulsory core functionality across implementing devices
\end{itemize}




\section{Packet Description}

\subsection{Packet Data Format}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline Packer header: 3 -- 6 bytes & Packet payload: 0 -- max bytes & CKS \\ 
\hline 
\end{tabular} 
\end{center}

A packet consists of a three to six byte header, a payload of variable length
from zero to implementation maximum payload size and a one byte checksum. This
format will be encapsulated in various lower level constructs depending on the
medium of transmission.

\subsection{Packet Header Layout}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline  Flags & Payload ID & Seq No. & Payload Length \\
\hline 
\end{tabular} 
\end{center}

\subsection{Packet Header Specification}

\begin{center}
\begin{tabular}{|l|l|l|}
\hline Byte sequence & Length & Data \\ 
\hline 1 & 1 & Header flags \\
\multirow{8}{*}{} & \multirow{8}{*}{} & Bit 0 - Has Length \\
&& Bit 1 - Is Negative Acknowledgment \\
&& Bit 2 - Has Sequence \\
&& Bit 3 - Reserved \\
&& Bit 4 - Reserved \\
&& Bit 5 - Reserved \\
&& Bit 6 - Reserved \\
&& Bit 7 - Reserved \\
\hline 2,3 & 2 & Payload ID \\ 
\hline 4 & 1 (opt) & Sequence number \\ 
\hline 4,5 or 5,6 & 2 (opt)* & Length of payload (excluding header/checksum) \\ 
\hline 
\end{tabular} 
\end{center}
  * Required for some packet types
  
\subsection{Payload}

Packets are just containers and the payload data is specific to the payload
type being sent. See Payload ID.

\subsection{Checksum}

The checksum is a simple 8 bit type as the EMS deals with bytes during uart
transmission. It is compulsory to include the checksum in each packet. It is
located at the end of the payload to ease load on the EMS during send and
receive operations and keep the code simple and compact. The checksum covers
all bytes in the message before any escaping and packetising occurs, including
the header but not including the start and stop delimiter bytes.




\section{Header Details}

\subsection{Protocol Packets}

Payload IDs of less than \texttt{256}/\texttt{0x0100} (\texttt{0x0000} -
\texttt{0x00FF} / \texttt{0} - \texttt{255}) are of type ``protocol'' and
defined payload IDs should be implemented in all devices using this protocol in
their application. (See appendix~\ref{app:payload})

All payload IDs of 256/0x0100 or more when the payload type bit of the header
flags is clear (\texttt{0}) the payload type is of the type ``firmware''. The firmware
payload IDs are covered in the FreeEMS Vanilla Specific protocol specification,
and are optional.

\subsection{Addressing Behaviour}

Addressing of packets will be handled by wrapping them in other protocols of
a different layer at a higher level. UART is a point to point protocol and
thus it does not make sense to use addressing at this layer of the protocol.
Wrapping could occur on CAN, SPI, Ethernet and other addressable bus type
protocols.

\subsection{Acknowledgments}

All packets received are responded to with an acknowledgement. The
acknowledgement payload ID is one greater than whatever was received. In the
header of the acknowledgement packet is a bit that specifies whether it is a
positive or negative acknowledgement. In the case of success, the payload will
either be empty (default) or contain the requested data in the format specified
for that payload ID. In the case of failure the payload contains a 16 bit error
code.

\subsection{Payload ID}

The payload ID is a unique identifier number that indicates the functionality
to be delivered by a request packet. All payload request IDs are to be even
and all response packets are the ID of the request plus one. See Apendix A for
protocol payload ID definitions.

\subsection{Sequence Numbers}

The protocol provides a facility for packets to be identified by sequence
number as well as just type and timing. If a sequence number is supplied and
flagged it will be returned just the same. Asynchronous packets will not carry
sequence numbers, and it is perfectly acceptable to not use them at all. They
are provided as an option for tuning authors to use.

\subsection{Length Of Payload}

Length from the end of the header to the end of the payload NOT including the
checksum, header or packet delimiters, and before escaping occurs.




\section{General Information}

\subsection{Flow Control}

Upon receiving a packet that requires a response the EMS will be responsible
for disabling receipt of further requests by disabling the receive register
full interrupt. By this mechanism it can ignore subsequent requests until the
current packet is dealt with and the receive buffer free again. The external
device should expect messages to be ignored up until the first byte of the
response packet comes back. If datalog streaming is enabled it will not be
possible to determine whether or not the packet is the response until it is
completely received. Instead, it is probably better to wait for some short
period after a packet is transmitted and delay sending further packets until
after that period is over. No harm can be done by sending packets while the
device is not listening, they will just be ignored. RS232 hardware flow control
is not used as it is rarely implemented correctly or even at all in common
physical implementations.

\subsection{Reliable Transmission}

In all cases the external software will be responsible for ensuring
unacknowledged messages sent to the EMS are resent. The EMS will work on a
``best effort'' basis to handle all requests promptly, however that can not be
guaranteed.

Several generic methods of recognising bad data exist:

\begin{itemize}
\item Checksum does not match that supplied.
\item Packet length in the header (if present) does not match the actual length received.
\item Packet is larger than the maximum allowed size.
\item The payload length does not match that of the payload ID (where fixed and defined)
\end{itemize}

In addition, the specifics of how each different physical layer operates will
provide further means of recognising bad packet data as will the different
payload types.

\subsection{Packet Size}

Packet size is only limited by the size of the payload size field and the
specific implementations hardware software setup. Currently FreeEMS Vanilla
only requires ~2k packet size to be handled, however up to 64k is supported by
the 16 bit size field. This will ensure it stays useful for a longer time and
will therefore allow tool re-use too. If larger data blocks need to be sent in
some future FreeEMS implementation then the data can be packetised down inside
this format without much difficulty. Each implementation should provide the
facility to request the maximum permissible packet size.

\subsection{Protocol Layers}

\begin{itemize}
\item Physical layers: UART over RS232 or USB, CAN, SPI, I$^2$C, etc
\item Data layer (UART): escaping/start/stop/bit format/etc
\item Data layer (CAN): yet to be determined
\item Packet layer: generic across all mediums
\item Application layer: device/firmware specific
\end{itemize}

\subsection{Response Times}

To be defined once implemented and tested. Thanks EdG!




\section{UART Specifc Information}

\subsection{Packet Data Format}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline  \texttt{STX} & Data Packet (Header Payload and Checksum): 4 -- max bytes & \texttt{ETX} \\
\hline 
\end{tabular} 
\end{center}

This is a diagram of a single packet of data, in a serial stream. \texttt{STX} and
\texttt{ETX} are single bytes that mark the start and end of the packet (the escaping
scheme). Should the data flow be interrupted, the data flow can be picked back
up again, by looking for the \texttt{STX} byte. This escaped and delimited packet scheme
affords a number of opportunities to catch corrupt data as seen in the Reliable
transmission section.

\subsection{Bitwise Byte Format}
\begin{center}
\begin{tabular}{|c|c|c|c|}
\hline  Start & 8 data bits & Parity & Stop \\
\hline 
\end{tabular} 
\end{center}

Low level 9 bit data format consisting of 11 bits total. One start bit (low),
8 data bits, one hardware generated odd parity bit, one stop bit (high). See
MC9S12XDP512V2.pdf section 11.4.3 page 495

\subsection{Escaping Scheme}

\begin{center}
\begin{tabular}{|c|c|c|}
\hline Description & Byte value & Escaped pair \\
\hline Start byte (\texttt{STX}) & \texttt{0xAA} & \texttt{0xBB} \texttt{0x55} \\
\hline Escape byte (\texttt{ESC}) & \texttt{0xBB} & \texttt{0xBB} \texttt{0x44} \\
\hline End byte (\texttt{ETX}) & \texttt{0xCC} & \texttt{0xBB} \texttt{0x33} \\
\hline
\end{tabular} 
\end{center}


All packets start with \texttt{STX}, and end with \texttt{ETX}. Should any of
\texttt{STX}, \texttt{ETX} or \texttt{ESC} occur within the packet contents
(including the header and checksum), it must be ``escaped'', with a preceding
\texttt{ESC} byte and the byte itself XOR'ed with the value 0xFF. This will
yield the following easy to recognise pairs in the stream:

If the character \texttt{0xBB} is found in the raw stream without
\texttt{0x55}, \texttt{0x44} or \texttt{0x33} following it, the stream and
therefore packet is corrupt and invalid and should be discarded.

\subsection{UART Specific Error Detection}

This section is supplementary to the things mentioned in the reliable delivery
section above. The following things should be checked for when interacting over
a UART connection of any sort:

\begin{itemize}
\item Start byte found before stop byte while inside a packet.
\item Escape byte not followed by an escaped special byte.
\item Parity bit errors at the byte level.
\item Framing errors
\item Overrun errors
\item Noise errors
\end{itemize}

Note, not all of these signals will be available on all devices, but whatever
is available should be used to ensure a highly robust serial solution is
implemented. After all, the EMS code can only confirm that the received data is
in good condition, it can not be responsible for the integrity of the data it
sends out as it could easily be corrupted in transit.

Note: The checksum doesn't cover the start, stop, escape or escaped bytes, only
the original packet data.




\section{CAN Specific Information}

Yet to be determined.




\section{Payload IDs}
\label{app:payload}
\footnotesize
\begin{tabular}{|l|p{2cm}|l|l|p{7cm}|}
\hline ID       & Name & Type & Size & Description and format \\
\hline 0        & Interface Version & Request & 0 & The unique number that identifies the firmware specific interface definition. If this is the same, there should be absolutely no difference in communications. Useful to allow tuning tools to support interface version ranges based on common functionality and different firmwares with the same interface type.\\
\hline 1        & Interface Version & Response & 16 - 256 & Details on the semantics of handling and using this field are not yet defined. For now it can simply be used to identify whether the API and data layout are AT ALL changed from previous versions. That is not good enough in the long term, though.\\
\hline 2        & Firmware Version & Request & 0 & The unique string that identifies the firmware version running on the device. Useful for associating behaviours, issues and bugs with specific code versions.\\
\hline 3        & Firmware Version & Response & 16 - 256 & An arbitrary string 16 to 256 bytes long uniquely describing the firmware version running on the device.\\
\hline 4        & Max Packet Size & Request & 0 & Ask the controller to tell us how large in bytes the maximum packet it can handle is. The size is measured from the end of start byte to beginning of stop byte including the header, payload and checksum regions.\\
\hline 5        & Max Packet Size & Response & 2 & 16 bit unsigned integer packet size.\\
\hline 6        & Echo Packet & Request & Any & Wrap and return the entire packet as the payload.\\
\hline 7        & Echoed Packet & Response & Any & The wrapped packet returned.\\
\hline 8        & Soft Reset * & Command & 0 & Instructs the device to software reset itself.\\
\hline 9        & $<$special case$>$ & Response & 2 & Only if the request is malformed you will get an error back.\\
\hline 10       & Hard Reset * & Command & 0 & Instructs the device to hardware reset itself. (complete)\\
\hline 11       & $<$special case$>$ & Response & 2 & Only if the request is malformed you will get an error back.\\
\hline 13       & Error/Exception & Assertion & 2 & 16 bit unsigned integer error number.\\
\hline 15       & Debug Data & Assertion & Any & Free form data of any type for debug purposes.\\
\hline
\end{tabular}\\
* Note, soft and hard reset behaviour is implementation
dependent and both may behave the same way as each other.
\normalsize


\subsubsection*{ID}
Payload ID number which uniquely identifies the purpose and format of the data
contained in the payload.

\subsubsection*{Type Key:}
\begin{itemize}
\item Request - A message sent from a tuning device or similar to an embedded
device or similar. Typically PC $\rightarrow$ EMS
\item Response - The reply to a request from an embedded device or similar to a
tuning device or similar. Typically EMS $\rightarrow$ PC
\item Command - A message sent from a tuning device or similar to an embedded
device or similar. Typically PC $\rightarrow$ EMS
\item Assertion - A message sent from an embedded device or similar to a tuning
device or similar. Typically EMS $\rightarrow$ PC
\end{itemize}

\subsubsection*{Size}
Payload size is in bytes from end of header to beginning of checksum.

\subsubsection*{Description And Format}
The purpose of this particular payload type and how the data is laid out.

Note: This table is liable to be extended in subsequent
versions.




\section{Copyright}

Copyright 2008--2011 Fred Cooke. All rights reserved.




\end {document}