Cleanup - 0.0.2

[libautomation.git] / README

# Status of this project

This is an early preview. I think the structure of the API is mostly
stable (ie the relationship of data types should not change radically),
but names might change for better consistency and semantics (how errors
are handled, side effects like logging, etc.) will change in subtle ways
too. And of course lot's of symbols will be added to make a consistent
and conventient interface.

What is currently completely missing, is algorithms for data processing
(eg filtering), standard feedback loops, etc. While these are clearly
within the scope of this project, it turns out many applications don't
need them. Therefore implementation (and API design) will happen later.
(Hopefully somehow fitting with the existing API.)



# What libautomation is: Design decisions and lessons learned

libautomation was created by abstracting common code from several
applications of industrial and home automation. It is assumed that this
code will be useful for writing more automation applications.

libautomation is intended to do as little as possible, relying on linux
functionalities and external libraries (like libev) as much as possible.
It is written entirely in C and every part of it should be easily
replaceable by custom code, if the application has special needs.

The goal here is to make easy tasks simple and complex projects
possible and easily managable.

As such libautomation is not only (maybe even not mainly) about the
code provided, but very much about the lessons learned from writing
automation applications: How to do things and what not to do because
it leads to problems down the road. These lessons have been turned
into design decisions.


## Use a system daemon instead of a watchdog

While it might seem tempting on embedded automation solutions to use the
watchdog directly in the main application, because this automatically
tests the entire software stack, this approach is too inflexible: Any
additional features need to be integrated into the main application,
which conflicts with trying to keep libautomation very modular.

Instead a system daemon (like procd from OpenWRT) should feed the
watchdog and in turn monitor all applications running on the system.


## An automation system should be resilent against crashing/hanging processes

libautomation aims for custom made (home brewn) automations solutions.
These often don't run on a 10k Euro PLC, yet might be used in an industrial
environment with high levels of EMI, that might cause occasional hardware
faults in cheap HW. It should be easy to write automation applications
in such a way, that the system can recover from such errors by restarting
processes or resetting the system.


## library functions might crash instead of fail

This might seem like an unconventional choice at first. But additional to
the remarks about reliability above, think about the following points:
 * Usually nobody attends the automation system to read error messages.
 * If a syscall returns an error, that can be handled, the library should
   do it. If it can't be handled, then restarting the application or the
   system seems to be a plausible fix.
 * The library API is a lot easier to use when functions cant fail ... ;)

Of course the above is only a guideline, not a principle. If in doubt
failing and not-failing versions of the same function should be provided.


## Multiple processes need to cooperate

There are many constraints on how an automation algorithm is split into
threads and processes:
* Some actions can cause interference with reading sensors, so you typically
  want to have everything in one thread, to control relativ timing of
  operations.
* However not every IO can be made non-blocking, e.g. reading a sensor
  via sysfs might block for a long time until it runs into some timeout.
  Therefore at least multiple threads are needed, if not multiple processes.
* Some input is expensive to read (might come over the network or a slow
  bus), so the effort should not be duplicated needlessly.
* Sometimes it becomes desireable two run the same thread twice, controlling
  two machines at the same times. When people write multithreaded applications,
  they often don't anticipate this case, and if they do, it usually is untested
  and more complicated then plain running the application twice.

To make it possible to meet all these contraints, clearly an ability to
share (sensor) data between multiple processes is necessary. To that end
libautomation provides a shared memory interface.



# What to find in the directory tree

The repository currently contains the sources and header files of the
library itself in the folder `lib' and some small demo applications,
that are either helpful utilities or small automation applications
in the folders `tools' and `examples'.


## Tools

### atmdump SHMID
The `atmdump' tool prints the contents of the libautomation shared memory
domain SHMID in human readable form to standard output. This is mostly
useful for ad-hoc testing and inspection, but also serves as a demo for
shared memory clients.

### atmd /path/to/configfile
The `atmd' (for libautomation daemon) reads any number of data source (ie
devices) from the configuration file specified on the command line and
makes their periodically updated values available via shared memory. This
is the demo for providing values in shared memory,

`atmd' is also a very important part of the libautomation ecosystem:
In the case where reading a sensor might block for an unacceptable long
time, `atmd' provides reading the sensor in a seperate process, without
any change to the primary application.

Furthermore `atmd' allows organizing data sources into so called
data source groups, which each can have their own policy regarding handling
of errors when reading a sensor. Each data source group lives in their
own config file. Config files can be loaded recursively with the
`dsgrp' keyword.


## Examples

### humiditycontrol /path/to/configfile
is a small air humidity regulator. It measures the relativ humidity and
temperature on two places (typically inside a building and out doors) and
calculates the absolute humidities. If the absolute humidity on the out side
is lower then insides, then a fan is turned on.

There are several configuration settings to control target values, allowed
energy loss when it is cold outsides, etc. This demo is actually useful
for real world applications and can easily get extended and customized to
specific needs, like adding a dehumidifier, etc.

### line_monitor /path/to/configfile
is a building block for an alert system. It monitors some input (typically
a gpio) indicating the status of some equipment. When the input changes
state, it triggers execution of some external programm like a script doing
a mobile call.

The alert command is executed periodically until either the error condition
is fixed or the operator acknowledges the error.

This application is actually used verbatim in a district heating plant.
Well, since I changed the libautomation API after deploying the system,
it isn't actually verbatim any longer ...



# API Overview

## Important data structures

### struct ATM_VALUE
A value (machine size integer) together with a timestamp, when the value
has been obtained.

### struct ATM_DS
Data Source: Descriptor how to obtain/update a single value. This is
mostly used internally. Users typically either update an ATM_VALUE
manually or have this managed completely by libautomation.

### struct ATM_DSGRP
A data source group is a list of data sources together with a policy, how
to handle errors on reading data sources. Some useful policies are
predefined, however it is possible to implement arbitrary policies in the
application via callback.

Also data source groups can be stacked by treating data source groups as
pseudo data sources. This allows building complex policies: E.g. you can
have an inner data source group, that retries reading a sensor three times
before failing, and an outer data source group, that resets the data bus
to recover from errors.

In the above example, you would have all sensors of the bus as children
of the data source group, to prevent sensor access while the bus is down.

### struct ATM_TASK
A task is a repeating timer together with a data source group and an
optional function. Every time the timer fires, all values in the data
source group are updated. As last step the optional function is called.

A task can be type cast to it's ev_timer and thus can be started and
stopped using the usual libev facilities.

libautomation automatically initializes a task with the global symbol
atm_main_task. This task is started when atm_main() is called.

### TODO: Write something about filters


## Shared memory interface

As noted above, sharing sensor data between applications is an important
requirement for automation applications. Therefore the shared memory
interface is a core part of libautomation and most features make use of
it.

Each shared memory region has an unique id. Typically an application will
create one shared memory region to export its data and connect to N other
memory regions for data input. The first shared memory region created, is
assigned to the global symbol `atm_shm_stdmem`.

Each shared memory region is organized as key-value database. Where keys
can be arbitrary strings and values are of type `struct ATM_VALUE`.

### Creating shared memory regions
`atm_shm_create(id)` creates a new shared memory region with unique id.
It is an error, if a memory regiion with the same id has already been
created by an other process. If `id` is NULL, then a memory region is
allocated on the heap instead, making the memory region effectifly private.

It is good policy to use the name of the config file as key. This is
under the assumption, that to instances of the same application surely
would need different config files, because they should not access the same
sensors direclty.

### Exporting locally calculated values
`atm_shm_register(shmr, key)` registers a new key-value-pair with shared
memory region `shmr` and returns a pointer to the value.

Use `atm_shm_update(struct ATM_VALUE *var, int value)` to automatically
update the timestamp with the value.

### Exporting sensor data
Registering a new data source with
`atm_ds_register(struct ATM_DSGRP *grp, const char *url, const char *key)'
automatically registers a key-value pair in `atm_shm_stdmem`.

### Reading data from shared memory
`atm_ds_register(struct ATM_DSGRP *grp, const char *url, const char *key)`
where `url` is of the form "shm:id/key" returns a pointer to the
associated `struct ATM_VALUE`. In this case, the value of `key` is
ignored and no key-value pair is exported.

Optionally there is also `atm_shm_get(const char *id, const char *key)`
with the same effect.


## Some notes on time keeping

There are three different conventions to handle time. Sorry for the
inconveniance:

### ev_time from libev
`ev_time` is a double precision floating point data type used by libev
and typically stores the time in something close to seconds since UNIX
epoch. The upside is: no overflow danger and good precision at the same
time. The downside is: floating point arithmetic. :-(

Since we are using libev, we have to use this.

### atm_time, atm_timestamp()
This uses native integer variables. `atm_timestamp()` returns the time
since booting the system in 1/10 seconds. This also is quite safe from
overflows (assuming at least 32bit integers) and precise enough to keep
track of typical hardware like relais.

`atm_time` is a global variable and automatically set from `atm_timestamp()`
each time an ATM_TASK is run. The idea is to have an rough estimate of the
current time without having to force a context switch (i.e. calling into the
OS) all the time.

### ATM_TIMER_RES
This macro is defined to 0.001, meaning one millisecond.

Data sources (or actually data source groups) return a positive integer
value when they need to get called again to complete their operation.
(E.g. because they had to reset some bus and need to wait for everything to
initialize.) This return value times `ATM_TIMER_RES` is the waiting time
until resuming the task.

This allows for more fine grained control then 1/10 of a second. Think of
1/10 second as the fastest sensible interval to repeat a task (but nothing
stops you, from repeating faster). Then obviously interruptions of tasks
have to support a much shorter time scale.



# Getting in touch

Libautomation is currently hosted at https://repo.or.cz/libautomation.git -
please ask me if you want push access. I'm easily reachable via e-mail.

If there is sufficient interest, I can open a mailinglist for this
project, but at the moment you need to send all questions and bug
reports to me personally.