monitor: refactor/rewrite to use new async API
In order to migrate to the upcoming Danga::Socket-based
HTTP API, we'll first refactor monitor to use the new API
(but preserve LWP usage behind-the-scenes).
DEBUG=1 users will see the elapsed time for all device refreshes
each time monitor runs.
While we're at it, also guard against race conditions on the
PUT/GET test by double-checking on failure. (A long-standing
TODO item)
also squashed the following commit:
use conn_timeout in monitor, node_timeout in other workers
This matches the behavior in MogileFS:Server 2.65.
It makes sense to use a different, lower timeout in monitor to
quickly detect overloaded nodes and avoid propagating their
liveness for a monitoring period.
It also makes sense to use a higher value for node_timeout in
other workers since other actions are less fault-tolerant.
For example, a timed-out size check in create_close may cause a
client to eventually reupload the file, creating even more load
on the cluster.