Monitor Server¶
Monitor Server is component responsible for providing infrastructure necessary for multiple redundant component execution. Additionally, Monitor Server provides centralized registry of running components in single system which can be used for service discovery.
Warning
Monitor Server functionality is defined under assumptions:
components running as part of single system are trusted and implemented according to specifications
communication channels between components on single computing node are reliable
Redundancy mechanisms provided by Monitor Server do not provide reliability in case of possible component malfunctions due to implementation errors.
Running¶
By installing Monitor Server from hat-monitor package, executable hat-monitor becomes available and can be used for starting this component.
usage: hat-monitor [-h] [--conf PATH]
options:
-h, --help show this help message and exit
--conf PATH configuration defined by hat-monitor://server.yaml (default
$XDG_CONFIG_HOME/hat/monitor.{yaml|yml|toml|json})
Additionally, hat-monitor package provides implementation of library which can be used as basis for communication between components and Monitor Server. This library is available in hat.monitor.component module.
Communication model¶
Monitor Server provides n-node redundancy architecture based on chatter protocol (structure of communication messages is defined in HatObserver package). It is based on server-client communication between components and monitor server. There also exists horizontal peer communication between multiple monitor servers which enables forming of single system based on multiple distributed computing nodes. It is assumed that, for each system, single computing node runs single instance of Monitor Server and unlimited number of components that connect to local Monitor Server. Each node’s Monitor Server establishes communication with other nodes’ Monitor Server through single master Monitor Server. Hierarchy of Monitor Servers which can perform functionality of master is configured statically with configuration options available to each Monitor Server on its startup.
Entities participating in ‘vertical’ communication between components and local monitor server:
Client
Component proclaiming its existence to the server, discovering other components and participating in redundancy algorithm.
Server
Local monitor server providing global components state to all local clients and providing user interface.
Entities participating in ‘horizontal’ communication between Monitor Servers:
Master
Monitor Server responsible for execution of the redundancy algorithm and notifying other Monitor Servers about the current global state of the entire system.
Slave
Monitor Server responsible for notifying master about its current local state and delegating master’s notifications to its server entity.
Monitor Server uses two different independent listening sockets for client-server and master-slave communication.
If we represent components with Cn and Monitor Servers with Mn, where master hierarchy of M1 > … > Mn is presumed, an example of a single system monitor communication can be viewed as:
Component information¶
Component information is basic structure of properties that describe each component included in system. It is initially created on local Monitor Server and later updated by master Monitor Server. Collection of all components information associated with clients connected to local Monitor Server and calculated by local Monitor Server is called local state. Collection of all components information in single system calculated by master Monitor server is called global state. Each Monitor Server provides global state to its local clients.
Properties included in a component information:
cid
Component id assigned to client by its local Monitor Server.
mid
Monitor id identifying local Monitor Server (assigned to local Monitor Server by master). Value
0
indicates Monitor Server which is master or is not connected to remote master.name
User provided identifier of component. This entry is used for UI presentation purposes, logging and rank caching. It is recommended to use unique identifiers for each component instance. This property is assigned by client.
group
String identifier by which components are grouped while blessing calculation algorithm is applied (see Blessing algorithm). This property is assigned by client.
data
JSON serializable data representing arbitrary information that correspond to the component. This property is assigned by client.
rank
Component’s rank - used by Blessing algorithm. This property is initially assigned by local Monitor Server. Changes of this property value is available as part of local Monitor Server’s UI.
blessing_req
Blessing request assigned and changed exclusively by master Monitor Server (see Component lifetime). It consists of two optional properties:
token is optional number, used as unique token with the purpose of assigning blessing to the component for its primary functionality. When token is
None
, it means component is not blessed. Hereafter this token is called request token.timestamp is optional number that represents Unix epoch timestamp. It is strongly related to request token since corresponds to point in time when master Monitor Server assigned request token to the component. When token is
None
, master also sets timestamp to``None``.
blessing_res
Blessing response assigned and changed exclusively by client (see Component lifetime). It consists of two properties:
token is optional number, used as unique token as a client’s response to master’s blessing request token. When response token is set to exactly the same value as the request token, it means component is active, that is, it started providing its primary functionality. When component is no more active, it revokes this token by setting it to
None
. Hereafter this token is called response token.ready is boolean indicating whether component is ready to provide its primary functionality.
Master slave communication¶
Horizontal communication between Monitor Servers is hierarchically ordered.
Each Monitor Server knows its superiors’ addresses. If M1 > M2 > M3
,
then M1
doesn’t know any other monitor address; M2
knows the address
of M1
; M3
knows addresses of M1
and M2
in that order.
Each Monitor Server’s configuration contains zero or more other Monitor Server addresses. These other servers are “superior” to the monitor server. A monitor server will always try to maintain an active connection with exactly one of its superiors. The addresses list is ordered by priority meaning that if the Monitor Server isn’t connected to a superior, it tries to connect to the first monitor server in the list with connect_timeout. If the connection fails, it tries the second one and so on. If it can’t connect to any of its superiors, it waits for connect_retry_delay and retries again from the first monitor server in the list. It will retry connect_retry_count times before it can proclaim itself as master. The connecting to master process continues until the Monitor Server connects to its first superior, even if the Monitor Server is master or connection to some other superior is established. Connection parameters connect_timeout, connect_retry_delay and connect_retry_count are defined with configuration.
Once a slave Monitor Server connects to the Master Monitor server it sends its local state to the master and keeps notifying the master about any change in its local state while the connection is active. The master gathers all local states and generates its global state which it then transmits to all its slaves and keeps notifying them of any change. Global state contains information from all components received from local states except for those where component’s name or group are not set. Master also identifies each Monitor Server with unique monitor identifier (mid) which is provided to slave together with global state. It is important to note that only master Monitor Server calculates blessing request blessing_req for each component.
Every Monitor Server is responsible for starting master listening socket immediately after startup. While Monitor Server isn’t operating in master mode, all connections made to master listening socket will be closed immediately after their establishment - this behavior will indicate to connecting Monitor Server that its superior is not currently master.
Messages used in master slave communications are defined in HatMonitor SBS module (see Chatter messages). These messages are:
Message
Conversation
Direction
First
Last
Token
MsgSlave
T
T
T
s > m
MsgMaster
T
T
T
m > s
where s > m represents slave to master communication and m > s represents master to slave communication. When new connection is established, master should immediately associate new mid with connection and wait for MsgSlave sent by slave. After master receives MsgSlave and calculates new global state, it will send MsgMaster to slave. Once initial exchange of MsgSlave followed by MsgMaster finished, each communicating entity (master or slave) should send new state message (MsgMaster or MsgSlave) if any data obtained from MsgSlave or MsgMaster changes. Sending of MsgMaster and MsgSlave should be implemented independent of receiving messages from associated entity.
Server client communication¶
Vertical communication between client and server enables bidirectional asynchronous exchange of component information data. Client is responsible for providing name, group, data and blessing_res properties initially and on every change. Server provides global state to each connected client and each client’s component id (cid) and monitor id (mid). If any part of state available to server changes (including token changes), server sends updated state to all clients. Client can also request change for information provided to server at any time.
Messages used in server client communications are defined in HatMonitor SBS module (see Chatter messages). These messages are:
Message
Conversation
Direction
First
Last
Token
MsgClient
T
T
T
c > s
MsgServer
T
T
T
s > c
MsgClose
T
T
T
s > c
where c > s represents client to server communication and s > c represents server to client communication. When new connection is established, each communicating entity (server or client) server immediately sends initial state message (MsgServer or MsgClient) and should send new state messages when any data obtained from MsgServer or MsgClient changes. Sending of MsgServer and MsgClient should be implemented independent of receiving messages from associated entity. Implementation of server should not be dependent on receiving initial MsgClient and should continue sending MsgServer on every state change even if no MsgClient is received.
Server always sends last known global state calculated by master monitor server (even in case when connection to master is not established).
Client can close connection at any time. If server wishes to terminate connection, it should send MsgClose message to client. Once client receives MsgClose it should close connection as soon as possible (with possibility of local resource cleanup). If client doesn’t close connection after configured timeout, server can close connection.
Component lifetime¶
For most components, connection to local Monitor Server is mandatory for providing implemented functionality. Because of this, component usually connects to local Monitor Server during startup and preserves this active connection during entire component run lifetime. If this connection is closed for any reason, process also terminates. This behavior is not mandated.
Components which connect to Monitor Server participate in redundancy supervised by master Monitor Server. Redundancy utilizes two tokens, the one from blessing_req, said as request token, and the other from blessing_res, said as response token:
request token
This token is controlled exclusively by master Monitor Server. Master gives “blessing for work” to a component by setting this token to an integer number. On the other hand, it revokes blessing by setting this token to
None
. If connection to master is not established, token’s value equals toNone
.response token
This token is controlled exclusively by client. Upon receiving request token component starts providing its functionality and sets response token to match the request token in order to signalize its activity. If, at any time, component stops its activity, it revokes token by setting it to
None
.
While component information has request and response tokens with the same same value, it means component is active. If, at any time, component losses blessing, that is, master revokes request token, component starts with stopping its activity. When component activity is stopped it indicates it by revoking the response token.
On behalf of ready property of blessing_res, each component informs whether it is ready to provide its functionality based on global state provided by local Monitor Server. While component is not ready, one does not expect that it gets the request token. In any case, component that is not ready never sets its response token.
Responsibility of each Monitor Server is to cache last known blessing_req associated with it’s local components. This information is provided to master Monitor Server thus enabling transfer of blessing_req state in cases of master switchover procedures.
Blessing algorithm¶
Blessing algorithm determines value of each component’s request token and associated timestamp. This calculation is performed on master Monitor Server and should be executed each time any part of global state changes. This calculation should be integrated part of state change and thus provide global state consistency. That is, one can say that master Monitor Server blessed a component when it set its request token to an integer value.
Monitor Server implements multiple algorithms for calculating value of request token. Each component group can have associated different blessing algorithm and all groups that don’t have associated blessing algorithm use default algorithm. Group’s associated algorithms and default algorithm are provided to Monitor Server as configuration parameters during its startup.
Calculation of request token values is based only on previous global state and new changes that triggered execution of blessing algorithm.
Currently supported algorithms:
BLESS_ALL
This simple algorithm provides blessing to all components in associated group that are ready (ready flag of blessing_res is
True
). Blessing is revoked only when ready flag is set toFalse
.BLESS_ONE
In each group with this algorithm associated, there can be only one highlander and only one blessed component. Only components that are ready (ready flag of blessing_res is
True
) are considered as candidates for receiving blessing. In case there is no any ready component, this algorithm will not give blessing to any component.For determining which component receives the blessing, multiple ordered criteria are applied sequentially until there is only one component left. If any of the criteria is satisfied by more than one component, the next criteria is applied. Criteria are the following, respectively:
the mathematically lowest rank
request token previously set
the lowest blessing timestamp
the lowest mid
Finally, when algorithm defined the component to be blessed, if that component doesn’t already have request token equal to its response token, and there is another component in the group with response token, master does not bless any component. It waits until all the components from the group have revoked their response tokens, and only then, issues new request token to the chosen component. In case component to be blessed already has request token equal to its response token, master keeps its request token even if there exists a component with response token in the same group.
Components rank¶
Association of component’s rank is responsibility of component’s local Monitor Server for all of its local components. Monitor Server should associate same rank as was last rank value associated with previously active client connection with same name and group values as newly established connection. If such previously active connection does not exist, default rank value, as specified by Monitor Server’s configuration, is applied. After initial rank value is associated with client and its ComponentInfo, local Monitor Servers can later change rank’s value. These changes should be cached by local Monitor Servers in case connection to component is lost and same component tries to establish new connection. This cache is maintained for duration of single Monitor Server process execution and is not persisted between different Monitor Server processes.
User interface¶
As secondary functionality, Monitor Server provides web-based user interface for monitoring global components state and controlling component’s rank. Implementation of this functionality is split into server-side web backend and web frontend exchanging communication messages based on juggler communication protocol.
Server state¶
Backend provides to frontends all information that is made available by server to clients. This information is provided as continuously updated server state.
State structure is defined by JSON schema
hat-monitor://juggler.yaml#/$defs/state
.
Request/response¶
Juggler request/response communication is used primary for enabling user control of components’ ranks.
Request data structures are defined by JSON schema
hat-monitor://juggler.yaml#/$defs/request
.
In case of successful request execution, response data is null
.
Future features¶
Todo
optional connection to monitor/event server
mapping of current status to events
listening for control events
Implementation¶
Documentation is available as part of generated API reference:
Chatter messages¶
module HatObserver
MsgClient = Record {
name: String
group: String
data: String
blessingRes: BlessingRes
}
MsgServer = Record {
cid: Integer
mid: Integer
components: Array(ComponentInfo)
}
MsgClose = None
MsgSlave = Record {
components: Array(ComponentInfo)
}
MsgMaster = Record {
mid: Integer
components: Array(ComponentInfo)
}
ComponentInfo = Record {
cid: Integer
mid: Integer
name: Optional(String)
group: Optional(String)
data: String
rank: Integer
blessingReq: BlessingReq
blessingRes: BlessingRes
}
BlessingReq = Record {
token: Optional(Integer)
timestamp: Optional(Float)
}
BlessingRes = Record {
token: Optional(Integer)
ready: Boolean
}
Juggler definitions¶
$schema: "https://json-schema.org/draft/2020-12/schema"
$id: "hat-monitor://juggler.yaml"
$defs:
state:
type: object
required:
- mid
- local_components
- global_components
properties:
mid:
type: integer
local_components:
type: array
items:
type: object
required:
- cid
- name
- group
- data
- rank
properties:
cid:
type: integer
name:
type:
- string
- "null"
group:
type:
- string
- "null"
rank:
type: integer
global_components:
type: array
items:
type: object
required:
- cid
- mid
- name
- group
- data
- rank
- blessing_req
- blessing_res
properties:
cid:
type: integer
mid:
type: integer
name:
type:
- string
- "null"
group:
type:
- string
- "null"
rank:
type: integer
blessing_req:
type: object
required:
- token
- timestamp
properties:
token:
type:
- integer
- "null"
timestamp:
type:
- float
- "null"
blessing_res:
type: object
required:
- token
- ready
properties:
token:
type:
- integer
- "null"
ready:
type: boolean
request:
set_rank:
type: object
required:
- cid
- rank
properties:
cid:
type: integer
rank:
type: integer
Configuration¶
$schema: "https://json-schema.org/draft/2020-12/schema"
$id: "hat-monitor://server.yaml"
type: object
required:
- default_algorithm
- group_algorithms
- server
- master
- slave
properties:
type:
const: monitor
description: configuration type identification
version:
type: string
description: component version
log:
$ref: "hat-json://logging.yaml"
default_algorithm:
$ref: "hat-monitor://server.yaml#/$defs/algorithm"
group_algorithms:
description: |
keys represent group names
type: object
patternProperties:
".+":
$ref: "hat-monitor://server.yaml#/$defs/algorithm"
server:
$ref: "hat-monitor://server.yaml#/$defs/server"
master:
$ref: "hat-monitor://server.yaml#/$defs/master"
slave:
$ref: "hat-monitor://server.yaml#/$defs/slave"
ui:
$ref: "hat-monitor://server.yaml#/$defs/ui"
$defs:
server:
title: Listening Orchestrator Server
type: object
required:
- host
- port
- default_rank
properties:
host:
type: string
default: '127.0.0.1'
port:
type: integer
default: 23010
default_rank:
type: integer
master:
title: Listening Orchestrator Master
type: object
required:
- host
- port
properties:
host:
type: string
default: '127.0.0.1'
port:
type: integer
default: 23011
slave:
type: object
required:
- parents
- connect_timeout
- connect_retry_count
- connect_retry_delay
properties:
parents:
title: parent addresses
type: array
items:
type: object
required:
- host
- port
properties:
host:
type: string
port:
type: integer
default: 23011
connect_timeout:
type: number
connect_retry_count:
type: integer
connect_retry_delay:
type: number
ui:
title: Listening UI Web Server
type: object
required:
- host
- port
properties:
host:
type: string
default: '127.0.0.1'
port:
type: integer
default: 23022
algorithm:
enum:
- BLESS_ALL
- BLESS_ONE