Voter Server Redundancy

Redundant Server Host with Automatic Failover
This is something that I have played with in a very basic sense but I have never properly implemented it. The only place I was able to find proper documentation on it is in the source code comments of the chan_voter.c module on GitHub. I happened to find this by accident when I was looking through the source code for the Asterisk Voter Channel Module. 
You will notice in the configuration of the RTCM/Voter Board an option for a secondary server. It seems simple enough, you simply need to setup a second host with an identical configuration with the exception of a different node number and the clients should stay on the Primary, and only switch to the Secondary if the Primary is lost and when the Primary comes back online again to switch back. Unfortunately in practice it doesn't work like this, what eventually happens it your system will "split" if some of the clients cannot see the primary but others can and it will stay that way unless you force them over to either one. 
However there is a proper way to have server redundancy and keep all your clients on the same page as it were. Like I said, I haven't tried this yet but I will be doing so in the future and documenting my findings here. In the mean time, here are the comments in the chan_voter.c source code to help understand how to do it:

Redundant "Proxy" Mode:
A "Redundant" (backup) server may be set up, so that if the "primary" server fails,
clients can detect this failure, and connect to the designated "backup" (or "secondary")
server.
Needless to say, since Internet connectivity is not by any means guaranteed to be consistent,
it is possible for some clients to have working connectivity to the "primary" server and not
others, even though the "primary" server is functional.
If this was to occur, actual voting and/or simulcast clients would have a "broken" system
(being that all the clients need to be on the same server for any sort of functional operation).
To eliminate this possibility, functionality has been added so that a "secondary" server
will "proxy" (forward) all of its VOTER packets to the "primary" (if the "primary" is
on line), and the "primary" will generate all of the outbound VOTER packets, which (for clients
"connected" to the "secondary" server) get sent to the "secondary" server to distribution to
its clients.
This allows for a "unity" of all of the clients on a network, even though they may be connected
to different servers.
In addition, it is assumed that "permanent linking" (at least of some sort) will be provided between
the channel side of the chan_voter instances (presumably through a "perma-link" provided by app_rpt).
When the "secondary" is "proxying" (to the "primary") it does not provide direct connectivity to/from
its locally-connected clients, thus allowing them to "connect" via the "primary" server instead. In
"normal" mode, it works "normally".
The operation is performed by more-or-less "encapsulating" the VOTER packets received by the "secondary"
server, and forwarding them on to the "primary" server, where they are "un-encapsulated" and appear to
that serer to be coming from clients connected directly to it (and keeps track of which ones are connected
in this manner, etc). When it needs to send VOTER packets to a client connected through the "secondary",
it "encapsulates" them, and sends them to the "secondary", where they get "un-enacpsulated" and sent
to their associated connected clients, based upon information in the "encapsulation".
If the "secondary" server loses (or does not make) connection to the "primary", it operates as normal, until
such time as it can make the connection.
The server redundancy feature is local to each chan_voter instance.
For each chan_voter instance served by both the "primary" and "secondary" servers, the client
list (parameters, etc) *MUST* be identical.
In addition, the following things must be added uniquely on each server:
In the "primary" server, there needs to be a "primary connectivity" client specified for each
"secondary" server for which it is "primary". Basicaly, this is a client that does NOTHING other
then providing a means by which the "secondary" can determine whether the "primary" is on line.
It is a stanard chan_voter client, with nothing else specified other then its password. Again,
although it is a "legitmate" client (technically), its only purpose *MUST* be to allow the secondary
server to connect to it.
The "primary" server also needs to have the following in all of its instances that require redundancy:
isprimary = y
The "secondary" server needs to have the following in all of its instances that require redundancy:
primary = 12.34.56.78:667,mypswd
(where 12.34.56.78:667 is the IPADDDR:PORT of the "primary" server, and mypswd is the password of the
"primary connectivity" client)
Note: Master timing sources *MUST* be local to their associated server, and therefore, can not be operated
in a redundant configuration. If a radio needs server redundancy, it CAN NOT be connected to a master timing
source. Also, the master timing source MUST be associated with a chan_voter instance that DOES NOT have
redundancy configured for it, even if a separate instance needs to be created just for this purpose.
Also, if Non-GPS-based operation is all that is needed, just the use of redundancy within the clients is
sufficient, and does not require any use of the server redundancy features.