Add Secondary Servers
The first SAFR Server you install will automatically become the primary server. All subsequent servers you install will be secondary servers. There are two types of secondary servers:
- Simple: Does not replicate the database data.
- Redundant: Replicates database data from the primary server, possibly providing failover functionality. (See the Database and Object Storage Redundancy topic for details.)
Note: Only Windows and Linux SAFR Servers can become redundant secondary servers.
If your system is connected to the Internet, do the following to add a secondary server:
- Download and install SAFR Platform on the additional machine.
- Log in to the SAFR auto-discovery process:
- Connect the Desktop Client to the primary server (for macOS and Windows) as described here.
- Connect your Web Console to the primary server as described here.
- During auto-discovery, the following automatically happens:
- The secondary server contacts a SAFR Licensing Server in the cloud to acquire a license.
- The SAFR Licensing Server authenticates the SAFR account credentials.
- The SAFR Licensing Server identifies the license and deployment type.
- A suitable license is returned to the secondary server and information about the primary server is returned to the secondary server, including the hostname.
- If your new secondary server is on a Windows or Linux machine, you will be prompted to choose which kind of secondary server you want: simple or redundant. If your new secondary server is on a macOS machine no prompt will occur; macOS secondary servers are always simple.
- Auto-discovery will now continue, with the following automatically occurring:
- The secondary server re-configures itself to reference the primary server.
- The secondary server registers itself with the primary server.
- The primary server updates its local database.
- If you're using a Software-Based Load Balancing Configuration, the primary server now adds the new secondary server to its load balancer configuration and uses it as an additional node in its cluster.
If you are not connected to the Internet, you can still connect your new secondary server to the primary server, but the auto-discovery process is not available. You must instead manually configure the newly installed secondary server to locate the primary server. If your new secondary server is on a Windows or Linux machine, you'll need to choose which kind of secondary server you want: simple or redundant. If your new secondary server is on a macOS machine no such decision is required; macOS secondary servers are always simple.
Download and install SAFR Platform on the second machine.
Run the safr-worker script on your secondary server by doing the following:
For macOS:
Open Terminal.
Run the following command, substituting the primary SAFR hostname for HOSTNAME:
sudo /Library/RealNetworks/SAFR/bin/safr-worker HOSTNAME
For Windows
On the primary server record the contents of C:\ProgramData\RealNetworks\SAFR\mongo\.adminpass
and C:\ProgramData\RealNetworks\SAFR\mongo\mongod.keyfile
On the new secondary server, open a command prompt by right-clicking on the Start menu, selecting Run, and entering cmd
.
If you want it to be a simple secondary server, in the new command prompt run the following command, substituting the password from .adminpass in Step 1 for PASSWORD and the primary server hostname for HOSTNAME:
python "C:\Program Files\RealNetworks\SAFR\bin\safr-worker.py" -p PASSWORD HOSTNAME
OR
If you want it to be a redundant secondary server, in the new command prompt run the following command, substituting the mongod.keyfile
contents from Step 1 for KEYFILE, the password from .adminpass in Step 1 for PASSWORD, and the primary server hostname for HOSTNAME:
python "C:\Program Files\RealNetworks\SAFR\bin\safr-worker.py" -s KEYFILE -p PASSWORD HOSTNAME
For Linux
On the primary server, record the contents of /opt/RealNetworks/SAFR/mongo/.adminpass
and /opt/RealNetworks/SAFR/mongo/mongod.keyfile
On the primary server, open /opt/RealNetworks/SAFR/virgo/config/virgo-factory.conf
Within that file, look for a section that looks something like:
"global":{
"environment": "CUSTOM",
"user-id": "ubuntu18int2tst",
"user-encrypted-password": "%qy4Effq2cxYUrEopuIFSY3LE22hDBMOG6NdeqTWfok4=",
"status-interval":5000,
"remote-control-enabled":true
},
Record the "user-id" and "user-encrypted-password" values. These will be your SAFRUSER and SAFRPASSWORD values in the steps below.
If you want it to be a simple secondary server, on the new secondary server run the following command.
sudo python /opt/RealNetworks/SAFR/bin/safr-worker.py -p PASSWORD -u SAFRUSER -x SAFRPASSWORD HOSTNAME
where:
- PASSWORD = the password from .adminpass in Step 1
- SAFRUSER = "user-id" from Step 4. Don't include the enclosing double quotation marks.
- SAFRPASSWORD = "user-encrypted-password" from Step 4. Don't include the enclosing double quotation marks.
- HOSTNAME = the primary server hostname
OR
If you want it to be a redundant secondary server, on the new secondary server run the following command.
sudo python /opt/RealNetworks/SAFR/bin/safr-worker.py -s KEYFILE -p PASSWORD -u SAFRUSER -x SAFRPASSWORD HOSTNAME
where:
- KEYFILE = the contents of
mongod.keyfile
from Step 1
- PASSWORD = the password from .adminpass in Step 1
- SAFRUSER = "user-id" from Step 4. Don't include the enclosing double quotation marks.
- SAFRPASSWORD = "user-encrypted-password" from Step 4. Don't include the enclosing double quotation marks.
- HOSTNAME = the primary server hostname
When attempting to join a new secondary server, you might encounter the following error messages:
System is offline |
Network or system connectivity issue. Attempt to access the system at a later time. |
SAFR master host is not reachable |
Ensure all servers are connected to the same network and try again. |
Improperly configured SSL certificate |
Installing non self-signed SSL certificates is recommended when setting up multiple servers. See the SSL Certificates page for information about how to install an SSL certificate. |
Secure connection error. Check server for valid SSL certificate |
Installing non self-signed SSL certificates is recommended when setting up multiple servers. See the SSL Certificates page for information about how to install an SSL certificate. |
Incomplete server connection |
Attempt to join again; a persistent issue may require either uninstalling and reinstalling SAFR Platform on your servers or contacting your SAFR support representative. |
- At startup each server, both primary and secondary, registers itself by posting its status to the database on the primary server.
- The primary server directs requests to all secondary servers in a least connection method that keeps the load evenly balanced among all secondary servers.
- As long as a given secondary server remains healthy, the primary server keeps that secondary server in its load balance rotation.
- Status information about all secondary servers is stored in the database on the primary server.
- Every minute all servers (the primary server as well as the secondary servers) send a status update to the database on the primary server.
- Every five seconds, the primary server attempts to ping all servers (the primary server as well as all secondary servers) via the SAFR heath check API.
- If the health check fails for a given secondary server for 15 seconds (i.e. for 3 health check API calls in a row), that secondary server is removed from load balancing rotation and face recognition requests are no longer routed to it. If the health check succeeds for the removed secondary server for ten seconds (i.e. for 2 health check API calls in a row), the secondary server is returned to the load balancing rotation and resumes accepting face recognition requests.
- If a secondary server's status has not been reported for over five minutes, it is removed from the load balancer configuration. In this case, it is no longer sent face recognition requests or health check API calls.
- If a secondary server has been pulled out of rotation for not responding to health checks, or is removed from the load balancer configuration for not reporting status for more than five minutes, it can still be put back in rotation through any of the following:
- If a network interruption prevents the secondary server from sending a request, the secondary server continues to send a status update at its regularly scheduled interval after it goes back online and its status is updated in the primary server.
- If the secondary server is restarted, it sends a status update after all services are started and ready.
- If the secondary server IP address is changed, the secondary server must be manually restarted to force it to send a status update to the primary server with the new IP address.