The first SAFR Server you install will automatically become the primary server. All subsequent servers you install will be secondary servers. There are two types of secondary servers:
With both types of secondary servers services such as feed management, reports, and the Web Console are not load-balanced and are always served from the primary server.
If there are at least two redundant secondary servers (three servers total), failover functionality is enabled. This means that if the primary server goes offline and both of the first two installed redundant secondary servers are still online, one of the redundant secondary servers will become the new primary server and the server cluster will continue to function as normal.
If additional redundant secondary servers are installed beyond the first two, database data will be replicated on them, but they don't count for the purpose of failover functionality.
Note: Object Storage Redundancy is only available on Windows and Linux.
The Object Storage Service is used for storing objects, such as profile and event images, as well as ephemeral data, such as event reply messages.
The service can operate in a redundant configuration when you have multiple SAFR servers running. All redundant secondary servers are load-balanced by the primary server for all Object Storage Service requests it receives.
Using shared object storage provides a shared location for each server to save and retrieve objects from. This provides each Object Storage Server with access to all of the objects, rather than just objects saved to their local storage.
Shared storage also provides an easier backup process, as you only have to run it from the primary server.
By default all redundant servers will save objects locally, and ask other Object Storage Servers for objects it does not have locally.
When you're using local object storage, you will lose access to all objects that are only stored by an offline Object Storage Server until the server becomes healthy again. If that server's objects are lost, and you do not have backups, they will be unrecoverable.
Backups must be run on every redundant server that has Object Storage enabled.
This section describes the functionality of a common large scale deployment configuration: an external load balanced system with 2 redundant secondary servers and a network-attached storage (NAS), a type of shared object storage.
When all the servers are working, this configuration has the following properties:
When one of the redundant secondary servers fail, the following occurs:
Impact:
When the primary server fails, the following occurs:
Impact:
If you start with local storage but later decide to move to shared storage, you will need to consolidate all of your objects to the new shared storage location, delete the local copies, and then mount the shared storage to the correct location. To do this, do the following:
python /opt/RealNetworks/SAFR/bin/backup.py
python /opt/RealNetworks/SAFR/bin/backup.py -o
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py"
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py" -o
/opt/RealNetworks/SAFR/bin/stop
"C:\Program Files\RealNetworks\SAFR\bin\stop.bat"
/opt/RealNetworks/SAFR/cv-storage
C:\ProgramData\RealNetworks\SAFR\cv-storage
/opt/RealNetworks/SAFR/cv-storage
C:\ProgramData\RealNetworks\SAFR\cv-storage
/opt/RealNetworks/SAFR/bin/start
"C:\Program Files\RealNetworks\SAFR\bin\start.bat"
On simple secondary servers, the Object Storage Service will operate in proxy mode.
Object Storage Servers operating in proxy mode will not attempt to use their own storage for objects, but will instead proxy the request to Object Storage Services that are running on either the primary server or on a redundant secondary server. If the redundant server it contacts doesn't have the object, the contacted redundant server will ask all other redundant servers for the object.
The list of servers that run the Object Storage Service is stored in the database and updated every minute. If a host does not respond within a timeout, it is de-prioritized.
On both the primary server and on redundant secondary servers the Object Storage Service stores new objects in storage.
When a server receives a request for a file it does not find in its storage, it will request the object from other Object Storage Servers via HTTPS, and return the object if found. (The same applies for DELETEs.) This allows multiple Object Storage Servers to operate without using shared network storage, with each server saving a subset of the total objects, and relaying requests for other objects to its neighbors.
Even when using shared network storage, sometimes a request will come in for a new object before it is visible to all systems on the shared storage. The Object Storage Service will ask all the other Object Storage Servers for the object until it finds one that has the object.
The SAFR backup and restore process when using shared network storage is straightforward - you just need to back up the primary server. This will back up all configs, database content, and Object Service Storage objects.
When using local storage, however, the objects are distributed to multiple servers, so the backup must be run on the primary server as well as all redundant secondary servers.
The primary server should run a regular backup, while the redundant secondary servers run an ‘objects only' backup. The difference is just the addition of the "-o" flag to the backup script.
When restoring multiple backups, you can restore them all to the primary server, or you can restore the ‘object only' backups back to the same servers that they were backed up from.
python /opt/RealNetworks/SAFR/bin/backup.py
python /opt/RealNetworks/SAFR/bin/backup.py -o
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py"
python "C:\Program Files\RealNetworks\SAFR\bin\backup.py" -o
python /opt/RealNetworks/SAFR/bin/restore.py BACKUPFILENAME
python /opt/RealNetworks/SAFR/bin/restore.py -o BACKUPFILENAME
python "C:\Program Files\RealNetworks\SAFR\bin\restore.py" BACKUPFILENAME
python "C:\Program Files\RealNetworks\SAFR\bin\restore.py" -o BACKUPFILENAME
Below are two example shared storage configurations.
Shared storage on Linux is very straightforward. Simply mount your shared storage to the /opt/RealNetworks/SAFR/cv-storage
location.
Stop SAFR.
/opt/RealNetworks/SAFR/bin/stop
Create a shared storage location. The example below uses Amazon's Elastic File System (EFS).
Edit /etc/fstab
to create a mount point of /opt/RealNetworks/SAFR/cv-storage
for your shared storage. The specific mount options should be provided by your specific storage service or device.
fs-12345678.efs.us-west-2.amazonaws.com:/ /opt/RealNetworks/SAFR/cv-storage nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0
Mount the remote share.
sudo mount -a
Start SAFR.
/opt/RealNetworks/SAFR/bin/start
Windows cannot mount a shared storage location directly to C:\ProgramData\RealNetworks\SAFR\cv-storage
. It must instead create a symbolic link by doing the following:
Stop SAFR.
"C:\Program Files\RealNetworks\SAFR\bin\stop.bat"
Create a shared storage location.
Delete the existing C:\ProgramData\RealNetworks\SAFR\cv-storage
by running rmdir /q /s C:\ProgramData\RealNetworks\SAFR\cv-storage
in an administrative command prompt. Deleting the existing cv-storage allows you to create a symbolic link from the cv-storage location to your shared storage location.
Note: Be sure you either followed the migration steps above to consolidate your objects onto the new shared storage location, or that you're doing this on a new system without any data.
Create the symbolic link from C:\ProgramData\RealNetworks\SAFR\cv-storage
to your shared storage location. To do this, run the appropriate command in an administrative command prompt:
If you're using a mapped network drive, run
mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage Z:\
If you're using an SMB share, run
mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage \\servername\share
Start SAFR.
"C:\Program Files\RealNetworks\SAFR\bin\start.bat"