Database and Object Storage Redundancy

Database Redundancy

The first SAFR Server you install will automatically become the primary server. All subsequent servers you install will be secondary servers. There are two types of secondary servers:

With both types of secondary servers services such as feed management, reports, and the Web Console are not load-balanced and are always served from the primary server.

Failover Functionality

If there are at least two redundant secondary servers (three servers total), failover functionality is enabled. This means that if the primary server goes offline and both of the first two installed redundant secondary servers are still online, one of the redundant secondary servers will become the new primary server and the server cluster will continue to function as normal.

If additional redundant secondary servers are installed beyond the first two, database data will be replicated on them, but they don't count for the purpose of failover functionality.

Object Storage Redundancy

Note: Object Storage Redundancy is only available on Windows and Linux.

The Object Storage Service is used for storing objects, such as profile and event images, as well as ephemeral data, such as event reply messages.

The service can operate in a redundant configuration when you have multiple SAFR servers running. All redundant secondary servers are load-balanced by the primary server for all Object Storage Service requests it receives.

Shared Object Storage (Network Storage)

Using shared object storage provides a shared location for each server to save and retrieve objects from. This provides each Object Storage Server with access to all of the objects, rather than just objects saved to their local storage.

Shared storage also provides an easier backup process, as you only have to run it from the primary server.

By default all redundant servers will save objects locally, and ask other Object Storage Servers for objects it does not have locally.

When you're using local object storage, you will lose access to all objects that are only stored by an offline Object Storage Server until the server becomes healthy again. If that server's objects are lost, and you do not have backups, they will be unrecoverable.

Backups must be run on every redundant server that has Object Storage enabled.

External Load Balancing (ELB) Walkthrough

This section describes the functionality of a common large scale deployment configuration: an external load balanced system with 2 redundant secondary servers and a network-attached storage (NAS), a type of shared object storage.

When all the servers are working, this configuration has the following properties:

Secondary Server Failure in an ELB Configuration

When one of the redundant secondary servers fail, the following occurs:

Impact:

Primary Server Failure in an ELB Configuration

When the primary server fails, the following occurs:

Impact:

Migrate from Local to Shared Storage

If you start with local storage but later decide to move to shared storage, you will need to consolidate all of your objects to the new shared storage location, delete the local copies, and then mount the shared storage to the correct location. To do this, do the following:

  1. Back up both the primary and redundant secondary servers to ensure you have a full backup of all SAFR content.
    • On Linux:
      • Primary: python /opt/RealNetworks/SAFR/bin/backup.py
      • Redundant Secondaries: python /opt/RealNetworks/SAFR/bin/backup.py -o
    • On Windows:
      • Primary: python "C:\Program Files\RealNetworks\SAFR\bin\backup.py"
      • Redundant Secondaries: python "C:\Program Files\RealNetworks\SAFR\bin\backup.py" -o
  2. Stop all primary and redundant secondary servers by using the stop command. This can be done by doing the following on each server:
    • On Linux: /opt/RealNetworks/SAFR/bin/stop
    • On Windows: "C:\Program Files\RealNetworks\SAFR\bin\stop.bat"
  3. Mount the new shared storage to a temporary location on primary and redundant secondary servers.
  4. Copy all files from the primary server and every redundant secondary server(s) to the temporary location of the shared storage. from within the following paths:
    • On Linux: /opt/RealNetworks/SAFR/cv-storage
    • On Windows: C:\ProgramData\RealNetworks\SAFR\cv-storage
  5. Delete or move the contents of the CV Storage folder on each primary and redundant secondary server as specified below.
    • On Linux: /opt/RealNetworks/SAFR/cv-storage
    • On Windows: C:\ProgramData\RealNetworks\SAFR\cv-storage
  6. Unmount the temporary location of the new shared storage.
  7. Mount the shared storage to the correct CV Storage location, or create a symlink to the shared storage location.
  8. Start the primary and redundant secondary servers by using the start command. On each server, do the following:
    • On Linux /opt/RealNetworks/SAFR/bin/start
    • On Windows "C:\Program Files\RealNetworks\SAFR\bin\start.bat"
  9. Disable any automatic backups on redundant secondary servers.
    • Now that you're using shared storage, only the primary server needs to be backed up. Disable any automatic backups you may have configured on your secondary servers.

Simple Secondary Server Behavior with Local Object Storage

On simple secondary servers, the Object Storage Service will operate in proxy mode.

Object Storage Servers operating in proxy mode will not attempt to use their own storage for objects, but will instead proxy the request to Object Storage Services that are running on either the primary server or on a redundant secondary server. If the redundant server it contacts doesn't have the object, the contacted redundant server will ask all other redundant servers for the object.

The list of servers that run the Object Storage Service is stored in the database and updated every minute. If a host does not respond within a timeout, it is de-prioritized.

Redundant Secondary Server Behavior with Local Object Storage

On both the primary server and on redundant secondary servers the Object Storage Service stores new objects in storage.

When a server receives a request for a file it does not find in its storage, it will request the object from other Object Storage Servers via HTTPS, and return the object if found. (The same applies for DELETEs.) This allows multiple Object Storage Servers to operate without using shared network storage, with each server saving a subset of the total objects, and relaying requests for other objects to its neighbors.

Even when using shared network storage, sometimes a request will come in for a new object before it is visible to all systems on the shared storage. The Object Storage Service will ask all the other Object Storage Servers for the object until it finds one that has the object.

Backup and Restore

The SAFR backup and restore process when using shared network storage is straightforward - you just need to back up the primary server. This will back up all configs, database content, and Object Service Storage objects.

When using local storage, however, the objects are distributed to multiple servers, so the backup must be run on the primary server as well as all redundant secondary servers.

The primary server should run a regular backup, while the redundant secondary servers run an ‘objects only' backup. The difference is just the addition of the "-o" flag to the backup script.

When restoring multiple backups, you can restore them all to the primary server, or you can restore the ‘object only' backups back to the same servers that they were backed up from.

Backup for Local Storage

Restore for Local Storage

Example Shared Storage Configurations

Below are two example shared storage configurations.

Linux

Shared storage on Linux is very straightforward. Simply mount your shared storage to the /opt/RealNetworks/SAFR/cv-storage location.

  1. Stop SAFR.

    /opt/RealNetworks/SAFR/bin/stop

  2. Create a shared storage location. The example below uses Amazon's Elastic File System (EFS).

  3. Edit /etc/fstab to create a mount point of /opt/RealNetworks/SAFR/cv-storage for your shared storage. The specific mount options should be provided by your specific storage service or device.

    fs-12345678.efs.us-west-2.amazonaws.com:/ /opt/RealNetworks/SAFR/cv-storage nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0

  4. Mount the remote share.

    sudo mount -a

  5. Start SAFR.

    /opt/RealNetworks/SAFR/bin/start

Windows

Windows cannot mount a shared storage location directly to C:\ProgramData\RealNetworks\SAFR\cv-storage. It must instead create a symbolic link by doing the following:

  1. Stop SAFR.

    "C:\Program Files\RealNetworks\SAFR\bin\stop.bat"

  2. Create a shared storage location.

  3. Delete the existing C:\ProgramData\RealNetworks\SAFR\cv-storage by running rmdir /q /s C:\ProgramData\RealNetworks\SAFR\cv-storage in an administrative command prompt. Deleting the existing cv-storage allows you to create a symbolic link from the cv-storage location to your shared storage location.
    Note: Be sure you either followed the migration steps above to consolidate your objects onto the new shared storage location, or that you're doing this on a new system without any data.

  4. Create the symbolic link from C:\ProgramData\RealNetworks\SAFR\cv-storage to your shared storage location. To do this, run the appropriate command in an administrative command prompt:

    • If you're using a mapped network drive, run

      mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage Z:\

    • If you're using an SMB share, run

      mklink /d C:\ProgramData\RealNetworks\SAFR\cv-storage \\servername\share

  5. Start SAFR.

    "C:\Program Files\RealNetworks\SAFR\bin\start.bat"

See Also