Configuring Archive Module

An archive module allows the Zoom archive manager to delegate the actual archive/restore operation to a specific archive handler. The following are currently supported:

Archive Module	Type	Implemented	Description
Basic	Internal	Built-in	Supports archiving to a single file system path on the network
Path-specific Archive	Internal	Built-in	Similar to basic, plus the ability to map Zoom project vaults to different file system directories. This allows each brand/business unit to have its own archive storage
File System Archive	External	Zoom 7.3 onwards – using Hub (as Hierarchical Archive)	Supports archiving to external file systems and media using Evolphin’s Hub Server
S3/BlobStore	External	Until Zoom 7.2 – using scripts Zoom 7.3 onwards – using Hub (as Hierarchical Archive)	Supports Object Blob Storage or Cloud-based Storage such as Google Cloud Storage, AWS S3, Cloudian S3, or Microsoft Azure Blob
FrontPorch	External	Using scripts	Supports Oracle FrontPorch DIVA
SGL	External	Using scripts	Supports SGL FlashNet

This post explains the steps involved in setting up archiving to external systems like S3 / SGL / DIVA or dedicated File-System partitions. The external systems act as a secondary archive storage for the Zoom internal, and Zoom referenced (or indexed) assets.

Understanding the External Archive Flow

Archive Pre Hook Flow

1. The user initiates an archive from Asset Browser, the control passes to the archive pre-hook installed on Zoom server. Metadata is updated on the asset as shown in the diagram below. The hook is responsible for marking assets in a  “Pending ..” state for them to be archived by the Archive Cron timer task

Checkpointing

If the asset is a Zoom direct asset (stored in the Zoom database), it needs to be checkpointed before it can be archived. Checkpointing will ensure all the versions are stored in an “Asset ID” file that will be copied to the external archive. Assets that have native or mid-res proxy on an external storage such as a SAN, can be copied without checkpointing but even for these assets the low-res proxy is often checked-in into Zoom database as a direct asset. if so the low-res proxy will also need to be check-pointed in order to archive.

Archive Cron Flow

The archive cron timer task will run periodically and submit assets that are ready for archiving to the external archive system such as S3 or SGL. The assets need to meet the following requirements:

1. Metadata needs to indicate a “Pending..” status
2. Direct asset/low-res proxy in Zoom database representing the asset needs to be check-pointed

Monitor Flow (for SGL and other Job Queue bases external archive systems)

Understanding the External Restore Flow

Restore Pre Hook Flow

1. User initiates  a restore from Asset Browser, the control passes to the restore pre-hook installed on Zoom server. Metadata is updated on the asset as shown in the diagram below. The hook is responsible for marking assets in a  “Pending Restore..” state for them to be restored by the Archive Cron timer task

Archive Cron Flow

The archive cron timer task will run periodically and copy assets that are ready for restore from the external archive system such as S3 or SGL. The assets need to meet the following requirements:

1. Metadata needs to indicate a “Pending restore..” status

SGL Monitor Cron Flow

For SGL external archive system the following flow is used:

Steps to install and configure the Archive components

1. Install Perl library components.
2. Install the Archive-specific modules.
3. Configure the archive and restore pre-hooks 
4. Update the various configuration files for your deployment.
5. Configure the recurring archive timer – Cron for Linux and for Windows a timer. 

Installing the Perl libraries and modules

Review and follow the instructions for installing the Perl SDK. You only need Perl and the modules so just follow the instructions for “Perl Installation”. 

Installing the Archive Modules

Download the archive.zip file from the support site or your mail. 

After unzipping you should have the following folder structure. 

Configuring the Archive / Restore Pre-hooks

• Login to web-based admin console
• Open the “Server Control Panel” menu
• Launch the Archive Settings menu option
• Check “Enable Archive”
• Create a folder on your server file system and copy/paste that path into “Archive Location”. 
• Put the path for the “Pre-script for Archive” in the box. In our example we are using C:/zoom-deploy/ArchivePreHook/archive-hook.bat
• Put the path for he “Pre-script for Restore” in the box. in our example we are using C:/zoom-deploy/RestorePreHook/RestorePreHook.bat
• Leave both “Post-script” fields blank
• Set your email list if you are using one, you can add it at a later date.
• Set “Limit on Arguments on Command line” to 0. 
• Click on the fields for archive and restore pre-processing, and point them to the Archive and Restore pre-hook batch files (if Windows) or perl scripts (if Linux). See screenshot below.
• Save and restart server for the changes to take effect. 

Configuration Files

System-wide Common Parameters

For every config.ini, ArchiveCron, ArchivePreHook and RestorePreHook we have common sections; these are detailed below:

[ZOOMSERVER]

serverURL=	http://<your host IP address>:8880	IP and port for your server
webminURL=	http://<your host IP address>:8443	IP and port for your server
serverUsername=	<your admin account>	Zoom admin user account to use
serverPassword=	<your admin password>	Password for the above account

 

 [METADATA]

proxy=	<your metadata group>_Proxy	Indicates whether or not, an asset is a proxy for a high-res external asset. [VideoFX]
hiResPath=	<your metadata group>_NativeFilePath	Indicates the path where the high-res external asset is placed. [VideoFX]
hiResArchiveStatus	<your metadata group>_HiResArchStatus	Indicates the current status of the external archive operation on the high-res asset. [VideoFX]
directArchiveStatus=	<your metadata group>_DirectArchivetatus	Indicates the current status of the external archive operation on the Zoom assets. [In VideoFX setup, specifically for non-proxy files]
lowresArchiveStatus=	<your metadata group_DirectArchiveStatus	Indicates the current status of the external archive operation on the low-resolution proxy asset. [VideoFX]
midresArchiveStatus=	<your metadata group_MidResArhchiveStatus	Indicates the current status of the external archive operation on the mid-res asset. [VideoFX]

Archive Pre-Hook config.ini

[TO ARCHIVE] – this section tells the system which categories of files are to be archived. Set 0 = disable and 1 = enable. 

hires=	0 or 1	Enable to archive high-res assets
midres=	0 or 1	Enable to archive mid-res assets
lowres=	0 or 1	Enable to archive proxy of external high-res assets
direct=	0 or 1	Enable to archive assets directly ingested into the Zoom repository

ArchiveCron config.ini

[DEFAULT UNC path]

root_UNC_PATH

Specify the location that the assets must be copied to, if no dedicated path is defined for the project to which an asset belongs

[Project UNC map]

<Zoom Project Root Path>	Location where assets belonging to this project must be archived to

[ARCHIVE]

archiveModule	Can be set to “AwsS3”, “FileSystemArchive”, “SGL”

Zoom Server Metadata.xml

If you choose to follow our metadata recommendation, add the following to your metadata.xml and restart the server. 

The Cron or Timer Task

A key component of the external archive module is a script that needs to be periodically run continuously, without user intervention. To achieve this, we configure the target script using a Cron tab in Linux, or using the Windows Task Scheduler in Windows. See details below. 

For Linux

When set, this will cycle and call the archiveCron.pl causing archive and/or restore events to run. 

For Linux you set up cron for the user account and the following line can be copied and pasted into your crontab. 

Note: the above will run every minute

For Windows 

To setup a task scheduler,  run the following steps as administrator:

• Open the control panel
• Choose “System and Security”
• Under “Administrative Tools”, select “Schedule tasks” 
• In the “Task Scheduler”, select “Create Task”

• in the “General” tab seen above, do the following:
• Provide a Name, example “zoom archive” and description
• Enter a Location \ do not change
• Using the “Change User or Group…” Do not change
• Select elect the “Run whether user is logged on or not”, we want it to run all the time 
• Configure for <your version of the OS>
• Now select the “Triggers” tab: and use the “New…” button to create a new task

• Select “Begin the task” and select “At startup”
• Select the “Repeat task every” checkbox
• Choose your duration, change “5 minutes” to “1 minute”
• Change “for a duration of” to “Indefinitely”
• Select “Ok”
• Now using the Actions tab, we need to select the program to run:

• Select “Start a program”
• Browse to <Your root directory> and select the appropriate batch file. 
• Leave the next two, optional fields blank
• Select “OK”
• Restart the system and your task should be active