SUMMARY
It may be necessary to replace an AppObject Server Node with a higher-end machine, or rebuild the machine because it has hardware/software issues.
This Tech Note describes the procedure to replace or rebuild a running Redundant AppObject Server Node.
The procedure can be completed in a production environment with minimum or negligible impact to the production system. Further, data collection and AppObject execution should not be impacted during a rebuilding or replacement process.
Note: This Tech Note assumes that production data is provided for InTouch® or ActiveFactory applications.
Application Versions
Industrial Application Server 2.1 and later
SITUATION
For this Tech Note a simple load sharing setup is configured to execute AppObjects between two Redundancy-enabled AppEngine pairs that are hosted to Primary and Backup Nodes. After failover, AppObjects execution on the Primary Node transition to the Backup Node.
The following graphic shows a typical setup where load is shared between two Redundancy-enabled AppEngines. AppObjects are shared between two Redundancy-enabled AppEngines called RE1 and RE2.
Data gets published/subscribed to/from: AppObjects <---> RDI <---> SL <---> DAServer.
Note: In this example, SuiteLink and DAServers are hosted on Primary and Backup Nodes in this configuration. These objects can be hosted to other platforms in a Galaxy.
Figure 1: Deployment View of Redundant AppObject Nodes
ACTION
AppObject Server Node Replacement/Rebuild Process
The following information describes the specific rebuild tasks and their steps.
Verify Redundancy Status
Before initiating a force failover verify the following:
Verify that Redundancy Partner Status for RE1 and RE2 (RE1.Redundancy.PartnerStatus/RE2.Redundancy.PartnerStatus) is Standby-Ready as shown in the ObjectViewer.
Verify that the connection status for Redundant DI Object (RDI.ConnectionStatus) is Connected in ObjectViewer. This means the both primary and backup sources for RDI are in Good state and RDI failover can be initiated.
Verify the total object counts for RE1 and RE2 (RE1.Scheduler.ObjectCnt/RE2.Scheduler.ObjectCnt) in ObjectViewer. This number should be the same after failover.
Verify the total object offScan count for RE1 and RE2 (RE1.Scheduler.ObjectsOffscanCnt/RE2.Scheduler.ObjectsOffScanCnt) in ObjectViewer. This number should be the same after failover.
Trigger a Force Failover
Initiate a Force Failover using the Redundancy.ForceFailoverCmd for RE1 or RE2 in ObjectViewer.
Failover RE1 if Backup Node needs to be rebuilt or failover RE2 if Primary Node needs to be rebuilt.
Initiate a Force Failover for Redundant DI Object (RDI.ForceFailoverCmd) in ObjectViewer.
Failover to SL1 if the Backup Node needs to be rebuilt or failover to SL2 if the Primary Node needs to be rebuilt.
Verify the Data Updates After the Failover
After the failover, verify that your data is updating in InTouch and historizing in Active Factory Trend. This ensures that a successful failover has occurred and the entire load is now running either on the Primary or Backup Node.
Remove the Platform from the SMC
This task removes the platform in runtime.
From the SMC, expand the PlatformManager item and right-click the GalaxyName.
Click Remove Platform.
Note: This operation needs to be executed in the SMC running on the node that needs to be rebuilt or replaced.
Execute the steps on the Primary Node if the Primary Node needs to be rebuilt.
Execute the steps on the Backup Node if the Backup Node needs to be rebuilt.
Disconnect the Network
Disconnect the Primay or Backup Node that needs to be rebuit/replaced from the Primary and RMC networks. Once the node is replaced/rebuilt verify that it has same nodename and IPAddress, if the IPAddress is configured to be static.
Reconnect the Network to the Rebuild Node
Once the rebuilt/replaced node is in place, reconnect Primary and RMC networks and verify network communications are good by executing ping command via nodename and IPAddress.
Redeploy the Rebuilt AppObject Server Node
From the IDE, cascade redeploy either from Primary or Backup Node that has been rebuilt as shown in the above snapshot. This will deploy all children objects and the platform. Once the deployment is completed verify the redundancy partner status transitions to Standby – Ready.
Note: In Step 1, objects hosted to RE1 will get redeployed if the Primay Node is rebuilt. Objects hosted to RE2 get redeployed if the Backup Node is redeployed. There will be lost data for these objects until they get redeployed.
If the AppObjects’ redeployment is not feasible before you remove the Platform from the SMC, you can undeploy the primary engine (RE1 or RE2) by itself. In a Redundant environment, the primary engine can be undeployed leaving children objects deployed. Relocate the primary engine along with children objects to a dummy platform.
Dummy platform – create a new instance from $WinPlatform. Double-click it and assign a virtual nodename in the network address and virtual RMC Network Address. Save the changes. Once the rebuild process is completed, this platform can be deleted.
Once Step 2 is completed, you can re-assign this primary engine along with objects to this rebuilt/replaced platform and just deploy the primary engine after a cascade redeploy from the Primary or Backup Platform.
By doing this, objects will not get undeployed in Step 1. Once this operation is completed, the partner status should sync up and go to Standby Ready state.