GoldenGate – Clusterware Installation with RAC

In this mini series of blog entries I’m demonstrating GoldenGate installation on a cluster with a RAC database, with Oracle Dataguard also thrown into the mix.  This first blog entry just covers the clusterware and RAC side.

I’ve set up a demonstration environment as follows –

  • Two node RAC cluster with nodes exa1db01, exa1db02.
  • Database named “ordersA” which will be the OLTP type order entry database that is the source for our GoldenGate configuration.
  • Database named “reportA” which is the data warehouse database that will receive data via GoldenGate from the ordersA database.
  • Both run on the exa1 cluster.
  • I’ll be using GoldenGate 12.1.2.1.0 on Linux 64 with GI 12.1.0.2 and RDBMS 11.2.0.4.

Install Oracle Grid Infrastructure Standalone Agents

The first thing I’d like to do is install the latest XAG agents.  This is what will control the GoldenGate processes at a cluster level.  A version of the XAG agents is shipped with the GI home, but it is generally quite old so we’ll benefit from using the latest version.  However, we can’t install this into the GI home so it goes elsewhere.

We download the latest version from –

http://www.oracle.com/technetwork/database/database-technologies/clusterware/downloads/index.html

At the time of writing, the latest version is 7.1.0 which we install to /u01/app/grid/xag.  The version of XAG bundled with GI 12.1.0.2 is actually version 3.1.0 so well worth using a more recent version.

DBFS

The next thing we want to do is create a DBFS file system for our GoldenGate shared files.  This will contain the GoldenGate recovery files and the trail files.  We actually have the option of using ACFS instead of DBFS, but we’re going to stick with DBFS.  Right now, this gives us more reliability with keeping database and filesystem files in sync with each other in the event of Dataguard switchover/failover scenarios.

We’re also making sure we use DBFS from within the database we’re extracting from, for the same reason.  Creating a separate “DBFS” database means that the GoldenGate source and file system could get out of sync again.

First I want to create a DBFS user in my source database (ordersA) –

And then create the DBFS file store (note that the first argument is the tablespace name, and the second argument will be the name of the directory that gets mounted later) –

 

Next we can think about mounting up that file system.  Firstly, we want to make sure we have a fuse group.  It should be there by default on Exadata, otherwise we need to create it on both nodes –

Then we need to add our oracle user to that group on both nodes –

And create a mount point to mount our file system into (again, both nodes) –

 

We also need to install the fuse package on both nodes if it isn’t already there –

We also need to make sure that the library libfuse.so exists, at least as a symbolic link.  Failure to do this will give us quite a frustrating “Fail to load library libfuse.so” error alongside “A dynamic linking error occurred: (libfuse.so: cannot open shared object file: No such file or directory).  Make sure this is set up as both nodes again.

Set an option in fuse.conf that we’ll use later.  Run this on both nodes as root.

We should also create a TNS entry that doesn’t use the SCAN address – we really want to make sure the traffic stays local for obvious performance reasons.  So create this entry on both nodes in the RDBMS home.

 

Now need to find a reliable and “cluster-aware” method to mount our DBFS file system.  This is achieved by using a script that Oracle provide called dbfs_mount.sh and is available as an attachment from Doc ID 1054431.1.

Follow the above article to install and configure that script, Personally I blank out TNS_ADMIN unless I’m using it.  For this example I’m not using the wallet so the password is stored in plain text within the script.  Oracle wallet is part of the advanced security pack which many customers don’t have, so I’m not demonstrating it here.  The other thing you may like to change in the “add-dbfs-resource.sh” script is to change the type from “local_resource” to “cluster_resource”.  The reason for this is that DBFS doesn’t provide any file locking mechanism, and therefore having it mounted on two nodes could be asking for trouble.

If you’ve done everything right, you should have run “add-dbfs-resource.sh” and the resource should be in CRS –

stat_res

 

We could actually test the script now – any errors are redirected to the OS messages file if it doesn’t work.

 

I need to stop and start CRS now because I haven’t done so since adding the oracle user to the fuse group.  Once that is done, our new cluster resource should start up without error.

Once last thing to watch out for – having the DBFS database password expire can be a pain.  You may need to manage that in whichever way your organisation manages system passwords, but for the purposes of my demonstration I’m going to create a new user profile that doesn’t expire passwords –

Virtual IP

We’ll only be running our “target” GoldenGate manager on one node at a time, and we need to make sure it is always addressed by the same IP wherever is running.  Therefore we need to create a VIP for it.

For the purposes of a demonstration I’m using the same VIP name (which will also be the CRS name) as the DNS entry.  I only have one network but you can check this as follows –

I then allow oracle to be able to start the VIP and start it up on the local node –

Software Installation

GoldenGate can be installed onto a shared file system, but at the moment Oracle recommend installing it on DBFS –

Oracle Database file system (DBFS). Do not install Oracle GoldenGate on DBFS, but you can store the subdirectories (that are created with CREATE SUBDIRS during installation) in a DBFS cluster that is only mounted to one server at a time.

Therefore we’ll perform a cluster installation onto local storage, but we’ll point certain “shared” directories at our DBFS shared storage.

The silent installation is straight forward and has very few options, but for the purposes of a demonstration we’ll stick with the graphical installer.  We run this as oracle on both nodes, one at a time.

XAG Registration

Now we can register GoldenGate with XAG for cluster administration.  Note that we’re using the full “cluster resource” name for the database to avoid an XAG-233 error.  We’re also not using the optional “vip_name” parameter since this is the source GoldenGate installation and won’t be listening for incoming connections.

I want to take this opportunity to stop CRS from automatically starting the DBFS mount point, since we’ll be using AGCTL to do that –

We can now use agctl to stop and start our GoldenGate related resources on one node at a time.

Note that the node running GoldenGate has the “mgr” process running, as well as the VIP and the /dbfs_direct mount.  The other node does not.  Relocating the resource stops those resources on one node and moves them elsewhere –

If we prevent /dbfs_direct from being umounted (just by sitting our bash session in there) then we can see the GoldenGate resource fails to move and it keeps everything running where it was.  This is likely to be a common problem when moving GoldenGate resources, so definitely worth checking for open files in /dbfs_direct prior to moving GoldenGate –

Repeat for target

Our GoldenGate configuration is going to have two databases – in our demonstration both running on the same RAC cluster (orders is the source database, and report is the target).  I’m going to repeat the steps above so that the target database has it’s own GoldenGate software installation, DBFS mount and AGCTL resource.

We’ll use port 7810 for the manager of the second GoldenGate installation.

Note this time we used the “vip_name” since this GoldenGate installation will have a manager process listening for incoming trail files –

Looking good –

Symbolic Links

We need to ensure that the shared GoldenGate directories are now pointing to DBFS storage.  On Linux we can use symbolic links to achieve that.

We’re only going to put dirdat, dirchk, dirprm and BR into shared storage.  We have the option of putting dirtmp into DBFS (purely to take advantage of potentially greater available diskspace) but that the recommendation is that DBFS would then be created on NOCACHE NOLOGGING tablespaces.  That’ll cause us problems later if we use Data Guard, so we’re keeping dirtmp on local storage.

We do this for both GoldenGate installations on both nodes.

Pump Setup

Our pump process is configured as follows –


EXTRACT PMPORDER
USERID GG_ADMIN, PASSWORD gg_admin
RMTHOST exa1-gg-vip, MGRPORT 7810
RMTTRAIL ./dirdat/ro
TABLE ORDERS_APP.*;

You can see that it creates the remote trail via the VIP we created earlier – this means it doesn’t matter which node the replicat process is running.

Fail Over

We could even test failing over the target GoldenGate installation to the second node –

The Pump process notices but handles it very well –

Summary

That’s it – we have everything in place now to run GoldenGate in a RAC-aware and HA configuration.

Part two of this series will cover throwing Data Guard into the mix.

Advertisements
Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: