Cryptographic Security Platform 1.6 - Vault 10.6.1 installation, administration, and user guide [PDF][Releases]

Purpose: Step-by-step guide to build a 2-node Oracle 19c Real Application Clusters (RAC) environment on AIX 7.3 LPARs in IBM Cloud Power Virtual Server (PowerVS): Grid Infrastructure, ASM, Oracle Database software and a RAC container database (CDB) with one pluggable database (PDB).
Audience: QA / Dev engineers who need an Oracle RAC test bed (for example, for TDE / KeyControl testing).
End result: A RAC CDB running on both nodes, with its PDB opening automatically after every restart.

1. Architecture

                     Clients (port 1521)
                            |
                     SCAN  <scan-name>
                            |
   ------------------- Public network (en0) -------------------
        |                                               |
   +-----------+                                   +-----------+
   |  Node 1   |  <== Private interconnect (en2) ==>|  Node 2   |
   |  AIX 7.3  |                                   |  AIX 7.3  |
   | GI + ASM  |                                   | GI + ASM  |
   | DB inst 1 |                                   | DB inst 2 |
   +-----------+                                   +-----------+
         \                                               /
          \--------- Shared disk (ASM disk group) -------/

Layer	Component
Clusterware	Oracle Grid Infrastructure 19c (Flex ASM)
Storage	ASM disk group `DATA` on one shared disk (external redundancy)
Database	Oracle Database 19c Enterprise Edition, RAC CDB with one PDB

2. Prerequisites

2.1 Compute (per LPAR, 2 LPARs)

Item	Minimum	Recommended
OS	AIX 7.3	AIX 7.3 latest TL/SP
vCPU	2	4
RAM	10 GB	16 GB or more

With less than 16 GB RAM, the database must use a small memory target (about 2 GB) and DBCA takes much longer.

2.2 Storage (per LPAR)

Each LPAR needs three disks: one OS disk, one local Oracle software disk, and the shared ASM disk.

Disk	Purpose	Size	Attached to
Disk 1 (`rootvg`)	AIX OS, `/tmp`, `/var`, paging space	100 GB or more	Each node (local)
Disk 2 (`u01vg`)	`/u01`: Grid home, DB home, Oracle base, inventory, staging	100 GB or more	Each node (local)
Disk 3 (shared)	ASM disk group `DATA` (database files, OCR, voting)	100 GB or more	Both nodes (shareable volume)

Required free space inside these disks:

Filesystem / area	Minimum
`/tmp`	5 GB
`/var`	2 GB
Paging space	Equal to RAM (up to 16 GB)
`/u01`	60 GB (Grid home ~12 GB, DB home ~8 GB, logs, staging zips)
Shared ASM disk	50 GB

The shared disk must be created in PowerVS as a shareable volume and attached to both LPARs. It must not belong to any AIX volume group.

2.3 Network

Network	Interface	Purpose	Addresses needed
Public subnet	`en0`	Host IPs, VIPs, SCAN	2 host IPs + 2 VIPs + 1 SCAN IP (all in the same subnet)
Private subnet	`en2`	Cluster interconnect and ASM traffic	1 IP per node

VIPs and the SCAN IP must be free (not assigned to any host) before the Grid install. Grid Infrastructure brings them up itself.

2.4 Software

Item	Notes
`AIX.PPC64_193000_grid_home.zip`	Oracle Grid Infrastructure 19c for AIX (Oracle eDelivery)
`AIX.PPC64_193000_db_home.zip`	Oracle Database 19c for AIX (Oracle eDelivery)
AIX fileset `xlfrte` 17.1.1.4 or later	Fortran runtime required by Oracle
NTP server reachable from both nodes	For example, `time.google.com`
Root SSH access to both LPARs	Required for setup

2.5 Planning sheet

Fill this in before you start. The commands in this guide use the Example values; replace them with yours.

Parameter	Example	Your value
Node 1 / Node 2 hostname	`rac1` / `rac2`
Node 1 / Node 2 public IP (`en0`)	`192.168.133.107` / `192.168.133.110`
Public subnet / netmask	`192.168.133.104` / `255.255.255.248`
Node 1 / Node 2 VIP	`192.168.133.106` (`rac1-vip`) / `192.168.133.109` (`rac2-vip`)
SCAN name / IP	`racscan` / `192.168.133.108`
Node 1 / Node 2 private IP (`en2`)	`192.168.100.2` (`rac1-priv`) / `192.168.100.3` (`rac2-priv`)
Private subnet / netmask	`192.168.100.0` / `255.255.255.248`
Cluster name	`raccluster`
Local Oracle disk	`hdisk3`
Shared ASM disk	`hdisk0` (`/dev/rhdisk0`)
Database name / PDB name	`orcl` / `orclpdb`
Passwords (ASM, SYS, SYSTEM, PDB admin)	`<PASSWORD>`

3. Prepare the operating system

Run every step on both nodes as root unless stated otherwise.

3.1 Host names (`/etc/hosts`)

cat >> /etc/hosts <<'EOF'
# Public
192.168.133.107  rac1
192.168.133.110  rac2
# VIP
192.168.133.106  rac1-vip
192.168.133.109  rac2-vip
# SCAN
192.168.133.108  racscan
# Private interconnect
192.168.100.2    rac1-priv
192.168.100.3    rac2-priv
EOF

Check the private network:

ifconfig en2 # correct private IP and netmask

ping -c 3 rac2-priv # from node 1 (ping rac1-priv from node 2)

3.2 Groups and users

Use the same IDs on both nodes.

mkgroup -A id=54321 oinstall
mkgroup -A id=54322 dba
mkgroup -A id=54327 asmdba
mkgroup -A id=54328 asmoper
mkgroup -A id=54329 asmadmin
 
mkuser id=54331 pgrp=oinstall groups=dba,asmadmin,asmdba,asmoper home=/home/grid   grid
mkuser id=54321 pgrp=oinstall groups=dba,asmdba                   home=/home/oracle oracle
passwd grid
passwd oracle
 
chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid
chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle

3.3 Check OS filesystem sizes

df -g /tmp /var

lsps -a

If /tmp or /var is below the minimum in section 2.2, grow it (rootvg must have free space):

chfs -a size=5G /tmp

chfs -a size=2G /var

3.4 Create `/u01` and the Oracle directories

mkvg -y u01vg hdisk3
crfs -v jfs2 -g u01vg -m /u01 -A yes -a size=95G
mount /u01
 
mkdir -p /u01/app/19.0.0/grid /u01/app/grid /u01/app/oraInventory \
         /u01/app/oracle/product/19.0.0/dbhome_1 /u01/tmp
chown -R grid:oinstall   /u01/app/19.0.0 /u01/app/grid /u01/app/oraInventory
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/app
chmod 1777 /u01/tmp

3.5 Required AIX settings

# I/O completion ports
mkdev -l iocp0
chdev -l iocp0 -P -a autoconfig=available
lsdev -Cc iocp                    # iocp0 Available
 
# Packet filter devices used by installer network checks
tcpdump -D >/dev/null 2>&1
ls -l /dev/bpf*                   # devices exist
 
# Network tunable
no -p -o ipqmaxlen=512
 
# Fortran runtime
lslpp -l xlfrte                   # 17.1.1.4 or later

3.6 Time synchronization (NTP with slewing)

cat > /etc/ntp.conf <<'EOF'
server time.google.com
driftfile /etc/ntp.drift
tracefile /etc/ntp.trace
EOF
 
stopsrc -s xntpd
startsrc -s xntpd -a "-x"
ntpq -p

To keep -x after reboot, edit the xntpd line in /etc/rc.tcpip:

start /usr/sbin/xntpd "$src_running" "-x"

3.7 `scp` compatibility (OpenSSH 9.x)

The Oracle installer copies files between nodes with scp. OpenSSH 9.x uses the SFTP protocol by default, which the installer does not support. Force the legacy protocol:

mv /usr/bin/scp /usr/bin/scp.orig
cat > /usr/bin/scp <<'EOF'
#!/bin/sh
exec /usr/bin/scp.orig -T -O "$@"
EOF
chmod 755 /usr/bin/scp

3.8 Shared ASM disk

The disk name can differ between nodes. Identify it by size or PVID.

lspv
chdev -l hdisk0 -a reserve_policy=no_reserve -a pv=clear
chown grid:asmadmin /dev/rhdisk0
chmod 660 /dev/rhdisk0
ls -l /dev/rhdisk0               # crw-rw----  grid  asmadmin

3.9 User environment

Add to both ~grid/.profile and ~oracle/.profile:

export TMP=/u01/tmp
export TMPDIR=/u01/tmp
export CV_DESTLOC=/u01/tmp
umask 022

Add to ~grid/.profile:

export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/19.0.0/grid
export PATH=$ORACLE_HOME/bin:$PATH

Add to ~oracle/.profile (use ORACLE_SID=orcl1 on node 1 and orcl2 on node 2):

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
export ORACLE_SID=orcl1
export PATH=$ORACLE_HOME/bin:$PATH

3.10 Passwordless SSH

Set up for grid and oracle, between both nodes and from each node to itself.

su - grid
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
# Append ~/.ssh/id_rsa.pub of BOTH nodes to ~/.ssh/authorized_keys on BOTH nodes
ssh rac1 date; ssh rac2 date      # no password prompt

Repeat as oracle.

4. Install Oracle Grid Infrastructure

4.1 Unzip the software (node 1, as `grid`)

cd /u01/app/19.0.0/grid

unzip -q /u01/tmp/AIX.PPC64_193000_grid_home.zip

As root on both nodes, run the AIX pre-install script (copy it to node 2 first):

/u01/app/19.0.0/grid/clone/rootpre.sh

4.2 Create the response file (node 1, as `grid`)

Save as /u01/tmp/grid.rsp and run chmod 600 on it.

oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
INVENTORY_LOCATION=/u01/app/oraInventory
oracle.install.option=CRS_CONFIG
ORACLE_BASE=/u01/app/grid
oracle.install.asm.OSDBA=asmdba
oracle.install.asm.OSOPER=asmoper
oracle.install.asm.OSASM=asmadmin
oracle.install.crs.config.scanType=LOCAL_SCAN
oracle.install.crs.config.gpnp.scanName=racscan
oracle.install.crs.config.gpnp.scanPort=1521
oracle.install.crs.config.ClusterConfiguration=STANDALONE
oracle.install.crs.config.configureAsExtendedCluster=false
oracle.install.crs.config.clusterName=raccluster
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.clusterNodes=rac1:rac1-vip:HUB,rac2:rac2-vip:HUB
oracle.install.crs.config.networkInterfaceList=en0:192.168.133.104:1,en2:192.168.100.0:5
oracle.install.crs.configureGIMR=false
oracle.install.asm.configureGIMRDataDG=false
oracle.install.crs.config.storageOption=FLEX_ASM_STORAGE
oracle.install.asm.SYSASMPassword=<PASSWORD>
oracle.install.asm.monitorPassword=<PASSWORD>
oracle.install.asm.diskGroup.name=DATA
oracle.install.asm.diskGroup.redundancy=EXTERNAL
oracle.install.asm.diskGroup.AUSize=4
oracle.install.asm.diskGroup.disks=/dev/rhdisk0
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/rhdisk0
oracle.install.asm.configureAFD=false
oracle.install.crs.rootconfig.executeRootScript=false

In networkInterfaceList, 1 = public and 5 = ASM + private interconnect.

4.3 Run the installer (node 1, as `grid`)

export SKIP_ROOTPRE=TRUE

/u01/app/19.0.0/grid/gridSetup.sh -silent -ignorePrereqFailure -responseFile /u01/tmp/grid.rsp

SKIP_ROOTPRE=TRUE stops the installer from waiting at the interactive "Has rootpre.sh been run?" question.

4.4 Run the root scripts on node 1 (as `root`)

/u01/app/oraInventory/orainstRoot.sh

/u01/app/19.0.0/grid/root.sh

Wait for CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded.

4.5 Apply the PowerVS network settings (node 1, as `root`)

Apply both settings before running root.sh on node 2.

a) Set the netmask and interface for the ASM network. Without this, the ASM network resource stays OFFLINE and node 2 cannot reach ASM.

/u01/app/19.0.0/grid/bin/crsctl modify res ora.asmnet1.asmnetwork \
    -attr "USR_ORA_NETMASK=255.255.255.248,USR_ORA_IF=en2" -unsupported
su - grid -c "srvctl start listener -listener ASMNET1LSNR_ASM -node rac1"

b) Disable HAIP. The PowerVS network does not pass the link-local (169.254.x.x) traffic that HAIP uses, so the interconnect must use the private IPs directly.

/u01/app/19.0.0/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
/u01/app/19.0.0/grid/bin/crsctl stop crs -f
/u01/app/19.0.0/grid/bin/crsctl start crs

4.6 Run the root scripts on node 2 (as `root`)

/u01/app/oraInventory/orainstRoot.sh

/u01/app/19.0.0/grid/root.sh

Then disable HAIP on node 2 as well, and restart its stack:

/u01/app/19.0.0/grid/bin/crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
/u01/app/19.0.0/grid/bin/crsctl stop crs -f
/u01/app/19.0.0/grid/bin/crsctl start crs

4.7 Finish the Grid configuration (node 1, as `grid`)

srvctl modify asm -count 2
 
export SKIP_ROOTPRE=TRUE
/u01/app/19.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /u01/tmp/grid.rsp -silent

NETCA and ASMCA must succeed. A failure of the final Cluster Verification Utility check (INS-43080) can be ignored.

4.8 Verify the cluster

olsnodes -n -s                 # both nodes Active
crsctl check cluster -all      # CRS, CSS, EVM online on both nodes
crsctl stat res -t             # ASM, DATA disk group, listeners, VIPs, SCAN ONLINE

5. Install the Oracle Database software

5.1 Unzip the software (node 1, as `oracle`)

cd /u01/app/oracle/product/19.0.0/dbhome_1

unzip -q /u01/tmp/AIX.PPC64_193000_db_home.zip

5.2 Create the response file (node 1, as `oracle`)

Save as /u01/tmp/db.rsp.

oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0
oracle.install.option=INSTALL_DB_SWONLY
UNIX_GROUP_NAME=oinstall
INVENTORY_LOCATION=/u01/app/oraInventory
ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
ORACLE_BASE=/u01/app/oracle
oracle.install.db.InstallEdition=EE
oracle.install.db.OSDBA_GROUP=dba
oracle.install.db.OSOPER_GROUP=dba
oracle.install.db.OSBACKUPDBA_GROUP=dba
oracle.install.db.OSDGDBA_GROUP=dba
oracle.install.db.OSKMDBA_GROUP=dba
oracle.install.db.OSRACDBA_GROUP=dba
oracle.install.db.rootconfig.executeRootScript=false
oracle.install.db.CLUSTER_NODES=rac1,rac2

5.3 Run the installer (node 1, as `oracle`)

export SKIP_ROOTPRE=TRUE
cd /u01/app/oracle/product/19.0.0/dbhome_1
./runInstaller -silent -ignorePrereqFailure -waitforcompletion -responseFile /u01/tmp/db.rsp

The installer copies the Oracle home to node 2 automatically. Expected result: Successfully Setup Software.

5.4 Run the root script (both nodes, as `root`)

/u01/app/oracle/product/19.0.0/dbhome_1/root.sh -silent

5.5 Check the `oracle` binary group (both nodes)

The oracle binary must belong to the asmadmin group so the database can open ASM files.

ls -l /u01/app/oracle/product/19.0.0/dbhome_1/bin/oracle

If the group is not asmadmin, fix it as grid:

su - grid -c "/u01/app/19.0.0/grid/bin/setasmgidwrap o=/u01/app/oracle/product/19.0.0/dbhome_1/bin/oracle"

6. Create the RAC database

6.1 Create the DBCA response file (node 1, as `oracle`)

Save as /u01/tmp/dbca_orcl.rsp and run chmod 600 on it.

responseFileVersion=/oracle/assistants/rspfmt_dbca_response_schema_v19.0.0
gdbName=orcl
sid=orcl
databaseConfigType=RAC
nodelist=rac1,rac2
policyManaged=false
createAsContainerDatabase=true
numberOfPDBs=1
pdbName=orclpdb
useLocalUndoForPDBs=true
pdbAdminPassword=<PASSWORD>
sysPassword=<PASSWORD>
systemPassword=<PASSWORD>
templateName=General_Purpose.dbc
emConfiguration=NONE
storageType=ASM
diskGroupName=+DATA
recoveryGroupName=+DATA
characterSet=AL32UTF8
nationalCharacterSet=AL16UTF16
listeners=LISTENER
automaticMemoryManagement=false
totalMemory=2048
databaseType=MULTIPURPOSE

Set totalMemory (in MB) to fit the free memory on each node: about 2048 for 10 GB LPARs, 4096 or more for 16 GB LPARs.

6.2 Run DBCA (node 1, as `oracle`)

nohup dbca -silent -createDatabase -responseFile /u01/tmp/dbca_orcl.rsp -ignorePreReqs \
      > /u01/tmp/dbca_orcl.log 2>&1 &
tail -f /u01/tmp/dbca_orcl.log

Expected result: 100% complete and Database creation complete.

6.3 Open the PDB automatically after restart (node 1, as `oracle`)

sqlplus / as sysdba

ALTER PLUGGABLE DATABASE orclpdb SAVE STATE INSTANCES=ALL;

SELECT con_name, instance_name, state FROM dba_pdb_saved_states;

6.4 Remove the response files

They contain passwords.

rm /u01/tmp/grid.rsp /u01/tmp/db.rsp /u01/tmp/dbca_orcl.rsp

7. Verify the installation

srvctl status database -d orcl -v

srvctl config database -d orcl | grep -iE "Management policy|Start option"

Expected:

Instance orcl1 is running on node rac1. Instance status: Open.
Instance orcl2 is running on node rac2. Instance status: Open.
Management policy: AUTOMATIC
Start options: open

SELECT inst_id, instance_name, host_name, status FROM gv$instance;

SELECT inst_id, name, open_mode FROM gv$pdbs ORDER BY name, inst_id;

Expected: both instances OPEN, and the PDB READ WRITE on both instances.

Restart test:

srvctl stop database -d orcl

srvctl start database -d orcl

The PDB must come back READ WRITE on both instances.

7.1 Connection strings

Target	Connect string
CDB	`sqlplus sys/<PASSWORD>@//racscan:1521/orcl as sysdba`
PDB	`sqlplus system/<PASSWORD>@//racscan:1521/orclpdb`
Local, on a node	`sqlplus / as sysdba` (with `ORACLE_SID` set)

8. Troubleshooting

Symptom	Cause	Fix
Silent installer waits at "Has rootpre.sh been run?"	Interactive AIX question	`export SKIP_ROOTPRE=TRUE` (section 4.3)
Installer fails while copying files to node 2	OpenSSH 9.x `scp` uses SFTP	`scp` wrapper (section 3.7)
Prerequisite check fails on `iocp0` or `/dev/bpf`	Devices not configured	Section 3.5
Installer or `root.sh` fails with "no space"	`/tmp`, `/var` or `/u01` below minimum	Size disks as in section 2.2
Node 2 `root.sh` fails with `CLSRSC-117` / `CRS-5055`; `ora.asmnet1.asmnetwork` OFFLINE	ASM network has no netmask/interface	Section 4.5 a)
ASM on node 2 stops about a minute after start (LMON error)	HAIP link-local traffic blocked on PowerVS	Section 4.5 b)
Database cannot open ASM files on one node	`oracle` binary group is not `asmadmin`	Section 5.5
DBCA warning `DBT-11209` (low memory)	Memory target larger than free RAM	Lower `totalMemory` or add RAM
PDB is `MOUNTED` after restart	PDB state not saved	Section 6.3
`PRKC-1213 ... group "root"` message in a log	AIX has no `root` group	Can be ignored

9. Quick reference

Task	Command
Cluster resource status	`crsctl stat res -t`
Cluster health (all nodes)	`crsctl check cluster -all`
Cluster nodes	`olsnodes -n -s`
ASM disk groups	`asmcmd lsdg` (as `grid`)
Interconnect configuration	`oifcfg getif`
Database status	`srvctl status database -d orcl`
Stop / start database	`srvctl stop database -d orcl` / `srvctl start database -d orcl`
Stop / start clusterware on one node	`crsctl stop crs -f` / `crsctl start crs` (as `root`)
Clusterware alert log	`/u01/app/grid/diag/crs/<node>/crs/trace/alert.log`
Database alert log	`/u01/app/oracle/diag/rdbms/orcl/<ORACLE_SID>/trace/alert_<ORACLE_SID>.log`