Adding Node to a Oracle RAC Cluster and Extending DB instance to the Node in 11.1.0.7 cluster

The article deals with Adding node and extending DB to the new node in a 11.1.0.7 cluster. The activity can be broadly divided into
1. Adding Node to the existing cluster
2. Adding instance to the existing DB

Adding Node to Existing Cluster

Adding node can be accomplished by two methods namely Interactive and Silent. In both the cases OUI is used in add node mode to add a new node. While adding the node you have to ensure that all the RAC pre-reqs are in place.

Interactive Method

1. From one of the node of the existing cluster we will launch the OUI and for that ensure ORACLE_HOME is set to the CRS.
2. Go to ORACLE_HOME/oui/bin and run addNode.sh. It will launch the OUI in Add Node mode and displays the welcome screen. Click Next and the Specify Cluster Nodes for Node Addition page displays.
3. Oracle Universal Installer displays the Node Selection Page on which you should enter the node or nodes that you want to add and click Next.
4. Verify the entries that Oracle Universal Installer displays on the Summary Page and click Next. Oracle now runs the usual RAC tests to ensure the new Node is fit to be clubbed into the cluster.
5. The Node Addition Summary page displays the following information showing the products that are installed in the Oracle Clusterware home that you are extending to the new nodes:

• The source for the add node process, which in this case is the Oracle Clusterware home
• The private node names that you entered for the new nodes
• The new nodes that you entered
• The required and available space on the new nodes
• The installed products listing the products that are already installed on the existing Oracle Clusterware home

6. Click Next and Oracle Universal Installer displays the Cluster Node Addition Progress page.
7. The Cluster Node Addition Progress page shows the status of the cluster node addition process.
8. From the ORACLE_HOME/install directory on an existing node, run the Oracle Notification Service configuration utility (ONSCONFIG)
./onsconfig add_config node2:remote_port
9. You can check the nodes using ORACLE_HOME/bin/olsnodes -n

Silent Method

1. From one of the node of the existing cluster we will launch the OUI and for that ensure ORACLE_HOME is set to the CRS.
2. Go to ORACLE_HOME/oui/bin and run addNode.sh using following syntax
./addNode.sh –silent "CLUSTER_NEW_NODES={node2}" "CLUSTER_NEW_PRIVATE_NODE_NAMES={node2-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node2-vip}"
3. From the ORACLE_HOME/install directory on an existing node, run the Oracle Notification Service configuration utility (ONSCONFIG)
./onsconfig add_config node2:remote_port

Oracle TNS Poison Vulnerability Detection and Remediation

Introduction

The Oracle TNS Poison vulnerability was discovered by Joxean Koret, and vulnerability is defined in Oracle’s security note CVE-2012-1675.
Affected Products and Versions

• Oracle Database 11g Release 2, versions 11.2.0.2, 11.2.0.3, 11.2.0.4
• Oracle Database 11g Release 1, version 11.1.0.7
• Oracle Database 10g Release 2, versions 10.2.0.3, 10.2.0.4, 10.2.0.5

The following metalink notes provide the solution and detailed explanation to the vulnerability.

• 1340831.1 : For Oracle RAC deployments
• 1453883.1 : For Standalone deployments

Detection

The idea is to register a service from a machine(rogue) to the listener of the deployment as a result of which the connections can be directed to this remote service and the TNS packet flowing through this remote service can be sniffed. As a result, the confidentiality and integrity of the Database system can be compromised.
The following step by step process is used to detect in a database.

• Create a DB instance in a local machine (desktop/laptop).
  
• Add a entry lie the following in tnsnames.ora

       POISON=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=hostname)(PORT=1521)))

                change the hostname and port of the DB host.

• Login to the DB and execute

ALTER SYSTEM SET REMOTE_LISTENER=’POISON’;

ALTER SYSTEM REGISTER;

• The listener running in the DB host will register the instance running from the local machine.  The following can be seen in the listener log.
  Dynamic address is already listened on (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=xxxxxx)(PORT=1521)))
  30-SEP-2015 09:14:15 * service_register * orcl * 0

lsnrctl status shows.

Service "orcl" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...

Remediation

There are three ways to remediate the vulnerability.

• Disable Dynamic registration of instances at listener.
• Using Class of Secure Transfer(COST) for single and RAC deployments.
• Enable Valid Node Checking for Registration(VNCR) for 11.2.0.4 and above.

Disable Dynamic Registration

Switch off dynamic registration at the listener in it’s listener.ora by using the following
dynamic_registration_LISTENER_NAME=off

COST for Single and RAC deployments

Technically the steps for RAC and Standalone are same as the principle for instance registration is same in both cases, however in case of RAC there is added complexity to introduce load balancing in SCAN listeners. Further in RAC TCPS will be used with 1523 port to ensure that registration happens over TCPS protocol. The best approach would be to create the wallet in one node and then copy the wallet file to all the nodes of the cluster. Please do note at only the wallet file(ewallet.p12) is copied and the SSO file is created at each node separately.

To introduce COST the following are the steps. Further ensure that the bug fix patch 12880299 or latest PSU is applied to the Oracle Home from where the Listener is running.

• Create a wallet and delete all the pre-configured certificates

        $ORACLE_HOME/bin/orapki wallet create -wallet <wallet_location>

        $ORACLE_HOME/bin/orapki wallet remove -trusted_cert_all -wallet <wallet_location>

• Add a self signed wallet entry. The following adds the certificate valid for 10 years (3650)

        $ORACLE_HOME/bin/orapki wallet add -wallet <wallet_location> -self_signed -dn "cn=secure_register" -keysize 1024 -validity 3650

• Create a SSO file

        $ORACLE_HOME/bin/orapki wallet create -wallet <wallet_location> -auto_login

• Add the parameter secure_register and wallet location to listener.ora

        SECURE_REGISTER_LISTENER = (IPC,TCP,TCPS)
        WALLET_LOCATION =(SOURCE =(METHOD = FILE)(METHOD_DATA = (DIRECTORY = <wallet_location>)))

• Add the wallet location to the sqlnet.ora of the instance

        WALLET_LOCATION =(SOURCE =(METHOD = FILE)(METHOD_DATA = (DIRECTORY = <wallet_location>)))

• Bounce the DB instance and Listener to take effect

After the remediation if an instance is tried to register with the listener without wallet information the following will be seen in the listener log.
Wed Sep 30 10:01:59 2015
30-SEP-2015 10:01:59 * service_register_NSGR * 1194
TNS-01194: The listener command did not arrive in a secure transport

RAC Specific steps

• Adding TCPS to the Secure_Register to ensure that registration happens only over IPC and TCPS

       SECURE_REGISTER_LISTENER_SCAN1 = (IPC,TCPS)
       SECURE_REGISTER_LISTENER_SCAN2 = (IPC,TCPS)
       SECURE_REGISTER_LISTENER_SCAN3 = (IPC,TCPS)

• Adding TCPS and 1523 to SCAN Listener configuration

       $CRS_HOME/bin/srvctl modify scan_listener -p TCP:1521/TCPS:1523

• Bounce the Scan Listeners

• Updates the REMOTE_LISTENER in the DB

       alter system set remote_listener='(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCPS)(HOST=IP_1)(PORT=1523))(ADDRESS=(PROTOCOL=TCPS)(HOST=IP_2)(PORT=1523))(ADDRESS=(PROTOCOL=TCPS)(HOST=IP_3)(PORT=1523)))' scope=both sid='*';

Valid Node Checking for Registration (VNCR)

VNCR is the solution provided by Oracle from Database version 11.2.0.4 and above. The solution is build in the listener binaries and can be switched on by enabling the following switches in listener.ora

In case of Standalone
              VALID_NODE_CHECKING_REGISTRATION_LISTENER=ON

In case of RAC, the SCAN listeners need to have the address of the remote nodes as well, so a parameter REGISTRATION_INVITED_NODES is used

VALID_NODE_CHECKING_REGISTRATION_LISTENER=ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=ON
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=ON
REGISTRATION_INVITED_NODES_LISTENER_SCAN1=(NODE_1,NODE_2)
REGISTRATION_INVITED_NODES_LISTENER_SCAN2=(NODE_1,NODE_2)
REGISTRATION_INVITED_NODES_LISTENER_SCAN3=(NODE_1,NODE_2)