Oracle Rac Node Removal

STEP BY STEP ORACLE 11G R2 NODE REMOVAL






Prepared by: Hayat Mohammad Khan (DBA) hayathk@hotmail.com  - +92-333-5193460
Maroof Ud Din (DBA) maroofuddinkhan@yahoo.com 

Saturday, March 01, 2014



Disclaimar The steps performed in below documents are our own and do not necessarily reflect the views of Oracle Corporation. Steps may vary to actual activity, environment, platform, and database and grid version and patches.



Assumptions DB Version Oracle 11.2.0.4 two nodes cluster
OS Version IBM AIX 6.1
Cluster File System IBM GPFS 3.4
DB Nodes Names SETTDB1, SETTDB2
DB OS Names SETT1,SETT2
DB-Home /oracle/app/grid/product/11.2.0.3/dbhome_3
Grid-Home /oracle/app/11.2.0.3/grid_2/

Homes Sharing Status Not Shared
Recommendations Others Node1 refered to remaining node, Node2 refered to deleting node
Others2 Please always read logs files as mentioned in each step in separate terminal window
Others3 Always paste command to notepad / text editor and verify it, otherwise there is chances that it may exeucte on all nodes
Higly Recommended Pl take proper Backup of Oracle homes and database software before start of this activity
High Level Steps


Remove Oracle instance of deleting node

Remove Oracle database Binaries  of deleting node

Remove deleting node from Oracle Grid 

Remove Oracle Grid software from deleting node



Step#1 NODE ON NODE 2

Purpose For Stoping the instnace db services which you want to remove

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path From any where

Substep#

Purpose-Substep

Command-Syntax srvctl stop instance -d settdb -i settdb2

Expected OutPut



Step#2 NODE ON NODE 1

Purpose Disabling Thread Which you want to remove

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path Connect from Sqlplus/ as sysdba    -----> From Node 1  --- settdb1

Substep#

Purpose-Substep

Command-Syntax alter database disable thread 2;         --we want to remove thread 2

Expected OutPut



Step#3 NODE ON NODE 2

Purpose Instance Will Be removed

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path .PROFILE must be set to oracle binaries

Substep#

Purpose-Substep

Command-Syntax srvctl remove instance -d settdb -i settdb2         --we want to remove node2 i.e settdb2

Expected OutPut



Step#4 NODE ON NODE 2

Purpose the below command will update Inventoy on this node

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/grid/product/11.2.0.3/dbhome_3/deinstall

Substep#

Purpose-Substep

Command-Syntax ./runInstaller -updateNodeList ORACLE_HOME=/oracle/app/grid/product/11.2.0.3/dbhome_3 "CLUSTER_NODES={sett2}" -local

Expected OutPut

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 16384 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /oracle/app/oraInventory

'UpdateNodeList' was successful.



Step#5 NODE  FROM NODE 2 

Purpose For Deinstalling ORACLE HOME software

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/grid/product/11.2.0.3/dbhome_3/deinstall

Substep#

Purpose-Substep

Command-Syntax ./deinstall -local

Expected OutPut Checking for required files and bootstrapping ...

Please wait ...

Location of logs /oracle/app/oraInventory/logs/



############ ORACLE DEINSTALL & DECONFIG TOOL START ############





######################### CHECK OPERATION START #########################

## [START] Install check configuration ##





Checking for existence of the Oracle home location /oracle/app/grid/product/11.2.0.3/dbhome_3

Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database

Oracle Base selected for deinstall is: /oracle/app/grid

Checking for existence of central inventory location /oracle/app/oraInventory

Checking for existence of the Oracle Grid Infrastructure home /oracle/app/11.2.0.3/grid_2

The following nodes are part of this cluster: sett2

Checking for sufficient temp space availability on node(s) : 'sett2'



## [END] Install check configuration ##





Network Configuration check config START



Network de-configuration trace file location: /oracle/app/oraInventory/logs/netdc_check2014-03-25_05-34-41-PM.log



Network Configuration check config END



Database Check Configuration START



Database de-configuration trace file location: /oracle/app/oraInventory/logs/databasedc_check2014-03-25_05-34-47-PM.log



Database Check Configuration END



Enterprise Manager Configuration Assistant START



EMCA de-configuration trace file location: /oracle/app/oraInventory/logs/emcadc_check2014-03-25_05-34-52-PM.log 



Enterprise Manager Configuration Assistant END

Oracle Configuration Manager check START

OCM check log file location : /oracle/app/oraInventory/logs//ocm_check723.log

Oracle Configuration Manager check END



######################### CHECK OPERATION END #########################





####################### CHECK OPERATION SUMMARY #######################

Oracle Grid Infrastructure Home is: /oracle/app/11.2.0.3/grid_2

The cluster node(s) on which the Oracle home deinstallation will be performed are:sett2

Since -local option has been specified, the Oracle home will be deinstalled only on the local node, 'sett2', and the global configuration will be removed.

Oracle Home selected for deinstall is: /oracle/app/grid/product/11.2.0.3/dbhome_3

Inventory Location where the Oracle home registered is: /oracle/app/oraInventory

The option -local will not modify any database configuration for this Oracle home.



No Enterprise Manager configuration to be updated for any database(s)

No Enterprise Manager ASM targets to update

No Enterprise Manager listener targets to migrate

Checking the config status for CCR

Oracle Home exists with CCR directory, but CCR is not configured

CCR check is finished

Do you want to continue (y - yes, n - no)? [n]: y

A log of this session will be written to: '/oracle/app/oraInventory/logs/deinstall_deconfig2014-03-25_05-34-31-PM.out'

Any error messages from this session will be written to: '/oracle/app/oraInventory/logs/deinstall_deconfig2014-03-25_05-34-31-PM.err'



######################## CLEAN OPERATION START ########################



Enterprise Manager Configuration Assistant START



EMCA de-configuration trace file location: /oracle/app/oraInventory/logs/emcadc_clean2014-03-25_05-34-52-PM.log 



Updating Enterprise Manager ASM targets (if any)

Updating Enterprise Manager listener targets (if any)

Enterprise Manager Configuration Assistant END

Database de-configuration trace file location: /oracle/app/oraInventory/logs/databasedc_clean2014-03-25_05-36-32-PM.log



Network Configuration clean config START



Network de-configuration trace file location: /oracle/app/oraInventory/logs/netdc_clean2014-03-25_05-36-32-PM.log



De-configuring Local Net Service Names configuration file...

Local Net Service Names configuration file de-configured successfully.



De-configuring backup files...

Backup files de-configured successfully.



The network configuration has been cleaned up successfully.



Network Configuration clean config END



Oracle Configuration Manager clean START

OCM clean log file location : /oracle/app/oraInventory/logs//ocm_clean723.log

Oracle Configuration Manager clean END

Setting the force flag to false

Setting the force flag to cleanup the Oracle Base

Oracle Universal Installer clean START



Detach Oracle home '/oracle/app/grid/product/11.2.0.3/dbhome_3' from the central inventory on the local node : Done



Delete directory '/oracle/app/grid/product/11.2.0.3/dbhome_3' on the local node : Done



The Oracle Base directory '/oracle/app/grid' will not be removed on local node. The directory is in use by Oracle Home '/oracle/app/11.2.0.3/grid'.



Oracle Universal Installer cleanup was successful.



Oracle Universal Installer clean END





## [START] Oracle install clean ##



Clean install operation removing temporary directory '/tmp/deinstall2014-03-25_05-34-09PM' on node 'sett2'



## [END] Oracle install clean ##





######################### CLEAN OPERATION END #########################





####################### CLEAN OPERATION SUMMARY #######################

Cleaning the config for CCR

As CCR is not configured, so skipping the cleaning of CCR configuration

CCR clean is finished

Successfully detached Oracle home '/oracle/app/grid/product/11.2.0.3/dbhome_3' from the central inventory on the local node.

Successfully deleted directory '/oracle/app/grid/product/11.2.0.3/dbhome_3' on the local node.

Oracle Universal Installer cleanup was successful.



Oracle deinstall tool successfully cleaned up temporary directories.

#######################################################################





############# ORACLE DEINSTALL & DECONFIG TOOL END #############









Step#6 NODE  From NODE 1

Purpose For updating Inventory

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/grid/product/11.2.0.3/dbhome_3

Substep#

Purpose-Substep

Command-Syntax ./runInstaller -updateNodeList ORACLE_HOME=/oracle/app/grid/product/11.2.0.3/dbhome_3 "CLUSTER_NODES={sett1}" 

Expected OutPut Starting Oracle Universal Installer...


Checking swap space: must be greater than 500 MB.   Actual 16384 MB    Passed


The inventory pointer is located at /etc/oraInst.loc


The inventory is located at /oracle/app/oraInventory


'UpdateNodeList' was successful.



Step#7 NODE  From any remaing Node  in our case it is NODE1

Purpose UNPIN NODE 2

Run-From User From ROOT user

Path

Substep#

Purpose-Substep

Command-Syntax crsctl unpin css -n sett2

Expected OutPut Node sett2 successfully unpinned.



Step#8 NODE  From deleting Node in this example from NODE2

Purpose Removing node from cluster

Run-From User From ROOT user

Path /oracle/app/11.2.0.3/grid/crs/install

Substep#

Purpose-Substep

Command-Syntax ./rootcrs.pl -deconfig -deinstall -force 

Expected OutPut Using configuration parameter file: ./crsconfig_params


Network exists: 1/XX.XX.XX.64/255.255.255.XX/en0, type static


VIP exists: /sett1-vip/XX.XX.XX.XX/XX.XX.XX.64/255.255.255.XX/en0, hosting node sett1


VIP exists: /sett2-vip/XX.XX.XX.XX/XX.XX.XX.64/255.255.255.XX/en0, hosting node sett2


GSD exists


ONS exists: Local port 6100, remote port 6200, EM port 2016


CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'sett2'


CRS-2673: Attempting to stop 'ora.crsd' on 'sett2'


CRS-2677: Stop of 'ora.crsd' on 'sett2' succeeded


CRS-2673: Attempting to stop 'ora.mdnsd' on 'sett2'


CRS-2673: Attempting to stop 'ora.crf' on 'sett2'


CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'sett2'


CRS-2673: Attempting to stop 'ora.ctssd' on 'sett2'


CRS-2673: Attempting to stop 'ora.evmd' on 'sett2'


CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'sett2' succeeded


CRS-2677: Stop of 'ora.mdnsd' on 'sett2' succeeded


CRS-2677: Stop of 'ora.evmd' on 'sett2' succeeded


CRS-2677: Stop of 'ora.crf' on 'sett2' succeeded


CRS-2677: Stop of 'ora.ctssd' on 'sett2' succeeded


CRS-2673: Attempting to stop 'ora.cssd' on 'sett2'


CRS-2677: Stop of 'ora.cssd' on 'sett2' succeeded


CRS-2673: Attempting to stop 'ora.gipcd' on 'sett2'


CRS-2677: Stop of 'ora.gipcd' on 'sett2' succeeded


CRS-2673: Attempting to stop 'ora.gpnpd' on 'sett2'


CRS-2677: Stop of 'ora.gpnpd' on 'sett2' succeeded


CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'sett2' has completed


CRS-4133: Oracle High Availability Services has been stopped.


This may take several minutes. Please wait ...


0518-307 odmdelete: 1 objects deleted.


0518-307 odmdelete: 1 objects deleted.


0518-307 odmdelete: 1 objects deleted.


Successfully deconfigured Oracle clusterware stack on this node



Step#9 NODE  NODE 1

Purpose Updating GRID HOME inventory

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/11.2.0.3/grid_2/oui/bin

Substep#

Purpose-Substep

Command-Syntax ./runInstaller -updateNodeList ORACLE_HOME=/oracle/app/11.2.0.3/grid_2 "CLUSTER_NODES={sett1}" CRS=TRUE

Expected OutPut Starting Oracle Universal Installer...


Checking swap space: must be greater than 500 MB.   Actual 16384 MB    Passed


The inventory pointer is located at /etc/oraInst.loc


The inventory is located at /oracle/app/oraInventory


'UpdateNodeList' was successful.



Step#10 NODE  From NODE 1

Purpose Deleting node2 from cluster

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/11.2.0.3/grid_2/bin

Substep#

Purpose-Substep

Command-Syntax ./crsctl delete node -n sett2

Expected OutPut CRS-4661: Node sett2 successfully deleted.



Step#11 NODE  From Node2

Purpose Updating Grid Inventory

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/11.2.0.3/grid_2/oui/bin

Command-Syntax ./runInstaller -updateNodeList ORACLE_HOME=/oracle/app/11.2.0.3/grid_2 "CLUSTER_NODES={sett2}" CRS=TRUE -local

Expected OutPut Starting Oracle Universal Installer...


Checking swap space: must be greater than 500 MB.   Actual 16384 MB    Passed


The inventory pointer is located at /etc/oraInst.loc


The inventory is located at /oracle/app/oraInventory


'UpdateNodeList' was successful.








Please Please..... At end it will asked u to execute scripts as root  from respective nodes........... please DONT RUSH...........and execute in sequential order and as per instructions............
Step#12 NODE  From NODE2

Purpose Deinstalling Grid HOME

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/11.2.0.3/grid_2/deinstall

Command-Syntax ./deinstall -local

Expected OutPut Checking for required files and bootstrapping ...


Please wait ...


Location of logs /oracle/app/oraInventory/logs/





############ ORACLE DEINSTALL & DECONFIG TOOL START ############








######################### CHECK OPERATION START #########################


## [START] Install check configuration ##








Checking for existence of the Oracle home location /oracle/app/11.2.0.3/grid_2


Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster


Oracle Base selected for deinstall is: /oracle/app/grid


Checking for existence of central inventory location /oracle/app/oraInventory


Checking for existence of the Oracle Grid Infrastructure home 


The following nodes are part of this cluster: sett2


Checking for sufficient temp space availability on node(s) : 'sett2'





## [END] Install check configuration ##





Traces log file: /oracle/app/oraInventory/logs//crsdc.log


Enter an address or the name of the virtual IP used on node "sett2"[sett2-vip]


 > 





The following information can be collected by running "/sbin/ifconfig -a" on node "sett2"


Enter the IP netmask of Virtual IP "xx.xx.xx.xx" on node "sett2"[255.255.255.0]


 > 





Enter the network interface name on which the virtual IP address "xx.xx.xx.xx" is active


 > 





Enter an address or the name of the virtual IP[]


 > 








Network Configuration check config START





Network de-configuration trace file location: /oracle/app/oraInventory/logs/netdc_check2014-03-25_06-18-27-PM.log





Specify all RAC listeners (do not include SCAN listener) that are to be de-configured [LISTENER,LISTENER_SCAN1]:





Network Configuration check config END





Asm Check Configuration START





ASM de-configuration trace file location: /oracle/app/oraInventory/logs/asmcadc_check2014-03-25_06-18-57-PM.log








######################### CHECK OPERATION END #########################








####################### CHECK OPERATION SUMMARY #######################


Oracle Grid Infrastructure Home is: 


The cluster node(s) on which the Oracle home deinstallation will be performed are:sett2


Since -local option has been specified, the Oracle home will be deinstalled only on the local node, 'sett2', and the global configuration will be removed.


Oracle Home selected for deinstall is: /oracle/app/11.2.0.3/grid_2


Inventory Location where the Oracle home registered is: /oracle/app/oraInventory


Following RAC listener(s) will be de-configured: LISTENER,LISTENER_SCAN1


Option -local will not modify any ASM configuration.


Do you want to continue (y - yes, n - no)? [n]: y


A log of this session will be written to: '/oracle/app/oraInventory/logs/deinstall_deconfig2014-03-25_06-17-52-PM.out'


Any error messages from this session will be written to: '/oracle/app/oraInventory/logs/deinstall_deconfig2014-03-25_06-17-52-PM.err'





######################## CLEAN OPERATION START ########################


ASM de-configuration trace file location: /oracle/app/oraInventory/logs/asmcadc_clean2014-03-25_06-19-17-PM.log


ASM Clean Configuration END





Network Configuration clean config START





Network de-configuration trace file location: /oracle/app/oraInventory/logs/netdc_clean2014-03-25_06-19-17-PM.log





De-configuring RAC listener(s): LISTENER,LISTENER_SCAN1





De-configuring listener: LISTENER


    Stopping listener on node "sett2": LISTENER


    Warning: Failed to stop listener. Listener may not be running.


Listener de-configured successfully.





De-configuring listener: LISTENER_SCAN1


    Stopping listener on node "sett2": LISTENER_SCAN1


    Warning: Failed to stop listener. Listener may not be running.


Listener de-configured successfully.





De-configuring Naming Methods configuration file...


Naming Methods configuration file de-configured successfully.





De-configuring backup files...


Backup files de-configured successfully.





The network configuration has been cleaned up successfully.





Network Configuration clean config END








---------------------------------------->





The deconfig command below can be executed in parallel on all the remote nodes. Execute the command on  the local node after the execution completes on all the remote nodes.



Pl carefully execute it on deleting node only
Run the following command as the root user or the administrator on node "sett2".



Subset 12.1 Node Execute from Deleting Node [Node2]

Run-From User From Root User

Substep#

Purpose-Substep

Command /tmp/deinstall2014-03-25_06-17-30PM/perl/bin/perl -I/tmp/deinstall2014-03-25_06-17-30PM/perl/lib


 -I/tmp/deinstall2014-03-25_06-17-30PM/crs/install /tmp/deinstall2014-03-25_06-17-30PM/crs/install/rootcrs.pl


 -force  -deconfig -paramfile "/tmp/deinstall2014-03-25_06-17-30PM/response/deinstall_Ora11g_gridinfrahome3.rsp"

Expected Output


sett2:/ >/tmp/deinstall2014-03-25_06-17-30PM/perl/bin/perl -I/tmp/deinstall2014-03-25_06-17-30PM/perl/lib -I/tmp/deinstall2014-03-25_06-17-30PM/crs/install /tmp/deinstall2014-03-25_06-17-30PM/crs/install/rootcrs.pl -force  -deconfig -paramfile "/tmp/deinstall2014-03-25_06-17-30PM/response/deinstall_Ora11g_gridinfrahome3.rsp"


Using configuration parameter file: /tmp/deinstall2014-03-25_06-17-30PM/response/deinstall_Ora11g_gridinfrahome3.rsp


Usage: srvctl <command> <object> [<options>]


    commands: enable|disable|start|stop|status|add|remove|modify|getenv|setenv|unsetenv|config|upgrade


    objects: database|service|asm|diskgroup|listener|home|ons


For detailed help on each command and object and its options use:


  srvctl <command> -h or


  srvctl <command> <object> -h


PRKO-2012 : nodeapps object is not supported in Oracle Restart


CRS-4047: No Oracle Clusterware components configured.


CRS-4000: Command Stop failed, or completed with errors.


Either /etc/oracle/ocr.loc does not exist or is not readable


Make sure the file exists and it has read and execute access


Either /etc/oracle/ocr.loc does not exist or is not readable


Make sure the file exists and it has read and execute access


CRS-4047: No Oracle Clusterware components configured.


CRS-4000: Command Modify failed, or completed with errors.


CRS-4047: No Oracle Clusterware components configured.


CRS-4000: Command Delete failed, or completed with errors.


CRS-4047: No Oracle Clusterware components configured.


CRS-4000: Command Stop failed, or completed with errors.


################################################################


# You must kill processes or reboot the system to properly #


# cleanup the processes started by Oracle clusterware          #


################################################################


This may take several minutes. Please wait ...


Either /etc/oracle/olr.loc does not exist or is not readable


Make sure the file exists and it has read and execute access


Removing Trace File Analyzer


Either /etc/oracle/olr.loc does not exist or is not readable


Make sure the file exists and it has read and execute access


Failure in execution (rc=-1, 256, No such file or directory) for command /etc/ohasd deinstall


Successfully deconfigured Oracle clusterware stack on this node


sett2:/ >





========================================================= PRESS ENTER AFTER EXECUTION OF ABOVE SCRIPT FROM ROOT USER :+++=======================





Press Enter after you finish running the above commands





<----------------------------------------





Remove the directory: /tmp/deinstall2014-03-25_06-17-30PM on node: 


Setting the force flag to false


Setting the force flag to cleanup the Oracle Base


Oracle Universal Installer clean START





Detach Oracle home '/oracle/app/11.2.0.3/grid_2' from the central inventory on the local node : Done





Delete directory '/oracle/app/11.2.0.3/grid_2' on the local node : Done





The Oracle Base directory '/oracle/app/grid' will not be removed on local node. The directory is in use by Oracle Home '/oracle/app/11.2.0.3/grid'.





Oracle Universal Installer cleanup was successful.





Oracle Universal Installer clean END








## [START] Oracle install clean ##





Clean install operation removing temporary directory '/tmp/deinstall2014-03-25_06-17-30PM' on node 'sett2'





## [END] Oracle install clean ##








######################### CLEAN OPERATION END #########################








####################### CLEAN OPERATION SUMMARY #######################


Following RAC listener(s) were de-configured successfully: LISTENER,LISTENER_SCAN1


Oracle Clusterware is stopped and successfully de-configured on node "sett2"


Oracle Clusterware is stopped and de-configured successfully.


Successfully detached Oracle home '/oracle/app/11.2.0.3/grid_2' from the central inventory on the local node.


Successfully deleted directory '/oracle/app/11.2.0.3/grid_2' on the local node.


Oracle Universal Installer cleanup was successful.





For complete clean up of Oracle Clusterware software from the system, deinstall the following old clusterware home(s). Refer to Clusterware Install guide of respective old release for details.


    /oracle/app/11.2.0.3/grid on nodes : sett2


Oracle deinstall tool successfully cleaned up temporary directories.


#######################################################################








############# ORACLE DEINSTALL & DECONFIG TOOL END #############



Step#13 NODE  From Node1 

Purpose Updating Grid Inventory on Node1

Run-From User Must be execute from Grid or Oracle user (OS USER)

Path /oracle/app/11.2.0.3/grid_2/oui/bin

Substep#

Purpose-Substep

Command-Syntax ./runInstaller -updateNodeList ORACLE_HOME=/oracle/app/11.2.0.3/grid_2 "CLUSTER_NODES={sett1}"  CRS=TRUE

Expected OutPut Starting Oracle Universal Installer...


Checking swap space: must be greater than 500 MB.   Actual 16384 MB    Passed


The inventory pointer is located at /etc/oraInst.loc


The inventory is located at /oracle/app/oraInventory


'UpdateNodeList' was successful.



Comments

Popular posts from this blog