Search
  • Sudipta Bhaskar

Apr 21 PSU fails "oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "install_srvm"

While applying 12.2 Apr 21 PSU (12.2.0.1.210420), it fails with "Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "install_srvm"


Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : oel7n2->/grid/app/oracle/12.2 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /grid/app/oracle/12.2, host: oel7n2.
Command failed:  /grid/app/oracle/12.2/OPatch/opatchauto  apply /dboracle/software/32540149 -oh /grid/app/oracle/12.2 -target_type cluster -binary -invPtrLoc /grid/app/oracle/12.2/oraInst.loc -jre /grid/app/oracle/12.2/OPatch/jre -persistresult /grid/app/oracle/12.2/opatchautocfg/db/sessioninfo/sessionresult_oel7n2_crs_1.ser -analyzedresult /grid/app/oracle/12.2/opatchautocfg/db/sessioninfo/sessionresult_analyze_oel7n2_crs_1.ser
Command failure output:
==Following patches FAILED in apply:

Patch: /dboracle/software/32540149/31802727
Log: /grid/app/oracle/12.2/cfgtoollogs/opatchauto/core/opatch/opatch2021-04-25_14-18-38PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target "install_srvm".

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sun Apr 25 14:20:15 2021
Time taken to complete the session 2 minutes, 0 second

 opatchauto failed with error code 42

Opatch logs has the following


[Apr 25, 2021 2:14:57 PM] [INFO]    Finish invoking 'patchgen' at Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [INFO]    Running make for target "install_srvm"
[Apr 25, 2021 2:14:57 PM] [INFO]    Start invoking 'make' at Sun Apr 25 14:14:57 EDT 2021Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [INFO]    Finish invoking 'make' at Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [WARNING] OUI-67200:Make failed to invoke "/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/grid/app/oracle/12.2 OPATCH_SESSION=napply"....'/bin/sh: /usr/bin/gcc: No such file or directory
                                    make: *** [/grid/app/oracle/12.2/rdbms/lib/config.o] Error 127
                                    '
[Apr 25, 2021 2:14:57 PM] [INFO]    Re-link fails on target "install_srvm".
[Apr 25, 2021 2:14:57 PM] [INFO]    Running make for target "install"
[Apr 25, 2021 2:14:57 PM] [INFO]    Start invoking 'make' at Sun Apr 25 14:14:57 EDT 2021Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [INFO]    Finish invoking 'make' at Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [INFO]    --------------------------------------------------------------------------------
                                    Failed to run make commands. Please contact Oracle Support.
[Apr 25, 2021 2:14:57 PM] [SEVERE]  OUI-67115:OPatch failed to restore OH '/grid/app/oracle/12.2'. Consult OPatch document to restore the home manually before proceeding.
[Apr 25, 2021 2:14:57 PM] [WARNING] OUI-67124:
                                    NApply was not able to restore the home.  Please invoke the following scripts:
                                      - restore.[sh,bat]
                                      - make.txt (Unix only)
                                    to restore the ORACLE_HOME.  They are located under
                                    "/grid/app/oracle/12.2/.patch_storage/NApply/2021-04-25_14-13-23PM"
[Apr 25, 2021 2:14:57 PM] [SEVERE]  OUI-67073:UtilSession failed: Re-link fails on target "install_srvm".
[Apr 25, 2021 2:14:57 PM] [INFO]    --------------------------------------------------------------------------------
[Apr 25, 2021 2:14:57 PM] [INFO]    The following warnings have occurred during OPatch execution:
[Apr 25, 2021 2:14:57 PM] [INFO]    1) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/grid/app/oracle/12.2 OPATCH_SESSION=napply"....'/bin/sh: /usr/bin/gcc: No such file or directory
                                    make: *** [/grid/app/oracle/12.2/rdbms/lib/config.o] Error 127
                                    '
[Apr 25, 2021 2:14:57 PM] [INFO]    2) OUI-67124:Re-link fails on target "install_srvm".
[Apr 25, 2021 2:14:57 PM] [INFO]    3) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=/grid/app/oracle/12.2 OPATCH_SESSION=napply"....'/bin/sh: /usr/bin/gcc: No such file or directory
                                    make: *** [/grid/app/oracle/12.2/rdbms/lib/config.o] Error 127
                                    '
[Apr 25, 2021 2:14:57 PM] [INFO]    4) OUI-67124:
                                    NApply was not able to restore the home.  Please invoke the following scripts:
                                      - restore.[sh,bat]
                                      - make.txt (Unix only)
                                    to restore the ORACLE_HOME.  They are located under
                                    "/grid/app/oracle/12.2/.patch_storage/NApply/2021-04-25_14-13-23PM"
[Apr 25, 2021 2:14:57 PM] [INFO]    --------------------------------------------------------------------------------
[Apr 25, 2021 2:14:57 PM] [INFO]    Finishing UtilSession at Sun Apr 25 14:14:57 EDT 2021
[Apr 25, 2021 2:14:57 PM] [INFO]    Log file location: /grid/app/oracle/12.2/cfgtoollogs/opatchauto/core/opatch/opatch2021-04-25_14-12-59PM_1.log
[root@OEL7N2 ~]#


Investigation :


According to Opatchauto: Applying GIRU fails with error " OPATCHAUTO-68061" (Doc ID 2390188.1) this is due to a internal Bug Bug:26413725.


Solution :


As we can't copy paste the solution from oracle support doc, I will be showing the steps by steps solution of fixing it.


1. Perform the following by root user. (on the server which opatchauto failed)

# $GRID_HOME/crs/install/rootcrs.sh -lock


[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -lock
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n2/crsconfig/crslock_oel7n2_2021-04-25_02-46-24PM.log
2021/04/25 14:46:27 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

2. Install gcc package. (on the server which opatchauto failed)

[root@OEL7N2 ~]# yum install gcc
Loaded plugins: langpacks, ulninfo
ol7_UEKR5                                                                                                                                        | 2.5 kB  00:00:00
ol7_latest                                                                                                                                       | 2.7 kB  00:00:00
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-44.0.3.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-44.0.3.el7 for package: gcc-4.8.5-44.0.3.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-44.0.3.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-44.0.3.el7.x86_64
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-44.0.3.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================================================================
 Package                             Arch                                Version                                          Repository                               Size
========================================================================================================================================================================
Installing:
 gcc                                 x86_64                              4.8.5-44.0.3.el7                                 ol7_latest                               16 M
Installing for dependencies:
 cpp                                 x86_64                              4.8.5-44.0.3.el7                                 ol7_latest                              5.9 M
 libmpc                              x86_64                              1.0.1-3.el7                                      ol7_latest                               49 k
 mpfr                                x86_64                              3.1.1-4.el7                                      ol7_latest                              198 k

Transaction Summary
========================================================================================================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 22 M
Installed size: 53 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/4): cpp-4.8.5-44.0.3.el7.x86_64.rpm                                                                                                           | 5.9 MB  00:00:01
(2/4): libmpc-1.0.1-3.el7.x86_64.rpm                                                                                                             |  49 kB  00:00:00
(3/4): mpfr-3.1.1-4.el7.x86_64.rpm                                                                                                               | 198 kB  00:00:00
(4/4): gcc-4.8.5-44.0.3.el7.x86_64.rpm                                                                                                           |  16 MB  00:00:02
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                   7.7 MB/s |  22 MB  00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : mpfr-3.1.1-4.el7.x86_64                                                                                                                              1/4
  Installing : libmpc-1.0.1-3.el7.x86_64                                                                                                                            2/4
  Installing : cpp-4.8.5-44.0.3.el7.x86_64                                                                                                                          3/4
  Installing : gcc-4.8.5-44.0.3.el7.x86_64                                                                                                                          4/4
  Verifying  : gcc-4.8.5-44.0.3.el7.x86_64                                                                                                                          1/4
  Verifying  : cpp-4.8.5-44.0.3.el7.x86_64                                                                                                                          2/4
  Verifying  : libmpc-1.0.1-3.el7.x86_64                                                                                                                            3/4
  Verifying  : mpfr-3.1.1-4.el7.x86_64                                                                                                                              4/4

Installed:
  gcc.x86_64 0:4.8.5-44.0.3.el7

Dependency Installed:
  cpp.x86_64 0:4.8.5-44.0.3.el7                            libmpc.x86_64 0:1.0.1-3.el7                            mpfr.x86_64 0:3.1.1-4.el7

Complete!
[root@OEL7N2 ~]#

3. If multibyte character environment, unset NLS_LANG. (on the server which opatchauto failed)

# unset NLS_LANG

Please refer 1991534.1


[root@OEL7N2 ~]# crsctl query crs releasepatch
Oracle Clusterware release patch level is [0] and no patches have been applied on the local node.
[root@OEL7N2 ~]# unset NLS_LANG

4. Apply the GIRU manually. (on the server which opatchauto failed)

See Section 5 in Doc ID 2246888.1: Supplemental Readme - Grid Infrastructure Release Update 12.2.0.1.x / 18c /19c

[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-25_02-57-31PM.log
2021/04/25 14:57:33 CLSRSC-378: Failed to get the configured node role for the local node

CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7n2'
CRS-2676: Start of 'ora.storage' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7n2'
CRS-2676: Start of 'ora.crf' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7n2'
CRS-2676: Start of 'ora.crsd' on 'oel7n2' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: oel7n2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n1'
CRS-2672: Attempting to start 'ora.chad' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n2'
CRS-2676: Start of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n2'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ons' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'oel7n2'
CRS-2676: Start of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.ARCH.dg' on 'oel7n2'
CRS-2676: Start of 'ora.ARCH.dg' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.orcl.db' failed to start automatically.
CRS-2807: Resource 'ora.orcl.pdborcl_srv.svc' failed to start automatically.
CRS-6016: Resource auto-start has completed for server oel7n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7n2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.VOTE.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel7n2'
CRS-2677: Stop of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.VOTE.dg' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ARCH.dg' on 'oel7n2'
CRS-2677: Stop of 'ora.ARCH.dg' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n1'
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n1'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.ons' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7n2'
CRS-2677: Stop of 'ora.net1.network' on 'oel7n2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7n2' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crf' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7n2'
CRS-2677: Stop of 'ora.crf' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7n2'
CRS-2677: Stop of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7n2'
CRS-2677: Stop of 'ora.cssd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7n2'
CRS-2677: Stop of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7n2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2021/04/25 14:59:34 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/04/25 14:59:44 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/04/25 14:59:44 CLSRSC-347: Successfully unlock /grid/app/oracle/12.2


[root@OEL7N2 ~]# /grid/app/oracle/12.2/OPatch/opatch apply -oh /grid/app/oracle/12.2 -local /dboracle/software/32540149/31802727
The user is root. OPatch cannot continue if the user is root.

OPatch failed with error code 255
[root@OEL7N2 ~]# su - oracle
Last login: Sun Apr 25 14:59:44 EDT 2021 on pts/0
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
ORACLE_HOME = [/home/oracle] ? ^C
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? ASM2
ORACLE_HOME = [/home/oracle] ? /grid/app/oracle/12.2
The Oracle base has been set to /u01/app/grid
[oracle@OEL7N2 ~]$ /grid/app/oracle/12.2/OPatch/opatch apply -oh /grid/app/oracle/12.2 -local /dboracle/software/32540149/31802727
Oracle Interim Patch Installer version 12.2.0.1.24
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /grid/app/oracle/12.2
Central Inventory : /u01/app/oraInventory
   from           : /grid/app/oracle/12.2/oraInst.loc
OPatch version    : 12.2.0.1.24
OUI version       : 12.2.0.1.4
Log file location : /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-25_15-04-00PM_1.log

Verifying environment and performing prerequisite checks...

--------------------------------------------------------------------------------
Start OOP by Prereq process.
Launch OOP...

Oracle Interim Patch Installer version 12.2.0.1.24
Copyright (c) 2021, Oracle Corporation.  All rights reserved.


Oracle Home       : /grid/app/oracle/12.2
Central Inventory : /u01/app/oraInventory
   from           : /grid/app/oracle/12.2/oraInst.loc
OPatch version    : 12.2.0.1.24
OUI version       : 12.2.0.1.4
Log file location : /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-25_15-04-14PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   31802727

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/grid/app/oracle/12.2')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '31802727' to OH '/grid/app/oracle/12.2'

Patching component oracle.rdbms, 12.2.0.1.0...

Patching component oracle.has.common.cvu, 12.2.0.1.0...

Patching component oracle.has.cvu, 12.2.0.1.0...

Patching component oracle.has.common, 12.2.0.1.0...

Patching component oracle.has.db, 12.2.0.1.0...

Patching component oracle.xag, 12.2.0.1.0...

Patching component oracle.has.deconfig, 12.2.0.1.0...

Patching component oracle.has.rsf, 12.2.0.1.0...

Patching component oracle.has.crs, 12.2.0.1.0...
Patch 31802727 successfully applied.
Log file location: /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-25_15-04-14PM_1.log

OPatch succeeded.
[oracle@OEL7N2 ~]$


[oracle@OEL7N2 ~]$ /grid/app/oracle/12.2//rdbms/install/rootadd_rdbms.sh
You must be logged in as user with UID as zero (e.g. root user) to run root.sh.
Log in as user with UID as zero (e.g. root user) and restart root.sh execution.
[oracle@OEL7N2 ~]$ exit
logout

[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -postpatch
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-25_03-10-02PM.log
2021/04/25 15:10:11 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/04/25 15:10:23 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/04/25 15:10:29 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7n2'
CRS-2676: Start of 'ora.storage' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7n2'
CRS-2676: Start of 'ora.crf' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7n2'
CRS-2676: Start of 'ora.crsd' on 'oel7n2' succeeded
CRS-6017: Processing resource auto-start for servers: oel7n2
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2672: Attempting to start 'ora.chad' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n2'
CRS-2676: Start of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n2'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ons' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'oel7n2'
CRS-2676: Start of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.ARCH.dg' on 'oel7n2'
CRS-2676: Start of 'ora.ARCH.dg' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.orcl.db' failed to start automatically.
CRS-2807: Resource 'ora.orcl.pdborcl_srv.svc' failed to start automatically.
CRS-6016: Resource auto-start has completed for server oel7n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].

Need to run the following in both the nodes.

Check the contents of $GRID_HOME/rdbms/lib/config.c and $ORACLE_HOME/rdbms/lib/config.c are same.

[root@OEL7N1 ~]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7n1'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7n1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel7n1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.chad' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'oel7n1'
CRS-2672: Attempting to start 'ora.orcl.db' on 'oel7n2'
CRS-5017: The resource action "ora.orcl.db start" encountered the following error:
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/dboracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl2.ora'
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/oel7n2/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.orcl.db' on 'oel7n2' failed
CRS-2679: Attempting to clean 'ora.orcl.db' on 'oel7n2'
CRS-2677: Stop of 'ora.qosmserver' on 'oel7n1' succeeded
CRS-2681: Clean of 'ora.orcl.db' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.orcl.pdborcl_srv.svc' on 'oel7n1'
CRS-2677: Stop of 'ora.orcl.pdborcl_srv.svc' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.orcl.db' on 'oel7n1'
CRS-2677: Stop of 'ora.orcl.db' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.ARCH.dg' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.cvu' on 'oel7n1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.ARCH.dg' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.VOTE.dg' on 'oel7n1'
CRS-2677: Stop of 'ora.DATA.dg' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.VOTE.dg' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n1'
CRS-2677: Stop of 'ora.cvu' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n1'
CRS-2677: Stop of 'ora.scan2.vip' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'oel7n1'
CRS-2677: Stop of 'ora.mgmtdb' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'oel7n1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n1' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.oel7n1.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.oel7n1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'oel7n2'
CRS-2672: Attempting to start 'ora.qosmserver' on 'oel7n2'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.cvu' on 'oel7n2'
CRS-2672: Attempting to start 'ora.oel7n1.vip' on 'oel7n2'
CRS-2676: Start of 'ora.cvu' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.oel7n1.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.scan3.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.MGMTLSNR' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'oel7n2'
CRS-2676: Start of 'ora.qosmserver' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.mgmtdb' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'oel7n2'
CRS-2676: Start of 'ora.chad' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7n1'
CRS-2677: Stop of 'ora.ons' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7n1'
CRS-2677: Stop of 'ora.net1.network' on 'oel7n1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7n1' has completed
CRS-2677: Stop of 'ora.crsd'