Softwarepatch level different in RAC nodes after GI PSU apply

Updated: Apr 28, 2021

crsctl query crs softwarepatch showing different output in different nodes.

[oracle@OEL7N1 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node oel7n1 is [1831702305].

[oracle@OEL7N1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].

[oracle@OEL7N2 ~]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node oel7n2 is [4182837753].

[oracle@OEL7N2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].

kfod op=patches shows node2 has lesser patches than node1 however Grid Opatch lsinventory shows exactly same patches on both nodes.

[oracle@OEL7N1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
32542421;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:RELEASE) (32542421)
32507738;Database Apr 2021 Release Update : 12.2.0.1.210420 (32507738)
32231681;ACFS JAN 2021 RELEASE UPDATE 12.2.0.1.210119 (32231681)
31802727;OCW OCT 2020 RELEASE UPDATE 12.2.0.1.201020 (31802727)
26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)

[oracle@OEL7N2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
32542421;TOMCAT RELEASE UPDATE 12.2.0.1.0(ID:RELEASE) (32542421)
32231681;ACFS JAN 2021 RELEASE UPDATE 12.2.0.1.210119 (32231681)
32507738;Database Apr 2021 Release Update : 12.2.0.1.210420 (32507738)
31802727;OCW OCT 2020 RELEASE UPDATE 12.2.0.1.201020 (31802727)
26839277;DBWLM RELEASE UPDATE 12.2.0.1.0(ID:170913) (26839277)

[oracle@OEL7N1 ~]$ kfod op=patches
---------------
List of Patches
===============
26839277 =======>> Missing in Node2
31802727
32231681
32507738
32542421

[oracle@OEL7N2 ~]$ kfod op=patches
---------------
List of Patches
===============
31802727
32231681
32507738
32542421

Oracle doc 2176790.1 says that Using different Opatch version in different node is the reason but in my case both node has same opatch verion.

[oracle@OEL7N1 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.24

OPatch succeeded.

[oracle@OEL7N2 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.24

OPatch succeeded.

Oracle Doc 2266613.1 says that oneoff patch difference can also be an issue. But here that is also not the case.

[oracle@OEL7N1 ~]$ ls -lrt $ORACLE_HOME/inventory/oneoffs
total 0
drwxr-xr-x 4 oracle oinstall 29 Apr 25 13:24 26839277
drwxr-xr-x 4 oracle oinstall 29 Apr 25 13:26 31802727
drwxr-xr-x 4 oracle oinstall 29 Apr 25 13:29 32231681
drwxr-xr-x 4 oracle oinstall 29 Apr 25 13:33 32507738
drwxr-xr-x 4 oracle oinstall 29 Apr 25 13:34 32542421

[oracle@OEL7N2 ~]$ ls -lrt $ORACLE_HOME/inventory/oneoffs
total 0
drwxr-xr-x 4 oracle oinstall 29 Apr 25 14:13 26839277
drwxr-xr-x 4 oracle oinstall 29 Apr 25 14:14 31802727
drwxr-xr-x 4 oracle oinstall 29 Apr 25 16:29 32507738
drwxr-xr-x 4 oracle oinstall 29 Apr 25 16:35 32231681
drwxr-xr-x 4 oracle oinstall 29 Apr 25 16:37 32542421

Let's try to fix it by unistalling the patch from GI home and installing it again.

From node 2

[oracle@OEL7N2 ~]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54331(asmdba),54332(asmadmin)
[oracle@OEL7N2 ~]$ echo $ORACLE_SID
orcl2
[oracle@OEL7N2 ~]$ echo $ORACLE_HOME
/dboracle/app/oracle/product/12.2.0/dbhome_1
[oracle@OEL7N2 ~]$
[oracle@OEL7N2 ~]$ srvctl stop home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n orcl2
PRKO-2006 : Invalid node name: orcl2
[oracle@OEL7N2 ~]$ srvctl stop home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n OEL7N2
[oracle@OEL7N2 ~]$

[root@OEL7N2 ~]# oraenv
ORACLE_SID = [+ASM1] ? +ASM2
The Oracle base has been set to /u01/app/grid
[root@OEL7N2 ~]# echo $ORACLE_HOME

[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -prepatch -rollback
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-26_04-34-50PM.log
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7n2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.ARCH.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel7n2'
CRS-2677: Stop of 'ora.ARCH.dg' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.VOTE.dg' on 'oel7n2'
CRS-2677: Stop of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.VOTE.dg' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n1'
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n1'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.ons' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7n2'
CRS-2677: Stop of 'ora.net1.network' on 'oel7n2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7n2' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crf' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7n2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.crf' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.storage' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2677: Stop of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7n2'
CRS-2677: Stop of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7n2'
CRS-2677: Stop of 'ora.cssd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7n2'
CRS-2677: Stop of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7n2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2021/04/26 16:35:42 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 16:37:22 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 16:37:23 CLSRSC-347: Successfully unlock /grid/app/oracle/12.2
[root@OEL7N2 ~]#

[root@OEL7N2 ~]# su - oracle
Last login: Mon Apr 26 16:37:28 EDT 2021 on pts/1
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM2
The Oracle base has been set to /u01/app/grid
[oracle@OEL7N2 ~]$ echo $ORACLE_HOME
/grid/app/oracle/12.2
[oracle@OEL7N2 ~]$ /grid/app/oracle/12.2/OPatch/opatch nrollback -local -id %26839277% TRACKING BUG -oh /grid/app/oracle/12.2
Oracle Interim Patch Installer version 12.2.0.1.24
Copyright (c) 2021, Oracle Corporation. All rights reserved.

Argument(s) Error... The pattern of the specified patch(es) are not allowed. Please enter valid patch(es).

Please check the arguments and try again.

OPatch failed with error code 135

[oracle@OEL7N2 ~]$ /grid/app/oracle/12.2/OPatch/opatch rollback -local -id 26839277 -oh /grid/app/oracle/12.2
Oracle Interim Patch Installer version 12.2.0.1.24
Copyright (c) 2021, Oracle Corporation. All rights reserved.

Oracle Home : /grid/app/oracle/12.2
Central Inventory : /u01/app/oraInventory
from : /grid/app/oracle/12.2/oraInst.loc
OPatch version : 12.2.0.1.24
OUI version : 12.2.0.1.4
Log file location : /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-26_16-43-52PM_1.log

Patches will be rolled back in the following order:
26839277
The following patch(es) will be rolled back: 26839277

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/grid/app/oracle/12.2')

Is the local system ready for patching? [y|n]
y
User Responded with: Y

Rolling back patch 26839277...

RollbackSession rolling back interim patch '26839277' from OH '/grid/app/oracle/12.2'

Patching component oracle.wlm.dbwlm, 12.2.0.1.0...
RollbackSession removing interim patch '26839277' from inventory
Log file location: /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-26_16-43-52PM_1.log

OPatch succeeded.
[oracle@OEL7N2 ~]$

[root@OEL7N2 ~]# /grid/app/oracle/12.2/rdbms/install/rootadd_rdbms.sh
[root@OEL7N2 ~]#
[root@OEL7N2 ~]#
[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -postpatch -rollback
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-26_04-45-56PM.log
2021/04/26 16:46:03 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 16:46:18 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 16:46:21 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7n2'
CRS-2676: Start of 'ora.storage' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7n2'
CRS-2676: Start of 'ora.crf' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7n2'
CRS-2676: Start of 'ora.crsd' on 'oel7n2' succeeded
CRS-6017: Processing resource auto-start for servers: oel7n2
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2672: Attempting to start 'ora.ons' on 'oel7n2'
CRS-2672: Attempting to start 'ora.chad' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n2'
CRS-2676: Start of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n2'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.ons' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'oel7n2'
CRS-2676: Start of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2664: Resource 'ora.orcl.db' is already running on 'oel7n1'
CRS-6016: Resource auto-start has completed for server oel7n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
[root@OEL7N2 ~]#

[root@OEL7N2 ~]# su - oracle
Last login: Mon Apr 26 16:53:29 EDT 2021 on pts/1
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [oracle] ? orcl2
The Oracle base has been set to /dboracle/app/oracle
[oracle@OEL7N2 ~]$ echo $ORACLE_HOME
/dboracle/app/oracle/product/12.2.0/dbhome_1
[oracle@OEL7N2 ~]$ srvctl start home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n OEL7N2
[oracle@OEL7N2 ~]$

Let's install it again.

[oracle@OEL7N2 ~]$ >/tmp/status.txt
[oracle@OEL7N2 ~]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54331(asmdba),54332(asmadmin)
[oracle@OEL7N2 ~]$ srvctl stop home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n OEL7N2
PRKO-3244 : The '-statfile <state_file>' option value "/tmp/status.txt" already exists
[oracle@OEL7N2 ~]$
[oracle@OEL7N2 ~]$ RM /tmp/status.txt
-bash: RM: command not found
[oracle@OEL7N2 ~]$ rm /tmp/status.txt
[oracle@OEL7N2 ~]$ srvctl stop home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n OEL7N2
[oracle@OEL7N2 ~]$
[oracle@OEL7N2 ~]$ exit
logout

[root@OEL7N2 ~]# . oraenv
ORACLE_SID = [+ASM1] ? +ASM2
The Oracle base has been set to /u01/app/grid
[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -prepatch
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-26_04-59-26PM.log
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [0].
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crsd' on 'oel7n2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'oel7n2'
CRS-2673: Attempting to stop 'ora.ARCH.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.VOTE.dg' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.chad' on 'oel7n2'
CRS-2677: Stop of 'ora.ARCH.dg' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.VOTE.dg' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n2'
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n1'
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n1'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.ons' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oel7n2'
CRS-2677: Stop of 'ora.net1.network' on 'oel7n2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oel7n2' has completed
CRS-2677: Stop of 'ora.crsd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.crf' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oel7n2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.crf' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.asm' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.evmd' on 'oel7n2'
CRS-2677: Stop of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oel7n2'
CRS-2677: Stop of 'ora.cssd' on 'oel7n2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oel7n2'
CRS-2677: Stop of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oel7n2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2021/04/26 17:00:34 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 17:02:14 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 17:02:15 CLSRSC-347: Successfully unlock /grid/app/oracle/12.2
[root@OEL7N2 ~]#

[root@OEL7N2 ~]# su - oracle
Last login: Mon Apr 26 17:02:18 EDT 2021 on pts/1
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM2
The Oracle base has been set to /u01/app/grid
[oracle@OEL7N2 ~]$
[oracle@OEL7N2 ~]$ /grid/app/oracle/12.2/OPatch/opatch apply -oh /grid/app/oracle/12.2 -local /dboracle/software/32540149/26839277
Oracle Interim Patch Installer version 12.2.0.1.24
Copyright (c) 2021, Oracle Corporation. All rights reserved.

Oracle Home : /grid/app/oracle/12.2
Central Inventory : /u01/app/oraInventory
from : /grid/app/oracle/12.2/oraInst.loc
OPatch version : 12.2.0.1.24
OUI version : 12.2.0.1.4
Log file location : /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-26_17-04-01PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 26839277

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/grid/app/oracle/12.2')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '26839277' to OH '/grid/app/oracle/12.2'

Patching component oracle.wlm.dbwlm, 12.2.0.1.0...
Patch 26839277 successfully applied.
Log file location: /grid/app/oracle/12.2/cfgtoollogs/opatch/opatch2021-04-26_17-04-01PM_1.log

OPatch succeeded.
[oracle@OEL7N2 ~]$

[oracle@OEL7N2 ~]$ exit
logout
[root@OEL7N2 ~]#
[root@OEL7N2 ~]# /grid/app/oracle/12.2/rdbms/install/rootadd_rdbms.sh
[root@OEL7N2 ~]# /grid/app/oracle/12.2/crs/install/rootcrs.sh -postpatch
Using configuration parameter file: /grid/app/oracle/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/oel7n2/crsconfig/crspatch_oel7n2_2021-04-26_05-06-00PM.log
2021/04/26 17:06:15 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 17:06:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/04/26 17:06:42 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.evmd' on 'oel7n2'
CRS-2676: Start of 'ora.mdnsd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.evmd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'oel7n2'
CRS-2676: Start of 'ora.gpnpd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'oel7n2'
CRS-2676: Start of 'ora.gipcd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'oel7n2'
CRS-2676: Start of 'ora.cssdmonitor' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'oel7n2'
CRS-2672: Attempting to start 'ora.diskmon' on 'oel7n2'
CRS-2676: Start of 'ora.diskmon' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cssd' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ctssd' on 'oel7n2'
CRS-2676: Start of 'ora.ctssd' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'oel7n2'
CRS-2676: Start of 'ora.storage' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'oel7n2'
CRS-2676: Start of 'ora.crf' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'oel7n2'
CRS-2676: Start of 'ora.crsd' on 'oel7n2' succeeded
CRS-6017: Processing resource auto-start for servers: oel7n2
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2'
CRS-2673: Attempting to stop 'ora.oel7n2.vip' on 'oel7n1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1'
CRS-2672: Attempting to start 'ora.chad' on 'oel7n2'
CRS-2672: Attempting to start 'ora.ons' on 'oel7n2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'oel7n1'
CRS-2677: Stop of 'ora.oel7n2.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.oel7n2.vip' on 'oel7n2'
CRS-2676: Start of 'ora.chad' on 'oel7n2' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'oel7n1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'oel7n2'
CRS-2676: Start of 'ora.oel7n2.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2'
CRS-2676: Start of 'ora.ons' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'oel7n2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'oel7n2' succeeded
CRS-2676: Start of 'ora.asm' on 'oel7n2' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'oel7n2'
CRS-2676: Start of 'ora.DATA.dg' on 'oel7n2' succeeded
CRS-2664: Resource 'ora.orcl.db' is already running on 'oel7n1'
CRS-6016: Resource auto-start has completed for server oel7n2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1831702305].
[root@OEL7N2 ~]#

root@OEL7N2 ~]# su - oracle
Last login: Mon Apr 26 17:15:55 EDT 2021 on pts/1
[oracle@OEL7N2 ~]$ . oraenv
ORACLE_SID = [oracle] ? orcl2
The Oracle base has been set to /dboracle/app/oracle

[oracle@OEL7N2 ~]$ srvctl start home -o /dboracle/app/oracle/product/12.2.0/dbhome_1 -s /tmp/status.txt -n OEL7N2
[oracle@OEL7N2 ~]$

Now, let's check the patch level again

[oracle@OEL7N1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1831702305].

[oracle@OEL7N2 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1831702305].

We can see that patch levels are same now. Issue resolved.

6920