Search
  • Sudipta Bhaskar

How To Upgrade RAC Grid Infrastructure from 12.2 to 19.8 in Silent Mode

Updated: May 3

I Searched over the internet and couldn't find many post on RAC Grid Upgrade in silent mode. That's why I thought of writing this post.



Right now I have a 12.2 RAC Setup and I will upgrade it to 19.8.


Following are the network configuration.


# Public
192.168.126.11   OEL7N1.localdomain        OEL7N1
192.168.126.21   OEL7N2.localdomain        OEL7N2
# Private
192.168.100.11   OEL7N1-priv.localdomain   OEL7N1-priv
192.168.100.21   OEL7N2-priv.localdomain   OEL7N2-priv
# Virtual
192.168.126.31   OEL7N1-vip.localdomain    OEL7N1-vip
192.168.126.41   OEL7N2-vip.localdomain    OEL7N2-vip
# SCAN
#192.168.126.50   scan.localdomain scan
#192.168.126.60   scan.localdomain scan
#192.168.126.70   scan.localdomain scan

I have created directory structure /grid/app/oracle/19.3/grid in both nodes. This will serve as the Grid Home for 19c.


Downloaded 19.3 Grid Binary from oracle website.


Let' s copy LINUX.X64_193000_grid_home.zip to the mentioned location in node 1 only.


sudip@DESKTOP-RDIUEUO MINGW64 /d/Softwares
$ scp LINUX.X64_193000_grid_home.zip oracle@192.168.126.11:/grid/app/oracle/19.3/grid
oracle@192.168.126.11's password:
LINUX.X64_193000_grid_home.zip                                                                                                                                100% 2755MB  28.6MB/s   01:36

Let's unzip the binary.


[oracle@OEL7N1 grid]$ pwd
/grid/app/oracle/19.3/grid
[oracle@OEL7N1 grid]$ ls -ltr
total 2821472
-rw-r--r-- 1 oracle oinstall 2889184573 May  2 13:25 LINUX.X64_193000_grid_home.zip
[oracle@OEL7N1 grid]$ unzip LINUX.X64_193000_grid_home.zip

This is 19.3 binary. We need to take it to 19.8.0.0.200714. We will patch this binary with the PSU.


For 19.8.0.0.200714, we need patch number 31305339. I have downloaded and copied it to /dboracle/software. Unzipped it after copying. Let's apply it.


Before that installed latest OPatch.


[oracle@OEL7N1 OPatch]$ unzip -d /grid/app/oracle/19.3/grid /dboracle/software/p6880880_190000_Linux-x86-64.zip
Archive:  /dboracle/software/p6880880_190000_Linux-x86-64.zip
replace /grid/app/oracle/19.3/grid/OPatch/emdpatch.pl? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
  inflating: /grid/app/oracle/19.3/grid/OPatch/emdpatch.pl
  
  [oracle@OEL7N1 OPatch]$ ./opatch version
OPatch Version: 12.2.0.1.24

OPatch succeeded.  

[oracle@OEL7N1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent -applyRU /dboracle/software/31305339
Preparing the home to patch...
Applying the patch /dboracle/software/31305339...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2021-05-02_02-18-58PM/installerPatchActions_2021-05-02_02-18-58PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-40426] Grid installation option has not been specified.
   ACTION: Specify the valid installation option.
Note : This error is because it was trying to install grid 19c but our intention here is only to apply the latest RU.

/grid/app/oracle/19.3/grid
[oracle@OEL7N1 grid]$
[oracle@OEL7N1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent -applyRU /dboracle/software/31305339
Preparing the home to patch...
Applying the patch /dboracle/software/31305339...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2021-05-02_02-54-04PM/installerPatchActions_2021-05-02_02-54-04PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

^C[oracle@OEL7N1 grid]$

Let's run the pre checks. Run it from 19c home.


[oracle@OEL7N1 grid]$
[oracle@OEL7N1 grid]$ /grid/app/oracle/19.3/grid/runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /grid/app/oracle/12.2 -dest_crshome /grid/app/oracle/19.3/grid -dest_version 19.8.0.0.200714 -fixupnoexec -verbose

Verifying Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        4.826GB (5060456.0KB)     8GB (8388608.0KB)         failed
  oel7n1        4.826GB (5060472.0KB)     8GB (8388608.0KB)         failed
Verifying Physical Memory ...FAILED (PRVF-7530)
Verifying Available Physical Memory ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        2.4753GB (2595580.0KB)    50MB (51200.0KB)          passed
  oel7n1        3.1558GB (3309116.0KB)    50MB (51200.0KB)          passed
Verifying Available Physical Memory ...PASSED
Verifying Swap Size ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        10GB (1.0485756E7KB)      4.826GB (5060456.0KB)     passed
  oel7n1        10GB (1.0485756E7KB)      4.826GB (5060472.0KB)     passed
Verifying Swap Size ...PASSED
Verifying Free Space: oel7n2:/usr,oel7n2:/var,oel7n2:/etc,oel7n2:/sbin,oel7n2:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              oel7n2        /             15.7295GB     25MB          passed
  /var              oel7n2        /             15.7295GB     5MB           passed
  /etc              oel7n2        /             15.7295GB     25MB          passed
  /sbin             oel7n2        /             15.7295GB     10MB          passed
  /tmp              oel7n2        /             15.7295GB     1GB           passed
Verifying Free Space: oel7n2:/usr,oel7n2:/var,oel7n2:/etc,oel7n2:/sbin,oel7n2:/tmp ...PASSED
Verifying Free Space: oel7n1:/usr,oel7n1:/var,oel7n1:/etc,oel7n1:/sbin,oel7n1:/tmp ...
  Path              Node Name     Mount point   Available     Required      Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  /usr              oel7n1        /             15.8174GB     25MB          passed
  /var              oel7n1        /             15.8174GB     5MB           passed
  /etc              oel7n1        /             15.8174GB     25MB          passed
  /sbin             oel7n1        /             15.8174GB     10MB          passed
  /tmp              oel7n1        /             15.8174GB     1GB           passed
Verifying Free Space: oel7n1:/usr,oel7n1:/var,oel7n1:/etc,oel7n1:/sbin,oel7n1:/tmp ...PASSED
Verifying User Existence: oracle ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  oel7n2        passed                    exists(54321)
  oel7n1        passed                    exists(54321)

  Verifying Users With Same UID: 54321 ...PASSED
Verifying User Existence: oracle ...PASSED
Verifying Group Existence: asmadmin ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  oel7n2        passed                    exists
  oel7n1        passed                    exists
Verifying Group Existence: asmadmin ...PASSED
Verifying Group Existence: asmdba ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  oel7n2        passed                    exists
  oel7n1        passed                    exists
Verifying Group Existence: asmdba ...PASSED
Verifying Group Existence: oinstall ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  oel7n2        passed                    exists
  oel7n1        passed                    exists
Verifying Group Existence: oinstall ...PASSED
Verifying Group Membership: asmadmin ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            yes           yes           yes           passed
  oel7n1            yes           yes           yes           passed
Verifying Group Membership: asmadmin ...PASSED
Verifying Group Membership: asmdba ...
  Node Name         User Exists   Group Exists  User in Group  Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            yes           yes           yes           passed
  oel7n1            yes           yes           yes           passed
Verifying Group Membership: asmdba ...PASSED
Verifying Group Membership: oinstall(Primary) ...
  Node Name         User Exists   Group Exists  User in Group  Primary       Status
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n2            yes           yes           yes           yes           passed
  oel7n1            yes           yes           yes           yes           passed
Verifying Group Membership: oinstall(Primary) ...PASSED
Verifying Run Level ...
  Node Name     run level                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        3                         3,5                       passed
  oel7n1        3                         3,5                       passed
Verifying Run Level ...PASSED
Verifying Hard Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            hard          65536         65536         passed
  oel7n1            hard          65536         65536         passed
Verifying Hard Limit: maximum open file descriptors ...PASSED
Verifying Soft Limit: maximum open file descriptors ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            soft          1024          1024          passed
  oel7n1            soft          1024          1024          passed
Verifying Soft Limit: maximum open file descriptors ...PASSED
Verifying Hard Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            hard          16384         16384         passed
  oel7n1            hard          16384         16384         passed
Verifying Hard Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum user processes ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            soft          16384         2047          passed
  oel7n1            soft          16384         2047          passed
Verifying Soft Limit: maximum user processes ...PASSED
Verifying Soft Limit: maximum stack size ...
  Node Name         Type          Available     Required      Status
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            soft          10240         10240         passed
  oel7n1            soft          10240         10240         passed
Verifying Soft Limit: maximum stack size ...PASSED
Verifying Oracle patch:28553832 ...PASSED
Verifying Oracle patch:27006180 ...
  Node Name     Applied                   Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  oel7n1        27006180                  27006180                  passed
  oel7n2        27006180                  27006180                  passed
Verifying Oracle patch:27006180 ...PASSED
Verifying This test checks that the source home "/grid/app/oracle/12.2" is suitable for upgrading to version "19.8.0.0.200714". ...PASSED
Verifying Architecture ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        x86_64                    x86_64                    passed
  oel7n1        x86_64                    x86_64                    passed
Verifying Architecture ...PASSED
Verifying OS Kernel Version ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        4.14.35-2047.502.4.1.el7uek.x86_64  4.1.12                    passed
  oel7n1        4.14.35-2047.502.4.1.el7uek.x86_64  4.1.12                    passed
Verifying OS Kernel Version ...PASSED
Verifying OS Kernel Parameter: semmsl ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            250           250           250           passed
  oel7n2            250           250           250           passed
Verifying OS Kernel Parameter: semmsl ...PASSED
Verifying OS Kernel Parameter: semmns ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            32000         32000         32000         passed
  oel7n2            32000         32000         32000         passed
Verifying OS Kernel Parameter: semmns ...PASSED
Verifying OS Kernel Parameter: semopm ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            100           100           100           passed
  oel7n2            100           100           100           passed
Verifying OS Kernel Parameter: semopm ...PASSED
Verifying OS Kernel Parameter: semmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            128           128           128           passed
  oel7n2            128           128           128           passed
Verifying OS Kernel Parameter: semmni ...PASSED
Verifying OS Kernel Parameter: shmmax ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            4398046511104  4398046511104  2590961664    passed
  oel7n2            4398046511104  4398046511104  2590953472    passed
Verifying OS Kernel Parameter: shmmax ...PASSED
Verifying OS Kernel Parameter: shmmni ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            4096          4096          4096          passed
  oel7n2            4096          4096          4096          passed
Verifying OS Kernel Parameter: shmmni ...PASSED
Verifying OS Kernel Parameter: shmall ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            1073741824    1073741824    1073741824    passed
  oel7n2            1073741824    1073741824    1073741824    passed
Verifying OS Kernel Parameter: shmall ...PASSED
Verifying OS Kernel Parameter: file-max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            6815744       6815744       6815744       passed
  oel7n2            6815744       6815744       6815744       passed
Verifying OS Kernel Parameter: file-max ...PASSED
Verifying OS Kernel Parameter: ip_local_port_range ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
  oel7n2            between 9000 & 65500  between 9000 & 65500  between 9000 & 65535  passed
Verifying OS Kernel Parameter: ip_local_port_range ...PASSED
Verifying OS Kernel Parameter: rmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            262144        262144        262144        passed
  oel7n2            262144        262144        262144        passed
Verifying OS Kernel Parameter: rmem_default ...PASSED
Verifying OS Kernel Parameter: rmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            4194304       4194304       4194304       passed
  oel7n2            4194304       4194304       4194304       passed
Verifying OS Kernel Parameter: rmem_max ...PASSED
Verifying OS Kernel Parameter: wmem_default ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            262144        262144        262144        passed
  oel7n2            262144        262144        262144        passed
Verifying OS Kernel Parameter: wmem_default ...PASSED
Verifying OS Kernel Parameter: wmem_max ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            1048576       1048576       1048576       passed
  oel7n2            1048576       1048576       1048576       passed
Verifying OS Kernel Parameter: wmem_max ...PASSED
Verifying OS Kernel Parameter: aio-max-nr ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            1048576       1048576       1048576       passed
  oel7n2            1048576       1048576       1048576       passed
Verifying OS Kernel Parameter: aio-max-nr ...PASSED
Verifying OS Kernel Parameter: panic_on_oops ...
  Node Name         Current       Configured    Required      Status        Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  oel7n1            1             1             1             passed
  oel7n2            1             1             1             passed
Verifying OS Kernel Parameter: panic_on_oops ...PASSED
Verifying Package: kmod-20-21 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        kmod(x86_64)-20-28.0.3.el7  kmod(x86_64)-20-21        passed
  oel7n1        kmod(x86_64)-20-28.0.3.el7  kmod(x86_64)-20-21        passed
Verifying Package: kmod-20-21 (x86_64) ...PASSED
Verifying Package: kmod-libs-20-21 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        kmod-libs(x86_64)-20-28.0.3.el7  kmod-libs(x86_64)-20-21   passed
  oel7n1        kmod-libs(x86_64)-20-28.0.3.el7  kmod-libs(x86_64)-20-21   passed
Verifying Package: kmod-libs-20-21 (x86_64) ...PASSED
Verifying Package: binutils-2.23.52.0.1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        binutils-2.27-44.base.0.3.el7  binutils-2.23.52.0.1      passed
  oel7n1        binutils-2.27-44.base.0.3.el7  binutils-2.23.52.0.1      passed
Verifying Package: binutils-2.23.52.0.1 ...PASSED
Verifying Package: compat-libcap1-1.10 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
  oel7n1        compat-libcap1-1.10-7.el7  compat-libcap1-1.10       passed
Verifying Package: compat-libcap1-1.10 ...PASSED
Verifying Package: libgcc-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libgcc(x86_64)-4.8.5-44.0.3.el7  libgcc(x86_64)-4.8.2      passed
  oel7n1        libgcc(x86_64)-4.8.5-44.0.3.el7  libgcc(x86_64)-4.8.2      passed
Verifying Package: libgcc-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libstdc++(x86_64)-4.8.5-44.0.3.el7  libstdc++(x86_64)-4.8.2   passed
  oel7n1        libstdc++(x86_64)-4.8.5-44.0.3.el7  libstdc++(x86_64)-4.8.2   passed
Verifying Package: libstdc++-4.8.2 (x86_64) ...PASSED
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libstdc++-devel(x86_64)-4.8.5-44.0.3.el7  libstdc++-devel(x86_64)-4.8.2  passed
  oel7n1        libstdc++-devel(x86_64)-4.8.5-44.0.3.el7  libstdc++-devel(x86_64)-4.8.2  passed
Verifying Package: libstdc++-devel-4.8.2 (x86_64) ...PASSED
Verifying Package: sysstat-10.1.5 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        sysstat-10.1.5-19.el7     sysstat-10.1.5            passed
  oel7n1        sysstat-10.1.5-19.el7     sysstat-10.1.5            passed
Verifying Package: sysstat-10.1.5 ...PASSED
Verifying Package: ksh ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        ksh                       ksh                       passed
  oel7n1        ksh                       ksh                       passed
Verifying Package: ksh ...PASSED
Verifying Package: make-3.82 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        make-3.82-24.el7          make-3.82                 passed
  oel7n1        make-3.82-24.el7          make-3.82                 passed
Verifying Package: make-3.82 ...PASSED
Verifying Package: glibc-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        glibc(x86_64)-2.17-323.0.1.el7_9  glibc(x86_64)-2.17        passed
  oel7n1        glibc(x86_64)-2.17-323.0.1.el7_9  glibc(x86_64)-2.17        passed
Verifying Package: glibc-2.17 (x86_64) ...PASSED
Verifying Package: glibc-devel-2.17 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        glibc-devel(x86_64)-2.17-323.0.1.el7_9  glibc-devel(x86_64)-2.17  passed
  oel7n1        glibc-devel(x86_64)-2.17-323.0.1.el7_9  glibc-devel(x86_64)-2.17  passed
Verifying Package: glibc-devel-2.17 (x86_64) ...PASSED
Verifying Package: libaio-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
  oel7n1        libaio(x86_64)-0.3.109-13.el7  libaio(x86_64)-0.3.109    passed
Verifying Package: libaio-0.3.109 (x86_64) ...PASSED
Verifying Package: libaio-devel-0.3.109 (x86_64) ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
  oel7n1        libaio-devel(x86_64)-0.3.109-13.el7  libaio-devel(x86_64)-0.3.109  passed
Verifying Package: libaio-devel-0.3.109 (x86_64) ...PASSED
Verifying Package: libvirt-devel-4.5.0-10.0 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libvirt-devel-4.5.0-36.el7_9.3  libvirt-devel-4.5.0-10.0  passed
  oel7n1        libvirt-devel-4.5.0-36.el7_9.3  libvirt-devel-4.5.0-10.0  passed
Verifying Package: libvirt-devel-4.5.0-10.0 ...PASSED
Verifying Package: libvirt-libs-4.5.0-10.0 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        libvirt-libs-4.5.0-36.el7_9.3  libvirt-libs-4.5.0-10.0   passed
  oel7n1        libvirt-libs-4.5.0-36.el7_9.3  libvirt-libs-4.5.0-10.0   passed
Verifying Package: libvirt-libs-4.5.0-10.0 ...PASSED
Verifying Package: nfs-utils-1.2.3-15 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        nfs-utils-1.3.0-0.68.0.1.el7  nfs-utils-1.2.3-15        passed
  oel7n1        nfs-utils-1.3.0-0.68.0.1.el7  nfs-utils-1.2.3-15        passed
Verifying Package: nfs-utils-1.2.3-15 ...PASSED
Verifying Package: smartmontools-6.2-4 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        smartmontools-7.0-2.el7   smartmontools-6.2-4       passed
  oel7n1        smartmontools-7.0-2.el7   smartmontools-6.2-4       passed
Verifying Package: smartmontools-6.2-4 ...PASSED
Verifying Package: net-tools-2.0-0.17 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        net-tools-2.0-0.25.20131004git.el7  net-tools-2.0-0.17        passed
  oel7n1        net-tools-2.0-0.25.20131004git.el7  net-tools-2.0-0.17        passed
Verifying Package: net-tools-2.0-0.17 ...PASSED
Verifying Users With Same UID: 0 ...PASSED
Verifying Current Group ID ...PASSED
Verifying Root user consistency ...
  Node Name                             Status
  ------------------------------------  ------------------------
  oel7n2                                passed
  oel7n1                                passed
Verifying Root user consistency ...PASSED
Verifying correctness of ASM disk group files ownership ...PASSED
Verifying selectivity of ASM discovery string ...PASSED
Verifying ASM spare parameters ...PASSED
Verifying Disk group ASM compatibility setting ...PASSED
Verifying Package: cvuqdisk-1.0.10-1 ...
  Node Name     Available                 Required                  Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
  oel7n1        cvuqdisk-1.0.10-1         cvuqdisk-1.0.10-1         passed
Verifying Package: cvuqdisk-1.0.10-1 ...PASSED
Verifying Host name ...PASSED
Verifying Node Connectivity ...
  Verifying Hosts File ...
  Node Name                             Status
  ------------------------------------  ------------------------
  oel7n1                                passed
  oel7n2                                passed
  Verifying Hosts File ...PASSED

Interface information for node "oel7n2"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s8 192.168.126.21  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:E7:37:1D 1500
 enp0s8 192.168.126.41  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:E7:37:1D 1500
 enp0s8 192.168.126.50  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:E7:37:1D 1500
 enp0s9 192.168.100.21  192.168.100.0   0.0.0.0         192.168.100.1   08:00:27:BE:71:77 1500

Interface information for node "oel7n1"

 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 enp0s8 192.168.126.11  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:8C:A2:5F 1500
 enp0s8 192.168.126.31  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:8C:A2:5F 1500
 enp0s8 192.168.126.60  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:8C:A2:5F 1500
 enp0s8 192.168.126.70  192.168.126.0   0.0.0.0         192.168.100.1   08:00:27:8C:A2:5F 1500
 enp0s9 192.168.100.11  192.168.100.0   0.0.0.0         192.168.100.1   08:00:27:E6:53:62 1500

Check: MTU consistency on the private interfaces of subnet "192.168.100.0"

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            enp0s9        192.168.100.21  192.168.100.0  1500
  oel7n1            enp0s9        192.168.100.11  192.168.100.0  1500

Check: MTU consistency of the subnet "192.168.126.0".

  Node              Name          IP Address    Subnet        MTU
  ----------------  ------------  ------------  ------------  ----------------
  oel7n2            enp0s8        192.168.126.21  192.168.126.0  1500
  oel7n2            enp0s8        192.168.126.41  192.168.126.0  1500
  oel7n2            enp0s8        192.168.126.50  192.168.126.0  1500
  oel7n1            enp0s8        192.168.126.11  192.168.126.0  1500
  oel7n1            enp0s8        192.168.126.31  192.168.126.0  1500
  oel7n1            enp0s8        192.168.126.60  192.168.126.0  1500
  oel7n1            enp0s8        192.168.126.70  192.168.126.0  1500

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  oel7n1[enp0s8:192.168.126.11]   oel7n2[enp0s8:192.168.126.41]   yes
  oel7n1[enp0s8:192.168.126.11]   oel7n2[enp0s8:192.168.126.50]   yes
  oel7n1[enp0s8:192.168.126.11]   oel7n2[enp0s8:192.168.126.21]   yes
  oel7n1[enp0s8:192.168.126.11]   oel7n1[enp0s8:192.168.126.31]   yes
  oel7n1[enp0s8:192.168.126.11]   oel7n1[enp0s8:192.168.126.60]   yes
  oel7n1[enp0s8:192.168.126.11]   oel7n1[enp0s8:192.168.126.70]   yes
  oel7n2[enp0s8:192.168.126.41]   oel7n2[enp0s8:192.168.126.50]   yes
  oel7n2[enp0s8:192.168.126.41]   oel7n2[enp0s8:192.168.126.21]   yes
  oel7n2[enp0s8:192.168.126.41]   oel7n1[enp0s8:192.168.126.31]   yes
  oel7n2[enp0s8:192.168.126.41]   oel7n1[enp0s8:192.168.126.60]   yes
  oel7n2[enp0s8:192.168.126.41]   oel7n1[enp0s8:192.168.126.70]   yes
  oel7n2[enp0s8:192.168.126.50]   oel7n2[enp0s8:192.168.126.21]   yes
  oel7n2[enp0s8:192.168.126.50]   oel7n1[enp0s8:192.168.126.31]   yes
  oel7n2[enp0s8:192.168.126.50]   oel7n1[enp0s8:192.168.126.60]   yes
  oel7n2[enp0s8:192.168.126.50]   oel7n1[enp0s8:192.168.126.70]   yes
  oel7n2[enp0s8:192.168.126.21]   oel7n1[enp0s8:192.168.126.31]   yes
  oel7n2[enp0s8:192.168.126.21]   oel7n1[enp0s8:192.168.126.60]   yes
  oel7n2[enp0s8:192.168.126.21]   oel7n1[enp0s8:192.168.126.70]   yes
  oel7n1[enp0s8:192.168.126.31]   oel7n1[enp0s8:192.168.126.60]   yes
  oel7n1[enp0s8:192.168.126.31]   oel7n1[enp0s8:192.168.126.70]   yes
  oel7n1[enp0s8:192.168.126.60]   oel7n1[enp0s8:192.168.126.70]   yes

  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  oel7n1[enp0s9:192.168.100.11]   oel7n2[enp0s9:192.168.100.21]   yes
  Verifying Check that maximum (MTU) size packet goes through subnet ...PASSED
  Verifying subnet mask consistency for subnet "192.168.126.0" ...PASSED
  Verifying subnet mask consistency for subnet "192.168.100.0" ...PASSED
Verifying Node Connectivity ...PASSED
Verifying Multicast or broadcast check ...
Checking subnet "192.168.100.0" for multicast communication with multicast group "224.0.0.251"
Verifying Multicast or broadcast check ...PASSED
Verifying ASMLib installation and configuration verification. ...
  Verifying '/etc/init.d/oracleasm' ...PASSED
  Verifying '/dev/oracleasm' ...PASSED
  Verifying '/etc/sysconfig/oracleasm' ...PASSED

  Node Name                             Status
  ------------------------------------  ------------------------
  oel7n1                                passed
  oel7n2                                passed
Verifying ASMLib installation and configuration verification. ...PASSED
Verifying ACFS Driver Checks ...PASSED
Verifying OCR Integrity ...PASSED
Verifying Network Time Protocol (NTP) ...
  Verifying '/etc/ntp.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  oel7n2                                yes
  oel7n1                                yes

  Verifying '/etc/ntp.conf' ...PASSED
  Verifying '/etc/chrony.conf' ...
  Node Name                             File exists?
  ------------------------------------  ------------------------
  oel7n2                                yes
  oel7n1                                yes

  Verifying '/etc/chrony.conf' ...PASSED
  Verifying Daemon 'chronyd' ...
  Node Name                             Running?
  ------------------------------------  ------------------------
  oel7n2                                yes
  oel7n1                                yes

  Verifying Daemon 'chronyd' ...PASSED
  Verifying NTP daemon or service using UDP port 123 ...
  Node Name                             Port Open?
  ------------------------------------  ------------------------
  oel7n2                                yes
  oel7n1                                yes

  Verifying NTP daemon or service using UDP port 123 ...PASSED
  Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
Verifying Network Time Protocol (NTP) ...FAILED (PRVG-13606)
Verifying Same core file name pattern ...PASSED
Verifying User Mask ...
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  oel7n2        0022                      0022                      passed
  oel7n1        0022                      0022                      passed
Verifying User Mask ...PASSED
Verifying User Not In Group "root": oracle ...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  oel7n2        passed                    does not exist
  oel7n1        passed                    does not exist
Verifying User Not In Group "root": oracle ...PASSED
Verifying Time zone consistency ...PASSED
Verifying Time offset between nodes ...PASSED
Verifying VIP Subnet configuration check ...PASSED
Verifying Voting Disk ...PASSED
Verifying resolv.conf Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  oel7n1                                passed
  oel7n2                                passed

checking response for name "oel7n2" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  oel7n2        192.168.126.102           IPv4                      passed

checking response for name "oel7n1" from each of the name servers specified in
"/etc/resolv.conf"

  Node Name     Source                    Comment                   Status
  ------------  ------------------------  ------------------------  ----------
  oel7n1        192.168.126.102           IPv4                      passed
Verifying resolv.conf Integrity ...PASSED
Verifying Single Client Access Name (SCAN) ...PASSED
Verifying DNS/NIS name service ...PASSED
Verifying Daemon "avahi-daemon" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  oel7n2        no                        passed
  oel7n1        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  oel7n2        no                        passed
  oel7n1        no                        passed
Verifying Daemon "avahi-daemon" not configured and running ...PASSED
Verifying Daemon "proxyt" not configured and running ...
  Node Name     Configured                Status
  ------------  ------------------------  ------------------------
  oel7n2        no                        passed
  oel7n1        no                        passed

  Node Name     Running?                  Status
  ------------  ------------------------  ------------------------
  oel7n2        no                        passed
  oel7n1        no                        passed
Verifying Daemon "proxyt" not configured and running ...PASSED
Verifying loopback network interface address ...PASSED
Verifying Privileged group consistency for upgrade ...PASSED
Verifying Cluster Manager Integrity ...
  Node Name                             Status
  ------------------------------------  ------------------------
  oel7n1                                running
  oel7n2                                running
Verifying Cluster Manager Integrity ...PASSED
Verifying Cluster Integrity ...
  Node Name
  ------------------------------------
  oel7n1
  oel7n2
Verifying Cluster Integrity ...PASSED
Verifying CRS Integrity ...
  Verifying Clusterware Version Consistency ...
    Verifying cluster upgrade state ...PASSED
  Verifying Clusterware Version Consistency ...PASSED
Verifying CRS Integrity ...PASSED
Verifying CRS user Consistency for upgrade ...PASSED
Verifying Clusterware Version Consistency ...
  Verifying cluster upgrade state ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying Node Application Existence ...

Checking existence of VIP node application (required)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  oel7n2        yes                       yes                       passed
  oel7n1        yes                       yes                       passed


Checking existence of NETWORK node application (required)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  oel7n2        yes                       yes                       passed
  oel7n1        yes                       yes                       passed


Checking existence of ONS node application (optional)

  Node Name     Required                  Running?                  Comment
  ------------  ------------------------  ------------------------  ----------
  oel7n2        no                        yes                       passed
  oel7n1        no                        yes                       passed
Verifying Node Application Existence ...PASSED
Verifying Check incorrectly sized ASM Disks ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying /boot mount ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...PASSED
Verifying User Equivalence ...PASSED
Verifying RPM Package Manager database ...INFORMATION (PRVG-11250)
Verifying Network interface bonding status of private interconnect network interfaces ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying DefaultTasksMax parameter ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED
Verifying Systemd login manager IPC parameter ...PASSED

Pre-check for cluster services setup was unsuccessful.
Checks did not pass for the following nodes:
        oel7n2,oel7n1


Failures were encountered during execution of CVU verification request "stage -pre crsinst".

Verifying Physical Memory ...FAILED
oel7n2: PRVF-7530 : Sufficient physical memory is not available on node
        "oel7n2" [Required physical memory = 8GB (8388608.0KB)]

oel7n1: PRVF-7530 : Sufficient physical memory is not available on node
        "oel7n1" [Required physical memory = 8GB (8388608.0KB)]

Verifying Network Time Protocol (NTP) ...FAILED
  Verifying chrony daemon is synchronized with at least one external time
  source ...FAILED
  oel7n2: PRVG-13606 : chrony daemon is not synchronized with any external time
          source on node "oel7n2".

  oel7n1: PRVG-13606 : chrony daemon is not synchronized with any external time
          source on node "oel7n1".

Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.


CVU operation performed:      stage -pre crsinst
Date:                         May 2, 2021 2:57:54 PM
CVU home:                     /grid/app/oracle/19.3/grid/
User:                         oracle
[oracle@OEL7N1 grid]$

We can see that almost everything have passed. While running the upgrade, these will be ignored.

Let's run the dryRun before starting actual upgrade.



[oracle@OEL7N1 grid]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54331(asmdba),54332(asmadmin)
[oracle@OEL7N1 grid]$ pwd
/grid/app/oracle/19.3/grid

[oracle@OEL7N1 grid]$ ./gridSetup.sh -silent -dryRunForUpgrade -responseFile /home/oracle/gridupgrade.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2021-05-02_03-47-07PM/gridSetupActions2021-05-02_03-47-07PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2021-05-02_03-47-07PM/gridSetupActions2021-05-02_03-47-07PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
   
 Reason : Probably this is because I forgot to use -ignorePrereqFailure

Let's run dryRun again with -ignorePrereqFailure argument

[oracle@OEL7N1 grid]$ ./gridSetup.sh -ignorePrereqFailure -silent -dryRunForUpgrade -responseFile /home/oracle/gridupgrade.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2021-05-02_03-55-57PM/gridSetupActions2021-05-02_03-55-57PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2021-05-02_03-55-57PM/gridSetupActions2021-05-02_03-55-57PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /grid/app/oracle/19.3/grid/install/response/grid_2021-05-02_03-55-57PM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2021-05-02_03-55-57PM/gridSetupActions2021-05-02_03-55-57PM.log


As a root user, execute the following script(s):
        1. /grid/app/oracle/19.3/grid/rootupgrade.sh

Execute /grid/app/oracle/19.3/grid/rootupgrade.sh on the following nodes:
[oel7n1]

Run the script on the local node.

Successfully Setup Software with warning(s).

This time it is succssful.


Now we need to run "/grid/app/oracle/19.3/grid/rootupgrade.sh" and review log. Only from node1.


[root@OEL7N1 ~]# /grid/app/oracle/19.3/grid/rootupgrade.sh
Check /grid/app/oracle/19.3/grid/install/root_OEL7N1.localdomain_2021-05-02_16-19-23-303067379.log for the output of root script
[root@OEL7N1 ~]#
[oracle@OEL7N1 ~]$ cat /grid/app/oracle/19.3/grid/install/root_OEL7N1.localdomain_2021-05-02_16-19-23-303067379.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/oracle/19.3/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Performing Dry run of the Grid Infrastructure upgrade.
Using configuration parameter file: /grid/app/oracle/19.3/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n1/crsconfig/rootcrs_oel7n1_2021-05-02_04-20-04PM.log
2021/05/02 16:22:25 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/05/02 16:27:25 CLSRSC-729: Checking whether CRS entities are ready for upgrade, cluster upgrade will not be attempted now. This operation may take a few minutes.
2021/05/02 16:32:23 CLSRSC-693: CRS entities validation completed successfully.

Log is too long. Just pasting few lines which shows that dry run is successful

2021-05-02 16:32:23: crsctl sandbox clean output =
2021-05-02 16:32:23: Executing cmd: /grid/app/oracle/19.3/grid/bin/clsecho -p has -f clsrsc -m 693
2021-05-02 16:32:23: Executing cmd: /grid/app/oracle/19.3/grid/bin/clsecho -p has -f clsrsc -m 693
2021-05-02 16:32:23: Command output:
>  CLSRSC-693: CRS entities validation completed successfully.
>End Command output
2021-05-02 16:32:23: CLSRSC-693: CRS entities validation completed successfully.
2021-05-02 16:32:23: Exiting upgrade dry run

Let's check the CRS active version from both the nodes.

[root@OEL7N1 ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1831702305].

[root@OEL7N2 ~]#  crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [1831702305].

Time for actual upgrade.


We will be running upgrade in silent mode but it will require a response file. I have generated the response file by running the upgrade in GUI mode and at the last step there will be an option to save response file. Let's start.



[oracle@OEL7N1 grid]$ pwd
/grid/app/oracle/19.3/grid
[oracle@OEL7N1 grid]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),54323(oper),54324(backupdba),54325(dgdba),54326(kmdba),54330(racdba),54331(asmdba),54332(asmadmin)
[oracle@OEL7N1 grid]$
[oracle@OEL7N1 grid]$ ./gridSetup.sh -silent -ignorePrereqFailure -waitForCompletion -responseFile /home/oracle/gridupgrade.rsp
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13013] Target environment does not meet some mandatory requirements.
   CAUSE: Some of the mandatory prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2021-05-03_05-41-37AM/gridSetupActions2021-05-03_05-41-37AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2021-05-03_05-41-37AM/gridSetupActions2021-05-03_05-41-37AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /grid/app/oracle/19.3/grid/install/response/grid_2021-05-03_05-41-37AM.rsp


As a root user, execute the following script(s):
        1. /grid/app/oracle/19.3/grid/rootupgrade.sh

Execute /grid/app/oracle/19.3/grid/rootupgrade.sh on the following nodes:
[oel7n2, oel7n1]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.

Successfully Setup Software with warning(s).
As install user, execute the following command to complete the configuration.
        /grid/app/oracle/19.3/grid/gridSetup.sh -executeConfigTools -responseFile /home/oracle/gridupgrade.rsp [-silent]

Let's run /grid/app/oracle/19.3/grid/rootupgrade.sh starting with local node.


Last login: Mon May  3 05:22:14 2021
[root@OEL7N1 ~]# /grid/app/oracle/19.3/grid/rootupgrade.sh
Check /grid/app/oracle/19.3/grid/install/root_OEL7N1.localdomain_2021-05-03_05-52-00-082794319.log for the output of root script
[root@OEL7N1 ~]#

[root@OEL7N1 ~]# tail -f /grid/app/oracle/19.3/grid/install/root_OEL7N1.localdomain_2021-05-03_05-52-00-082794319.log
    ORACLE_HOME=  /grid/app/oracle/19.3/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /grid/app/oracle/19.3/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n1/crsconfig/rootcrs_oel7n1_2021-05-03_05-53-19AM.log
2021/05/03 05:53:31 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2021/05/03 05:53:31 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/05/03 05:53:31 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2021/05/03 05:53:31 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/05/03 05:53:33 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2021/05/03 05:53:33 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/05/03 05:58:26 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
2021/05/03 06:09:51 CLSRSC-693: CRS entities validation completed successfully.
2021/05/03 06:10:21 CLSRSC-515: Starting OCR manual backup.
2021/05/03 06:10:45 CLSRSC-516: OCR manual backup successful.
2021/05/03 06:11:09 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2021/05/03 06:11:09 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.
2021/05/03 06:11:09 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2021/05/03 06:11:29 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2021/05/03 06:11:29 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
2021/05/03 06:11:32 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
2021/05/03 06:11:46 CLSRSC-363: User ignored prerequisites during installation
2021/05/03 06:12:14 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
2021/05/03 06:12:15 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.
2021/05/03 06:17:17 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2021/05/03 06:17:18 CLSRSC-482: Running command: '/grid/app/oracle/12.2/bin/crsctl start rollingupgrade 19.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2021/05/03 06:17:24 CLSRSC-482: Running command: '/grid/app/oracle/19.3/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /grid/app/oracle/12.2 -oldCRSVersion 12.2.0.1.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2021/05/03 06:17:38 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2021/05/03 06:17:51 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2021/05/03 06:18:48 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2021/05/03 06:18:52 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
2021/05/03 06:18:55 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
2021/05/03 06:19:30 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
2021/05/03 06:20:06 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
2021/05/03 06:20:32 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
2021/05/03 06:20:54 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
2021/05/03 06:20:54 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2021/05/03 06:22:11 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
2021/05/03 06:25:57 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
2021/05/03 06:26:16 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
2021/05/03 06:28:55 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2021/05/03 06:30:03 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
2021/05/03 06:30:14 CLSRSC-474: Initiating upgrade of resource types
2021/05/03 06:33:45 CLSRSC-475: Upgrade of resource types successfully initiated.
2021/05/03 06:34:19 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
2021/05/03 06:34:44 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded


Let's run it in the second node.



[root@OEL7N2 ~]# /grid/app/oracle/19.3/grid/rootupgrade.sh
Check /grid/app/oracle/19.3/grid/install/root_OEL7N2.localdomain_2021-05-03_06-50-59-532517072.log for the output of root script
[root@OEL7N2 ~]#

[root@OEL7N2 ~]# tail -100f /grid/app/oracle/19.3/grid/install/root_OEL7N2.localdomain_2021-05-03_06-50-59-532517072.log
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/oracle/19.3/grid
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /grid/app/oracle/19.3/grid/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/oel7n2/crsconfig/rootcrs_oel7n2_2021-05-03_06-52-34AM.log
2021/05/03 06:53:06 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2021/05/03 06:53:06 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/05/03 06:53:07 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2021/05/03 06:53:07 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/05/03 06:53:10 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2021/05/03 06:53:11 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/05/03 06:57:24 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2021/05/03 06:57:24 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
2021/05/03 06:57:24 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
2021/05/03 06:57:27 CLSRSC-363: User ignored prerequisites during installation
2021/05/03 06:57:33 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
2021/05/03 06:57:33 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.

ASM configuration upgraded in local node successfully.

2021/05/03 06:58:00 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2021/05/03 06:58:44 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2021/05/03 07:02:32 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
2021/05/03 07:02:34 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
2021/05/03 07:02:57 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
2021/05/03 07:02:57 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
2021/05/03 07:03:04 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
2021/05/03 07:03:09 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
2021/05/03 07:03:10 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2021/05/03 07:04:06 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
2021/05/03 07:07:12 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
2021/05/03 07:07:17 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
2021/05/03 07:10:33 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 19 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2021/05/03 07:11:24 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
Start upgrade invoked..
2021/05/03 07:11:39 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2021/05/03 07:11:39 CLSRSC-482: Running command: '/grid/app/oracle/19.3/grid/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Oracle ASM was successfully upgraded.
Started to upgrade CRS.
CRS was successfully upgraded.
Started to upgrade Oracle ACFS.
Oracle ACFS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 19.0.0.0.0.
2021/05/03 07:12:51 CLSRSC-479: Successfully set Oracle Clusterware active version
2021/05/03 07:12:52 CLSRSC-476: Finishing upgrade of resource types
2021/05/03 07:13:20 CLSRSC-477: Successfully completed upgrade of resource types
2021/05/03 07:16:25 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
Successfully updated XAG resources.
2021/05/03 07:17:23 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded