1. Environment
-
Database: Oracle 19c (19.14 / 19.26)
-
Configuration: Oracle Restart (Standalone) & 6-Node RAC
-
Grid Infrastructure: 19c
-
OS: AIX 7.3 (64-bit)
2. Problem Statement
Issue 1: CRS Startup Failure
CRS-6706: Oracle Clusterware Release patch level does not match Software patch level
CRS-4000: Command Start failed, or completed with errors
Issue 2: ASM Disk Add Failure
ORA-15032: not all alterations performed
ORA-15137: The ASM cluster is in rolling patch state
3. Root Cause
-
Grid patches were applied successfully on all nodes.
-
However, Clusterware patch level was not updated correctly in OCR/OLR.
-
Cluster remained stuck in ROLLING PATCH state instead of transitioning to NORMAL.
-
Patch level mismatch observed:
-
Software Patch Level:
3284843566 -
Active Cluster Patch Level:
1179688039
4. Implementation / Solution
1. CRS-6706: Oracle Clusterware Release patch level does not match Software patch level
Environment:
Current Database: Oracle Restart Database 19.14 (Standalone Grid Infrastructure)
Operating System : AIX 7.3 64 Bit
Error:
CRS-6706: Oracle Clusterware Release patch level ('1211422631') does not match Software patch level ('2018016837'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
Change:
Oracle 19.25 Grid Patches were applied to GRID HOME.
Issue Description:
There were no errors in Patching log file. Patches were applied successfully to the GRID HOME. When tried to lock the GRID HOME binaries after patching activity, HAS was failed to start with above error messages.
After patching activiyt, by root user, below lock comamnd was executed.
root@testdb:/u01/app/grid/19.0.0/grid/crs/install >ls -ltr *root*.sh
-rwxr-x--- 1 grid oinstall 8051 Jun 19 17:20 rootcrs.sh
-rwxr-x--- 1 grid oinstall 3931 Jun 19 17:20 roothas.sh
root@testdb:/u01/app/grid/19.0.0/grid/crs/install >./roothas.sh -lock
Using configuration parameter file: /u01/app/grid/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/testdb/crsconfig/halock__2025-06-23_12-04-35AM.log
2025/06/23 12:04:36 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
When checked, the PMON and d.bin processes were not started automatically.
grid@testdb:/home/grid >ps -ef | grep pmon
grid 6160860 22872420 0 12:05:08 pts/0 0:00 grep pmon
grid@testdb:/home/grid >ps -ef | grep d.bin
grid 25821454 22872420 0 12:05:12 pts/0 0:00 grep d.bin
Then, tried to start HAS manually and below error messages were observed.
grid@testdb:/home/grid >./crsctl start has
CRS-6706: Oracle Clusterware Release patch level ('1211422631') does not match Software patch level ('2018016837'). Oracle Clusterware cannot be started.
CRS-4000: Command Start failed, or completed with errors.
Solution:
Step 1: Execute below command to fix the issue.
root@testdb:> cd /u01/app/grid/19.0.0/grid/bin
root@testdb:/u01/app/grid/19.0.0/grid/bin >./clscfg -localpatch
clscfg: EXISTING configuration version 0 detected.
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
Step 2: Now try to start HAS service by grid user. You can see now DB and ASM are started successfully.
grid@testdb:/home/grid >crsctl start has
CRS-4123: Oracle High Availability Services has been started.
grid@testdb:/home/grid >ps -ef | grep pmon
grid 25821606 1 0 12:11:59 - 0:00 asm_pmon_+ASM
grid 29294998 22872442 0 12:12:26 pts/0 0:00 grep pmon
grid 33358106 1 0 12:12:22 - 0:00 ora_pmon_oemdb
Login as oracle user and check the DB status:
root@testdb:/u01/app/grid/19.0.0/grid/bin >su - oracle
oracle@testdb:/home/oracle >sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jun 23 12:14:29 2025
Version 19.14.1.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.14.1.0.0
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- ------------
OEMDB READ WRITE
2. ORA-15137: The ASM cluster is in rolling patch state
Incident Summary:
Error encountered while adding a disk to the ASM diskgroup.
SQL> alter diskgroup DG_testdb_DB_XP8 add disk '/dev/dg_testdb_db_xp8_60267/testdb_db_3412';
alter diskgroup DG_testdb_DB_XP8 add disk '/dev/dg_testdb_db_xp8_60267/testdb_db_3412'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15137: The ASM cluster is in rolling patch state.
Changes in the environment:
This is a six-node RAC setup where the GRID Bug fix patch "37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]" was recently applied on all of the cluster nodes post which post-patch step failed and the issue was overlooked by the DBA.
Observation:
A few days later, there was a requirement to add a new disk to the ASM diskgroup. During this activity, the above error message was encountered. Based on the error, it appears that the cluster patch level is not consistent across all nodes. Let’s proceed with troubleshooting the issue.
Steps performed to troubleshoot the issue :
Step1 : Lets check opatch lspatches output on all nodes.
Node1:
grid@testdbnode1:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
Node2:
grid@testdbnode2:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
Node3:
grid@testdbnode3:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
Node4:
grid@testdbnode4:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
Node5:
grid@testdbnode5:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
Node6:
grid@testdbnode6:/oracle_19c_grid/app/oracle/product/19.3/OPatch#./opatch lspatches
37328497;ASM INSTANCE TERMINATED WITH ORA-00600 [KFCBPING05], [7543]
37268031;OCW RELEASE UPDATE 19.26.0.0.0 (37268031)
37260974;Database Release Update : 19.26.0.0.250121 (37260974)
29517247;ACFS RELEASE UPDATE 19.3.0.0.0 (29517247)
OPatch succeeded.
The opatch lspatches command confirms that the bug-fix patch 37328497 is listed in the lsinventory on all nodes.
Step2 : Let's check the opatch logs for all nodes.
Node1:
grid@testdbnode1:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode1:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch# grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_02-08-53AM_1.log:[Dec 10, 2025 2:18:30 AM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode1:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#egrep -i "error|fail|make" opatch2025-12-10_02-08-53AM_1.log
[Dec 10, 2025 2:08:53 AM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 2:16:24 AM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 2:16:24 AM] [INFO] Start invoking 'make' at Wed Dec 10 02:16:24 IST 2025Wed Dec 10 02:16:24 IST 2025
[Dec 10, 2025 2:16:24 AM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 2:18:29 AM] [INFO] Finish invoking 'make' at Wed Dec 10 02:18:29 IST 2025
[Dec 10, 2025 2:18:29 AM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 2:18:29 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_02-08-53AM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 2:18:30 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_02-08-53AM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 2:18:30 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_02-08-53AM/make.txt"
Node2:
grid@testdbnode2:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode2:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_05-13-56AM_1.log:[Dec 10, 2025 5:22:54 AM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode2:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#egrep -i "error|fail|make" opatch2025-12-10_05-13-56AM_1.log
[Dec 10, 2025 5:13:57 AM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 5:20:58 AM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 5:20:58 AM] [INFO] Start invoking 'make' at Wed Dec 10 05:20:58 IST 2025Wed Dec 10 05:20:58 IST 2025
[Dec 10, 2025 5:20:58 AM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 5:22:53 AM] [INFO] Finish invoking 'make' at Wed Dec 10 05:22:53 IST 2025
[Dec 10, 2025 5:22:53 AM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 5:22:53 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 5:22:54 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 5:22:54 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/make.txt"
Node3:
grid@testdbnode3:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode3:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_05-20-55AM_1.log:[Dec 10, 2025 5:29:40 AM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode3:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#egrep -i "error|fail|make" opatch2025-12-10_05-20-55AM_1.log
[Dec 10, 2025 5:20:56 AM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 5:27:53 AM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 5:27:53 AM] [INFO] Start invoking 'make' at Wed Dec 10 05:27:53 IST 2025Wed Dec 10 05:27:53 IST 2025
[Dec 10, 2025 5:27:53 AM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 5:29:38 AM] [INFO] Finish invoking 'make' at Wed Dec 10 05:29:38 IST 2025
[Dec 10, 2025 5:29:38 AM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 5:29:39 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-20-55AM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 5:29:39 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-20-55AM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 5:29:39 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-20-55AM/make.txt"
Node4:
grid@testdbnode4:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode4:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_05-13-56AM_1.log:[Dec 10, 2025 5:23:02 AM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode4:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#egrep -i "error|fail|make" opatch2025-12-10_05-13-56AM_1.log
[Dec 10, 2025 5:13:57 AM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 5:21:08 AM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 5:21:08 AM] [INFO] Start invoking 'make' at Wed Dec 10 05:21:08 IST 2025Wed Dec 10 05:21:08 IST 2025
[Dec 10, 2025 5:21:08 AM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 5:23:01 AM] [INFO] Finish invoking 'make' at Wed Dec 10 05:23:01 IST 2025
[Dec 10, 2025 5:23:01 AM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 5:23:01 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 5:23:01 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 5:23:01 AM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_05-13-56AM/make.txt"
Node5:
grid@testdbnode5:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode5:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_18-12-24PM_1.log:[Dec 10, 2025 6:21:33 PM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode5:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch# egrep -i "error|fail|make" opatch2025-12-10_18-12-24PM_1.log
[Dec 10, 2025 6:12:24 PM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 6:19:39 PM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 6:19:39 PM] [INFO] Start invoking 'make' at Wed Dec 10 18:19:39 IST 2025Wed Dec 10 18:19:39 IST 2025
[Dec 10, 2025 6:19:39 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 6:21:31 PM] [INFO] Finish invoking 'make' at Wed Dec 10 18:21:31 IST 2025
[Dec 10, 2025 6:21:31 PM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 6:21:31 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-12-24PM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 6:21:31 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-12-24PM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 6:21:31 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-12-24PM/make.txt"
Node6:
grid@testdbnode6:/home/grid#cd /oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch
grid@testdbnode6:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#grep -i "Patch 37328497 successfully applied" *.log
opatch2025-12-10_18-26-45PM_1.log:[Dec 10, 2025 6:35:38 PM] [INFO] Patch 37328497 successfully applied.
grid@testdbnode6:/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch#egrep -i "error|fail|make" opatch2025-12-10_18-26-45PM_1.log
[Dec 10, 2025 6:26:46 PM] [INFO] Runtime args: [-Xmx1536m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/oracle_19c_grid/app/oracle/product/19.3/cfgtoollogs/opatch, -DCommonLog.LOG_SESSION_ID=, -DCommonLog.COMMAND_NAME=apply, -DOPatch.ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3, -DOPatch.DEBUG=false, -DOPatch.MAKE=false, -DOPatch.RUNNING_DIR=., -DOPatch.MW_HOME=, -DOPatch.WL_HOME=, -DOPatch.COMMON_COMPONENTS_HOME=, -DOPatch.OUI_LOCATION=/oracle_19c_grid/app/oracle/product/19.3/oui, -DOPatch.FMW_COMPONENT_HOME=, -DOPatch.OPATCH_CLASSPATH=, -DOPatch.WEBLOGIC_CLASSPATH=, -DOPatch.SKIP_OUI_VERSION_CHECK=, -DOPatch.NEXTGEN_HOME_CHECK=false, -DOPatch.PARALLEL_ON_FMW_OH=]
/usr/ccs/bin/make /oracle_19c_grid/app/oracle/product/19.3/lib/libasmclntsh19.so -f /oracle_19c_grid/app/oracle/product/19.3/rdbms/lib/ins_rdbms.mk
make libasmclntsh19.so returned code 0
[Dec 10, 2025 6:33:49 PM] [INFO] OUI-67050:Running make for target ioracle
[Dec 10, 2025 6:33:49 PM] [INFO] Start invoking 'make' at Wed Dec 10 18:33:49 IST 2025Wed Dec 10 18:33:49 IST 2025
[Dec 10, 2025 6:33:49 PM] [INFO] Running make command: /usr/ccs/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/oracle_19c_grid/app/oracle/product/19.3 OPATCH_SESSION=apply
[Dec 10, 2025 6:35:37 PM] [INFO] Finish invoking 'make' at Wed Dec 10 18:35:37 IST 2025
[Dec 10, 2025 6:35:37 PM] [INFO] OPatch will clean up 'restore.sh,make.txt' files and 'scratch,backup' directories.
[Dec 10, 2025 6:35:37 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-26-45PM/backup/inventory/Components21/oracle.crs/19.0.0.0.0/UnixActions/makedeps.xml"
[Dec 10, 2025 6:35:38 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-26-45PM/backup/inventory/make/makeorder.xml"
[Dec 10, 2025 6:35:38 PM] [INFO] Deleted the file "/oracle_19c_grid/app/oracle/product/19.3/.patch_storage/NApply/2025-12-10_18-26-45PM/make.txt"
Based on the patching logs, it is evident that the bug-fix patch was successfully applied on all cluster nodes.
Step3 : Now check the kfod patches and patch level on all nodes.
Node1:
grid@testdbnode1:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode1:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
Node2:
grid@testdbnode2:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode2:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
Node3:
grid@testdbnode3:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode3:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
Node4:
grid@testdbnode4:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode4:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
Node5:
grid@testdbnode5:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode5:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
Node6:
grid@testdbnode6:/home/grid#kfod op=patchlvl
-------------------
Current Patch level
===================
3284843566
grid@testdbnode6:/home/grid#kfod op=patches
---------------
List of Patches
===============
29517247
37260974
37268031
37328497
The kfod patches and patch levels appear to be identical across all nodes. So what’s next? What could be causing the issue?
Step4 : Now check the cluster active version across all the cluster nodes.
The command "crsctl query crs activeversion -f" returns the active version for the entire cluster, so it does not need to be executed on every node. In this case, I executed it on each node only for informational purposes.
Node1:
grid@testdbnode1:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Node2:
grid@testdbnode2:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Node3:
grid@testdbnode3:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Node4:
grid@testdbnode4:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Node5:
grid@testdbnode5:/home/grid#crsctl query crs activeversion -f software
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Node6:
grid@testdbnode6:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [1179688039].
Although the cluster active patch level [1179688039]is the same across all nodes, the actual patch level must be [3284843566]since the cluster is operating in the [ROLLING PATCH] mode. This is the root cause of the issue.
The cluster patch level must be accurate, and the cluster upgrade state should remain in the [NORMAL] mode.
Step5 : Let's check the softwarepatch on all nodes.
grid@testdbnode1:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode1 is [3284843566].
grid@testdbnode2:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode2 is [3284843566].
grid@testdbnode3:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode3 is [1179688039].
grid@testdbnode4:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode4 is [1179688039].
grid@testdbnode5:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode5 is [3284843566].
grid@testdbnode6:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode6 is [1179688039].
From the above output, you can observe that the clusterware patch level on Node1, Node2, and Node5 is 3284843566. Ideally, this value should match the patch level reported by the "crsctl query crs activeversion -f" command. However, in our case, it does not match.
On the other hand, the clusterware patch level on Node3, Node4, and Node6 is 1179688039, which matches the active version output. This also indicates that the updated clusterware patch level 1179688039 was not applied correctly on all nodes. The actual cluster patch level should be [3284843566]in both activeversion output and softwarepatch output.
This indicates that the clusterware patch level was not updated properly on all nodes. As a result, the cluster is stuck in the [ROLLING PATCH] state instead of returning to [NORMAL] mode. In the [ROLLING PATCH] state, the cluster continues to function normally; however, any CRS-level operations such as adding a new disk to an ASM diskgroup will fail, as these actions are not permitted while the cluster is in [ROLLING PATCH] mode.
What would be the solution to fix this issue ???
Step5 : Use below commands to update the patch level in OLR and OCR.
clscfg -localpatch :-
This command updates the Oracle Local Registry (OLR) on the local node where it is run to update the new software patch level. CRS should be down to execute this command.
When to use: Only one node or local node has an incorrect patch level.
clscfg -patch :-
This command updates the Oracle Cluster Registry (OCR) which stores the clusterware configuration. This command is run to update the clusterware patch level. CRS should be up to execute this command.
When to use: Multiple nodes differ or complete mismatch across cluster.
In this case, we will use the second command, "clscfg -patch", because the clusterware patch level is not consistent across all nodes.
On problematic node, login as root user and execute below command.
Node3:
root@testdbnode3:/#/oracle_19c_grid/app/oracle/product/19.3/bin/clscfg -patch
Now execute the following command to transition the cluster upgrade state from [ROLLING PATCH] to [NORMAL].
You only need to run this command from any one node. Running it once is sufficient, as it updates the cluster upgrade state across the entire cluster.
root@testdbnode1:/#/oracle_19c_grid/app/oracle/product/19.3/bin/crsctl stop rollingpatch
CRS-1161: The cluster was successfully patched to patch level [3284843566]
Step6: Now check the clusterware activeversion and softwarepatch level across all the nodes.
Node1:
grid@testdbnode1:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node2:
grid@testdbnode2:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node3:
grid@testdbnode3:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node4:
grid@testdbnode4:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node5:
grid@testdbnode5:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node6:
grid@testdbnode6:/home/grid#crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3284843566].
Node1:
grid@testdbnode1:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode1 is [3284843566].
Node2:
grid@testdbnode2:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode2 is [3284843566].
Node3:
grid@testdbnode3:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode3 is [3284843566].
Node4:
grid@testdbnode4:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode4 is [3284843566].
Node5:
grid@testdbnode5:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode5 is [3284843566].
Node6:
grid@testdbnode6:/home/grid#crsctl query crs softwarepatch
Oracle Clusterware patch level on node testdbnode6 is [3284843566].
You can now see that the clusterware patch level and the software patch level are matching. Additionally, the cluster upgrade state has successfully transitioned from [ROLLING PATCH] to [NORMAL].
Issue has been resolved by updating OCR by clscfg -patch command and ASM disk was added to the diskgroup successfully withou any futher errors.
5. Key Learnings
-
Successful patching does not guarantee OCR/OLR update.
-
Always validate:
-
activeversion -
softwarepatch -
kfod patchlvl
-
-
Cluster must be in NORMAL state for ASM and CRS operations.
-
ROLLING PATCH mode restricts cluster-level changes.
6. Conclusion
This issue highlights a critical gap between binary patching and cluster metadata synchronization. Even though patches were successfully applied across all nodes, failure to update the OCR caused the cluster to remain in a ROLLING PATCH state, leading to operational failures.
By executing:
-
clscfg -patch -
crsctl stop rollingpatch
the cluster was successfully brought back to a consistent and stable state, resolving both CRS startup issues and ASM disk operation failures.