KBHS-00712: ORA-29024 received from local HTTP service during backup in RAC Oracle cloud

KBHS-00712: ORA-29024 received from local HTTP service during backup in RAC Oracle cloud

The backup of Oracle RAC DBaaS may fail during manual as well automatic backup.
If you manually invoke following command,The backup may not get successful.
Login to opc.
[opc@pocracdemo1 ~]$ raccli describe job 17
{
  “requestStatus” : “SUCCESS”,
  “jobStatus” : “FAILURE”,
  “message” : “Failed to run Rman Backup statement.”,
  “response” : [ {
    “startTime” : “Fri Apr 13 03:28:23 UTC 2018”,
    “endTime” : “Fri Apr 13 03:28:24 UTC 2018”,
    “status” : “SUCCESS”,
    “taskId” : “TaskZJsonRpcExt_2851”,
    “taskResult” : “backup validations done successfully”,
    “taskName” : ” backup validations”
  }, {
    “startTime” : “Fri Apr 13 03:28:24 UTC 2018”,
    “endTime” : “Fri Apr 13 03:28:49 UTC 2018”,
    “status” : “SUCCESS”,
    “taskId” : “TaskZJsonRpcExt_2853”,
    “taskResult” : “DB backup validations done successfully”,
    “taskName” : ” db backup validations”
  }, {
    “startTime” : “Fri Apr 13 03:28:49 UTC 2018”,
    “endTime” : “Fri Apr 13 03:29:07 UTC 2018”,
    “status” : “SUCCESS”,
    “taskId” : “TaskZJsonRpcExt_2855”,
    “taskName” : “cross check database backup”
  }, {
    “startTime” : “Fri Apr 13 03:29:07 UTC 2018”,
    “endTime” : “Fri Apr 13 03:29:57 UTC 2018”,
    “status” : “FAILURE“,
    “taskId” : “TaskZJsonRpcExt_2857”,
    “taskResult” : “Failed to run Rman Backup statement.”,
    “taskName” : “Database Backup”
  } ]
}
Now you need to check RMAN log for exact error:-

tail -100f  /home/oracle/bkup/logs/rman/rman_2018-04-13_02-16-15.0242.log

Starting backup at 04/13/2018 02:16:21
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 04/13/2018 02:16:21
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27023: skgfqsbi: media manager protocol error
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
KBHS-00715: HTTP error occurred ‘oracle-error’
KBHS-00712: ORA-29024 received from local HTTP service

Recovery Manager complete.

Solution:-
Please refer to below metalink note:-
RMAN Backup to Oracle Database Backup Cloud Service fails with KBHS-00715 ORA-29024 (Doc ID 2360941.1)
Till the bug is fixed, below workaround can be used:
1. Set below parameter in the config file (opcPROD.ora) and retry backup:

vi /u02/app/oracle/admin/orcl/oss_config/opc_orcl.ora

_OPC_VALIDATE_CERT=TRUE

2. If above does not help, re-install the cloud backup module with -trustedCerts option and provide opc.cert file attached in this document i.e.
You need to download opc.cert and opc_install.jar mentioned in above metalink note (Doc ID 2360941.1)
[oracle@pocracdemo1 ~]$ java -jar opc_install.jar -host https://xxx.eu.storage.oraclecloud.com/v1/Storage-xxx -opcId 'xxx@gmail.com' -opcPass 'xxx' -walletDir /home/oracle/opc_wallet -libDir $ORACLE_HOME/lib -libPlatform linux64 -debug -trustedCerts /home/oracle/opc.cert
Oracle Database Cloud Backup Module Install Tool, build 12.2.0.1.0DBBKPCSBP_2017-11-28
Debug: os.name        = Linux
Debug: os.arch        = amd64
Debug: os.version     = 4.1.12-112.14.10.el6uek.x86_64
Debug: file.separator = /
Debug: Platform = PLATFORM_LINUX64
Debug: OPC Account Verification: <?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?><account name=”Storage-62b84aec8de8478ba17a6e1f84c3475c”><container><name>DBaaS</name><count>1</count><bytes>279552</bytes><accountId><id>31714</id></accountId><deleteTimestamp>0.0</deleteTimestamp><containerId><id>7665965</id></containerId></container></account>
Oracle Database Cloud Backup Module credentials are valid.
Debug: Certificate Success: file = /home/oracle/opc.cert
Debug: Certificate Success:
       Subject  : CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
       Validity : Fri Nov 10 00:00:00 UTC 2006 – Mon Nov 10 00:00:00 UTC 2031
       Issuer   : CN=DigiCert Global Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
Oracle Database Cloud Backup Module wallet created in directory /home/oracle/opc_wallet.
Oracle Database Cloud Backup Module initialization file /u01/app/oracle/product/12.2.0.1/dbhome_1/dbs/opcorcl1.ora created.
Downloading Oracle Database Cloud Backup Module Software Library from file opc_linux64.zip.
Debug: Temp zip file = /tmp/opc_linux646570785615594303284.zip
Debug: Downloaded 27314069 bytes in 4 seconds.
Debug: Transfer rate was 6828517 bytes/second.
Download complete.
Debug: Delete RC = true
After that,I faced another issue:-
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
KBHS-01404: See trace file /u01/app/oracle/product/12.1.0/dbhome_1/rdbms/log/sbtio_8654_139790089015904.log for details
KBHS-00719: Error ‘unknown’; You are not allowed to perform a tagging operation on a container that has objects in it, accountName:
KBHS-00700
Recovery Manager complete.
Solution :-
Metalink note:-Backup fails with error ‘KBHS-00719: ERROR ‘UNKNOWN’; YOU ARE NOT ALLOWED TO PERFORM A TAGGING OPERATION’ (Doc ID 2371818.1)
Empty the container by moving the existing files in a different one before taking the first RMAN backup, or just create a new empty container and update the opc parameter file to point to it.
Now the backup succeeds

[opc@pocracdemo1 ~]$ raccli describe job 20

{
“requestStatus” : “SUCCESS”,
“jobStatus” : “SUCCESS”,
“response” : [ {
“startTime” : “Fri Apr 13 06:39:39 UTC 2018”,
“endTime” : “Fri Apr 13 06:39:39 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3885”,
“taskResult” : “backup validations done successfully”,
“taskName” : ” backup validations”
}, {
“startTime” : “Fri Apr 13 06:39:39 UTC 2018”,
“endTime” : “Fri Apr 13 06:40:04 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3887”,
“taskResult” : “DB backup validations done successfully”,
“taskName” : ” db backup validations”
}, {
“startTime” : “Fri Apr 13 06:40:04 UTC 2018”,
“endTime” : “Fri Apr 13 06:41:08 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3889”,
“taskName” : “cross check database backup”
}, {
“startTime” : “Fri Apr 13 06:41:08 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:29 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3891”,
“taskResult” : “Resource { id: 97761124902577, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Database Backup”
}, {
“startTime” : “Fri Apr 13 06:49:29 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskParallel_3893”,
“taskName” : “DB Config files backup”
}, {
“startTime” : “Fri Apr 13 06:49:29 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3894”,
“taskResult” : “Resource { id: 98261108067814, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “DB config files backup on node_1”
}, {
“startTime” : “Fri Apr 13 06:49:29 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3896”,
“taskResult” : “Resource { id: 97712211539712, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “DB config files backup on node_2”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:45 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskParallel_3899”,
“taskName” : “GI Config files backup”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:45 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskParallel_3904”,
“taskName” : “OS Config files backup”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:38 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3900”,
“taskResult” : “Resource { id: 98265677604650, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Gi config files backup on node_1”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:41 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3905”,
“taskResult” : “Resource { id: 98269544616295, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Os config files backup on node_1”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:48 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskParallel_3909”,
“taskName” : “Derby DB files backup”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:42 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3902”,
“taskResult” : “Resource { id: 97721137057959, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Gi config files backup on node_2”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:45 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3907”,
“taskResult” : “Resource { id: 97725142391225, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Os config files backup on node_2”
}, {
“startTime” : “Fri Apr 13 06:49:33 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:42 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3910”,
“taskResult” : “Resource { id: 98272448241014, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Derby Db config files backup on node_1”
}, {
“startTime” : “Fri Apr 13 06:49:41 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:48 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3912”,
“taskResult” : “Resource { id: 97728013660122, name: rdbaas_backupComponent, type: BackupComponent }”,
“taskName” : “Derby Db config files backup on node_2”
}, {
“startTime” : “Fri Apr 13 06:49:48 UTC 2018”,
“endTime” : “Fri Apr 13 06:49:48 UTC 2018”,
“status” : “SUCCESS”,
“taskId” : “TaskZJsonRpcExt_3914”,
“taskResult” : “Resource { id: 98279542295230, name: dcs_backup, type: Backu p }”,
“taskName” : “Persisting Backup metadata”
} ]
}

 

 

 

 

 

 

ORA-01111: name for data file is unknown – rename to correct file

Error in dataguard alert log/start managed recovery process:-

SYS@XXX>alter database recover managed standby database;
alter database recover managed standby database
*
ERROR at line 1:
ORA-00283: recovery session canceled due to errors
ORA-01111: name for data file 61 is unknown – rename to correct file
ORA-01110: data file 61: ‘/u01/app/oracle/product/12.1.0/db_1/dbs/UNNAMED00061’
ORA-01157: cannot identify/lock data file 61 – see DBWR trace file
ORA-01111: name for data file 61 is unknown – rename to correct file
ORA-01110: data file 61: ‘/u01/app/oracle/product/12.1.0/db_1/dbs/UNNAMED00061’

 

Solution:-

Check the exact size of the datafile in primary for file_id=61

In Standby,

SYS@XXX>alter system set standby_file_management=manual;

System altered.

SYS@XXX>Alter database create datafile ‘/u01/app/oracle/product/12.1.0/db_1/dbs/UNNAMED00061’ as ‘+DATA/’ size 34358689792;

Database altered.

SYS@XXX>alter system set standby_file_management=auto;

System altered.

Now you will be able to start managed recover process.

Undo Advisor help to estimate the undo tablespace size and undo retention to avoid ORA-1555

Prepare by: Nurullah Sharif

Scope: Undo Advisor

 

Undo Advisor help to estimate the undo tablespace size and also advise of undo retention.

SQL> @db
NAME      OPEN_MODE
--------- --------------------
COLLPROD  READ WRITE

 

Undo retention is 900 sec which 15 min

SQL> sho parameter undo
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_management                      string      AUTO
undo_retention                       integer     900
undo_tablespace                      string      UNDOTBS2

Undo tablespace Size:

SQL> @tblspc
TABLESPACE_NAME                ALLOCATED_MB    USED_MB FREE_SPACE_MB PERCENT_USED
------------------------------ ------------ ---------- ------------- ------------
SYSAUX                                  300    -88.625       388.625       -29.54
SYSTEM                                  300    -55.625       355.625       -18.54
DATA                                  30720        109         30611          .35
UNDOTBS2                                125       5.75        119.25          4.6
UNDOTBS1                                140    18.8125      121.1875        13.44
USERS                                     5     1.3125        3.6875        26.25
SYSTEM                                  870    514.375       355.625        59.12
SYSAUX                                 1350    961.375       388.625        71.21

8 rows selected.

 

Note: To make the undo tablespace fixed-size, Oracle suggests that you first allow enough time after database creation to run a full workload, thus allowing the undo tablespace to grow to its minimum required size to handle the workload. Then, you can use the Undo Advisor to determine, if desired, how much larger to set the size of the undo tablespace to allow for long-running queries and Oracle Flashback operations.

Historical information in memory:

SQL> set serveroutput on
 SQL> DECLARE
 utbsiz_in_MB NUMBER;
 BEGIN
 utbsiz_in_MB := DBMS_UNDO_ADV.RBU_MIGRATION;
 dbms_output.put_line('=================================================================');
 dbms_output.put_line('The Minimum size of the undo tablespace required is : '||utbsiz_in_MB||'
 MB');
 dbms_output.put_line('=================================================================');
 end;
 /
==============================================
The Minimum size of the undo tablespace required is : 72 MB
==============================================

PL/SQL procedure successfully completed.

Function undo_info is used to get information about undo tablespace of the current instance.

it returns undo tablespace name, max possible size, if it is auto extensible, current undo retention value and if the undo tablespace has guaranteed undo retention.

SQL> set serveroutput on
 SQL> DECLARE
 tbs_name VARCHAR2(30);
 tbs_size NUMBER(10);
 tbs_autoextend BOOLEAN;
 tbs_retention NUMBER(5);
 tbs_guarantee BOOLEAN;
 undo_adv BOOLEAN;
 BEGIN
 dbms_output.put_line('=====================================================================');
 undo_adv := dbms_undo_adv.undo_info(tbs_name, tbs_size, tbs_autoextend, tbs_retention, tbs_guarantee);
 If undo_adv=TRUE then
 dbms_output.put_line('UNDO Tablespace Name : ' || tbs_name);
 dbms_output.put_line('UNDO tablespace is '|| CASE WHEN tbs_autoextend THEN 'Auto Extensiable' ELSE 'Fixed Size' END);
 If tbs_autoextend=TRUE then dbms_output.put_line('UNDO Tablespace Maximum size (MB) is : ' || TO_CHAR(tbs_size));
 else dbms_output.put_line('UNDO Tablespace Fixed size (MB) is : ' || TO_CHAR(tbs_size));
 end if;
 dbms_output.put_line('Undo Retention is ' || TO_CHAR(tbs_retention)||' Seconds' ||' Equivelant to ' ||round((tbs_retention/60),2) ||' Minutes');
 dbms_output.put_line('Retention : '||CASE WHEN tbs_guarantee THEN 'Guaranteed ' ELSE 'Not Guaranteed' END);
 else dbms_output.put_line('Function undo_info can only run if parameters undo_management is auto');
 end if;
 dbms_output.put_line('=====================================================================');
 END;
 /

==============================================

UNDO Tablespace Name : UNDOTBS2

UNDO tablespace is Auto Extensiable

UNDO Tablespace Maximum size (MB) is : 32768

Undo Retention is 900 Seconds Equivelant to 15 Minutes

Retention : Not Guaranteed

==============================================

PL/SQL procedure successfully completed.

 

Function longest_query returns the length of the longest query for a given period

SQL> SELECT 'The Length of the Longest Query in Memory is ' || dbms_undo_adv.longest_query LONGEST_QUERY FROM dual;
LONGEST_QUERY
--------------------------------------------------------------------------------
The Length of the Longest Query in Memory is 1472

 

The Output using Start/End time :

SQL> SELECT 'The Length of the Longest Query During This Time Range is ' ||dbms_undo_adv.longest_query(SYSDATE-1/24, SYSDATE) LONGEST_QUERY FROM dual;

LONGEST_QUERY

The Length of the Longest Query During This Time Range is 1080

Function required_retention  returns the required value for parameter undo_retention to satisfy longest query based on undo statistics available

SQL> SELECT 'The Required undo_retention using Statistics In Memory is ' || dbms_undo_adv.required_retention required_retention FROM dual;

REQUIRED_RETENTION

——————————————————————————–

The Required undo_retention using Statistics In Memory is 1472

Current retention is 900,  and the required retention is 1472

 

SQL> SELECT 'The Required undo_retention During This Time Range is ' ||dbms_undo_adv.required_retention(SYSDATE-1/24, SYSDATE) required_retention FROM dual;

REQUIRED_RETENTION

——————————————————————————–

The Required undo_retention During This Time Range is 1080

 

SQL> SELECT 'The Required undo tablespace size using Statistics In Memory is ' || dbms_undo_adv.required_undo_size(900) || ' MB' required_undo_size FROM dual;

REQUIRED_UNDO_SIZE

——————————————————————————–

The Required undo tablespace size using Statistics In Memory is 80 MB

 

Function undo_health is used to check if there is any problem with the current setting of undo_retention and undo tablespace size based on the historical information of given period , and provide recommendation to fix the problem.

 

If the return value is 0, no problem is found. Otherwise, parameter “problem” and “recommendation” are the problem and recommendation on fixing the problem.

 

The Output Parameters are :

 

problem: problem of the system. It can be for example : “long running query may fail” or “undo tablespace cannot satisfy undo_retention”.

recommendation: recommendation on fixing the problem found.

rationale: rationale for the recommendation.

retention: numerical value of retention if recommendation is to change retention.

utbsize: numberical value of undo tablespace size in MB if recommendation is to change undo tablespace size.

 

DECLARE
 prob VARCHAR2(100);
 reco VARCHAR2(100);
 rtnl VARCHAR2(100);
 retn PLS_INTEGER;
 utbs PLS_INTEGER;
 retv PLS_INTEGER;
 BEGIN
 retv := dbms_undo_adv.undo_health(prob, reco, rtnl, retn, utbs);
 dbms_output.put_line('=====================================================================');
 If retv=0 Then dbms_output.put_line('Problem: ' || prob || ' The undo tablespace is OK');
 ELSIF retv=2 Then dbms_output.put_line('Long running queries may fail , The recommendation is :' || reco);
 dbms_output.put_line('rationale: ' || rtnl);
 dbms_output.put_line('retention: ' || TO_CHAR(retn));
 ELSIF retv=3 Then
 dbms_output.put_line('The Undo tablespace cannot satisfy the longest query ,The recommendation is : ' || reco);
 dbms_output.put_line('rationale: ' || rtnl);
 dbms_output.put_line('undo tablespace size in MB : ' || TO_CHAR(utbs));
 dbms_output.put_line('retention: ' || TO_CHAR(retn));
 ELSIF retv=4 Then
 dbms_output.put_line('The System does not have an online undo tablespace , The recommendation is : ' || reco);
 dbms_output.put_line('rationale: ' || rtnl);
 dbms_output.put_line('undo tablespace size in MB : ' || TO_CHAR(utbs));
 ELSIF retv=1 Then
 dbms_output.put_line('The Undo tablespace cannot satisfy the specified undo_retention or The Undo tablespace cannot satisfy auto tuning undo retention , The recommendation is :' || reco);
 dbms_output.put_line('rationale: ' || rtnl);
 dbms_output.put_line('undo tablespace size in MB : ' || TO_CHAR(utbs));
 end if;
 dbms_output.put_line('=====================================================================');
 END;
 /

PL/SQL procedure successfully completed.

Needed undo tablespace for specified undo retention:-

SELECT d.undo_size/(1024*1024) "ACTUAL UNDO SIZE [MByte]",
 SUBSTR(e.value,1,25) "UNDO RETENTION [Sec]",
 (TO_NUMBER(e.value) * TO_NUMBER(f.value) *
 g.undo_block_per_sec) / (1024*1024)
 "NEEDED UNDO SIZE [MByte]"
 FROM (
 SELECT SUM(a.bytes) undo_size
 FROM v$datafile a,
 v$tablespace b,
 dba_tablespaces c
 WHERE c.contents = 'UNDO'
 AND c.status = 'ONLINE'
 AND b.name = c.tablespace_name
 AND a.ts# = b.ts#
 ) d,
 v$parameter e,
 v$parameter f,
 (
 SELECT MAX(undoblks/((end_time-begin_time)*3600*24))
 undo_block_per_sec
 FROM v$undostat
 ) g
 WHERE e.name = 'undo_retention'
 AND f.name = 'db_block_size'
 /

Oracle time stamp showing wrong timezone connecting TNS remote

Time stamp problem:-

sqlplus system/****@ORCL

SQL>alter session set nls_date_format=’dd-mm-yyyy hh:mi:ss’;

SQL>select instance_name from v$instance;

SQL>select   sysdate,   current_timestamp,   systimestamp,   localtimestamp from   dual;

The above query will show different time .

How to Fix:-

1.1                               Stop the apps and DB cluster/HAS.

su – root

cd /orasw/app/grid/product/11.2.0/grid/bin

./crsctl stop crs –f

su – grid

cd /orasw/app/grid/product/11.2.0/grid/bin
./crsctl stop has

1.2                               Correct the TZ setting

Correct the TZ setting in the file $GRID_HOME/crs/install/s_crsconfig_<hostname>_env.txt.

Please comment the old TZ value(#TZ=GMT-08:00) in s_crsconfig_<hostname>_env.txt.

Add new TZ value(TZ=US/Pacific).Do not modify any other value.
su – root
cd /orasw/app/grid/product/11.2.0/grid/crs/install/
vi s_crsconfig_<hostname>_env.txt
TZ=US/Pacific
#TZ=GMT-08:00

1.3                               Restart the cluster and DB/HAS

 

su – root
cd /orasw/app/grid/product/11.2.0/grid/bin
./crsctl start crs

For single instance HAS

su – grid

cd /orasw/app/grid/product/11.2.0/grid/bin
./crsctl start has

Checking timestamp now:-

sqlplus system/****@ORCL

SQL>alter session set nls_date_format=’dd-mm-yyyy hh:mi:ss’;

SQL>select instance_name from v$instance;

SQL>select   sysdate,   current_timestamp,   systimestamp,   localtimestamp from   dual;

The above query will show same time .

Error undefined symbol: JNU_ThrowByName during oracle 12c R2 installation

We faced this error during Oracle12cR2 installation.

During our installation of oracle 12.2 in new home on existing Oracle 11g installation,we faced the error

 

INFO: Executing [/opt/oracle/product/12.2.0.1/db1/bin/diagsetup, clustercheck=false, basedir=/opt/oracle/product, oraclehome=/opt/oracle/product/12.2.0.1/db1]

Solution

We found there was old 11g Database installed and all environment variables were pointing to old installation.After resetting variables to new installed ORACLE_HOME and other variable,it got resolved.

 

ORA-12545: Connect failed because target host or object does not exist using SERVICE_NAME in tnsnames.ora

Today I observed that we were getting following error when we were going to connect using SERVICE_NAME from (DR site) to new exadata server.

The connection was working fine with SID in tnsnames.ora to individual instances.

[oracle@XXX admin]$ sqlplus dba/XXX@PROD_CDV

SQL*Plus: Release 12.1.0.2.0 Production on Mon Nov 27 09:08:38 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-12545: Connect failed because target host or object does not exist

Enter user-name:

I was using following tns:-

PROD_CDV =
(description=
(load_balance=on)
(address=(protocol=tcp)(host= x.x.x.x )(port=1521))
(address=(protocol=tcp)(host= x.x.x.y )(port=1521))
(connect_data=
(service_name= PROD.world)
)
)

The ping,tnsping were absolutely fine.

I enable trace in sqlnet.ora from client site

vi $ORACLE_HOME/network/admin

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)
TRACE_LEVEL_CLIENT=USER
TRACE_FILENO_CLIENT=6
TRACE_FILELEN_CLIENT=51200
TRACE_UNIQUE_CLIENT=ON
TRACE_TIMESTAMP_CLIENT=ON
TRACE_DIRECTORY_CLIENT=/home/oracle/client_trace
LOG_DIRECTORY_CLIENT=/home/oracle/client_trace
DIAG_ADR_ENABLED=OFF
ADR_BASE = /u01/app/oracle

and found below issue in trace file generated under /home/oracle/client_trace:-

(3267593728) [000001 27-NOV-2017 09:08:48:587] nsc2addr: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=prod01vm03-vip)(PORT=1521)))
(3267593728) [000001 27-NOV-2017 09:08:48:587] snlinGetAddrInfo: getaddrinfo() failed with error -2
(3267593728) [000001 27-NOV-2017 09:08:48:587] nttbnd2addr: looking up IP addr for host: prod01vm03-vip
(3267593728) [000001 27-NOV-2017 09:08:58:608] snlinGetAddrInfo: getaddrinfo() failed with error -3
(3267593728) [000001 27-NOV-2017 09:08:58:608] nttbnd2addr: *** hostname lookup failure! ***
(3267593728) [000001 27-NOV-2017 09:08:58:608] nserror: nsres: id=0, op=77, ns=12545, ns2=12560; nt[0]=515, nt[1]=110, nt[2]=0; ora[0]=0, ora[1]=0, ora[2]=0

So it seems IP is getting implicitly converted to hostname .But this IP is not in DNS in DR site.

I added below entry of exadata VIP and physical IP in /etc/hosts in all 4 DR nodes

10.1.14.145 prod01vm03.tdeprdcl.world.com prod01vm03
10.1.14.147 prod02vm03.tdeprdcl.world.com prod02vm03
10.1.14.146 prod01vm03-vip.tdeprdcl.world.com prod01vm03-vip
10.1.14.148 prod02vm03-vip.tdeprdcl.world.com prod02vm03-vip

After that connection worked fine.

 

Reference:-

  • 454927.1 Using and Disabling the Automatic Diagnostic Repository (ADR) with Oracle Net for 11g
  • 219968.1 SQL*Net & Oracle Net Services – Tracing and Logging at a Glance
  • 834822.1 Oracle Net Diagnostics
  • 1076022.1 Examples of Troubleshooting Slow Oracle Net Connections

Oracle database instance recovery is taking too much time after shutdown immediate

1.We were going to open DB after shutdown immediate but it went hang.In alert log , we could only notice following message for 40 minutes.

 

Completed: ALTER DATABASE MOUNT
Tue Oct 31 01:08:39 2017
ALTER DATABASE OPEN
Beginning crash recovery of 1 threadsparallel recovery started with 9 processes
Started redo scanTue Oct 31 01:08:58 2017
Completed redo scan
read 3733362 KB redo, 2824480 data blocks need recovery
Started redo application at
Thread 1: logseq 39596, block 3434354
Recovery of Online Redo Log: Thread 1 Group 10 Seq 39596 Reading mem 0
Mem# 0: /oracle/data/san_oradata01/eocmdb/redofiles/log10b.rdo
Recovery of Online Redo Log: Thread 1 Group 11 Seq 39597 Reading mem 0
Mem# 0: /oracle/data/san_oradata01/eocmdb/redofiles/log11b.rdo
Tue Oct 31 01:09:18 2017
Recovery of Online Redo Log: Thread 1 Group 13 Seq 39598 Reading mem 0
Mem# 0: /oracle/data/san_oradata01/eocmdb/redofiles/log13b.log
Tue Oct 31 01:09:28 2017
Completed redo application of 628.90MB

 

2.Troubleshooting using following queries:-

export NLS_DATE_FORMAT=’dd-mon-yyyy hh24:mi:ss’

set feedback on

set heading on

set echo on

set pagesize 100

set linesize 150

col checkpoint_change# format 999999999999999999999

set echo on

set linesize 400

set pagesize 20000

col name format a75

col MEMBER format a100

col file_name format a100

col PLATFORM_NAME format a30

col TABLESPACE_NAME format a30

set numformat 999999999999999

col hxfil format 9999

col fhsta format 9999

col fhscn format 999999999999999999999999

col fhtmn format a30

col fhrba_Seq format 9999999999999

col FHBCP_THR format 9999

col fhthr format 999

col fhrba_seq format 99999999

col fhscn format 999999999999999999

set linesize 175

set pagesize 50

alter session set nls_date_format = ‘DD-MON-RRRR HH24:MI:SS’;

Spool recover.lst

select name,platform_name,open_mode,controlfile_type,log_mode,flashback_on,RESETLOGS_TIME,RESETLOGS_CHANGE# from v$database;

select name, status from v$datafile;

select name, recover, fuzzy, checkpoint_change#,RESETLOGS_TIME,RESETLOGS_CHANGE# from v$datafile_header;

select * from v$backup;

select name, open_mode, checkpoint_change#, ARCHIVE_CHANGE# from v$database;

select GROUP#,THREAD#,SEQUENCE#,MEMBERS,ARCHIVED,STATUS,FIRST_CHANGE# from v$log;

select GROUP#,member from v$logfile;

select * from v$recover_file;

select * from v$recovery_log;

select hxfil FILE#, fhsta STATUS, fhscn SCN, fhrba_Seq SEQUENCE, fhtnm TABLESPACE, FHBCP_THR Thread, fhafs AFUZZY from x$kcvfh order by hxfil;

— Min PITR ABSSCN is minimun SCN needed to open

select min(FHSCN) “LOW FILEHDR SCN”, max(FHSCN) “MAX FILEHDR SCN”, max(FHAFS) “Min PITR ABSSCN” from X$KCVFH;

— Min max SEQUENCE on file headers

SELECT fhthr thread, fhrba_seq sequence, fhscn scn, fhsta status, count(*) FROM x$kcvfh group by fhthr,fhrba_seq,fhscn,fhsta;

select min(FHRBA_SEQ) “LOW FILEHDR SEQUENCE”, max(FHRBA_SEQ) “MAX FILEHDR SEQUENCE” from X$KCVFH;

select HXFIL File_num,substr(HXFNM,1,140) File_name,FHTYP Type,HXERR Validity, FHSCN SCN, FHTNM TABLESPACE_NAME,FHSTA status ,FHRBA_SEQ Sequence, FHTHR Thread from X$KCVFH;

SELECT fhthr thread, fhrba_seq sequence, fhscn scn, fhsta status, count(*) FROM x$kcvfh group by fhthr,fhrba_seq,fhscn,fhsta;

 

3. Please execute following command to check recovery.

RMAN>backup validate check logical database;

SQL>alter system dump logfile ‘/oracle/data/san_oradata01/eocmdb/redofiles/log10b.rdo’ validate;

RMAN>recover database test;

 

4.Re-creating control file and run recover from RMAN

connect target /;
alter session set tracefile_identifier = “10046_trace_file”;
alter session set events ‘10046 trace name context forever, level 3’;
set echo on;
debug on;
run {
allocate channel d1 type disk;
recover database;
}
debug off;
exit

But this was stuck with following:-

RMAN> recover database;

Starting recover at 31-OCT-17
using channel ORA_DISK_1

starting media recovery

unable to find archived log
archived log thread=1 sequence=39598
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 10/31/2017 06:29:09
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 39598 and starting SCN of 12444621699598

 

5.Now we need to validate archive log 39597 for consistency.

RMAN> validate archivelog sequence 39597 thread 1;

6.We need to do following if recover stuck

a)Open one more session and shutdown abort the instance

b)Startup mount

c)Try with no parallel recovery

Now, you are applying the redolog named /oracle/data/san_oradata01/eocmdb/redofiles/log13b.log

with sequence 39598

Which is the current redo log:

SQL> select GROUP#,THREAD#,SEQUENCE#,MEMBERS,ARCHIVED,STATUS,FIRST_CHANGE# from v$log;

GROUP# THREAD# SEQUENCE# MEMBERS ARC STATUS FIRST_CHANGE#
—————- —————- —————- —————- — —————- —————-
8 1 39593 1 YES INACTIVE 12444097046518
9 1 39595 1 YES INACTIVE 12444097257724
10 1 39596 1 YES ACTIVE 12444097354077
15 1 39592 1 YES INACTIVE 12444096949452
12 1 39594 1 YES INACTIVE 12444097140637
13 1 39598 1 NO CURRENT 12444621699598 >>>>>>>>>>>>>>>>>>
14 1 39591 1 YES INACTIVE 12444095658402
11 1 39597 1 YES ACTIVE 12444621392892

SQL> recover database noparallel using backup controlfile;

If it ask for log sequence 39598, then provide the full path as below and then press <Enter>
/oracle/data/san_oradata01/eocmdb/redofiles/log13b.log  (This is current redolog file)

If it says… “Media Recovery complete” then open the db with resetlogs:

SQL> alter database open resetlogs;

Note:-Following views are very useful to monitor instance recovery progress.

Reference:-Monitoring Restore/Recovery Progress (Doc ID 1335910.1)

 Session Waits:-

Are there any sessions in wait and what is it waiting for?

Run the following queries at least 3 times, at 5 minute intervals to see progress/change.

set linesize 200 trimspool oncol event form a25
col p1text form a15
col p1 form 999999
col p2text form a15
col p2 form 999999
col p3text form a10
col p3 form 9999
col waited form 9999
col waiting form 9999select sid, event, p1text, p1, p2text, p2, p3text, p3,
wait_time waited, seconds_in_wait waiting
from gv$session_wait
where event not like ‘SQL*Net%’
and event not like ‘%timer%’
and event not like ‘rdbms%’
and event not like ‘pipe%’
and event not like ‘DIAG%’
and event not like ‘Streams AQ%’
and event not like ‘VKTM%’
and state = ‘WAITING’
order by seconds_in_wait
/

Recovery Progress:-

What is the recovery progress?  V$RECOVERY_PROGRESS is only populated when RECOVERY is in progress. A restore operation will not populate this view. So if you think a recovery process is slow – is it really at the recovery phase, or still restoring from RMAN backuppieces?

This is an example of a recovery progress:

 

START_TIME ITEM UNITS SOFAR TOTAL TIMESTAMP
——————– ——————————– ——————————– ———- ———- ——————–
31-OCT-2017 15:52:22 Active Apply Rate KB/sec 164 0
31-OCT-2017 15:52:22 Average Apply Rate KB/sec 167 0
31-OCT-2017 15:52:22 Maximum Apply Rate KB/sec 5460 0
31-OCT-2017 15:52:22 Redo Applied Megabytes 330 0
31-OCT-2017 15:52:22 Last Applied Redo SCN+Time 0 0 30-OCT-2017 13:45:57
31-OCT-2017 15:52:22 Active Time Seconds 1963 0
31-OCT-2017 15:52:22 Elapsed Time Seconds 2018 0
31-OCT-2017 15:14:17 Log Files Files 1 1
31-OCT-2017 15:14:17 Active Apply Rate KB/sec 40 40
31-OCT-2017 15:14:17 Average Apply Rate KB/sec 45 45
31-OCT-2017 15:14:17 Maximum Apply Rate KB/sec 341 341START_TIME ITEM UNITS SOFAR TOTAL TIMESTAMP
——————– ——————————– ——————————– ———- ———- ——————–
31-OCT-2017 15:14:17 Redo Applied Megabytes 89 89
31-OCT-2017 15:14:17 Last Applied Redo SCN+Time 0 0 30-OCT-2017 13:39:19
31-OCT-2017 15:14:17 Active Time Seconds 1903 1903
31-OCT-2017 15:14:17 Apply Time per Log Seconds 1903 1903
31-OCT-2017 15:14:17 Elapsed Time Seconds 2005 2005

 

6.Root Cause of slowness:-

  • Hang analyzer log says:-

is waiting for ‘parallel recovery slave idle wait’ with wait info:
{
p1: ”=0x1001ffff
p2: ”=0x1
p3: ”=0x182f945d00
time in wait: 1.004650 sec
timeout after: never
wait id: 57508
blocking: 0 sessions
current sql: <none>

  • IO stat report:-

The all local disk (not in SAN) was showing await.

AWR report :-

it looks like local disk issue not SAN issue. If you look into Av rd(ms) ,it is 1 -2 ms for SAN and for local it is 20 ms to 60 ms. Oracle suggestion is it should be less than 10 ms. So all datafiles should immediately be moved to SAN from Local mount point (Local does not have good striping capacity compare to SAN which will raise bottleneck) .

Troubleshoot connection issue using standalone jdbc driver to oracle

Hi All

Please download following file from Google drive and keep in unix directory say /home/oracle

https://drive.google.com/open?id=0B8-3R318F39hQUJOcnVPMUdrNzg

The files to be downloaded:-

Please use ojdbc7.jar downloaded for 12c.For 11g,you can use ojdbc6.jar.

 

Please provide execute permissions

Now you can test connection using below:-

[oracle@SBJ1ITESBDBV01 ~]$ sh sqlScript_new.sh
2017-10-26 09:45:53.0

Please note you may get following error if you use ojdbc6.jar to connect 12c database.

[oracle@SBJ1ITESBDBV01 ~]$ sh sqlScript_new.sh
Exception in thread “main” java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at sqlconnect.main(sqlconnect.java:20)