Pod EJGK

Customer KPMG
Em Site OCI-FRA-FA-1
Version 11.13.24.01.0
Line of Business GSI
Lifecycle Status Production
Pod Size M8

ISSUE : Authentication Errors occurred for brief period of time

SEQUENCE OF INCIDENT:
1) RATS ALERT.
Open BI Analytics Page at : https://ejgk.fa.em2.oraclecloud.com/analytics/saw.dll?
bieehome&startPage=1 : A page error has been found: You are not currently signed in

2) Users getting Authentication error.

3) We do see that Authentication error was spiked for Crawler Job

(BISearchSessionManager). The crawls are configured to run at time : 30 mins post
the search
plugin run. If at that time the OBIPS is not up, the crawl will fail.

4) Found DB Instances were restarted at same time.

DB Restart Time:

TO_CHAR(STARTUP_TIME,'DD-MO INSTANCE_NUMBER
--------------------------- ---------------
23-APR-24 10:12:50 7
23-APR-24 12:14:51 8

After alert the complete BI was bounced by OCSC Team during MI, which resolved the
issue.

-----------------------------
OBIPS Killed:
-----------------------------
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <The server
'obips1' with process id 4020410 is no longer alive; waiting for the process to
die.>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <34991374
ServerMonitor.runMonitor waitForProcessDeath, RotateServerOutOnShutdown: false,
rotateServerFiles if true>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Process
died.>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <34991374
ServerMonitor.runMonitor, AFTER process died , serverMgr:
weblogic.nodemanager.server.ServerManager@3c435123, serverMgr.getServerDir:
/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/system_components/
OBIPS/obips1, serverMgr.getServerDir().getOutFile:
/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/system_components/
OBIPS/obips1/logs/obips1.out, conf:
weblogic.nodemanager.common.SystemComponentStartupConfig@150ab4ed,
conf.getServerOutFile: null, serverDir.exists? true>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Process is
killed 4020410 with Server was killed>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <Server was
killed>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <runMonitor
returned, setting finished=true and notifying waiters>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Finished
killing process>

------------------------------
OBIPS Started:
------------------------------
<May 23, 2024 10:50:22 AM UTC> <INFO> <NodeManager ComponentManager>
<Execute:/u01/APPLTOP/fmw/bi/bifoundation/web/bin/sawserver>
<May 23, 2024 10:50:22 AM UTC> <INFO> <NodeManager ComponentManager> <Executing:
/u01/APPLTOP/fmw/bi/bifoundation/web/bin/sawserver>
<May 23, 2024 10:50:22 AM UTC> <FINE> <NodeManager ComponentManager> <After
super.specifyEnvironmentValues...

This is NOT a BI Issue.

{
"server": "bi",
"time": "2024-05-23T10:40:53.384",
"component": "bi_serverHA",
"severity": "ERROR",
"module": "oracle.biee.search",
"tid": " BISearchPushCrawler-thread-3] [userId: anonymous",
"ecid": "00656i^x^UD6IRMMyMmZMG00BUXH0006wt,0:7:40:2:1:6",
"message": "[APP: bisearch] [partition-name: DOMAIN] [tenant-name: GLOBAL] Failed
to create system session due to exception: Authentication error. An error occurred
during authentication. Try again later or contact your service administrator.[[\n
javax.xml.ws.soap.SOAPFaultException: Authentication error. An error occurred
during authentication. Try again later or contact your service administrator.\n\tat
com.siebel.analytics.web.jaxwsutils.DirectSoapCaller.invokeSoapMethod(DirectSoapCal
ler.java:263)\n\tat
com.siebel.analytics.web.jaxwsutils.SAWSOAPTcpConnector.invokeSoapMethod(SAWSOAPTcp
Connector.java:79)\n\tat
com.siebel.analytics.web.jaxwsutils.SOAPInterfaceProxy.invoke(SOAPInterfaceProxy.ja
va:206)\n\tat com.sun.proxy.$Proxy518.logonex(Unknown Source)\n\tat
oracle.biee.search.security.BISearchSessionManager$BIPSSoapSession.createSessionID(
BISearchSessionManager.java:568)\n\tat
oracle.biee.search.security.BISearchSessionManager$BIPSSoapSession.createSession(BI
SearchSessionManager.java:432)\n\tat
oracle.biee.search.security.BISearchSessionManager$BIPSSoapSession.<init>(BISearchS
essionManager.java:415)\n\tat
oracle.biee.search.security.BISearchSessionManager$BIPSSoapSession.<init>(BISearchS
essionManager.java:395)\n\tat
oracle.biee.search.security.BISearchSessionManager.lambda$getSession$0(BISearchSess
ionManager.java:283)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
oracle.biee.search.security.BISearchSessionManager.getSession(BISearchSessionManage
r.java:294)\n\tat
oracle.biee.search.security.BISearchSessionManager.getWebcatIndexingSession(BISearc
hSessionManager.java:183)\n\tat
oracle.biee.search.security.BISearchSessionManager.getWebcatIndexingSession(BISearc
hSessionManager.java:177)\n\tat
oracle.biee.search.services.crawl.CatalogPushCrawler.call(CatalogPushCrawler.java:4
8)\n\tat
oracle.biee.search.services.crawl.CatalogPushCrawler.call(CatalogPushCrawler.java:2
2)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\
tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\
tat java.lang.Thread.run(Thread.java:767)\n\n]]",
"log_path":
"/u01/APPLTOP/instance/domains/fa-ha1.oracleoutsourcing.com/FADomain/servers/
bi_serverHA/logs/bi_serverHA-diagnostic.log",
"log_name": "bi_serverHA-diagnostic.log"
}

First:

/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/servers/obips1/
logs/sawjsonlog4.log

{"timestamp":"1715268732972","datetime":"1715268732.972","time":"2024-05-
09T15:32:12Z","application":"OBIPS","loglevel":"NOTIFICATION","severity":"1","categ
ory":"saw.securitysubsystem.session.logoff","ecid":"00656i^x^UD6IRMMyMmZMG00BUXH000
6wt","rid":"0:7:40:2:1:40:1:4:1:3502","tread":"763347792","sik":"ssi","username":"h
r8ks05ymxcuwj6zfgonvpeltdib7qa3","message":"SystemUser
hr8ks05ymxcuwj6zfgonvpeltdib7qa3 logged out (session ID: .1d6dd.6ho4ljs, service
instance context: ssi, active: 7173
seconds).","filename":"masterclientsession.cpp","fileline":"615","logsources":"saw.
securitysubsystem.session.logoff saw.soap.sessionrequesthandler saw.SOAP
saw.httpserver.request.soaprequest saw.rpc.server.responder saw.rpc.server
saw.rpc.server.handleConnection saw.rpc.server.dispatch
saw.threadpool.socketrpcserver
saw.threads","stacktrace":"","log_path":"/u01/APPLTOP/instance/domains/
fa.oracleoutsourcing.com/FADomain/servers/obips1/logs/
sawjsonlog4.log","log_name":"sawjsonlog4.log"}

/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/servers/obis1/
logs/obis1-diagnostic.log","log_name":"obis1-diagnostic.log

{"timestamp":"1715269587604","datetime":"1715269587.604","time":"2024-05-
09T15:46:27Z","application":"OBIPS","loglevel":"NOTIFICATION","severity":"1","categ
ory":"saw.securitysubsystem.session.logon","ecid":"00656i^x^UD6IRMMyMmZMG00BUXH0006
wt","rid":"0:7:40:2:1:7:9169:1","tread":"797201232","sik":"ssi","username":"hr8ks05
ymxcuwj6zfgonvpeltdib7qa3","message":"SystemUser hr8ks05ymxcuwj6zfgonvpeltdib7qa3
logged in (session ID: .1d7d2.39fq1ic, service instance context:
ssi).","filename":"masterclientsession.cpp","fileline":"485","logsources":"saw.secu
ritysubsystem.session.logon saw.securitysubsystem.checkauthentication.runimpl
saw.threadpool.asynclogon
saw.threads","stacktrace":"","log_path":"/u01/APPLTOP/instance/domains/
fa.oracleoutsourcing.com/FADomain/servers/obips1/logs/
sawjsonlog4.log","log_name":"sawjsonlog4.log"}

{"timestamp":"1716461030493","datetime":"1716461030.493","time":"2024-05-
23T10:43:50Z","application":"OBIPS","loglevel":"ERROR","severity":"16","category":"
saw.threadpool.asynclogon","ecid":"00656i^x^UD6IRMMyMmZMG00BUXH0006wt","rid":"0:7:4
0:2:1:40:1:11:11395:1","tread":"1093018448","sik":"","username":"","message":"Threa
dPool\u003cAsyncLogon:1\u003e: Job will not be executed. Job was waiting in the
queue for too long before it was scheduled for processing (2510709 mls). Start
timeout for this job was 900000
mls","filename":"threadpool.cpp","fileline":"905","logsources":"saw.threadpool.asyn
clogon saw.threads","stacktrace":"","log_path":"/u01/APPLTOP/instance/domains/
fa.oracleoutsourcing.com/FADomain/servers/obips1/logs/
sawjsonlog2.log","log_name":"sawjsonlog2.log"}

-----------------------------
OBIPS Killed:
-----------------------------
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <The server
'obips1' with process id 4020410 is no longer alive; waiting for the process to
die.>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <34991374
ServerMonitor.runMonitor waitForProcessDeath, RotateServerOutOnShutdown: false,
rotateServerFiles if true>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Process
died.>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <34991374
ServerMonitor.runMonitor, AFTER process died , serverMgr:
weblogic.nodemanager.server.ServerManager@3c435123, serverMgr.getServerDir:
/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/system_components/
OBIPS/obips1, serverMgr.getServerDir().getOutFile:
/u01/APPLTOP/instance/domains/fa.oracleoutsourcing.com/FADomain/system_components/
OBIPS/obips1/logs/obips1.out, conf:
weblogic.nodemanager.common.SystemComponentStartupConfig@150ab4ed,
conf.getServerOutFile: null, serverDir.exists? true>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Process is
killed 4020410 with Server was killed>
<May 23, 2024 10:43:51 AM UTC> <INFO> <NodeManager ComponentManager> <Server was
killed>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <runMonitor
returned, setting finished=true and notifying waiters>
<May 23, 2024 10:43:51 AM UTC> <FINEST> <NodeManager ComponentManager> <Finished
killing process>

------------------------------
OBIPS Started:
------------------------------
<May 23, 2024 10:50:22 AM UTC> <INFO> <NodeManager ComponentManager>
<Execute:/u01/APPLTOP/fmw/bi/bifoundation/web/bin/sawserver>
<May 23, 2024 10:50:22 AM UTC> <INFO> <NodeManager ComponentManager> <Executing:
/u01/APPLTOP/fmw/bi/bifoundation/web/bin/sawserver>
<May 23, 2024 10:50:22 AM UTC> <FINE> <NodeManager ComponentManager> <After
super.specifyEnvironmentValues...