Tech Support Team Handbook for Techonlin

12/24/2019

© 2018 EdgeVerve Systems Limited. Strictly private and confidential. No part of this
document should be reproduced or distributed without the prior permission of EdgeVerve
Systems Limited.
Tech Support Team Handbook for Techonline call lodgment - FAS

Confidential Information
This document is confidential to EdgeVerve Systems Limited ("EdgeVerve") and the Bank. This
document contains information and data that EdgeVerve considers confidential and proprietary
(“Confidential Information”).
Confidential Information includes, but is not limited to, the following:
 Corporate, employee and infrastructure information about EdgeVerve
 EdgeVerve project management, support and quality processes
Any disclosure of Confidential Information to or use of it by a third party (i.e., a party other than the
Bank), will be damaging to EdgeVerve. Ownership of all Confidential Information, no matter in what
media it resides, remains with EdgeVerve.

Page 2 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

Version Control

Version Author Reviewer Date Changes

0.1 Supriya Baji Saravanan_p06 23-Sep- Techonline call lodgment
2019 guideline and help
document

1.0 Supriya Baji Saravanan_p06 24-Dec- Baseline

2019

Page 3 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

Table of Contents
1. What this document is about............................................................................................... 5
2. Required Details....................................................................................................................6
2.1 Basic Information:..................................................................................................................6
2.2 Standard Definition of Severity Levels:...............................................................................7
2.3 Detailed components of ‘L2 Initial Analysis’.......................................................................8
2.4 Check list while lodging the call:..........................................................................................9
2.5 Issue Classification..............................................................................................................10
2.6 General Database queries:................................................................................................12

3. Logs to be collected............................................................................................................ 13
3.1 FAS specific logs.................................................................................................................13
3.2 Server Logs..........................................................................................................................14
3.3 SSO Logs.............................................................................................................................15
3.4 For Performance issues......................................................................................................16

Page 4 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

1. What this document is about

This document is intended for IT technical team of the bank. IT team of the bank should refer to this
document to understand the guideline for lodging a call in Techonline.
The document will assist those who lodge a call in Techonline – the issue management system for
Finacle. This document explains the information that is expected to be captured while lodging the
Techonline call.

Page 5 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

2. Required Details
This section explains on the details that have to be included in TechOnline when a request is raised
by bank. The type of information is broadly classified into:
 Basic Information
 Detailed Information
 Attachments

1.1 Basic Information:

These are the very basic type of information regarding the environment in which the issue is
observed.

Information Details
Product* Product in which the issue is observed [e.g. FINALERT/ FAS]
Product Version* Finacle Alerts Solution version in which the issue is observed. [e.g.
10.5.01, 10.5.02 ]
Module name* Module in which the issue is reported [e.g. Application , Batches, FI ,
SSO ]
Menu option* The menu options involved while the issue occurred. [e.g. XXX] or batch
job or services this issue is occurring

Mode* The mode of Finacle Alerts Solution application [ Batches / WEB]

Issue Classification* The type of issue this is related to? [e.g. Clarification/ Customization /
Product]

CFO* Customer Impact / Financial Impact / Operational Impact

Location Status* The area or the location of the Finacle Alerts Solution application where
the issue is observed[e.g. Production/Customization/UAT/SIT/offshore]

Category* Classification of the type of issue. [e.g. Customization, Clarification,

Performance, Product Bug etc.]
Severity* When an issue is logged the correct severity has to be assigned to the
issue. When the bank users logs the call in defect tracking system,
request to follow the guidelines provided under ‘Definition of Severity
Levels’ below.
Subject* Describe the problem in brief.
Problem Description* Describe the problem in detail. Mention exception coming on screen / FI
API / batches
Initial L2 Analysis* Details of first level of analysis captured by the Bank IT team. For details,
please check section 2.3

Page 6 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

1.2 Standard Definition of Severity Levels:

Severity Business Impact

1. Directly reducing Bank’s ability to conduct business.
Severity – 1
2. Essential function not working.
3. Critical program aborts or stops.
Major Impact 4. Critical application unavailable.

1. Directly causing partial loss of Bank’s ability to conduct

Severity – 2
business.
2. Essential function severely restricted.
Moderate Impact 3. Severe service degradation.

1. Problem encountered has low impact on business.

Severity – 3
2. A non-critical problem in the System where the bank will be
able to continue to run the application or a workaround is
Minimal Impact available.

1. Causing only inconvenience.

Severity – 4
2. No business impact.
3. No Bank regulatory or revenue impact but alternate
No Impact mechanism available.

Note: Above severities are descriptive and subject to change as per SLA agreement with the
respective customer.

Page 7 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

1.3 Detailed components of ‘L2 Initial Analysis’

This section mentions the detailed information and the first level analysis of the issue / requirement
occurred at bank. This will help on faster understanding and quicker resolution of the issue.

Information Details
Simulation Steps If the issue is simulatable at bank, please mention the exact steps with
screen shots. All required logs also to be shared and attached in
Techonline. Also clearly mention the area where the simulation is
done? (Production / UAT etc.)
Impact How many users are impacted? What is the level of financial/business
impact?

Frequency How frequently the error is occurring in production and the time when
this issue is first reported?

Recent Deployment Is the issue observed after any recent deployment? If yes, please share
details of the deployment.

Application logs Application logs, Batches logs, system out logs, server startup logs etc.
For details, please check the log section

Miscellaneous Here bank can provide any other details bank team has noticed /
suspecting for causing the error, as per their analysis. Any recent
changes in the system level or bank level settings / set-up or resources
can be mentioned here.
Data Details Related table data, batch related table details and transaction details to
be shared

Page 8 of 16
 Tech Support Team Handbook for Techonline call lodgment - FAS

1.4 Check list while lodging the call:

1 Is the issue simulatable in TEST/SIT/Offshore/UAT environment? (Yes/No/NA)

If Yes, request to share detailed simulation steps along with screen snapshots.(Simulation performed in
2 TEST/UAT)
3 If No, request to provide banks initial observations/analysis on the issue reported.
4 The time when the issue occurred .
5 Can you recall any such similar problem faced earlier?
6 Was any solution suggested to the problem that you had faced earlier? If yes, mention the same.
Is the issue occurring after any recent changes –service pack / patch deployments or customization
patch movement/configuration changes/set-up changes in production? If yes, mention the service pack/
7 patch id.
8 Any recent environmental changes performed? (Webserver/Appserver/OS/DB/hardware etc.)
Any recent changes done at OS/DB level (E.g. Memory tuning , Network related, Permissions at table
9 level ,Installation of patches)?
1 Reason for the performing the changes (at Application / Environment /OS/DB level) (TOL id or
0 reference)
1
1 Request to specify the issue is specific to a particular User. If so, please share user details.
1
2 Is the issue occurring always or intermittent in nature?
1
3 Please share any pattern observed if the issue is intermittent.

1 Based on the initial analysis performed bank or L2, Kindly zip the required files and upload the same in
4 Techonline based on the issue classification. Please refer the Issue Classification section.
1
5 Please provide the last APDM patch id deployed in environment.
1
6 Please confirm if UAT and Production are in Sync.

Page 9 of 16
 Tech Support Team Handbook for Techonline call lodgment - FAS

1.5 Issue Classification

Issue Classification Information required.

1. Application logs/Batch logs (Refer Type of logs section for the required
logs.)
2. Server logs (Refer Type of logs section for the required logs.)
3. Database logs (Refer Type of logs section for the required logs.)
4. Heap dump, Thread dump to be shared based on the situations below.
a. System/Particular use Case Slowness:
Application side - Five sets of Thread dumps which should be taken at 5 sec-10
sec interval at application server level. Thread dumps need to be taken before
restart.
Batch side – 5 to 7 sets of Thread dumps which should be taken at 10 sec-
20sec interval at JVM level. Thread dumps need to be taken before killing or
restarting batch.
b. Out of Memory case:
1. Performance issue
Thread dump & Heap dump that got generated at the working folder.
c. Database Performance:
i. Query.
ii. Execution Plan
iii. Table_stats (for Oracle DB)
iv. Table details along with available index
Performance issue at webserver side:
1. httpd.conf file
2. access logs / error log

1.Screen Shot
2. Application logs/Batch logs (Refer Type of logs section for the required
logs.)
2. Product Issue
3. Server logs
a. Systemerr.log
b. Systemout.log
c. Server startup logs

Page 10 of 16
 Tech Support Team Handbook for Techonline call lodgment - FAS

1.Screen shots – Non-Mandatory

2. Application logs/Batch logs (Refer Type of logs section for the required
logs.)- Non-Mandatory
3. Exact business requirement in detail
3. Clarification
4. Server logs (Refer Type of logs section for the required logs.) - Non-
Mandatory
a. Systemerr.log
b. Systemout.log
c. Sever startup logs
1. Screen shots.
2. Custom flow written and share the custom files.
4. Customization Issue
(only if customization 3. Application logs/Batch logs (Refer Type of logs section for the required
support is with logs.)
EV/Infosys)
3. Server logs
a. Systemerr.log
b. Systemout.log
c. Server startup logs
5. Enhancement
Requirement document
Request.(SER)

Page 11 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

1.6 General Database queries:

Please refer the below attached GeneralQueries.doc. This will have details of table
involved for some frequently used menu and batches.

Page 12 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

3. Logs to be collected
There are various types of logs present in the Finacle Alerts Solution. These logs are to be shared
along with the issue description and initial analysis to identify the cause of the defect based on the
type of the issue being raised. There are 3 varieties of logs
1) Application logs
2) FI logs
3) Batches logs

1.7 FAS specific logs

a) APPLICATION LOGS
Websphere application server
Location of the Log: FE/PROFILE/log/fas
a) logdebug.out

b) loginfo.out

c) logerror.out

Jboss Application server

Location of the Log: CUT/FDM+/scripts/log/fas
a) logdebug.out

b) loginfo.out

c) logerror.out

Weblogic Application server

Location of the Log: ../<Domain_Name>/log/fas
a) logdebug.out

b) loginfo.out

c) logerror.out

B) BATCH LOGS

HOST BATCHES
Location of host batches log :
FE/AlertsSolution/fas/hostbatch/host/log /fas
FE/AlertsSolution/fas/hostbatch/subscription/log /fas

Page 13 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

a) logdebug.out

b) loginfo.out

c) logerror.out

ALERT BATCHES
Location of alert batch log:
FE/AlertsSolution/fas/alertbatch/alertoverride/log/fas
FE/AlertsSolution/fas/alertbatch/archieval/log/fas
FE/AlertsSolution/fas/alertbatch/autosubscription/log/fas
FE/AlertsSolution/fas/alertbatch/delivery/log/fas
FE/AlertsSolution/fas/alertbatch/publish/log/fas
FE/AlertsSolution/fas/alertbatch/purge/log/fas
FE/AlertsSolution/fas/alertbatch/userupload/log/fas
d) logdebug.out

e) loginfo.out

f) logerror.out

C) FI LOGS

HTTP ROUTE

<FI_APP_PATH>/ COMMON/LOG/FSB/FSBHTTP
<FI_APP_PATH>/ COMMON/LOG/FININT/FEASMB
N OTE: FI_APP_PATH: FE folder path like /app1/FAS/FE/common/log

1.8 Server Logs

Location of Logs: Application server installed path from where you start and stop the service.

WebSphere application server:

1. StartServer.log
2. StopServer.log
3. Systemerr.log

Page 14 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

4. Systemout.log
5. FFDC logs

Jboss Application server:

1.startJBossStandalone.log
Weblogic Application Server:
1. <ProfilePath>/bin/nohup.out

1.9 SSO Logs

If issue is coming in SSO login or before selecting the FinacleAlerts then share the below logs.

Path: FE/common/log/fininfra/SSOServerLog.txt

Page 15 of 16
Tech Support Team Handbook for Techonline call lodgment - FAS

1.10 For Performance issues

1. 5-6 sets of Thread dumps which should be taken at 5-10 sec interval at server level where
issue is coming ( application / batches).
2. Heap dump or thread dumps that got generated at the working folder automatically by the
WAS. (Heap dumps will be big in size, hence bank should have heap analyzer in any
machine along with live meeting setup so that our Archie team will analyze and provide
suitable suggestion.)
Procedure for dumps generation is covered in the below document.

3. Utilization details from each server (both CPU and memory)

Page 16 of 16