MindStream Analytics

Webinar Series

Recent Issues/Fixes and

Monitoring
Recommendations for
Hyperion EPM
9.04.14

Overview
Introducing MindStream Analytics
Why do Organizations need a well defined application
management structure?
Recent issues and fixes

Monitoring Recommendations
Questions

MINDSTREAM

MindStream Analytics:
Business Intelligence in the 21st Century
MindStream delivers premier consulting and
managed services solutions to clients by
enhancing technology and aligning resources
through a systemic process to harness insight
and enable financial and operational fact-based
decision making.

MINDSTREAM

About MindStream Analytics

Boston based consulting and managed services provider

Oracle Platinum Partner with Hyperion Specialization
Support Team: Consultant Experience Dedicated Administrators
Committed to prioritizing timely and accurate responses to fit in with your
reporting timetable, ensuring that your business critical applications remain fully
operational.
Complete EPM environment

Infrastructure issues
Application maintenance
Oracle support
User queries
Change requests
Application enhancements
Training and process control

MINDSTREAM

Case Specific Management Services

MindStream Analytics and its Management Services

is focused on providing a level of personalized care to
each client
An experienced team of service providers is on hand
to give individualized responses to client issues
Escalation process ensures that the right people
address each and every issue

MINDSTREAM

Overview
Introducing MindStream Analytics
Why do Organizations need a well defined application
management structure?
Recent issues and fixes

Monitoring Recommendations
Questions

MINDSTREAM

Application Management Process

Why do organizations need a well defined application
management process?

Compliance and Audits

Change management logs
Security Access reports
Incident management reports

Application Stability/Upgrade recommendations/Oracle

support schedule
Know when Oracle comes out with a new release and what they
decide to do with YOUR version going forward.
Understanding what pieces of your infrastructure and applications are
working for you or against you.
Minimize system downtime.
MINDSTREAM

Overview
Introducing MindStream Analytics
Why do Organizations need a well defined application
management structure?
Recent issues and fixes

Monitoring Recommendations
Questions

MINDSTREAM

Issue: Disk space used

A Foundation server had 95% of disk space used.
MindStream discovered that the increase in used disk
space was related to a rebuild of QA Essbase
applications migrated through the server and not
cleaned up
Removing the aforementioned extracts decreased
disk space utilization to 80%.
Lesson learned: In order to quickly avoid disk space
issues, it is very important to know which directories
are necessary to run your EPMSystem and which can
be removed.

MINDSTREAM

Issue: Unable to start application

in Essbase
[Mon Feb 24 09:30:12 2014]Local/ActualPL///47593571703872/Error(1120006)
Unable to lock file
[/data/Oracle/Middleware/user_projects/epmsystem5/EssbaseServer/essbases
erver1/app/ActualPL/ActualPL/ActualPL.esm]. Essbase will try again after a
short delay.
[Mon Feb 24 09:30:13
2014]Local/ActualPL///47593571703872/Warning(1007089) Unable to open
[/data/Oracle/Middleware/user_projects/epmsystem5/EssbaseServer/essbases
erver1/app/ActualPL/ActualPL/ActualPL.esm]for database [ActualPL]

MINDSTREAM

Troubleshooting Steps
MindStream discovered that there was a hung process
specific to the application after stopping Essbase.
Found the process associated with it on the Essbase
server by running a grep on the essbase process (Unix)
Needed to kill the specific process ID associated with the
application and restart the app through EAS. Once the
process was killed it freed up the locked application.
Lesson learned: Always be sure all processes are
stopped before restarting.

MINDSTREAM

Issue: HFM Journal Report will not run

Journal report was aborting with no specific errors

MINDSTREAM

Issue: HFM Journal Report will not run

MindStream found that the hsveventlog.log and
hfm.odl.log files had grown beyond the 25MB limit
recommended by Oracle.
After stopping all HFM services, we archived the log files on
all 3 HFM servers while making sure no HFM related
processes were running (specifically)
HyS9FinancialManagementWeb.exe, HsvDataSource.exe
and CASSecurity.exe)

Restarting HFM recreated the logs and allowed the

journal report to run.
Lesson learned: Dont allow HFM logs to get too big, it
can effect system performance and functionality.
Bonus: We created a script that archived the files
regularly during scheduled restarts to prevent the
issue in the future.
MINDSTREAM

Issue: Hyperion QA environment

unstable and locking up on users
A calculation in Essbase that typically took five
minutes to run had been running for over two hours.
Because of application monitoring and benchmarking,
MindStream was able to identify large metadata
additions that were not previously communicated and
recommended that the admin reorder the outline.
Lesson learned: Adding a large amount of metadata
to any application can largely effect performance of
the application.

MINDSTREAM

Issue: Essbase application stopping

abruptly in Production.
Application was aborting after start. After examining
the EAS logs, MindStream found that when EAS
crashed and a restart was attempted, there were still
processes running that EAS was trying to start and
therefore could not start correctly
Recommended that the process HyS9eas.exe on the
Essbase server is completely stopped before a restart
of the service as it was still trying to call on essbase.
Lesson learned: Always double check for hung
processes when restarting or if you run into issues
after a restart.
MINDSTREAM

Issue: A rule in HFM was not working

properly
A rule developed by a MindStream client was not
providing desired results their HFM application.
The rule did not reference every dimension.
MindStream recommended that the rule reference the
necessary members in the Custom 1 dimension which
fixed the issue.
Lesson learned: be sure that all dimensions are
referenced if having a rule issue in HFM.

MINDSTREAM

Issue: HFM System performance

MindStream restarted and triaged the system on
automated restart.
Found the task that was set to run the restart would not do so
without a logged in user in the windows Task Scheduler and
therefore, the restarts were not running on schedule

Restarted the environment and fixed the restart

scripts to increase system speeds and repaired task
issue.
Lesson: be sure to set your scheduled tasks to run
when the user is NOT logged into the server. Also,
regular restarts of HFM are extremely important to
system performance.
MINDSTREAM

Issue: Essbase Server approaching

disk space limit
MindStream alert notified us of an Essbase database
growing quickly. We could either add disk space or
remove data. The database was 145gb.
We reached out to the business to find out if there was data
they would never use again and archived a Level0 export of
the database at a point in time then cleared the
year/period/scenarios specified by the business.
Tested the cleardata calc and defragment process in
Development and applied the same steps to production.
Process decreased database size from 145GB to 77GB.

MINDSTREAM

Issue: Production Essbase down

End users encountering issues when logging into
Workspace or Smart View.
Errors claiming these services were failing against Shared
Services Server. Essbase also would not start.

MindStream discovered the Oracle database Flash

Recovery area was at maximum capacity.
MindStream wrote a recovery manager script to keep only the
last 7 days of backups in the FRA.
By running the scheduled RMAN backup/Archive script, we
were able to successfully backup and reboot all services.
Allocated space to the backup pool and created a new script
that manages backups in order to optimize the storage space.
Lesson Learned: Always setup FRA archives for your
relational database when using Oracle database.
MINDSTREAM

Issue: Corrupted database files

Client stated that dense restructure of database in
production environment was failing.
Restructure would not kick off in either background or
foreground.
Start/stop of the application would not work either.
Dummy accounts created to prompt restructure unsuccessful.

Stopped database
Removed .inn and .pan files from database directory and
restarted application.
Restructure was successful.

Lesson learned: Restructures can abort from

corrupted temporary files. The database can be
recovered without losing anything by removing the
temporary files.
MINDSTREAM

Issue: Unable to Email Financial Reporting

When attempting to email a report as part of the
output of a Financial Reporting Batch, the email is not
received. The following error is seen in the "Show
Details" output after the batch completes: "Error
sending e-mail".

MINDSTREAM

Issue: Unable to Email Financial Reporting

To change the SMTPMailServer value:
1.In Workspace, navigate to the Shared Services Console (under Navigate > Administer)
2.Navigate to the appropriate section to access the entry where the SMTPMailServer value is
located: Application Groups/Foundation/Deployment Metadata/Foundation Services/Shared
Services/Properties
3.Right-click on the Properties and select to Export to Edit
4.Make a copy of the Component.Properties file
5.Edit the following values to have the correct server name and save the file.
SMTPHostName =
SMTPMailServer =
6.Import the updated file to the Shared Services Registry
7.Restart the following services: Reporting and Analysis Framework, Reporting and Analysis
Framework Web, Financial Reporting Web, Financial Reporting Print

Lesson learned: How to configure FR emailing by configuring SMTP

server in Shared Services
MINDSTREAM

Issue: Planning Database Refresh

Fails With Error:

MINDSTREAM

Issue: Planning Database Refresh Fails

This issue can be resolved by one of the following options:
A.Edit the Planning Application's Data source and configure an Essbase
user with Essbase Administrator role. Then, restart the Planning web
application service and refresh the database.
Or
B.In Shared Services provision the pre-existing Essbase user id
configured in the Data source with the Administrator role in Essbase. If
using versions earlier than 11.1.2.x, then refresh security from Shared
Services via EAS console or via MaxL. Then, restart the Planning web
application service and refresh the database.

MINDSTREAM

Smart View Error XML Load Error

Login to Planning DB to delete a row using:

DELETE FROM <appname>_USERPARAMS
WHERE Username like '<username>@%' and
Parameterkey = 'UserMbrSel'
< appname> = (The company's HFM App name)
<username> = (The affected user's User ID)
This will purge the affected record that is causing the
XML Load Error
MINDSTREAM

How to Increase Smart View drill down

capacity

On the APS server navigate to

Oracle/user_projects/epmsystem1/aps/bin. Find the
User Essbase property file and change the max
SmartView rows to the necessary max row number.
Disclaimer This can put a large strain on APS and
can cause the service to crash. Test to max number
of rows to be sure there are no negative
consequences. You can also increase the Java heap
size on the APS server to allow for more refresh
capacity.
MINDSTREAM

Overview
Introducing MindStream Analytics
Why do Organizations need a well defined application
management structure?
Recent issues and fixes

Monitoring Recommendations
Questions

MINDSTREAM

General Infrastructure Monitoring

Recommendations

Disk space for all drives on all servers

CPU usage
Physical Memory usage
Virtual Memory usage
Server Uptime (we recommend at least monthly
server reboots if not completed during patching
period)

MINDSTREAM

Important terms as they relate to Essbase

Monitoring
Percentage of maximum blocks existing:
Compares, as a percentage, the number of blocks
that exist and the number of potential blocks. The
percentage is a measure of the sparsity of the
database. It is not uncommon for this number to be
very small (less than 1%) but could potentially
indicate a need for tuning.
Compression ratio: Indicates the compression
efficiency of blocks stored on disk, usually from using
bitmap compression. #MISSING cells are typically
compressed with this method. Therefore, the
compression ratio usually tracks block density.
MINDSTREAM

Important terms
Average clustering ratio: Indicates the
fragmentation level of data (.pag) files. The maximum
value, 1, indicates no fragmentation. If you are
experiencing degraded retrieval, calculation, or data
load performance and the clustering ratio value is
significantly less than 1, consider performing an
export and reload to rewrite the data files. Essbase
also rewrites the data files during outline restructures
that are triggered by changes to dense dimensions.
Rewriting the data files defragments them, resulting in
a clustering ratio closer to 1.

MINDSTREAM

Important terms:
Hit ratio on index cache: Indicates the success rate
of the Essbase Server kernel in locating index
information in the index cache without having to
retrieve another index page from disk. The cache hit
ratio indicates the percentage of time that a requested
piece of information is already in the cache. A higher
hit ratio indicates that the data is in the cache more
often. This improves performance, because the
requested data need not be retrieved from disk for the
next process. A hit ratio of 1.0 indicates that every
time data is requested, it is found in the cache. This is
the maximum performance possible from a cache
setting.
MINDSTREAM

Important terms:
Hit ratio on data cache: Indicates the success rate
of the Essbase Server kernel in locating data blocks
in the data cache without having to retrieve the blocks
from disk. The Hit Ratio on Data Cache setting
indicates the Essbase success rate in locating data
blocks in the data cache without having to retrieve the
block from the data file cache.

MINDSTREAM

Essbase Monitoring

Number of existing blocks

Block size (B)
Potential number of blocks
Existing level 0 blocks
Existing upper-level blocks
Block density (%)
% of maximum blocks existing
Compression ratio
Average clustering ratio
Run-times of calculations
Hit ratio on index cache
Hit ratio on data cache

Database Size (GB)

160
120
80
40
0

Number of existing
blocks
10500000
10000000
9500000
9000000
8500000

MINDSTREAM

HFM Stat General Recommendations

Consolidation Time (Specify Entity)

CPU Usage @ point in time
Number of Users on the system @ point in time
Physical Memory used
Max Database Connections
Data Audit report size
Track all dimension sizes over time

MINDSTREAM

HFM Stat General Recommendations

HSVEventLog.log and hfm.odl.log Size on Application
Servers
Potential Subcubes
Potential Records
Task Audit activities by date range FreeLRU's - Too
many messages means max cache is not sizing
correctly

MINDSTREAM

Accelatis for Application and

Infrastructure optimization for
Hyperion
What to Track
User Activity
System Events
Activity and Event Patterns
Cause-And-Effect Relationships

Different Perspectives
Local User Response
Remote User Response
System Response
Component Connectivity
Network Connectivity
Benchmark Task Results

MINDSTREAM

36

Accelatis continued:

Modules include:
Data and Application Montoring
and optimization.
Component Health for services,
network and servers.
Fully customizable KPIs that
allow you to set thresholds and
alerts on any part of the
environment.
Log monitoring/alerting/parsing to
identify patterns.

MINDSTREAM

37

Scott Ritter
Director of Managed Services

MindStream Analytics
617.848.4542 office
617.694.4548 cell
mailto:[email protected] m
http://www.mindstreamanalytics.com

MindStream Webinar Series Presents:

Think outside the box - New ideas in Financial Data Management
March 1, 2013 Noon 1pm eastern
MINDSTREAM

39

Questions

MINDSTREAM