Weblogic authentication denied – WLS 10

Problem description

The Weblogic admin server and / or managed server(s) are unable to start properly and throwing an authentication denied error message.

Weblogic errors observed

Error #1

<Jul 30, 2011 3:13:18 AM PST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed.

Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid;

The user name and/or password from the boot identity file (boot.properties) is not valid.

The boot identity may havenbeen changed since the boot identity file was created. Please edit and update

the boot identity file with the proper values of username and password. The first time the updated boot identity file

is used to start the server, these new values are encrypted.

weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid;

The user name and/or password from the boot identity file (boot.properties) is not valid.

The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file

with the proper values of username and password. The first time the updated boot identity file is used to start the server,

these new values are encrypted.

at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(Unknown Source)

at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(Unknown Source)

at weblogic.security.service.SecurityServiceManager.initialize(Unknown Source)

at weblogic.security.SecurityService.start(SecurityService.java:141)

at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

Truncated. see log file for complete stacktrace

Error #2

<Jul 30, 2011 5:11:55 AM PST> <Critical> <Security> <BEA-090403> <Authentication for user <user> denied>

<Jul 30, 2011 5:11:55 AM PST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed.

Reason: weblogic.security.SecurityInitializationException: Authentication for user <user> denied

weblogic.security.SecurityInitializationException: Authentication for user <user> denied

at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(Unknown Source)

at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(Unknown Source)

at weblogic.security.service.SecurityServiceManager.initialize(Unknown Source)

at weblogic.security.SecurityService.start(SecurityService.java:141)

at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

Truncated. see log file for complete stacktrace

Possible root causes and solutions

Root cause #1

The Weblogic boot.properties file is corrupted or contains invalid principal and credentials

Solution >> boot.properties reset

·         Backup and clear the cache and data directories under <WL Domain>/servers/<Admin & Managed server>

·         Recreate boot.properties (put back your plain text username and password) under <WL Domain>/servers/<Admin & Managed server>/security directory and restart the affected server(s)

Root cause #2

The Weblogic boot.properties file is valid but the security realm is corrupted or in an invalid state

Solution >> Weblogic Admin username and password reset

·         Backup your Weblogic server domain

·         Rename or delete <WL Domain>/security/DefaultAuthenticatorInit.ldift

·         Run the following Java command:

        java weblogic.security.utils.AdminAccount <new-admin-user-name> <new-admin-user-pwd> <<WL Domain>/security >

·         Delete the contents inside the file boot.properties under <WL Domain>/servers/< AdminServer>/security

·         Add the following contents inside the boot.properties

        username=<new-admin-user-name>

        password=<new-admin-user-pwd>

·         Backup and delete the folder: <WL Domain>/servers/<AdminServer>/data/ldap

·          Restart your Weblogic server 

Read More

Java DNS Cache Reference Guide

Development of complex distributed Java EE systems quite often involves integration with multiple downstream systems. Such system business service(s) can be exposed via HTTP or other protocols, Internet facing or secured within its own private network zone. The most common approach is to centralize the platform access via a front door DNS (Domain Name System) name. When supporting a Java EE production system, it is important to understand the JDK DNS management; especially its default caching policy.

This article will provide you an overview and comparison matrix of the DNS cache policy between JDK 1.4, 1.5 and 1.6 and how you can override the default behaviour when necessary.

Default DNS cache policy and DNS spoofing attack

The default JDK DNS cache policy TTL (time to live) value is -1 (caching forever). You may wonder why as this can cause some problems when network DNS re-pointing changes are required, forcing any Java client to shutdown and restart its JVM / Java EE server.

The main reason for this default behaviour is security. As mentioned in Sun documentation, no DNS caching or any positive value below 30 seconds could expose your environment to DNS spoofing attack; especially for Internet facing Java EE environments vs. applications deployed within a secured and private network zone.

A DNS spoofing attack is an attempt by an attacker to fool a DNS server and re-point a specific DNS entry to a different IP (hacker IP). The DNS server then remains “poisoned” until it refreshes its cache. This means all Java InetSocketAddress DNS lookup requests to the effected DNS Server during that time period will also be “poisoned“ and return an unexpected / hacked IP address.

Now find below the default DNS cache behavior between the different JDK versions and override methods.

DNS cache override and JDK comparison matrix

	JDK 1.4 & 1 .5	JDK 1.6, 1.7 & 1.8
Default value	-1 (caching forever) *JVM restart required to flush the DNS cache	30 secs (When a security manager is not set) -1           (When a security manager is set) * DNS Cache is refreshed every 30 seconds
Editable	yes	yes
Default Value Printing	System.out.println("DEFAULT DNS TTL: "+sun.net.InetAddressCachePolicy.get());	System.out.println("DEFAULT DNS TTL: "+sun.net.InetAddressCachePolicy.get());
Override Option #1	<JDK_HOME>/jre/lib/security/java.security #networkaddress.cache.ttl=-1 * Uncomment the above parameter and change as per your desired positive value in seconds	<JDK_HOME>/jre/lib/security/java.security #networkaddress.cache.ttl=-1 * Uncomment the above parameter and change as per your desired positive value in seconds
Override Option #2	* Execute the code below on JVM start-up java.security.Security.setProperty ("networkaddress.cache.ttl" , TTL_SECS); // TTL_SECS represents your configured TTL value	* Execute the code below on JVM start-up java.security.Security.setProperty ("networkaddress.cache.ttl" , TTL_SECS); // TTL_SECS represents your configured TTL value
Java reference classes	sun.net.InetAddressCachePolicy	sun.net.InetAddressCachePolicy

Read More

SecurityInitializationException when starting Weblogic Integration 9.2

This case study describes the root cause analysis and resolution of an authentication problem (BEA-000386) with Weblogic Integration Admin server.

Environment specifications

·         Java EE server: Oracle Weblogic Integration 9.2 MP2 (WLI)

·         JDK: Java version: 1.5.0 SR6, IBM VM

·         RDBMS: Oracle Database 10g Enterprise Edition Release 10.2.0.2.0 - 64bit Production

·         Platform type: Middle tier  - Order Provisioning System

Problem description

weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied observed at start-up of the WLI Admin server.

<Jan 21, 2011 7:07:41 AM EST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>

<Jan 21, 2011 7:07:41 AM EST> <Critical> <Security> <BEA-090403> <Authentication for user <WL user> denied>

<Jan 21, 2011 7:07:41 AM EST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied

weblogic.security.SecurityInitializationException: Authentication for user <WL user> denied

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:947)

        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1029)

        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:854)

        at weblogic.security.SecurityService.start(SecurityService.java:141)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        Truncated. see log file for complete stacktrace

Gathering and validation of facts

As usual, a Java EE problem investigation requires gathering of technical and non technical facts so we can either derived other facts and/or conclude on the root cause. Before applying a corrective measure, the facts below were verified in order to conclude on the root cause:

·         Recent change of the platform? Yes, deployment was performed few hours before the problem

·         Sudden crash of the Weblogic admin server and/or managed servers? No (important to determine any corruption type of event)

·         Any change done at the WLI database level? Yes, deployment process did include the creation of a new WLI domain and re-use of the existing WLI database schema

·         Any change of the affected application / WLI domain at file system level? No, files were only modified/created on the new WLI domain which is not sharing the same file system of the existing WLI domain

Root cause

Based on the above facts, we ruled out a local WLI domain security LDAP corruption problem and moved to investigation at the WLI schema / database level where other facts were gathered.

What we found is that the creation process of the new WLI domain, sharing the same WLI schema of the existing domain, removed one entry within USERS & GROUPMEMBERS tables.

Such data is mandatory and is verified on WLI server start-up as Weblogic is looking for a match between such users and group vs. the local security LDAP data.

Solution

Once the root cause was confirmed, we applied the following resolution procedure:

1) Manually insert the missing data within the USERS & GROUPMEMBERS WLI schema tables:

INSERT INTO USERS (U_NAME, U_PASSWORD, U_DESCRIPTION) values ('<WL user>', '<Encrypted Pwd>', 'This user is the default administrator.'); 

INSERT INTO GROUPMEMBERS (G_NAME, G_MEMBER) values ('Administrators', '<WL user>'); 

2) Restart the Admin server

3) Restart the managed servers

4) Log monitoring and successful deployment status did confirm the resolution

Recommendations and best practices

Lessons learned below were identified and documented to prevent such problem in the near future:

·         Take proper backup and monitor critical WLI schema and tables such as USERS & GROUPMEMBERS for any unexpected data update or removal

·         From an architecture perspective, consider the creation of a separate WLI schema for any new creation of WLI domain which would provide better flexibility and eliminate the risk of any data overlap

Read More