Oracle18c/19c & Centrally

Managed Users (CMU)

DB User Management Made Easy

Simon Pane
December 4, 2019
Simon Pane
Pythian Principal Consultant

• ~25 years Oracle experience

• Community Volunteer
• Oracle ACE
• Oracle Certified

© Pythian Services Inc., 2019 2

Conference and/or Webcast Speaker For

© Pythian Services Inc., 2019 3

 PYTHIAN
A global IT company that helps businesses leverage disruptive technologies to better compete.

Our services and software solutions unleash the power of cloud, data and analytics to drive better
business outcomes for our clients.

Our 20 years in data, commitment to hiring the best talent, and our deep technical and business expertise
allow us to meet our promise of using technology to deliver the best outcomes faster.

© Pythian Services Inc., 2019 4

 22 400+ 350+
Years in Experts in 35 Clients
Business Countries Globally

© Pythian Services Inc., 2019 5

 PYTHIAN TIMELINE

1997-2012 2013-2014 2015 2016 2017

Remote Database Cloud emerges, Expanded Open Competencies 11,000 database

Management DevOps practice Source– grow with Cloud systems under
Services–Oracle, established databases partners–Data, Pythian
Microsoft SQL Cassandra, Machine Learning, management
Server, MySQL Hadoop practice MongoDB Migrations,
established DevOps Analytics as a
Cloud partnerships Service launches
with Google, AWS,
First Cloud Microsoft
Managed Service Completed one of
the world’s most
Analytics practice complex Cloud
established Migrations

© Pythian Services Inc., 2019 6

Quick Definitions

• CMU support both “Directory Server Authentication” and “Single Sign-on”

https://en.wikipedia.org/wiki/Single_sign-on © Pythian Services Inc., 2019 7

What is Achievable

© Pythian Services Inc., 2019 8

AGENDA

• Overview of CMU
• One-time Active Directory Config
• One-time RDBMS Home Config
• User and Role Mapping and Testing
• Troubleshooting & Common Issues Reference

© Pythian Services Inc., 2019 9

Background & How it Works
Oracle Possibilities with Directory Services
1. Federate OCI with an IdP
2. Centralise Net Naming Services in AD, OID, or any LDAP compliant directory
3. User management through Enterprise User Security (EUS) and OUD HARD !!!

4. NEW: Oracle Database 18c authentication and authorisation for multiple 18c+
databases within Microsoft Active Directory

• No additional licenses required

• No additional software tiers to add
• Compatible with 11g and 12c clients

© Pythian Services Inc., 2019 11

Some Foundational Basics …
• “Active Directory” (AD) is Microsoft’s customised LDAP Directory Service
• Supports many common LDAP features and tools
• Is based on the concept of an AD “schema” which holds properties of objects

• Runs on one or more “Domain Controllers” (DCs)

• Other services such as DNS often run on the same DCs

• Minimum version for CMU is Microsoft Windows Server 2008 R2

© Pythian Services Inc., 2019 12

 The Difference is Profound
Oracle Directory
Services

EUS / OUD and

syncing of AD
users/groups

VERY COMPLEX

Database 18c
talks directly
to AD

© Pythian Services Inc., 2019 13

Why Do We Want To Do This?
• Centralise (some) DB user management
• If organisationally using Active Directory, then users are almost certainly
added/maintained there anyway
• Removes user account and user password layer from the database
• Can leverage Active Directory security groups – map to database roles/privileges
• Reduced DBA administration workload
• With shared DB schemas, no onboarding or offboarding at the DB level

• What’s not included

• Integration with any other LDAP directory service – only Active Directory currently
© Pythian Services Inc., 2019 14
Similar to SQL Server Integrated Logons

Now we really have the same options with Oracle Database 18c+

© Pythian Services Inc., 2019 15

“… IDENTIFIED GLOBALLY AS …”

Exclusive User

USER

Shared Schema
USER

Global Role
ROLE

NOTE: Oracle User/Role names don’t need to match AD User/Group names © Pythian Services Inc., 2019 16
Logical Connection Flow

Found AD DN
USER

No Match – LDAP Query AD

dsi.ora/ldap.ora
& wallet Send Credential to DB

© Pythian Services Inc., 2019 17

Authentication and Authorisation Options
• Oracle Database 18c provides several AD authentication options:
1. Password Compatibility

2. Kerberos Recommended

3. SSL Certificate (PKI)

• Oracle Database 18c provides several AD authorisation options:

• Normal Oracle Database built-in technologies (roles, privileges, etc.)
• Active Directory Security Groups

© Pythian Services Inc., 2019 18

To Put it Simply
• Using the “Password” configuration option: PASSWORD
• Database connections still require credentials (username & password)
• Password is validated against Active Directory instead of the database
• Essentially “re-prompting” – compromised desktop != DB access

• Using the “Kerberos” configuration option: KERBEROS

• Active Directory issues Kerberos “tickets” (TGT)
• Tickets are used for authentication – no credential (no username or password)
required for DB connections

© Pythian Services Inc., 2019 19

 PASSWORD
Summary of Implementation Steps
!?!?!?
1. Extend the AD Schema and install the Oracle “Password Verifiers”

2. Create an “Oracle Service Directory User” (for DB <-> AD communication)

3. Configure the RDBMS home to integrate with AD via the Service Directory User

and the AD’s “Public Certificate”

4. Create “... IDENTIFIED GLOBALLY ...” database users and/or roles

© Pythian Services Inc., 2019 20

 KERBEROS
Summary of Implementation Steps

1. Create a “service principal” for the DB server in Active Directory

2. Extract the “key table” for the “service principal” and copy to the DB server

3. Configure Kerberos settings and SQLNET.ORA on DB server

4. Create “... IDENTIFIED EXTERNALLY ...” database users and/or groups

5. Configure client Kerberos settings and SQLNET.ORA

!?!?!?
© Pythian Services Inc., 2019 21
Explaining “Password” authentication
going forward as it seems to usually
be the most applicable

But paper explaining “Kerberos”

setup is available upon request
© Pythian Services Inc., 2019 22
Active Directory
Implementation Steps
 This might seem a little
complicated at first but
really is not. And is only a
one-time setup!

© Pythian Services Inc., 2019 24

Test Environment Summary – OCI Based
• Oracle Linux 7.7 database server with Oracle 18c XE RDBMS home:
• Using default locations for certain files such as dsi.ora and Oracle Wallet

• XE means a CDB database with one pluggable database XEPDB1:

• CMU works fine with PDB or non-CDB database

• One Windows 2016 Standard Edition Domain Controller (DC)

© Pythian Services Inc., 2019 25

Prerequisites
• An Active Directory (AD) forest and domain controller (DC)
• Administrative access to the DC – AD schema will be extended
• Easy to setup your own PoC / test lab using a cloud environment (OCI):
• Provision new Windows 2016 Server (Standard edition on VM will suffice)
• Install and configure Active Directory Domain Services Easy to follow
step-by-step
• Install and configure Active Directory Certificate Services instructions

• Step-by-step blog series (for DBAs to implement) coming out soon.

Easy to setup a complete test environment in OCI or on-prem VM

© Pythian Services Inc., 2019 26

Overview of the Active Directory Setup
1. Create an Oracle “Service Directory User” (in AD)
● Is the credential that the database software will use to interact with (query) AD

2. Install the Oracle “Password Filter” into each Domain Controller (DC)
● Will allow AD to capture a password hash compatible with Oracle queries

3. Extract the DC Public Certificate for the Oracle DB to connect with

4. Create AD groups (and optionally new users)
5. Configure the Database users based on the AD “Distinguished Name” (DN)

© Pythian Services Inc., 2019 27

Creating the Oracle Service Directory User
• An AD user that the Oracle Database software will use for AD interaction
• Sample Windows PowerShell script:

A Personal Preference:
• Alternatively use “Users”
• This path is referenced in later commands © Pythian Services Inc., 2019 28
The Oracle Service Directory User

© Pythian Services Inc., 2019 29

Permissions for the Oracle Service Directory User
• Not very clear in the official documentation
• Actual AD implementation steps:
All tasks -> Delegate Control
Select the Oracle Services Directory User
Choose the "Create a custom task to delegate" radio box
Select the "Only the following objects in the folder" radio box, then the "User objects" check-box
Choose both the "General" and "Property specific" check-boxes
Select the "Read" and "Write lockout Time" permissions.

• Or from Windows PowerShell:

© Pythian Services Inc., 2019 30

Copy the Password Filter Installer to the DC
Remember:
• Extends the Active Directory Schema: Not required
with
• Adds the “orclCommonAttribute” for user accounts Kerberos!
• Creates three new AD groups that will use the password filter
• Must install on every DC (reboot required)
• Copy the ${ORACLE_HOME}/bin/opwdintg.exe file from an RDBMS home
• Must be an Oracle18c+ home
• Can be copied from a Linux home (same endian)

© Pythian Services Inc., 2019 31

Install the Password Filter into AD
• IMPORTANT: a Domain Controller reboot is required!

© Pythian Services Inc., 2019 32

The Result: New AD Groups
• Three new AD groups for the Oracle Database 11g password verifier, 12c
password verifier, and WebDAV client:

© Pythian Services Inc., 2019 33

 A Quick Test/Verification
• LDAP utilities have been in RDMS homes for many releases

To test two-way
authentication using
certificate

Location of Oracle Wallet containing cert

© Pythian Services Inc., 2019 34
Export the Server’s Public Certificate
• From the GUI or a PowerShell cmdlet:

• Recommend the above command over the certsrv.msc (GUI) for reliability
• Manually copy the exported public certificate to the database server

© Pythian Services Inc., 2019 35

Database Home
Configuration
Specifying the Active Directory Servers
• List AD servers in a dsi.ora file (use of an ldap.ora is not recommended)

Can use hostname or FQDN

and list multiple DCs

Optional

© Pythian Services Inc., 2019 37

Create a Wallet File
• To hold the “Service Directory User’s” credential and the certificate

© Pythian Services Inc., 2019 38

Database Configuration
• Initialisation Parameter adjustments:

• Instance restart required for the optional ldap_directory_sysauth change

© Pythian Services Inc., 2019 39
Database User / Role
Configuration
Database Catalog Differences
• Normal database authenticated users – DB stored credentials:

• New Active Directory authenticated users – AD stored credentials:

Active Directory “Distinguished Name” © Pythian Services Inc., 2019 41

Recommended AD Query Tool
• AD Explorer - Windows Sysinternals
• Single executable utility
• Useful for obtaining the user’s “Distinguished Name” and checking the “orclCommonAttribute”

© Pythian Services Inc., 2019 42

Command Line Alternatives
• Command shell example:
dsquery user -name simon -o dn

• PowerShell example:
Get-ADUser -Identity "simon" -properties DistinguishedName,orclCommonAttribute

© Pythian Services Inc., 2019 43

Create Users and Roles
• Use “... INDENTIFIED GLOBALLY AS ...”
• Obtain the “Distinguished Names” from Active Directory

• Existing database users can also be migrated via “ALTER USER ... IDENTIFIED
GLOBALLY AS ...”
• Administrative users and connections are also supported

© Pythian Services Inc., 2019 44

Connection Options
• Can perform the database connection using:
1. Using the “down-level logon name” (or “SAMAccountName”, “pre-Windows 2000 logon
name”) : DOMAIN\User
2. Using the “User Principal Name” (or “UPN”): User@Domain
3. Just using the “User Login Name” : User

• Local BEQ and TNS connections supported

• Examples:
SQL> connect "STAGECOACH\simon"@ORCL
SQL> connect "[email protected]"@ORCL
SQL> connect simon
© Pythian Services Inc., 2019 45
Other Suggestions
• Use a good nomenclature to make AD users/groups easily identifiable:

What appears in v$session and other views

© Pythian Services Inc., 2019 46

Must Change AD Password After Creation
• Because the AD “Password Verifier” groups are assigned after creation
• Need to be part of the verifier groups to store hash in orclCommonAttribute
• Users usually have to change their password on first (Windows) login anyway

Reminder:
Not required
with
Kerberos!

© Pythian Services Inc., 2019 47

Example Shared Schema Configuration

© Pythian Services Inc., 2019 48

The Typical Session Properties
• Only shows the Shared Schema details:

© Pythian Services Inc., 2019 49

Authentication and Identity Properties
• Does show all of the pertinent information:

© Pythian Services Inc., 2019 50

A Simple Auditing Test
• Audit create session and connect using a Shared Schema:

© Pythian Services Inc., 2019 51

Audit Records

© Pythian Services Inc., 2019 52

Both Exclusive and Shared Schema Matches?
• Connects as the Exclusive User over the Shared Schema

© Pythian Services Inc., 2019 53

Member of Multiple AD Groups? TIP: Don’t Do!

• Only connects as the one Shared Schema based on lowest USER_ID

© Pythian Services Inc., 2019 54
Global Roles
• Create a Database Role that maps to an AD group:
CREATE ROLE ad_sales_role IDENTIFIED GLOBALLY AS
'CN=DB_PRD01_Sales_Group,CN=Users,DC=STAGECOACH,DC=NET';

© Pythian Services Inc., 2019 55

Global Role Membership
• Can’t grant in the DB – membership assigned through AD group:

• Effectively grant through AD Group membership

© Pythian Services Inc., 2019 56

Global Roles – Activated When Connected
• After connecting:

© Pythian Services Inc., 2019 57

Issues and
Troubleshooting
What does ORA-01017 Actually Mean?
• Error ORA-01017 is commonly returned due to a wide variety of causes

• Really means: could not validate that the credential is valid:

• Bad Password
• DC unreachable (due to setup, networking, routing, permissions, or server down)
© Pythian Services Inc., 2019 59
First Check the Obvious: Verify the Password
• Test AD user password:

• AD user may have:

• Expired password
The Oracle Database
• Locked account due to failed login attempts honors these

© Pythian Services Inc., 2019 60

Verify the Connection: Test Using an LDAP Query

• Check firewalls
• At the network level, the DB server level, and Domain Controller level
• ICMP (ping) tests
© Pythian Services Inc., 2019 61
Firewall Rules – Common LDAP Ports Required

© Pythian Services Inc., 2019 62

Active Directory Policies and Passwords
• Oracle DB prevents connections when the AD status is:
• “password expired”
• “password must change”
• “account locked out”
• “account disabled”
• Remember to change the AD password after adding the user to the Oracle
password verifier group(s) in AD

© Pythian Services Inc., 2019 63

Connection Tracing
• Additional details can be obtained using tracing:
alter system set events='trace[gdsi] disk low';
off
• Then review the resulting trace file in the ADR:
[28994890]kzlg discovered server type: AD
[28994890]kzlg AD user name: STAGECOACH\simon
[28994890]kzlg found dn in wallet
[28994890]kzlg found pwd in wallet
[28994890]kzlg found usr in wallet
[28994890]kzlg discovered ldaptype: AD
[28994890]kzlg ldap_open 10.0.0.12:636
[28994890]kzlg DB-LDAP init SSL succeeded.
...

© Pythian Services Inc., 2019 64

Lacking Critical Detail in Oracle Return Codes

• Within Active Directory (and associated trace file messages)

• “User must change password at next logon”: kzlg polerr=28223
• “Account disabled”: kzlg polerr=28052
• “Password incorrect”: kzlg polerr=0 ; KZLG_ERR: LDAPERR=49, OER=28043
• Cannot contact AD DC: KZLG_ERR: 28030 from kzlgOpenBind
© Pythian Services Inc., 2019 65
User “locked” in Active Directory

• Usually associated with new AD accounts that have never logged into Windows
• Unlock within AD

© Pythian Services Inc., 2019 66

User “locked” in Active Directory
• Is a group policy causing them to lock too easily?

© Pythian Services Inc., 2019 67

Difficult Problem: LDAP Bind Errors
• SYMPTOM / ERROR from SQLPlus:
• ORA-01017: invalid username/password; logon denied

• SYMPTOM / ERROR from alert log:

• ORA-28043: invalid bind credentials for DB-OID connection

• SYMPTOM / ERROR from trace file:

• KZLG_ERR: failed to sasl bind to LDAP server. err=49

© Pythian Services Inc., 2019 68

Trace File from Network Issue

© Pythian Services Inc., 2019 69

Two Possible Causes
1. Check that the Oracle Directory User’s credentials in the wallet are valid:
orapki wallet display -wallet .
mkstore -wrl . -viewEntry ORACLE.SECURITY.DN
mkstore -wrl . -viewEntry ORACLE.SECURITY.USERNAME
mkstore -wrl . -viewEntry ORACLE.SECURITY.PASSWORD

2. Networking resolution / firewall / routing

● Check resolution in DNS server or in local /etc/hosts file as a workaround if
needed

© Pythian Services Inc., 2019 70

 Networking Solution
• On Domain controller determine the internal (private) IP, hostname, and FQDN. From
Windows Command Prompt:
hostname
hostname | nslookup

• On DB Server ensure that LDAP port 636 can be reached for the IP, hostname, and
FQDN (output from all three above):
(echo > /dev/tcp/10.0.0.12/636) >/dev/null 2>&1 && echo "OPEN" || echo "CLOSED"

(echo > /dev/tcp/DC2/636) >/dev/null 2>&1 && echo "OPEN" || echo "CLOSED"

(echo > /dev/tcp/DC2.STAGECOACH.net/636) >/dev/null 2>&1 && echo "OPEN" || echo "CLOSED"
Testing:
1) Private network IP
2) Hostname
3) FQDN © Pythian Services Inc., 2019 71
Authorisation is Still Database Based

• Still need to setup grants, roles, etc within the database via a normal role, global
role or direct grant
• Granting to either the Exclusive User or Shared Schema

© Pythian Services Inc., 2019 72

A Few More Experienced Errors

ORA-12638: Credential retrieval failed

ORA-12641: Authentication service failed to initialize

• Usually related to SQLNET.ORA, specifically SQLNET.AUTHENTICATION_SERVICES

• ORA-12638 is really a “catch-all” error – SQLNET tracing might be required

© Pythian Services Inc., 2019 73

One Final Error / Solution

ORA-12638: ORA-28276: Invalid ORACLE password attribute

• No shadow password in orclCommonAttribute in Active Directory

• Change AD password to create shadow hash

• Ensure user is part of ORA_VFR_... AD Security Group

© Pythian Services Inc., 2019 74

WRAP UP!

© Pythian Services Inc., 2019 75

Summary…

• CMU finally means authorisation and authentication can finally be easily

offloaded to Microsoft Active Directory:
• If using AD organisationally, new users will need to be in AD anyway

• Some initial one-time setup is required:

• AD schema needs to be extended & password filter installed (for “password” option)
• RDBMS home requires dsi.ora, Oracle Wallet, and initialisation parameters
• Less AD setup required for “Kerberos” based authentication

• Actual Database user and role management is easy

© Pythian Services Inc., 2019 76

Interesting: Even Included with Amazon RDS

• Not personally tested

• Reference: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/oracle-kerberos.html
© Pythian Services Inc., 2019 77
MOS Notes
• How To Configure Authentication For The Centrally Managed Users In An 18c Database
(Doc ID 2462012.1)
• Tracing CMU connection issues (Doc ID 2470608.1)
• 18c Active Directory Password Authentication Fails With ORA-28276 for Client
Connections Below 12c (Doc ID 2472256.1)
• How To Configure Kerberos Authentication In A 12c Database (Doc ID 1996329.1)
• Configuring ASO Kerberos Authentication with a Microsoft Windows 2008 R2 Active
Directory KDC (Doc ID 1304004.1)
• Kerberos Troubleshooting Guide (Doc ID 185897.1)
• Bug 28994890 : CMU-AD: CUMULATIVE FIXES FOR DATABASE 18C

© Pythian Services Inc., 2019 78

THANK YOU
http://bit.ly/OraCMU-UKOUG19

[email protected]

© Pythian Services Inc., 2019 80