Dell ObjectScale 1.

3 Administration Guide

September 2023
Rev. 1.0
Notes, cautions, and warnings

NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid
the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

© 2023 Dell Inc. or its subsidiaries. All rights reserved. Dell Technologies, Dell, and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners.
 Contents
Revision history......................................................................................................................................................................... 11
Document feedback........................................................................................................................................................... 11
About using this guide............................................................................................................................................................ 12

Chapter 1: Overview.....................................................................................................................13
About Dell ObjectScale .................................................................................................................................................... 13
Product Description...........................................................................................................................................................13
Basic Terminology.............................................................................................................................................................. 14
ObjectScale User Interfaces........................................................................................................................................... 15
ObjectScale Portal for Dell ObjectScale................................................................................................................. 16
Grafana Dashboards.................................................................................................................................................... 16
ObjectScale and Kubernetes........................................................................................................................................... 16
ObjectScale Storage Classes.....................................................................................................................................17
ObjectScale components........................................................................................................................................... 18
Data protection with ObjectScale Erasure Coding schemes...................................................................................19
Customer Feedback..........................................................................................................................................................20
Provide general feedback..........................................................................................................................................20

Chapter 2: Getting Started with ObjectScale............................................................................... 21

Accessing the ObjectScale Portal user interface.......................................................................................................21
Access the ObjectScale Portal user interface ..................................................................................................... 21
Using the ObjectScale Portal user interface.............................................................................................................. 22
View Dashboard........................................................................................................................................................... 22
ObjectScale Portal user interface........................................................................................................................... 23
ObjectScale Portal Menu ..........................................................................................................................................23
Link ObjectScale Appliance Hardware with Software License.........................................................................24
ObjectScale Performance Dashboard ................................................................................................................... 24
Administration.............................................................................................................................................................. 25
Security configuration......................................................................................................................................................26
Role-based security.......................................................................................................................................................... 26

Chapter 3: IAM accounts and account entities ............................................................................ 27

Introduction to Identity and Access Management.................................................................................................... 27
Accounts..............................................................................................................................................................................28
Creating and mainitaining ObjectScale accounts...................................................................................................... 28
New Accounts.............................................................................................................................................................. 28
View the summary of an account............................................................................................................................29
Edit Account................................................................................................................................................................. 29
Enable or Disable an Account................................................................................................................................... 30
Delete an account....................................................................................................................................................... 30
Configuring ObjectScale account entities...................................................................................................................30
Users...............................................................................................................................................................................30
Groups............................................................................................................................................................................33
Roles............................................................................................................................................................................... 34
Policies........................................................................................................................................................................... 38

Contents 3
 Identity Provider.......................................................................................................................................................... 59
Root Access Keys........................................................................................................................................................ 61
Notification Destinations........................................................................................................................................... 62

Chapter 4: Object stores............................................................................................................. 64

About ObjectScale object stores...................................................................................................................................64
Object store naming conventions............................................................................................................................65
Creating and maintaining object stores....................................................................................................................... 66
Creating a new object store..................................................................................................................................... 66
Edit the settings of an object store........................................................................................................................ 69
Delete an object store................................................................................................................................................ 69
Managing object stores................................................................................................................................................... 70
View the Summary of an object store.................................................................................................................... 70
View the Dashboard of an Object Store................................................................................................................ 70
Understand object store space reclamation......................................................................................................... 72
Set capacity alerts for an object store...................................................................................................................72
Managing accounts associated with object stores............................................................................................. 73
View the certificates for an object store............................................................................................................... 74
Monitor and manage replication for an object store...........................................................................................74
View the health of an object store..........................................................................................................................76
Metrics........................................................................................................................................................................... 76

Chapter 5: Buckets......................................................................................................................77
About ObjectScale buckets.............................................................................................................................................77
Bucket and object naming conventions................................................................................................................. 77
Bucket versioning and Object Lock.........................................................................................................................77
About ObjectScale Bucket Logging........................................................................................................................ 79
Creating and managing buckets using ObjectScale..................................................................................................82
Create a bucket........................................................................................................................................................... 82
Edit a bucket.................................................................................................................................................................84
View the summary of a bucket................................................................................................................................ 85
Delete a bucket............................................................................................................................................................ 86
Configure Bucket Logging.........................................................................................................................................86
About bucket policies................................................................................................................................................. 87
Setting up bucket event notifications.................................................................................................................... 94
Configuring bucket entities........................................................................................................................................... 102
View the Bucket Summary...................................................................................................................................... 102
Managing Bucket Replication................................................................................................................................. 103

Chapter 6: Federate ObjectScale Systems................................................................................. 104

Federating ObjectScale Systems.................................................................................................................................104
Create a federation of ObjectScale systems............................................................................................................ 105
Add additional ObjectScale instances to an existing ObjectScale federation ..................................................107

Chapter 7: ObjectScale Replication............................................................................................ 110

Introduction to ObjectScale Replication..................................................................................................................... 110
ObjectScale Replication overview and configuration details............................................................................ 111
Required IAM permissions ....................................................................................................................................... 112
Bucket Replication Policy............................................................................................................................................... 113

4 Contents
 Configuring replication rules using the ObjectScale Portal UI......................................................................... 113
Configure replication using the S3 API..................................................................................................................114
Replication Rules.........................................................................................................................................................115
Bucket Replication to multiple destinations............................................................................................................... 115
Delete marker replication on versioning-enabled buckets .....................................................................................117
Manage a Bucket Replication Policy using ObjectScale UI.................................................................................... 117
Configure a new bucket replication rule................................................................................................................117
Edit an existing bucket replication rule................................................................................................................. 119
Delete a bucket replication rule..............................................................................................................................120
Working with bucket replication rules...................................................................................................................120
Change the priority of bucket replication rules................................................................................................... 121
Check replication rule statuses ............................................................................................................................. 122
Configure the destination bucket to receive objects........................................................................................122
Set up ObjectScale Replication using the ObjectScale API...................................................................................124
Create and configure an account and an IAM role............................................................................................ 124
Set up the ObjectScale to ObjectScale Replication.......................................................................................... 126
Replication status............................................................................................................................................................ 127
Monitor for failed replications on the ObjectScale Portal UI.......................................................................... 128
Get replication failure reasons, failed destinations, and remediation hints ................................................ 128

Chapter 8: Platform settings .................................................................................................... 130

ObjectScale hardening overview................................................................................................................................. 130
List of protected actions .............................................................................................................................................. 130
Hardening with Federation............................................................................................................................................ 132
Privileged actions approval system (PAAS)..............................................................................................................132
Enable PAAS............................................................................................................................................................... 133
Manage Approver Users...........................................................................................................................................135
Create and manage requests.................................................................................................................................. 137
Platform protection mode............................................................................................................................................. 140
Enable platform protection mode.......................................................................................................................... 140
Disable platform protection mode.......................................................................................................................... 141
Escalated request for operating system access in Platform Protection Mode........................................... 141
Account Protection Mode.............................................................................................................................................. 141
Enable Account Protection Mode on an account.............................................................................................. 142
Disable account protection on an account.......................................................................................................... 142
Create account requests ........................................................................................................................................ 143
Complete approved S3 requests on protected accounts................................................................................ 143

Chapter 9: Management Users and Roles in ObjectScale ........................................................... 145

Management Users in ObjectScale Software Bundle............................................................................................. 145
Roles for Management Users................................................................................................................................. 146
Default user ................................................................................................................................................................ 147
Password management for local users................................................................................................................. 147
Managing Local Management Users .................................................................................................................... 148
Management User audit logs and alerts............................................................................................................... 151
Approver users................................................................................................................................................................. 154

Chapter 10: Authentication Providers in ObjectScale Software Bundle.......................................155

Configuring external authentication providers......................................................................................................... 155

Contents 5
 Create an external authentication provider........................................................................................................ 155
Manage external authentication providers.......................................................................................................... 158
Map ObjectScale roles to external users............................................................................................................. 158
Manage role mappings..............................................................................................................................................159

Chapter 11: ObjectScale Administration..................................................................................... 160

About the Administration section of the ObjectScale Portal................................................................................160
Installing and maintaining the health of ObjectScale...............................................................................................161
Licensing ObjectScale..................................................................................................................................................... 161
About ObjectScale Licensing................................................................................................................................... 161
Manage ObjectScale licenses................................................................................................................................. 163
System support................................................................................................................................................................ 163
About SupportAssist................................................................................................................................................. 164
Monitoring system health........................................................................................................................................ 168
SAML Service Provider Metadata............................................................................................................................... 168
Generate SAML Service Provider Metadata.......................................................................................................168
Manage ObjectScale Certificates................................................................................................................................ 168
Security settings .............................................................................................................................................................169
Set password complexity rules...............................................................................................................................169
Set lockout rules........................................................................................................................................................ 170
Active sessions................................................................................................................................................................. 170
View active sessions.................................................................................................................................................. 171
Cancel active sessions...............................................................................................................................................171
ObjectScale Upgrades..................................................................................................................................................... 171
ObjectScale 1.3 upgrade considerations...............................................................................................................172
Upgrade ObjectScale on OpenShift...................................................................................................................... 172
Upgrade ObjectScale on Software Bundle.......................................................................................................... 176
Upgrade ObjectScale on Appliance........................................................................................................................177
Troubleshooting Object store upgrade status in progress after failed node repair.......................... 186

Chapter 12: ObjectScale Management REST API........................................................................ 187

ObjectScale Management REST API introduction...................................................................................................187
ObjectScale Management REST API summary........................................................................................................ 187
Authenticate with the ObjectScale Management REST API................................................................................ 189
Download and set up CURL.................................................................................................................................... 190
Log in and obtain the Access Token for the ObjectScale-level APIs............................................................190
Log in and obtain the Access Token for the object store-level APIs.............................................................191
ObjectScale Management Service.........................................................................................................................193
Logout...........................................................................................................................................................................194

Chapter 13: Accessing data with IAM and S3 ............................................................................. 195

ObjectScale IAM overview............................................................................................................................................ 195
IAM Account Management......................................................................................................................................195
Access Management.................................................................................................................................................201
Security Token Service............................................................................................................................................205
IAM SAML support................................................................................................................................................... 207
IAM Resource ARNs...................................................................................................................................................211
Amazon S3 API support in ObjectScale...................................................................................................................... 211
S3 support in ObjectScale........................................................................................................................................ 211

6 Contents
 Multipart upload (MPU) support for huge objects ...........................................................................................212
ObjectScale S3 error codes.....................................................................................................................................213
Authenticating with the S3 service...................................................................................................................... 228
Use SDKs to access the S3 service......................................................................................................................229
Working with S3 workloads in ObjectScale........................................................................................................ 232

Chapter 14: Alerts......................................................................................................................234

About ObjectScale instance event and issue monitoring...................................................................................... 234
View ObjectScale health issues and events....................................................................................................... 234
View the health of an object store....................................................................................................................... 235
Hardware Alerts.........................................................................................................................................................235
Monitoring Events, Audits, and Alerts....................................................................................................................... 238
CSI-01...........................................................................................................................................................................238
CSI-03..........................................................................................................................................................................238
CSI-05..........................................................................................................................................................................238
DECKS-HC-1000....................................................................................................................................................... 239
DECKS-LIC-1002.......................................................................................................................................................239
DECKS-LIC-1005.......................................................................................................................................................239
DECKS-LIC-1006.......................................................................................................................................................239
DECKS-LIC-1008.......................................................................................................................................................240
DECKS-LIC-1011........................................................................................................................................................ 240
DECKS-SA-1023........................................................................................................................................................240
DECKS-SA-1024........................................................................................................................................................240
KAHM-HC-1000......................................................................................................................................................... 241
OBJSC-CO-0000.......................................................................................................................................................241
OBJSC-CO-0001........................................................................................................................................................241
OBJSC-CO-0002....................................................................................................................................................... 241
OBJSC-CO-0003...................................................................................................................................................... 242
OBJSC-CO-0004...................................................................................................................................................... 242
NVMF-1389................................................................................................................................................................ 242
NVMF-1390................................................................................................................................................................ 242
NVMF-1393.................................................................................................................................................................242
NVMF-1395................................................................................................................................................................ 243
NVMF-1396................................................................................................................................................................ 243
OBJPRECHK-2000...................................................................................................................................................243
OBJPRECHK-2001....................................................................................................................................................244
OBJPRECHK-2002................................................................................................................................................... 244
OBJPRECHK-2003................................................................................................................................................... 244
OBJPRECHK-2004................................................................................................................................................... 244
OBJPRECHK-2005...................................................................................................................................................245
OBJPRECHK-2006...................................................................................................................................................245
OBJPRECHK-2007................................................................................................................................................... 245
OBJPRECHK-2008...................................................................................................................................................245
OBJPRECHK-2009...................................................................................................................................................246
OBJPRECHK-2010....................................................................................................................................................246
OBJPSTCHK-3000................................................................................................................................................... 246
OBJPSTCHK-3001....................................................................................................................................................246
OBJPSTCHK-3002................................................................................................................................................... 247
OBJPSTCHK-3003................................................................................................................................................... 247
OBJSC-FED-1001...................................................................................................................................................... 247

Contents 7
 OBJSC-IAM-1004......................................................................................................................................................247
OBJSC-LIC-0004...................................................................................................................................................... 248
OBJSC-MGR-3000...................................................................................................................................................248
OBJSC-MGR-HC-1000............................................................................................................................................248
OBJSC-MON-1111......................................................................................................................................................248
OBJSC-MON-1112.....................................................................................................................................................249
OBJSC-MON-1113.....................................................................................................................................................249
OBJSC-MON-3002.................................................................................................................................................. 249
OBJSC-MON-3003.................................................................................................................................................. 249
OBJSC-MON-4019................................................................................................................................................... 250
OBJSC-MON-4020.................................................................................................................................................. 250
OBJSC-MON-4021................................................................................................................................................... 250
OBJSC-MON-4022.................................................................................................................................................. 250
OBJSC-MON-4025................................................................................................................................................... 251
OBJSC-MON-4028................................................................................................................................................... 251
OBJSC-SP-0000....................................................................................................................................................... 251
OBJSC-SP-0001........................................................................................................................................................ 251
OBJSC-SP-0002.......................................................................................................................................................252
OBJSC-SP-0003.......................................................................................................................................................252
OBJSC-SP-0004.......................................................................................................................................................252
OBJSC-TARGET-01..................................................................................................................................................252
OBJSOP-1000............................................................................................................................................................253
OBJSOP-1001.............................................................................................................................................................253
OBJSOP-1002............................................................................................................................................................253
OBJSOP-1003............................................................................................................................................................253
OBJSOP-1004............................................................................................................................................................254
OBJSOP-1005............................................................................................................................................................254
OBJSOP-1006............................................................................................................................................................254
OBJSOP-2001............................................................................................................................................................254
OBJSOP-2002........................................................................................................................................................... 255
OBJST-1006...............................................................................................................................................................255
OBJST-1008............................................................................................................................................................... 255
OBJST-12001............................................................................................................................................................. 255
OBJST-12003.............................................................................................................................................................256
OBJST-12004.............................................................................................................................................................256
OBJST-12005.............................................................................................................................................................256
OBJST-12006.............................................................................................................................................................256
OBJST-12007............................................................................................................................................................. 257
OBJST-12008.............................................................................................................................................................257
OBJST-12010..............................................................................................................................................................257
OBJST-12011...............................................................................................................................................................257
OBJST-13000.............................................................................................................................................................258
OBJST-13001............................................................................................................................................................. 258
OBJST-13002.............................................................................................................................................................258
OBJST-13003.............................................................................................................................................................258
OBJST-13004.............................................................................................................................................................259
OBJST-13005.............................................................................................................................................................259
OBJST-13006.............................................................................................................................................................259
OBJST-13007.............................................................................................................................................................260
OBJST-13008.............................................................................................................................................................260

8 Contents
OBJST-13009.............................................................................................................................................................260
OBJST-13010............................................................................................................................................................. 260
OBJST-13011.............................................................................................................................................................. 260
OBJSTEPUPD-4000................................................................................................................................................. 261
OBJSTEPUPD-4001.................................................................................................................................................. 261
OBJSTEPUPD-4002................................................................................................................................................. 261
OBJSTEPUPD-4003................................................................................................................................................. 261
OBJSTEPUPD-4004.................................................................................................................................................262
OBJSTEPUPD-4005.................................................................................................................................................262
OBJSTEPUPD–4006................................................................................................................................................262
OBJSTEPUPD–4007................................................................................................................................................263
OBJSTEPUPD–4008................................................................................................................................................263
OBJST-1320............................................................................................................................................................... 263
OBJST-1321................................................................................................................................................................ 263
OBJST-1324............................................................................................................................................................... 264
OBJST-1325............................................................................................................................................................... 264
OBJST-1328............................................................................................................................................................... 264
OBJST-1329............................................................................................................................................................... 264
OBJST-1332............................................................................................................................................................... 265
OBJST-1333............................................................................................................................................................... 265
OBJST-1336............................................................................................................................................................... 265
OBJST-1337............................................................................................................................................................... 265
OBJST-1340............................................................................................................................................................... 265
OBJST-1341................................................................................................................................................................ 266
OBJST-1344............................................................................................................................................................... 266
OBJST-1345............................................................................................................................................................... 266
OBJST-1352............................................................................................................................................................... 266
OBJST-1354............................................................................................................................................................... 266
OBJST-1364............................................................................................................................................................... 267
OBJST-1365............................................................................................................................................................... 267
OBJST-1366............................................................................................................................................................... 267
OBJST-1389............................................................................................................................................................... 267
OBJST-1390............................................................................................................................................................... 268
OBJST-1392............................................................................................................................................................... 268
OBJST-1600............................................................................................................................................................... 268
OBJST-1601................................................................................................................................................................268
OBJST-1602............................................................................................................................................................... 268
OBJST-1603............................................................................................................................................................... 269
OBJST-1604............................................................................................................................................................... 269
OBJST-1605............................................................................................................................................................... 269
OBJST-1700............................................................................................................................................................... 270
OBJST-1701................................................................................................................................................................ 270
OBJST-2100............................................................................................................................................................... 270
OBJST-2101................................................................................................................................................................ 270
OBJST-MON-4016....................................................................................................................................................270
OBJST-MON-4019.................................................................................................................................................... 271
OBJST-MON-4020....................................................................................................................................................271
OBJSTORE-HC-1000................................................................................................................................................ 271
OBJUPD-1000.............................................................................................................................................................271
SNMPNOTI-1000...................................................................................................................................................... 272

Contents 9
 TEST TRAP................................................................................................................................................................. 272

Chapter 15: Metrics for ObjectScale and object stores.............................................................. 273

ObjectScale metrics........................................................................................................................................................273
Metering details within an ObjectScale instance...............................................................................................273
ObjectScale metrics in Grafana................................................................................................................................... 275
Grafana dashboards overview................................................................................................................................275
Navigating Grafana................................................................................................................................................... 277
View the Metrics dashboards for the ObjectScale instance...........................................................................277
View the Metrics dashboards for an object store.............................................................................................278

Chapter 16: Maintain ObjectScale..............................................................................................280

About ObjectScale service procedures..................................................................................................................... 280
About ObjectScale capacity expansion procedures......................................................................................... 280
ObjectScale Maintenance Modes..........................................................................................................................283
Service procedures for cluster disks.................................................................................................................... 293
Service procedures for cluster nodes.................................................................................................................. 298
Troubleshooting Service Procedures................................................................................................................... 326
Updating iDRAC IPs Using Server Patch API for ObjectScale Appliance................................................... 328
Shutting Down and Restarting ObjectScale....................................................................................................... 332

10 Contents
 Revision history
Table 1. Revision history
Revision Date Revision Number Description of change
October 11, 2023 1.0 Initial release for ObjectScale 1.3.0.

Document feedback
Submit feedback or suggestions about this document to [email protected].

Revision history 11
 About using this guide
CAUTION: Many PDF viewers add a line break to the end of each line of text in a PDF. Adobe Acrobat (Reader,
Standard, and Pro) and other common PDF viewers, including Google Chrome and Microsoft Edge, insert these
line breaks. As a result, when you copy commands that wrap across multiple lines in a PDF, the copied command
is in the wrong format. (It contains erroneous line breaks.) If you copy and paste commands, the line breaks
cause issues during the installation and administration of ObjectScale.

To address this known limitation and ensure that copied commands do not contain unintentional line breaks, do
one of the following:

● Paste the copied commands into a text editor and remove the line breaks.
● Use the HTML version of this document when you are copying commands.

12 About using this guide

 1
Overview
This chapter contains:
Topics:
• About Dell ObjectScale
• Product Description
• Basic Terminology
• ObjectScale User Interfaces
• ObjectScale and Kubernetes
• Data protection with ObjectScale Erasure Coding schemes
• Customer Feedback

About Dell ObjectScale

ObjectScale uses a software-defined, containerized architecture to deliver enterprise-class, high-performance object storage in
a native Kubernetes package. ObjectScale empowers organizations to move faster and respond more effectively to rapidly
changing business needs. This next generation of object storage software is lighter, faster, and deployable on existing
infrastructure. You can deploy ObjectScale on your Kubernetes (Red Hat OpenShift Container Platform) or SUSE Linux
Enterprise Server (SLES) infrastructure. ObjectScale is also available as a preconfigured Appliance (ObjectScale XF960).
ObjectScale supports the storage, manipulation, and analysis of unstructured data on a massive scale.
With rich S3 compatibility and self-service APIs, you can quickly spin up object storage containers. These containers can service
many types of applications, from big data and analytics to ephemeral development or test sandboxes.
ObjectScale allows any organization to deliver scalable cloud services with the reliability and control of a private cloud
infrastructure. ObjectScale enables convenient management for a globally distributed storage infrastructure.
ObjectScale is built with certain design principles, such as:
● Global namespace with eventual consistency
● Scale-out capabilities
● Secure multitenancy
● Superior performance for small, large, and huge objects
The platform was built as a distributed system following the microservices principle of cloud applications. ObjectScale has a
layered architecture, with every function in the system built as an independent layer, making them horizontally scalable across
all nodes and enabling high availability. The S3-compatible ObjectScale software forms the underlying cloud storage service,
providing protection, geo-replication, and data access.

Product Description
ObjectScale is an enterprise-grade object storage system from Dell that runs efficiently on shared infrastructure and in multi-
tenant environments.
ObjectScale gives organizations the power to put data closer to the applications they support, reducing latency and improving
the user experience. In addition, object storage from disparate platforms can cross-replicate for greater access, reliability, and
redundancy.
ObjectScale offers the following major functionality:
● Simple, S3-compatible enterprise-grade object storage
● Kubernetes-based, customer-deployable
● Scaled-out, software-defined architecture
Other important features of ObjectScale include:

Overview 13
● Improved data protection with new erasure coding schemes
● New replication model with eventual consistency for greater availability during hardware failure
● Integrated management of bucket or object events, enabling bucket notifications, ObjectScale replication, and metering
● A complete multi-tenant IAM service with IAM accounts, with each account supporting IAM entities such as users, groups,
roles, policies, and service providers
ObjectScale runs in Kubernetes. Building ObjectScale for Kubernetes allowed Dell to deliver a simplified product where
Kubernetes handles the OS- and hardware-level layers. ObjectScale handles the storage and storage management.
With this underlying Kubernetes architecture, ObjectScale gives you segmented control of the storage, compute, and network
services. The architecture allows for dynamic provisioning of resources. You can control when new services are started on an as
needed basis. These new resources are tied to the underlying resources upon creation.
Here are some of the benefits Kubernetes provides for ObjectScale:
● Predictable application deployment using a declarative method
● Dynamic scaling of application resources
● Deployment using only required resources
● Highly portable across deployment models
● Self-healing: Autoplacement, auto restart, and autoreplication
In Kubernetes, each resource can be affinitized to run on one host. Affinization of resources to hosts allows ObjectScale to
behave as its own fault domain.
Flexible deployment environments support customer-built and maintained object storage systems. ObjectScale deployment
platforms are:
● ObjectScale Appliance
● ObjectScale Software Bundle
● ObjectScale for Red Hat OpenShift
ObjectScale performance and maintenance tasks benefit from the collocation of the compute and storage infrastructure.

Basic Terminology
The following terms are basic to understand ObjectScale.

Account A logical construct that corresponds to a customer business unit, tenant, project, and so on, which are
relevant to the account admin role and end users that belong to an account.
Admin Admin of an ObjectScale or a federation of ObjectScale instances.
Buckets Buckets are object containers that are used to control access to objects.
Chunk A Chunk is the basic unit in ObjectScale for data storage. A chunk is 128MiB of logical storage that is
erasure-coded and written to multiple disks across multiple nodes in the instance.
Custom Resource Custom Resource Definitions are extensions to Kubernetes API resources. ObjectScale adds CRDs that
Definition create custom resources with the specified name and schema.
DECKS Dell Common Kubernetes Services created by Dell.
Federation A federation joins multiple ObjectScale instances together. Global information like endpoints or global
accounts are replicated throughout an ObjectScale federation.
Horizontal Object stores can be expanded through horizontal expansion by adding more Storage Servers to the
Expansion object store.
IAM Role An IAM Role (role) is an IAM identity that you can create in your account that has specific permissions.
An IAM Role is similar to an IAM user, in that it is an ObjectScale identity with permission policies
that determine what the identity can and cannot do in ObjectScale. However, instead of being uniquely
associated with one person, a role is intended to be assumable by anyone who needs it. An IAM role
does not have any credentials and cannot make direct requests, and IAM roles need to get short term
credentials by assuming role. IAM roles with temporary credentials are used in the certain situations like,
federated user access, temporary IAM user permissions , cross-account access, and cross-service access.
IAM User IAM user has permanent long-term credentials and is used to directly interact with ObjectScale data
services. An IAM user is an identity with permission policies that determine what the identity can and
cannot do in ObjectScale.
KAHM Kubernetes Application Health Management created by Dell.

14 Overview
Kubernetes Kubernetes (K8s) is an open-source container-orchestration system for automating application
deployment, scaling, and management.
Large Size Object A large object is a conceptual distinction, and is an object where the size of the file is such that the time
spent moving the payload data of the file is the dominant component of the overall response time. This
is in contrast to a small object, where the dominant component of the overall response time would be
the transactional overhead. The distinction between these two is useful in understanding the factors of
overall system performance.
Namespace In Kubernetes, namespaces act as a mechanism for isolating groups of resources within a single cluster.
Object Attribute An object attribute is an aspect of an object version that can be updated and replicated separately, such
as an object tag, ACL, or lock.
Object Data Data of an object version.
Object Data The data locations of an object version on chunks.
Index
ObjectScale ObjectScale is deployed in a Kubernetes cluster. The deployment is termed as an ObjectScale Instance
Instance (OSI). ObjectScale, or the ObjectScale instance, is a software bundle of management services that
contains everything that is needed to deploy and consume Dell object storage. The ObjectScale instance
is deployed once per Kubernetes cluster and provides management and shared object store services.
Including:
● IAM
● Federation service
● Serviceability features such as SupportAssist.
Object Stores A unique and independent storage system with an individualized life cycle. One or more object stores are
deployed by each ObjectScale instance. Object stores are created, updated, and deleted independently
from all other object stores managed by the shared ObjectScale instance. Kubernetes cluster resources
such as storage, CPU, and RAM are defined for each object store based on workload demand inputs that
are specified at object store creation. Resources that are reserved for an object store at creation may be
adjusted at any time.
Object Metadata The system or user metadata of an object. Object metadata is a part of the object version, and it cannot
be updated separately. As a result object metadata is not a part of ObjectScale replication attributes.
Object metadata is replicated whenever the object data is replicated.
Object Version All data or metadata or attribute belongs to a specific version of an object.
Resource Names Resource Names (RNs) are names that uniquely identify resources. Resource Names (RNs) are required
when user must specify a resource unambiguously in an ObjectScale.
Small Size Object A small object is a conceptual distinction, and is an object where the size of the file is such that most
of the I/O time is spent accessing metadata and thus, is bounded by the performance of the metadata
services.
Storage Class Storage Class determines which driver is used to create a persistent volume. At a per ObjectScale level,
the admin can map storage classes to storage tiers.
Storage Servers Storage Servers (SS) in ObjectScale interact with storage media. In ObjectScale, each physical server is a
(SS) Kubernetes node, and each SS pod instance is an ObjectScale node.
SupportAssist SupportAssist provides a network based connection to Dell Support. SupportAssist enables Dell Support
to receive telemetry and issues, events, and alerts from your ObjectScale instance, and to perform
remote troubleshooting, resulting in a fast and efficient time to resolution.
Tenant A tenant is a logical construct resulting from the binding of an IAM account to an object store. When an
IAM account is added to an object store, that account becomes a tenant within that object store.
Vertical ObjectScale can be expanded through vertical expansion by increasing the number of volumes per
Expansion Storage Server replica in the object store.

ObjectScale User Interfaces

ObjectScale provides the following interfaces.
● ObjectScale Portal for Dell ObjectScale
● Grafana Dashboards

Overview 15
ObjectScale Portal for Dell ObjectScale
The ObjectScale Portal user interface is used to manage deployments of ObjectScale.
The ObjectScale Portal allows you to easily manage ObjectScale and its features, as well as object stores, accounts, and account
entities.

Grafana Dashboards
ObjectScale includes the collection, storage, and visualization of detailed metrics in Grafana dashboards. Administrators can use
these dashboards to drill into problems or identify developing problems with ObjectScale or problems with underlying storage
resources.
Similar metrics are also available at the ObjectScale-level, from the Dashboard page (must have admin permissions to see
them).
Grafana is an open-source metrics visualization tool. The ObjectScale installation deploys Grafana.
See Grafana for basic details of navigation in Grafana dashboards.

ObjectScale and Kubernetes

Dell ObjectScale is an object storage software of management services that contains everything that it must deploy to consume
Dell object storage. ObjectScale is deployed in a Kubernetes cluster allowing Kubernetes to handle the necessary orchestration.
ObjectScale supports deployment as a preconfigured Appliance (ObjectScale XF960), an application within a Red Hat OpenShift
cluster environment, or as a software bundle (ObjectScale Software Bundle). You can deploy the ObjectScale Software Bundle
on a cluster that is configured with a supported Operating System. The ObjectScale Software Bundle includes ObjectScale and
the necessary Kubernetes and platform management components.
One ObjectScale instance with one object store in Non-Volatile Memory express (NVMe) is deployed per Kubernetes cluster.
Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management.
A Kubernetes cluster consists of physical or virtual nodes. Each Kubernetes node runs a process that is named kubelet.
ObjectScale is built on Kubernetes clusters using physical server infrastructure.
In ObjectScale, Kubernetes provides the connective glue between physical infrastructure, such as disk and network, and the
application services running in containers. ObjectScale leverages the efficient resource management capabilities of Kubernetes
and relies on it to handle operating system and hardware interaction.

ObjectScale Kubernetes components

ObjectScale includes these software components:
1. ObjectScale Manager—Installs and manages the custom ObjectScale resources.
2. Dell EMC Common Kubernetes Services (DECKS)—DECKS is a suite of tools that performs log collection and gathers
telemetry information about ObjectScale licensed resource usage. The Dell SupportAssist Embedded Support Enabler (ESE)
is part of DECKS.
3. Kubernetes Application Health Management (KAHM)—KAHM handles event persistence management, notifications, and
complex event routing rules.
4. User Interfaces:
● ObjectScale Portal user interface
● Grafana, with preconfigured dashboards for monitoring the ObjectScale instance
● Kubectl plug-in at CLI
● Helm binary at CLI
● ObjectScale and object store management APIs

Operators and ObjectScale

Kubernetes has a concept that is called an Operator. An Operator is an application-specific controller and contains all the
operational considerations of an application. Operator resources are defined in YAML files as Kubernetes Custom Resource

16 Overview
Definitions (CRD). Custom resources define actions available to users of the Operator. Kubernetes manages custom resources
like it manages its own integrated resources.
The ObjectScale Operator is a custom resource that creates object stores. The ObjectScale Operator connects object stores to
the management services, and orchestrates operations, such as upgrades and deletions.
The ZooKeeper operator is a custom resource that manages all the ZooKeeper clusters for ObjectScale.
The Atlas operator is a custom resource that:
● Provides Atlas services to implement a new key-value store
● Provides stability, predictability, and efficiency for per-operation overhead (key-value operations), system operation
overhead (node replacement), and overall CPU and memory use

Other Kubernetes resources to know

Here is a list of additional common Kubernetes resources for administrators of ObjectScale:
● Annotations are key-value maps that attach arbitrary nonidentifying metadata to objects such as Pods. Tools and libraries
use annotations.
● Labels are key-value pairs that are attached to objects. Labels are used to organize and to select subsets of objects.
● Pods are a unit of application running in Kubernetes. Each pod consists of one or more containers. A set of pods makes up a
Kubernetes application. ObjectScale deploys multiple types of pods for each object store.
● A Deployment provides declarative updates for Pods and ReplicaSets. A deployment describes a wanted state. The
deployment controller tracks and maintains the actual state to the wanted state.
● A ReplicaSet is a deployment model available in Kubernetes. A ReplicaSet is one or more of a single type of pod. ReplicaSets
are used to guarantee the availability of the service they provide. An example of a ReplicaSet used in ObjectScale is
GraphQL. The number of replicas in a set may be adjusted as required.
● A StatefulSet is a deployment model in Kubernetes. StatefulSets are used for deploying stateful applications. StatefulSets
manage the deployment and scaling of a set of Pods and provide guarantees about the ordering and uniqueness of these
Pods. StatefulSets maintain a sticky identity to a Kubernetes node for each pod in the set.
● A Service is an abstract way to expose an application running on a set of Pods as a network service. Networking services
are provided for Kubernetes environments that allow for ingress, egress, and load balancing of traffic in and out of the
Kubernetes environment. Services provide Client connectivity to ObjectScale.
● A PersistentVolume (PV) is storage that is provisioned on available storage.
● A PersistentVolumeClaim (PVC) is a request for PV resources. PVCs request and consume specific size and access modes.
A PVC, or claim for short, is bound to a persistent volume. Persistent Volumes and associated provisioned virtual disks are
deleted at PVC deletion. Pod creation and destruction have no effect on PVC or PV.

ObjectScale Storage Classes

ObjectScale utilizes these Dell Bare-Metal CSI Driver storage classes (SC).

Bare-metal CSI Driver Storage Classes

Name Reclaim Policy Volume Allow Volume Disk Micro Highly Media Types
Binding Mode Expansion Partitioning Available
csi-baremetal-sc Delete WaitForFirstCon No No No Any
(default) sumer
csi-baremetal- Delete WaitForFirstCon No No No HDD
sc-hdd sumer
csi-baremetal- Delete WaitForFirstCon Yes Yes No HDD
sc-hddlvg sumer
csi-baremetal- Delete WaitForFirstCon No No No NVMe
sc-nvme sumer
csi-baremetal- Delete WaitForFirstCon No No No SSD
sc-ssd sumer

Overview 17
Name Reclaim Policy Volume Allow Volume Disk Micro Highly Media Types
Binding Mode Expansion Partitioning Available
csi-baremetal- Delete WaitForFirstCon Yes Yes No SSD
sc-ssdlvg sumer
csi-baremetal- Delete WaitForFirstCon Yes Yes No Any
sc-syslvg sumer

ObjectScale components
An ObjectScale deployment contains these components to support ObjectScale features and functionality.
The following table describes ObjectScale and object store components that are deployed with an ObjectScale instance. Several
columns show the component size and expected storage class.

Table 2. ObjectScale Stateful components

Name Level Highly Minimum Volume Size (large SSD Storage Class (SC)
Available replicas profile)
count
rsyslog ObjectScal No Number of Default 200Gi No csi-baremetal-sc-
e nodes hddlvg
iam-atlas ObjectScal Yes 3 Default 10Gi Yes csi-baremetal-sc-
e ssdlvg
dcm-atlas ObjectScal Yes 3 Default 1Gi Yes csi-baremetal-sc-
e ssdlvg
federation- ObjectScal Yes 3 Default 10Gi Yes csi-baremetal-sc-
atlas e ssdlvg
influxdb ObjectScal Yes 3 Default 20Gi Yes csi-baremetal-sc-
e ssdlvg
db-kahm ObjectScal Yes 3 Default 8G i No csi-baremetal-sc-
e ssdlvg
decks- ObjectScal No 1 Default 200Gi No csi-baremetal-sc-
support- e ssdlvg
store
supportassis ObjectScal No 1 Default 2Gi No csi-baremetal-sc-
t var/config e ssdlvg

supportassis ObjectScal No 1 Default 50Gi No csi-baremetal-sc-

t support- e ssdlvg
store
ss object No 3 Multiple Varies No csi-baremetal-sc-
store hdd
atlas object Yes 3 Default 32Gi Yes csi-baremetal-sc-
store ssdlvg
zookeeper object Yes SS<3: 1 Default 2Gi No csi-baremetal-sc-
store ssdlvg
3<SS<5: 3
SS>4: 5

influxdb object Yes 3 Default 20Gi Yes csi-baremetal-sc-

store ssdlvg
bookie object Yes SS<3: 1 index 3Gi Yes csi-baremetal-sc-
store ssdlvg/hddlvg
SS=3: 3

18 Overview
Table 2. ObjectScale Stateful components (continued)
Name Level Highly Minimum Volume Size (large SSD Storage Class (SC)
Available replicas profile)
count

SS>4: journal 50Gi Yes csi-baremetal-sc-

max(4, 0.5 ssdlvg/hddlvg
* SS)
ledger 300Gi Yes csi-baremetal-sc-
ssdlvg/hddlvg

ObjectScale contains the following stateless components:

objectscale-rest-service REST API

objectscale-lcm Life-cycle manager

Data protection with ObjectScale Erasure Coding

schemes
ObjectScale uses various Erasure Coding schemes for data protection. Erasure coding (EC) is a method of data protection in
which data is broken into fragments, expanded and encoded with redundant data pieces. The data pieces are stored across
different locations or storage media.
The goal of erasure coding is to enable data that becomes corrupted at some point in the disk storage process to be
reconstructed by using information about the data that is stored elsewhere in ObjectScale. Erasure code schemes are often
used instead of traditional RAID because of their ability to reduce the time, overhead required to reconstruct data, and greater
data resiliency, depending on the EC scheme used.
During the object store creation process using ObjectScale Portal, the available EC schemes that are presented within the
New Object Store wizard is based on the number of Kubernetes nodes, either physical servers or worker nodes, in the cluster.
ObjectScale uses the Kubernetes anti-affinity rules to ensure that the storage server (SS) instances are properly placed across
the nodes in the cluster. The New Object Store wizard ensures that the number of SS instances for the new object store is not
below the minimum for the selected EC scheme.
ObjectScale implements Reed Solomon error correction using these schemes:
● 12+4 - Chunk is broken into 12 data segments, and four coding (parity) segments are created.
For each EC scheme, the resulting data and coding segments of each chunk are equally distributed across the nodes in the
Kubernetes cluster.
Upon a Kubernetes node permanent failure, the copies of lost data segments are recreated using remaining data and coding
segments. During temporary Kubernetes node failure, data services continue with data and coding segments that are being used
to recreate data when needed.
ObjectScale minimum disk requirements vary based on object store EC requirements. When an object store is created, the total
raw capacity and EC scheme are specified. Administrators choose the topology based on input to provide optimal protection and
SS size. The number and size of SS instances in an object store represent the persistent storage capacity allocated for raw user
data. SS instances attach to Kubernetes persistent volumes (PVs) on disks using Kubernetes persistent volume claims (PVCs).
ObjectScale writes data for best protection considering number of volumes on disk, disks per SS, and SS instances across the
cluster.

Supported ObjectScale Erasure Coding Schemes

Erasure Coding Number of nodes Data availability during component
Scheme failures
12+4 4–5 nodes One node failure
Disk failures from a single node

12+4 6–9 nodes One node failure

Overview 19
Erasure Coding Number of nodes Data availability during component
Scheme failures

Disk failures from up to two different nodes

and up to a maximum of four disk failures in
total

12+4 >= 10 nodes Two node failures

Disk failures from two different nodes
One node failure and disk failures from
another node

Customer Feedback
Use the customer feedback feature in the ObjectScale Portal to report your satisfaction with ObjectScale, provide feedback,
and send requests for enhancements. Customer feedback is used to improve the customer experience.

Provide general feedback

Submit feedback on ObjectScale and the ObjectScale Portal to help Dell determine possible issues or enhancements.

About this task

From within the ObjectScale Portal, report your satisfaction with ObjectScale and the ObjectScale Portal and provide your
valuable feedback.

Steps
1. Click Feedback, located in the bottom of left hand navigation panel.
The ObjectScale customer feedback survey opens in a new window in your internet browser.
NOTE: In environments with limited external connectivity, such as dark sites, an error appears in the web browser and
the customer feedback survey is not displayed.

2. Complete the desired fields in the customer feedback survey, and when finished, click Submit.
You have the option to rate your satisfaction with ObjectScale and make a recommendation for how to improve the
customer experience. You also have the option to provide an email address so that Dell can follow up with you regarding your
feedback.

NOTE: Customer contact information will not be used for marketing purposes.

20 Overview
 2
Getting Started with ObjectScale
Use these sections to begin using ObjectScale following installation.
Topics:
• Accessing the ObjectScale Portal user interface
• Using the ObjectScale Portal user interface
• Security configuration
• Role-based security

Accessing the ObjectScale Portal user interface

ObjectScale users can access the ObjectScale Portal user interface to manage ObjectScale, object stores, and accounts, as well
as other monitoring and management tasks.

Access the ObjectScale Portal user interface

Follow these steps to connect to the ObjectScale Portal with a supported Internet browser.

Prerequisites
If you have not already done so, obtain the network address (EXTERNAL-IP) of the ObjectScale Portal user interface:

kubectl -n <OBJECTSCALE_NAMESPACE> get svc objectscale-portal-external

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

objectscale-portal-external LoadBalancer 10.55.66.100 10.x.y.z 4443:30436/TCP 9m2s

Steps
1. Open a supported web browser and enter the External IP address and port for the objectscale-portal service:
https://<EXTERNAL_IP_ADDRESS>:4443

The ObjectScale Portal login page appears.

2. Use the Username and Password credentials to log in to the ObjectScale Portal.
For the ObjectScale Software Bundle or the ObjectScale Appliance, your credentials are configured with certain permissions
for accessing and maintaining the ObjectScale instance and object stores.
● You can log in as a local Management User. These accounts are created in ObjectScale. Local Management Users have
permissions that define specific access to all or portions of ObjectScale.
● If external identity providers are integrated into ObjectScale, you can log in as an externally defined AD or LDAP end user.
Identity provider end user accounts are mapped to ObjectScale permissions that define specific access to all or portions
of ObjectScale.
● When the Privileged Actions Approval System (PAAS) is enabled, you can log in as an Approver User. Approvers are
created in ObjectScale. Their purpose is to approve or reject requests for performing protected actions in PAAS.
For ObjectScale on OpenShift, obtain your credentials using the OpenShift identity provider user that is configured for
accessing the ObjectScale instance or object store.
● You can log in as kubeadmin and can have access to the entire ObjectScale instance and you can create object stores
in any namespace you have configured.
● You can log in as an OpenShift Identity Provider User end user and can access only the namespaces to which you have
permissions.

Getting Started with ObjectScale 21

 ○ If you log in as an end user who has edit permissions to a specific namespace, you can manage the object store in that
namespace. All other ObjectScale features are not available.
○ If you log in as an end user who has view permissions to a specific namespace, you can view the Object Store in that
namespace. All other ObjectScale features are not available.

Using the ObjectScale Portal user interface

Refer to the sections below for details on navigating within ObjectScale.

View Dashboard
The Dashboard tab provides an overview of ObjectScale.
● Object Store Performance Select the namespace containing the object store from the namespace drop-down on the
upper right corner of the ObjectScale Portal. Once you select a namespace ObjectScale Portal displays data on one or more
object stores within that namespace.
○ Name
○ State
○ Latency
○ Compression Ratio
You can use the timeframe filter on the right side of the section to choose between the following options:
○ Last 24 hours
○ Last 7 days
○ Last 30 days
○ Custom Range
● ObjectScale Summary
○ Health - Monitor the ObjectScale system alerts. Click links in the row to see related alerts.
○ System Data - The capacity that is used by the ObjectScale processes that track and describe the data in the system.
Hover over the category names to know more.
○ Capacity Utilization - Monitor all capacities at the ObjectScale level. Hover over the category names to know more.
○ Data Management - Monitor the capacity reclaimed, user data reclaimed, and system metadata reclaimed in
ObjectScale level. Hover over the category names to know more.

Table 3. Dashboard field details

Field Description
Name Lists the object stores present in the selected namespace. Select a namespace with the
dropdown menu in the upper right corner of the object store performance section.
State Lists the states of object stores
Latency Latency consists of:
● Read First Byte(p50)
● Write Last Byte(p50)
● Read First Byte(p99)
● Write Last Byte(p99)
Compression Ratio Lists the compression ratios of object stores
Health The Health section shows information about number of unacknowledged issues with severity:
● Critical
● Error
● Warning
System Data The System Data section shows information that is related to data such as:
● Data Protection
● Metadata
● Metadata Protection
● Data pending for EC
● Rate of EC per Second

22 Getting Started with ObjectScale

Table 3. Dashboard field details (continued)
Field Description
Capacity Utilization The Capacity Utilization section shows information that is related to capacity such as:
● Physical Used
● Available
● Reserved
● Total
● % Full
● Days until Full (Est)
Data Management The Data Management section shows information that is related to data such as:
● Data Being Reclaimed
● Unreclaimable Metadata
● Unreclaimable User Data
● Reclaimable Metadata
● Reclaimable User Data
● Capacity Reclaimed

ObjectScale Portal user interface

When launched, the ObjectScale Portal user interface Dashboard page appears.

Navigating within ObjectScale Portal

You can view different portions of the ObjectScale Portal user interface by selecting a section from the left-side navigation
panel. ObjectScale only presents users with the sections of the interface that they can view or edit based on their assigned
permissions.
After selecting a section of ObjectScale Portal, you can move to another section by clicking its name within the ObjectScale
navigation panel. ObjectScale also creates a clickable breadcrumb navigation, which is presented at the top of the ObjectScale
Portal user interface.

Notifications for completed user actions

ObjectScale Portal displays ephemeral notifications at the top of the Internet browser window after completing certain actions.
These notifications are temporary alerts to the current user. They provide information about recently completed user actions or
anything that needs the attention of the user within the ObjectScale Portal user interface.
These notifications are not stored elsewhere within ObjectScale Portal. More detailed messages on these actions can be found
on the Alerts and Logs pages.

ObjectScale Portal Menu

Overview of different menu options in the ObjectScale Portal.
The menu on the left side of the ObjectScale Portal provides access to different options:
● Accounts - View or manage ObjectScale identity and access management accounts
● Buckets - View or manage ObjectScale buckets with options to filter by namespace, Object store, Account, and Bucket
name.
● Nodes - View or manage available Nodes
● Disks - View or manage available Disks with options to filter by Status, Disk Name, and Storage Type.
● Alerts - View or manage alerts. To display all alerts including hidden ones, enable Show All. There are filters to display alerts
by specific time periods and by Severity, Message ResourceID, SymptomID, Reason, and Component.
● Logs - View or manage logs. There are filters to display alerts by specific time periods and by Severity, Message,
Occurrences, SymptomID, and Reason.
● Event Settings - View or manage SNMP and Hardware alerts.

Getting Started with ObjectScale 23

● Performance Dashboard - Displays the performance of the selected Object store with parameters such as Latency,
Bandwidth, Throughput, Request By Response Codes, and Requests By Methods.
● Administration - Opens submenus for ObjectScale, Licensing, SupportAssist, SAML Metadata, Requests, Platform Settings,
Security Settings, Active Sessions, and Upgrades.

Link ObjectScale Appliance Hardware with Software License

Use the ObjectScale Portal user interface to activate an ObjectScale license and apply the license file to the ObjectScale
instance.

About this task

Linking of ObjectScale appliance with Software licensing is an important step for sales and support.
To add a license:

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Licensing.
3. Click Generate Activation XML.
The Generate Activation XML box appears.
4. Enter file name in the Download file name field, and click Download.
The Activation XML is generated and downloaded.
5. Generate License on SLC.
See Software Licensing Central: Activation, Entitlements, Rehost and Regeneration Guide "Chapter 2 - Activate-by-File" on
Dell Support for the detailed procedures.
The activated license is saved to a file.
6. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
7. Click Licensing.
The page displays available license information.
8. Click Apply.
The Apply License box appears.
9. Click Select to browse and upload the ObjectScale license file. Once uploaded, click Apply.
10. Expand the license in the Licensing table to display details about the ObjectScale license and its enabled features and
capacities.

ObjectScale Performance Dashboard

This section provides an overview of the Performance Dashboard in the ObjectScale Portal.
When you specify the namespace and the Object Store from the drop-down on the right side, the performance dashboard is
displayed. For each section, you can filter the data using the following timeframe options:
● Last 24 hours
● Last 7 days
● Last 30 days
● Custom Range
Each section has a slider to adjust the time displayed.

Table 4. Performance Dashboard Overview

Section Description
Latency (millisecond) Latency can be viewed as a graph for
the following parameters that you can
select using the radio button next to it:

24 Getting Started with ObjectScale

Table 4. Performance Dashboard Overview (continued)
Section Description
● Read First Byte(p99)
● Write Last Byte(p99)
● Read First Byte(p50)
● Write Last Byte(p50)
● Total
Bandwidth (MBs) Bandwidth can be viewed for the
following options:
● Read
● Write
● Total
Throughput (bps) Throughput displays the graph for the
following options:
● Read
● Write
● Total
Requests by Response Codes (count) The following options are available:
● Server Error Requests
● Client Error Requests
● Info Requests
● Redirection Requests
● Success Requests
● All
Requests by Method (count) You can view the graph for the following
options:
● Get Requests
● Post Requests
● Put Requests
● Delete Requests
● Head Requests
● All

Administration
View the ObjectScale Portal Administration section.
The Administration section of the ObjectScale Portal provides access to:
● ObjectScale
● Licensing
● SupportAssist
● SAML Metadata
● Requests
● Platform Settings
● Security Settings
● Active Sessions
● Upgrades

ObjectScale
View or manage Object Stores, Federation, Remote Instance and ObjectScale Certificates.

Getting Started with ObjectScale 25

Licensing
View or manage one or more of the licenses that are applied to ObjectScale.

SupportAssist
View or manage the SupportAssist configuration for ObjectScale from the SupportAssist section of the ObjectScale Portal.

SAML Metadata
View or manage SAML Service Provider Metadata details for ObjectScale from the SAML Service Provider Metadata section
of the ObjectScale Portal.

Requests
View or manage requests with options to filter by Status, Request ID, Type, Requestor, and Approver.

Platform Settings
View or manage Privileged Actions Approval System and Platform Protection Mode.

Security Settings
View or manage Management Users, Authentication Providers, Approvers, and User Settings.

Active Sessions
View or manage active sessions by Management Users, Authentication Providers, and Approvers.

Upgrades
View or manage available upgrade and upgrade history.

Security configuration
A separate guide provides some configuration tasks which are intended for security administrators, whose role may be separate
from the system administrator.
The ObjectScale Security Configuration Guide provides detailed instructions for security-related tasks.

Role-based security
ObjectScale provides predefined user roles that control access to areas of the user interface and to protected operations.
Some of the functionality in this guide is reserved for particular roles and may not be accessible from every user account.
By using the predefined roles, you can limit access to ObjectScale, and the object stores and buckets by applying the principle of
least privilege.
When ObjectScale is deployed on a Red Hat OpenShift Container Platform, the OpenShift cluster administrator user can
set up end-user namespaces and users with specialized permissions for access to object stores deployed within an end-user
namespace.

26 Getting Started with ObjectScale

 3
IAM accounts and account entities
This chapter contains:
Topics:
• Introduction to Identity and Access Management
• Accounts
• Creating and mainitaining ObjectScale accounts
• Configuring ObjectScale account entities

Introduction to Identity and Access Management

In ObjectScale, Identity and Access Management (IAM) a shared service within a single ObjectScale instance used to manage
Accounts and the Account's IAM entities.
IAM provides an AWS-compatible authentication and authorization mechanism that are availed by other ObjectScale services
such as:
● Datahead (S3)
● Geoservice
● Object store management service
In this release the top-most level of the ObjectScale IAM hierarchy is an Account. Several Accounts can be defined within a
single ObjectScale instance. When an IAM account is added to an object store, that account becomes a tenant within that
object store. A tenant is a logical construct resulting from the binding of the IAM account to the object store.
Every Account has a globally unique identifier assigned to it by the IAM service at the time of creation. An IAM Account contains
other IAM entities like Users, Groups, Roles, Policies, and Service Providers associated with it. You cannot create or modify an
Account to have another Account associated with it.

NOTE: ECS Object users are not supported in ObjectScale.

In ObjectScale, each account consists of replicated IAM entities and ObjectScale local IAM entities. Local IAM entities remain
local within the ObjectScale instance and are not replicated. Global entities are replicated to other ObjectScale instances.
Replicated IAM entities and ObjectScale local IAM entities have separate APIs.
The ObjectScale instance where the Account was created initially owns that Account and is known as the Account Owner. That
account is a primary account on that ObjectScale instance. Within ObjectScale there can be only one Account Owner for any
given Account and its underlying IAM entities.
Multiple ObjectScale instances can be connected to each other forming an ObjectScale Federation. Within this federation
all ObjectScale instances have a trust relationship established with each other. Any Federation member knows about other
Federation members through ObjectScale Federation Service. Any Federation member knows about all existing Accounts across
the Federation, i.e. there is a shared Account Registry. When a primary IAM account is replicated from its ObjectScale instance
to another ObjectScale instance(s) within the federation, it becomes a secondary on these other ObjectScale instances. For
more information about federating ObjectScale instances, see Federate ObjectScale Systems.
An Account can be changed by an authorized user. The user can add, update, or delete any of the entities associated with that
account. However, such operations must always be performed on the ObjectScale instance that owns the Account. When the
IAM entity is changed, the effects of those changes may not take effect immediately.
If a user tries to change an Account from an ObjectScale that doesn't own that Account, the user will get HTTP 301 or 308
message along with the URL that corresponds to the Account Owner.
The ObjectScale Management Rest API ZIP file with the supported IAM APIs at available on the Drivers & Downloads tab
of your model and version (https://www.dell.com/support/home/en-us/product-support/product/objectscale-product-family/
overview).

IAM accounts and account entities 27

Accounts
An ObjectScale Account is a logical construct that corresponds to a customer business unit, tenant, project, and so on, which
are relevant to the account admin role and end users that belong to an account. ObjectScale users with the Admin role can
create accounts in an ObjectScale instance.
After an account is created, the Admin assigns the account to object stores. The account can then create buckets in the
object store. A dedicated object store contains only one tenant account. A shared object store can have more than one tenant
account.
After an account creates a bucket, that account owns the bucket and can assign permissions to other accounts for cross-
account access.
NOTE: An account is not required to be assigned to the object store for it to create objects in a bucket in the object store.

Also, an account can be assigned to multiple object stores.

On the Accounts page, a properly credentialed user can see a list of accounts with the following details for each account:
● Account ID
● Alias
● Labels
● ObjectScale
● Encryption
● Created On
● Status
● Description
Also, the user can perform the following Accounts actions using the ObjectScale Portal user interface:
● Create an account.
● Edit existing account.
● Enable the account.
● Disable the account.
After the user selects an ObjectScale account from the Accounts page of the ObjectScale Portal user interface, they can:
● View a Summary of the account.
● Create and manage account users.
● Create and manage account groups.
● Create and manage account policies.
● Manage account identity providers.
● Create and manage account root access keys.
● Configure notification destinations.
● View account metrics.

Creating and mainitaining ObjectScale accounts

New Accounts
This task describes how to create an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select NEW ACCOUNT.
New Account window opens.
3. Fill all the required fields in the New Account page.

28 IAM accounts and account entities

 Field Description
Alias An informal name for the new account.
Description Enter details about the new account.
Encryption Encryption is disabled by default.
Click to enable or disable encryption.

Labels Optionally, enter up to five labels on the account.

4. Click SAVE.
The new Account is saved with added authorization for buckets.

View the summary of an account

Select an account to view a detailed summary of that account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. To view details of an Account, click the name of an account that is listed in the accounts table.
The Summary tab is displayed and shows details about the account and the account data, such as
● Aggregate Metrics
● Hour Metrics

Edit Account
This task describes how to edit an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account and click EDIT.
A new window opens.
3. Edit the fields in the new window.
NOTE: All fields are not editable.

Field Edit option

Alias Alias of the account is not editable.
Description Description about the new account is editable.
Encryption Encryption is not editable.
Status Status is not editable.
Labels Labels is editable.

4. Click SAVE.
The Account is saved with updated fields.

IAM accounts and account entities 29

Enable or Disable an Account
This task describes how to enable or disable an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account and click ENABLE or DISABLE.

Delete an account
You cannot delete an account using the ObjectScale Portal user interface, accounts can only be deleted using the IAM API.

Steps
1. Follow the steps in Delete a bucket to delete a bucket and its objects.
2. Follow the steps in Delete an IAM user account to delete all the IAM user accounts from the IAM account.
3. Finally, use the IAM API to remove the IAM account.
The ObjectScale Management Rest API with the supported APIs is available at https://www.dell.com/support/home/
product-support/product/objectscale/drivers.

Configuring ObjectScale account entities

Users
In ObjectScale, an IAM User is a person or application in the account.
Use the following tasks to manage ObjectScale IAM users.

View existing IAM users within an account

Use the ObjectScale Portal user interface to view existing IAM users within an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Users tab.
The list of Users within that account appears.

Create an IAM user within an account

Use the ObjectScale Portal user interface to create an IAM user within the selected account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Users tab.
The Users list appears.
3. Click New User.
The New User window opens.
4. Fill all the required fields in the New User window.

30 IAM accounts and account entities

 Field Descriptions
Name a. Enter name of the user.
b. To go to Permissions tab, click Next.
Permissions a. You can add permissions to the new user in one of the following ways:
● Permission boundary and copy permission from an existing user to the current account.
● Add existing group of current account and permission boundary.
● Add existing policies of current account and permission boundary.
b. To go to Tags tab, click Next.
Tags You can add one or more tags to a User.
(Optional) a. Enter the details for Key and Value for a tag.
b. To go to Review tab, click Next.
Review a. Review details of the user.
b. Click CREATE USER.
Secret Keys The Secret Keys tab consists of:
● A list of users that are created along with permission, Access Key ID, and Access Secret keys.
● To download the user table in CSV format, click DOWNLOAD.CSV button.
NOTE: This is the only time that you will be able to download this CSV for this user.

If this access secret key is lost, delete and create a new secret key.

Figure 1. New User - Secret Keys tab

5. Click Complete.
A new IAM user is added in ObjectScale.

IAM accounts and account entities 31

Edit an IAM user account
Use the ObjectScale Portal user interface to edit the details of an IAM user in an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and select Users. Select a user from the user list.
The Summary tab opens by default, and is not editable. The other tabs are:
● Groups
● Permissions
● Tags
● Secret Key
3. Edit the fields.
NOTE: All fields are not editable.

Table 5. Edit User

Field Edit option
Summary ● The Summary tab opens by default, and is not editable.
● You can copy the User ARN.
Groups Groups tab consists of:
● Groups that are associated with the selected user.
● ADD GROUPS
○ To add one or more groups, select > ADD GROUPS. In the wizard that appears,
select the group to add and then click SAVE.
● REMOVE USER FROM GROUPS
○ To remove a user from one or more groups, select one or more groups from the
Group list and click REMOVE USER FROM GROUPS.
Permissions ● Permissions tab consists of:
○ MANAGED POLICIES
○ INLINE POLICY
○ BOUNDARIES
● MANAGED POLICIES tab is displayed by default.
○ To attach a policy, select:
￭ ATTACH POLICY > Copy permissions from user > User list > SAVE, or
￭ ATTACH POLICY > Policies > select one or more policies > SAVE
○ To detach a policy, select one or more policies > DETACH > SAVE
● INLINE POLICY tab allows you to:
○ ADD INLINY POLICY
○ DETACH
● BOUNDARIES tab allows you to:
○ CHANGE
○ REMOVE
Tags ● Tags tab consists of:
○ ADD TAGS
○ EDIT
○ DELETE
Secret Key ● Secret Key tab consists of:
○ ADD KEY
○ REMOVE
○ ACTIVATE
○ DEACTIVATE

32 IAM accounts and account entities

Delete an IAM user account
Use the ObjectScale Portal user interface to delete an IAM user from the selected account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and select Users. Select a user from the user list.
The Summary tab opens by default.
3. Select Secret Key
The user Secret Key tab appears and displays the keys of the user.
4. Select all the keys and click Remove.
5. Click < Users at the top of the user tab to return to the list of users.
NOTE: In order to delete a user, you should first delete the associated permissions and policies that are attached in
addition to deleting the secret key.

6. Select the IAM user from the User list to remove from the account and click DELETE.

Groups
A Group is a collection of Users. You can use groups to specify permissions for a collection of users.
Use the following tasks to manage ObjectScale IAM groups.

New Group
This task describes how to add groups to an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Groups tab.
The Groups list appears.
3. Click NEW GROUP.
The NEW GROUP window opens.
4. Fill all the required fields in the NEW GROUP window.
● Name
● Policies
5. Click SAVE.
A New Group is created for the account.

Edit Group
This task describes how to edit groups of an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Groups tab.
The Groups list appears.
3. Select a group, and click EDIT.
● The Summary tab opens, by default, and is not editable.
● The other two tabs are:
○ Users, and

IAM accounts and account entities 33

 ○ Permissions.
4. To add or remove users from the group, do the following:
● To add a user to the group, click Users > ADD USER > select one or more users > SAVE.
● To remove a user from the group, click Users > select one or more users > REMOVE > SAVE.
5. To edit permissions for the group, select Permissions.
● Permissions for groups consists of:
○ MANAGED POLICIES
○ INLINE POLICIES

Delete Groups
This task describes how to delete groups from an account with the ObjectScale Portal user interface.

Prerequisites
Before you delete a group first remove all the users who are attached to the group, along with the permissions and policies.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Groups tab.
The Groups list appears.
3. Select one more or more Groups, and click DELETE.
A confirmation window opens that displays the selected Groups to be deleted.
4. Click YES.
The selected Groups are deleted from the account.

Roles
A role is similar to a user, in that it is an identity with permission policies that determine what the identity can and cannot do.
Instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it. Also, a role
does not have any credentials (password or access keys) associated with it. Instead, if a user is assigned to a role, access keys
are created dynamically and provided to the user.
Use the following tasks to manage ObjectScale IAM roles.

New Role
This task describes how to add role to an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Roles tab.
The Roles list appears.
3. Click NEW ROLE.
The New Role window opens.

34 IAM accounts and account entities

 Figure 2. New Role - General tab

General tab is selected by default.

4. Fill all the required fields in the General tab.
● The General tab consists of:
○ Name. Name of the new role.
○ Description. Add details about the new role.
○ Session Duration. The default selected value is one hour.
5. Click NEXT.
The Trust tab opens.
6. Fill all the required fields in the Trust tab.
● The Trust tab consists of:
○ Set Effect, click Allow or Deny.
○ Account. The Account tab is selected by default.
a. To Add Principal ARN, click ADD PRINCIPAL ARN and provide the Principal ARN value to the text field.
b. To Add Service Principal, click the slider to enable.
c. To go to the Permissions tab, click NEXT.
○ SAML2.0 Federation.
a. From the drop-down menus, select:
￭ SAML Provider
￭ Attribute
￭ Value
￭ Conditions, with Key, Condition, and Value.
b. To go to the Permissions (optional) tab, click NEXT.
7. Select policies to be associated with the new rule in the Permissions tab.
● You can select one or more policies from the tab
○ All policies tab. The All policies tab is selected by default.
○ System Managed policies tab.
○ Customer Managed policies tab.
, and click NEXT.

IAM accounts and account entities 35

 ● Optionally, you can add Permissions Boundary, by enabling the Permission Boundary slider and selecting the required
policies.
8. Fill all the required fields in the Tags (optional) tab.
● The Tags (optional) tab allows you to:
○ Enter values in the Key and Value fields for a tag.
○ To delete a tag, select the tag and click Delete.
○ To add a tag, click ADD TAG, and enter the values in the Key and Value fields.
9. Click NEXT.
The Review tab opens.
10. Review all the required fields in the Review tab.
11. Click SAVE.
The New Role is saved with all the provided information and policies.

Edit Roles
This task describes how to edit roles with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Roles tab.
The Roles list appears.
3. Select a role from the list.
The Roles details for the select role appear.
● Summary
● Trust
● Permissions
● Tags
4. To edit the fields in the Summary tab, click EDIT.
● You cannot edit the Name field.
● You can only edit the Description, and Session Duration fields.
5. Click SAVE.
6. To edit the fields in the Trust tab, click Trust.
● NOTE: The ACCOUNT tab is displayed by default.

The Trust tab consists of:

○ ACCOUNT
○ SAML 2.0 FEDERATION

Table 6. Account
Field Action
Effect To edit Effect:
a. Click EDIT.
b. Select Allow or Deny.
c. Click SAVE.
Principal ARN. To add Principal ARN:
a. Click ADD PRINCIPAL ARN.
b. Add one or more principle ARN.
c. Click SAVE.
To edit Principal ARN:
a. Select a principal ARN.
b. Click EDIT.
c. Edit the principal ARN.

36 IAM accounts and account entities

 Table 6. Account (continued)
Field Action
d. Click SAVE.
To delete Principal ARN:
a. Select a principal ARN.
b. Click DELETE.
Service ARN To delete Service ARN:
a. Select a Service ARN.
b. Click DELETE.

Table 7. SAML 2.0 Federation

Field Action
SAML 2.0 FEDERATION To edit SAML 2.0 FEDERATION:
a. Select a SAML Provider.
b. Select an Attribute.
c. Select a Value.
d. Click SAVE.
To add a condition:
a. Select a ADD CONDITION.
b. Provide the condition information.
c. Click SAVE.
To edit a condition:
a. Select a condition from the table.
b. Click EDIT.
c. Modify the condition values.
d. Click SAVE.
To delete a condition:
a. Select a condition from the table.
b. Click DELETE.

7. To edit the policies that are attached to the role, click Permissions.
Select a policy from the MANAGED POLICIES, INLINE POLICIES, or BOUNDARY tab and click ATTACH POLICY or
DETACH POLICY.
8. To edit the fields in the Tags tab, click Tags.
You can ADD TAGS, EDIT, or DELETE from the selected role.

Delete Roles
This task describes how to delete roles with the ObjectScale Portal user interface.

Prerequisites
Before you delete a role, first remove all the permissions and policies that are attached to it.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Roles tab.
The Roles list appears.
3. Select one or more roles, and click DELETE.
A confirmation window opens that displays the selected roles to be deleted.
4. Click YES.
The selected roles are deleted from the account.

IAM accounts and account entities 37

Policies
IAM policies are documents in JSON format that define permissions for an operation regardless of the method that you use to
perform the operation.
The table below describes the policy types that are designed for use in ObjectScale.

Table 8. IAM Policies

Identity-based Identity-based policies grant permissions to an IAM entity to control what actions an entity (users, groups
policies of users, and roles) can perform, on which resources, and under what conditions.

In ObjectScale, resource-based policies are further categorized as:

ObjectScale managed policies Created and managed by ObjectScale. These policies cannot be modified
or deleted.
Customer-managed policies Managed policies that users create and manage in account.
Inline policies Policies that are added to a single user, group, or role.
Resource-based Attached inline policies to resources. Resource-based policies grant permissions to the principal that is
policies specified in the policy. Principals can be in the same account as the resource or in other accounts.

In ObjectScale, resource-based policies are further categorized as:

● S3 bucket policies
● IAM role trust policies
Permissions Sets the maximum permissions that an identity-based policy can grant to an IAM entity (user or role). When
boundaries you set a permissions boundary for an entity, the entity can perform only the actions that are allowed by
both its identity-based policies and its permissions boundaries. Resource-based policies that specify the
user or role as the principal are not limited by the permissions boundary. An explicit deny in any of these
policies overrides the allow.
Session policies Session policies are advanced policies that you enter a parameter when you programmatically create a
temporary session for a role. The permissions for a session are the intersection of the identity-based
policies for the IAM entity (user or role) used to create the session and the session policies. Permissions
can also come from a resource-based policy. An explicit deny in any of these policies overrides the allow.

Use the following tasks to manage ObjectScale IAM policies.

NOTE: Only customer-managed policy documents can be edited or deleted.

Create a new customer-managed policy

This task describes how to add policies to an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Click NEW POLICY.
The New Policy window opens.
4. Create the policy using the New Policy wizard.
a. On the General tab, add Name and Description for the new policy.
b. On the Editor tab, click Visual or JSON to fill out the policy editor.
Click ADD POLICY STATEMENT to add additional statements.

38 IAM accounts and account entities

 Figure 3. New Policy - Editor tab

c. On the Review tab, verify that the previewed policy statement is accurate and then click SAVE.
New policy is created for the account.

Edit a customer-managed policy

Use this task to edit customer-managed policies that are attached to an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select the customer-managed policy to modify.
The policy details are displayed, and the policy Summary tab is shown by default.
4. Edit the aspects of the customer-managed policy.
● To edit the policy permissions, select the Permissions tab.
On the Permissions tab, you can CLONE or DELETE policy statements. You can also edit the Service, Action,
Resources, and Request Condition values for this policy.

IAM accounts and account entities 39

 Figure 4. Policy Permissions tab
● To edit the policy usage, select the Usage tab.

Figure 5. Policy Usage tab

● To manage the versions of the policy, select the Versions tab.

40 IAM accounts and account entities

 Figure 6. Policy Permissions tab

Delete a customer-managed policy

This task describes how to delete a customer-managed policy from an account with the ObjectScale Portal user interface.

Prerequisites
To avoid a conflict from the deletion, you should first remove all the subordinate entities that are attached to the IAM-managed
policy before deleting the policy.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select one or more policies, and click DELETE.
A confirmation window opens that displays the selected policies to be deleted.
4. Click YES.
The selected policies are deleted from the account.

Attach a policy to an account entity

Use the ObjectScale Portal user interface to attach policies to an account entity.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select a policy, and click Usage and then select Permissions.
4. Click ATTACH
● The Attach Policy window opens, and consists of:
○ The User tab opens by default and lists all the Users in the Account.
○ The Group tab lists all the Groups in the Account.
○ The Role tab lists all the Roles in the Account.
● Any User, Group, or Role that are attached to policy are pre-selected.
5. Select one or more Users, Groups, or Roles, and click SAVE.

IAM accounts and account entities 41

 The selected User, Group, and Roles are added to the policy.

Detach a policy from an account entity

Use the ObjectScale Portal user interface to detach a policy from an account entity.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select a policy, and click Usage and then select Permissions.
4. Click Detach
The Detach Policies window opens, and displays details on the selected entity.
5. If you are sure you want to detach the displayed entity or entities, click Yes. Otherwise, click No.
The policy is now detached from all selected entities.

Attach a permission boundary to an account entity

Use the ObjectScale Portal user interface to attach a permission boundary to an account entity.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select a policy, and click Usage and then select Permissions Boundary.
4. Click ATTACH
● The Attach Permission Boundary window opens, and consists of:
○ The User tab opens by default and lists all the Users in the Account.
○ The Role tab lists all the Roles in the Account.
● User or Role that are attached to Policy are pre-selected.
5. Select one or more Users or Roles, and click SAVE.
The selected User and Roles are added to selected Policy.
6. Select User.
7. Select one or more Users and click SAVE.
The selected Users are added to selected Policy.
8. Select Role.
9. Select one or more Roles and click SAVE.
The selected Roles are added to selected Policy.

Detach a permission boundary from an account entity

Use the ObjectScale Portal user interface to detach a permission boundary from an account entity.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Policies tab.
The Policies list appears.
3. Select a policy, and click Usage and then select Permissions Boundary.
4. Click Detach
The Detach Permission Boundary window opens, and consists of the name of the entity to detach from the selected
policy.

42 IAM accounts and account entities

5. If you are sure you want to delete the permissions boundary for the entity, click Yes. Otherwise, click No.
The permissions boundary is now detached from all selected entities.

Actions in IAM Policy

This section describes all the supported actions in IAM policies that allow system account users or IAM users to perform
operations.

Actions supported for system account user

Table 9. Account
Action Description Access Resource Type Condition Keys
Level (* required)
account:* All account actions. N/A - -
account:CreateAccount Create an account in ObjectScale Write - -
instance.
account:UpdateAccount Update account configuration. Write - -
account:ListAccounts List all accounts created in all List - -
ObjectScale instances.
account:GetAccount Retrieves information about the Read - -
specified account.
account:DeleteAccount Delete the specified account. Write - -
account:AssociateAccou Associate account to object store. Write - If required, can
ntToObjectStore support below
condition key.
account:object
StoreId
account:UnassociateAcc Disassociate account from object Write - -
ountToObjectStore store.

Table 10. Grafana

Action Description Access Resource Condition Keys
Level Type (*
required)
grafana:* Grant all access for all object store * - -
operations.

Table 11. Object store

Action Description Access Resource Condition Keys
Level Type (*
required)
objectstore:* Grant access for all object store * - -
operations.
objectstore:Get* Grant read access to object store. Read - -
objectstore:Write* Grant write access to object store. Write - -

Table 12. InfluxDB

Action Description Access Resource Condition Keys
Level Type (*
required)
influxdb:* Grant all influxdb operations. * - -

IAM accounts and account entities 43

Table 12. InfluxDB (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
influxdb:Get* Grant influxdb read. Read - -
influxdb:Write* Grant influxdb write. Write - -

Table 13. Alert

Action Description Access Resource Condition Keys
Level Type (*
required)
Alert:* Allows system account user to * - -
perform all alert operations.

Actions supported for IAM identities

IAM user, role, and group support these actions.

Table 14. IAM entity management

Action Description Access Resource Condition Keys
Level Type (*
required)
iam:AddUserToGroup Adds an IAM user to the specified IAM Write group* -
group.
iam:AttachGroupPolicy Attaches a specified managed policy Permissions group* iam:PolicyARN
to the specified IAM group. management
iam:AttachRolePolicy Attaches a specified managed policy Permissions role* iam:PolicyARN
to the specified IAM role. management iam:Permission
sBoundaryiam:R
esourceTag/$
{TagKey}
iam:AttachUserPolicy Attaches a specified managed policy Permissions user* iam:PolicyARN
to the specified IAM user. management iam:Permission
sBoundary
iam:ResourceTa
g/${TagKey}
iam:CreateAccessKey Creates a new secret access Write user* -
credential for a specified IAM user.
iam:CreateGroup Creates a IAM group in the Write group* -
namespace.
iam:CreatePolicy Creates a new managed policy in the Permissions policy* -
namespace. management
iam:CreatePolicyVersio Creates a version of the specified Permissions policy* -
n managed policy in namespace. management
iam:CreateRole Creates a IAM role in the namespace. Write role* iam:Permission
sBoundary
iam:CreateSAMLProvider Creates a SAML 2.0 identity provider Write saml-provider* -
(IdP) in the namespace.
iam:CreateUser Creates an IAM user in namespace. Write user* iam:Permission
sBoundary

44 IAM accounts and account entities

Table 14. IAM entity management (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
iam:DeleteAccessKey Deletes the specified access key Write user* -
credentials that are associated with
the specified IAM user.
iam:DeleteGroup Deletes the specified IAM group from Write group* -
the namespace.
iam:DeleteGroupPolicy Deletes the specified inline policy from Permissions group* -
its group. management
iam:DeletePolicy Deletes the specified managed policy. Permissions policy* iam:PolicyARN
management
iam:DeletePolicyVersio Deletes the specified version from the Permissions policy* -
n managed policy. management
iam:DeleteRole Grants permission to delete the Write role* -
specified role.
iam:DeleteRolePermissi Deletes the permissions boundary for Permissions role* iam:Permission
onsBoundary the specified IAM role. management sBoundaryiam:R
esourceTag/$
{TagKey}
iam:DeleteRolePolicy Deletes the specified inline policy from Permissions role* iam:Permission
its role. management sBoundary
iam:DeleteSAMLProvider Deletes a specified SAML provider. Write saml-provider* -
iam:DeleteUser Deletes the specified IAM user from Write user* iam:ResourceTa
the namespace. g/${TagKey}
iam:DeleteUserPermissi Deletes the permissions boundary for Permissions user* iam:Permission
onsBoundary the specified IAM user. management sBoundaryiam:R
esourceTag/$
{TagKey}
iam:DeleteUserPolicy Deletes the specified inline policy from Permissions user* iam:Permission
its user. management sBoundary
iam:ResourceTa
g/${TagKey}
iam:DetachGroupPolicy Detach a specified managed policy Permissions group* -
from the specified IAM group. management
iam:DetachRolePolicy Detach a specified managed policy Permissions role* iam:PolicyARN
from the specified IAM role. management iam:Permission
sBoundaryiam:R
esourceTag/$
{TagKey}
iam:DetachUserPolicy Detach a specified managed policy Permissions user* iam:PolicyARN
from the specified IAM user. management iam:Permission
sBoundaryiam:R
esourceTag/$
{TagKey}
iam:GetAccessKeyLastUs Retrieves best effort information Read user* -
ed about when the specified access key
was last used.
iam:GetContextKeysForC Retrieves a list of all the context Read - -
ustomPolicy keys that are referenced in the input
policies.

IAM accounts and account entities 45

Table 14. IAM entity management (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
iam:GetContextKeysForP Detach a specified managed policy Read user, group, -
rincipalPolicy from the specified IAM entity. role
iam:GetGroup Retrieves a list of IAM users that Read group* -
are in the specified IAM group. You
can paginate the results using the
MaxItems and Marker parameters.
iam:GetGroupPolicy Gets the specified inline policy Read group* -
document from the specified IAM
group.
iam:GetPolicy Retrieves information about the Read policy* -
specified managed policy.
iam:GetPolicyVersion Retrieve information about a version Read policy* -
of the specified managed policy.
iam:GetRole Retrieves information about the Read role* iam:ResourceTa
specified role. g/${TagKey}
iam:GetPolicy Retrieves information about specified Read policy* -
managed policy.
iam:GetPolicyVersion Retrieves information about specified Read policy* -
version of the managed policy.
iam:GetRolePolicy Retrieves the specified inline policy Read role* iam:ResourceTa
document that is embedded with the g/${TagKey}
specified IAM role.
iam:GetSAMLProvider Retrieves the SAML provider Read saml-provider* -
metadata document that is associated
with the IAM SAML provider resource.
iam:GetUser Retrieves information about the Read user* iam:ResourceTa
specified IAM user. g/${TagKey}
iam:GetUserPolicy Retrieves the specified inline policy Read user* iam:ResourceTa
document of the specified IAM user. g/${TagKey}
iam:ListAccessKeys Lists information about the access List user* -
key IDs that are associated with the
specified IAM user.
iam:ListAttachedGroupP Lists all managed policies that are List group* -
olicies attached to the specified IAM group.
iam:ListAttachedRolePo Lists all managed policies that are List role* iam:ResourceTa
licies attached to the specified IAM role. g/${TagKey}
iam:ListAttachedUserPo Lists all managed policies that are List user* iam:ResourceTa
licies attached to the specified IAM user. g/${TagKey}
iam:ListEntitiesForPol Lists all entities (IAM users, groups, List policy* -
icy and roles) that are attached to the
specified managed policy.
iam:ListGroupPolicies List the names of the inline policies List group* -
that are in the specified IAM group.
iam:ListGroups List the IAM groups that have the List - -
specified path prefix.
iam:ListGroupsForUser List the IAM groups that the provided List user* iam:ResourceTa
IAM user belongs to. g/${TagKey}

46 IAM accounts and account entities

Table 14. IAM entity management (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
iam:ListPolicies Lists all managed policies that are List - -
associated with the namespace.
iam:ListPolicyVersions Lists information about the versions of List policy* -
the requested managed policy.
iam:ListRolePolicies Lists the names of the inline policies List role* iam:ResourceTa
that are in the specified IAM role. g/${TagKey}
iam:ListRoles Lists the IAM roles that have the List - -
specified path prefix.
iam:ListRoleTags Lists the tags that are attached to the List role* iam:ResourceTa
specified role. g/${TagKey}
iam:ListSAMLProviders Lists the SAML providers that are in List - -
the namespace.
iam:ListUserPolicies Lists the names of the inline policies List user* iam:ResourceTa
that are in the specified IAM user. g/${TagKey}
iam:ListUsers Lists the IAM users that have the List - -
specified path prefix.
iam:ListUserTags Lists the tags that are attached to the List user* iam:ResourceTa
specified user. g/${TagKey}
iam:PutGroupPolicy Adds or updates an inline policy Permissions group* -
document to the specified IAM group. management
iam:PutRolePermissions Sets or updates the provided managed Permissions role* iam:Permission
Boundary policy as the roles permissions management sBoundaryiam:R
boundary. esourceTag/$
{TagKey}
iam:PutRolePolicy Adds or updates an inline policy Permissions role* iam:Permission
document to the specified IAM role. management sBoundaryiam:R
esourceTag/$
{TagKey}
iam:PutUserPermissions Sets or updates the provided managed Permissions user* iam:Permission
Boundary policy as the permissions boundary of management sBoundaryiam:R
a user. esourceTag/$
{TagKey}
iam:PutUserPolicy Adds or updates an inline policy Permissions user* iam:Permission
document to the specified IAM user. management sBoundaryiam:R
esourceTag/$
{TagKey}
iam:RemoveUserFromGrou Removes an IAM user from the Write group* -
p specified group.
iam:SetDefaultPolicyVe Sets the specified version of the Permissions policy* -
rsion policy as default. management
iam:SimulateCustomPoli Simulates how IAM policies and Read - -
cy optionally a resource-based policy
works with a list of API operations and
ObjectScale resources to determine
the effective permissions of the
policy.
iam:SimulatePrincipalP Simulates how IAM policies that are Read user, group, -
olicy attached to an IAM entity (user, role

IAM accounts and account entities 47

Table 14. IAM entity management (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
group, or role) works with a list of API
operations and ObjectScale resources
to determine the effective permissions
of the policy.
iam:TagRole Adds tags to an IAM role. Tagging role* -
iam:TagUser Adds tags to an IAM user. Tagging user* -
iam:UntagRole Removes tags from specified IAM role. Tagging role* iam:ResourceTa
g/${TagKey}
iam:UntagUser Removes tags from specified IAM Tagging user* iam:ResourceTa
user. g/${TagKey}
iam:UpdateAccessKey Updates the status of specified access Write user* -
keys as Active or Inactive.
iam:UpdateAssumeRolePo Updates the policy that grants an IAM Permissions role* iam:ResourceTa
licy entity permission to assume a role. management g/${TagKey}
iam:UpdateRole Updates the description or maximum Write role* iam:ResourceTa
session duration setting of an IAM g/${TagKey}
role.
iam:UpdateSAMLProvider Updates the metadata document for Write saml-provider* -
an existing SAML provider.

Table 15. STS

Action Description Access Resource Condition Keys
Level Type (*
required)
sts:AssumeRole Returns temporary security credentials Write role*
for accessing ObjectScale resources aws:RequestT
ag/${TagKey}
that you might not have access to.
aws:TagKeys
aws:Principa
lTag/$
{TagKey}

sts:AssumeRoleWithSAML Returns temporary security credentials Write role*

for users who authenticated using a aws:RequestT
ag/${TagKey}
SAML authentication response.
aws:TagKeys
aws:Principa
lTag/$
{TagKey}
saml:iss
saml:aud
saml:sub
saml:sub_typ
e
saml:edupers
onorgdn
saml:namequa
lifier

sts:GetFederationToken Returns temporary security credentials Read user*

(consisting of an access key ID, a aws:Principa
lTag/$
secret access key, and a security
{TagKey}
token). aws:Principa
lArn

48 IAM accounts and account entities

Table 15. STS (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

aws:username
aws:userid
aws:Principa
lAccount

Table 16. S3
Action Description Access Resource Condition Keys
Level Type (*
required)
New operations supported by S3 service:
s3:GetReplicationConfi Grants permission to get the Read bucket*
guration replication configuration information s3:authType
s3:signature
set on an amazon S3 bucket.
version
s3:x-amz-
content-
sha256

s3:PutReplicationConfi Grants permission to create a Write bucket*

guration replication configuration or replace an s3:authType
s3:signature
existing one.
version
s3:x-amz-
content-
sha256

s3:DeleteReplicationCo Grants permission to delete a Write bucket* -

nfiguration replication configuration.
s3:GetBucketObjectLock Grants permission to get the object Read bucket*
Configuration lock configuration of an amazon S3 s3:authType
s3:signature
bucket.
version

s3:PUTBucketObjectLock Grants permission to get the object Write bucket*

Configuration lock configuration of an amazon S3 s3:authType
s3:signature
bucket.
version

s3:GetObjectLegalHold Grants permission to get the current Read object*

legal hold status of an object. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:PutObjectLegalHold Grants permission to apply a legal hold Write object*

configuration to a specified object. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256
s3:object-
lock-legal-
hold

IAM accounts and account entities 49

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)
s3:GetObjectRetention Grants permission to retrieve the Read object*
retention settings for an object. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:PutObjectRetention Grants permission to place an object Write object*

retention configuration on an object. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256
s3:object-
lock-mode
s3:object-
lock-retain-
until-date
s3:object-
lock-
remaining-
retention-
days

s3:BypassGovernanceRet Grants permission to allow Permission object*

ention circumvention of governance-mode Management s3:RequestOb
jectTag/
object retention settings.
<key>
s3:RequestOb
jectTagKeys
s3:authType
s3:signature
version
s3:x-amz-
acl s3:x-
amz-content-
sha256 s3:x-
amz-copy-
source s3:x-
amz-grant-
full-
control
s3:x-amz-
grant-read
s3:x-amz-
grant-read-
acp s3:x-
amz-grant-
write s3:x-
amz-grant-
write-acp
s3:x-amz-
metadata-
directive
s3:x-amz-
server-side-
encryption
s3:x-amz-
storage-
class
s3:object-
lock-mode
s3:object-
lock-retain-

50 IAM accounts and account entities

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

until-date
s3:object-
lock-
remaining-
retention-
days
s3:object-
lock-legal-
hold

Existing S3 operations supported by S3 service:

s3:AbortMultipartUploa Grants permission to cancel a Write object*
d multipart upload. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:CreateBucket Grants permission to create a bucket. Write bucket*

s3:authType
s3:signature
version
s3:x-amz-acl
s3:x-amz-
content-
sha256
s3:x-amz-
grant-full-
control
s3:x-amz-
grant-read
s3:x-amz-
grant-read-
acp
s3:x-amz-
grant-write
s3:x-amz-
grant-write-
acp

s3:DeleteBucket Grants permission to delete the Write bucket*

bucket named in the URI. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:DeleteBucketPolicy Grants permission to delete policy on a Permission bucket*

specified bucket. Management s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:DeleteObject Grants permission to remove the null Write object*

version of an object and insert a s3:authType
s3:signature
delete marker, which becomes the
version
current version of the object. s3:x-amz-

IAM accounts and account entities 51

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

content-
sha256

s3:DeleteObjectTagging Grants permission to use the tagging Tagging object*

subresource to remove the entire tag s3:ExistingO
bjectTag/
set from the specified object.
<key>
s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:DeleteObjectVersion Grants permission to remove a Write object*

specific version of an object. s3:authType
s3:signature
version
s3:versionid
s3:x-amz-
content-
sha256

s3:DeleteObjectVersion Grants permission to remove the Tagging object*

Tagging entire tag set for a specific version of s3:ExistingO
bjectTag/
the object.
<key>
s3:authType
s3:signature
version
s3:versionid
s3:x-amz-
content-
sha256

s3:GetBucketAcl Grants permission to use the ACL Read bucket*

subresource to return the access s3:authType
s3:signature
control list (ACL) of an Amazon S3
version
bucket. s3:x-amz-
content-
sha256

s3:GetBucketCORS Grants permission to return the CORS Read bucket*

configuration information set for an s3:authType
s3:signature
Amazon S3 bucket.
version
s3:x-amz-
content-
sha256

s3:GetBucketPolicy Grants permission to return the policy Read bucket*

of the specified bucket. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:GetBucketTagging Grants permission to return the tag Read bucket*

set associated with an Amazon S3 s3:authType
s3:signature
bucket.
version

52 IAM accounts and account entities

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

s3:x-amz-
content-
sha256

s3:GetBucketVersioning Grants permission to return the Read bucket*

versioning state of an Amazon S3 s3:authType
s3:signature
bucket.
version
s3:x-amz-
content-
sha256

s3:GetLifecycleConfigu Grants permission to return the life- Read bucket*

ration cycle configuration information set on s3:authType
s3:signature
an Amazon S3 bucket.
version
s3:x-amz-
content-
sha256

s3:GetObject Grants permission to retrieve objects Read object*

from Amazon S3. s3:ExistingO
bjectTag/
<key>
s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:GetObjectAcl Grants permission to return the Read object*

access control list (ACL) of an object. s3:ExistingO
bjectTag/
<key>
s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:GetObjectTagging Grants permission to return the tag Read object*

set of an object. s3:ExistingO
bjectTag/
<key>
s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:GetObjectVersion Grants permission to retrieve a Read object*

specific version of an object. s3:ExistingO
bjectTag/
<key>
s3:authType
s3:signature
version
s3:versionid
s3:x-amz-

IAM accounts and account entities 53

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

content-
sha256

s3:GetObjectVersionAcl Grants permission to return the Read object*

access control list (ACL) of a specific s3:ExistingO
bjectTag/
object version.
<key>
s3:authType
s3:signature
version
s3:versionid
s3:x-amz-
content-
sha256

s3:GetObjectVersionTag Grants permission to return the tag Read object*

ging set for a specific version of the object. s3:ExistingO
bjectTag/
<key>
s3:authType
s3:signature
version
s3:versionid
s3:x-amz-
content-
sha256

s3:ListAllMyBuckets Grants permission to list all buckets List -

owned by the authenticated sender of s3:authType
s3:signature
the request.
version
s3:x-amz-
content-
sha256

s3:ListBucket Grants permission to list some or all List bucket*

the objects in an Amazon S3 bucket s3:authType
s3:delimiter
(up to 1000).
s3:max-keys
s3:prefix
s3:signature
version
s3:x-amz-
content-
sha256

s3:ListBucketMultipart Grants permission to list in-progress Read bucket*

Uploads multipart uploads. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:ListBucketVersions Grants permission to list metadata Read bucket*

about all the versions of objects in an s3:authType
s3:delimiter
Amazon S3 bucket.
s3:max-keys
s3:prefix
s3:signature
version
s3:x-amz-

54 IAM accounts and account entities

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

content-
sha256

s3:ListMultipartUpload Grants permission to list the parts Read object*

Parts that have been uploaded for a specific s3:authType
s3:signature
multipart upload.
version
s3:x-amz-
content-
sha256

s3:PutBucketAcl Grants permission to set the Permission bucket*

permissions on an existing bucket Management s3:authType
s3:signature
using access control lists (ACLs).
version
s3:x-amz-acl
s3:x-amz-
content-
sha256
s3:x-amz-
grant-full-
control
s3:x-amz-
grant-read
s3:x-amz-
grant-read-
acp
s3:x-amz-
grant-write
s3:x-amz-
grant-write-
acp

s3:PutBucketCORS Grants permission to set the CORS Write bucket*

configuration for an Amazon S3 s3:authType
s3:signature
bucket.
version
s3:x-amz-
content-
sha256

s3:PutBucketPolicy Grants permission to add or replace a Permission bucket*

bucket policy on a bucket. Management s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:PutBucketTagging Grants permission to add tags to an Tagging bucket*

existing Amazon S3 bucket. s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:PutBucketVersioning Grants permission to set the Write bucket*

versioning state of an existing Amazon s3:authType
s3:signature
S3 bucket.
version
s3:x-amz-

IAM accounts and account entities 55

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

content-
sha256

s3:PutLifecycleConfigu Grants permission to create a life- Write bucket*

ration cycle configuration for the bucket s3:authType
s3:signature
or replace an existing life-cycle
version
configuration. s3:x-amz-
content-
sha256

s3:PutObject Grants permission to add an object to Write object*

a bucket. s3:RequestOb
jectTag/
<key>
s3:RequestOb
jectTagKeys
s3:authType
s3:signature
version
s3:x-amz-acl
s3:x-amz-
content-
sha256
s3:x-amz-
copy-source
s3:x-amz-
grant-full-
control
s3:x-amz-
grant-read
s3:x-amz-
grant-read-
acp
s3:x-amz-
grant-write
s3:x-amz-
grant-write-
acp
s3:x-amz-
metadata-
directive
s3:x-amz-
server-side-
encryption
s3:x-amz-
server-side-
encryption-
aws-kms-key-
id
s3:x-amz-
storage-
class
s3:object-
lock-mode
s3:object-
lock-retain-
until-date
s3:object-
lock-
remaining-
retention-
days
s3:object-

56 IAM accounts and account entities

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

lock-legal-
hold

s3:PutObjectAcl Grants permission to set the access Permission object*

control list (ACL) permission for an Management s3:ExistingO
bjectTag/
object that exists in a bucket.
<key>
s3:authType
s3:signature
version
s3:x-amz-acl
s3:x-amz-
content-
sha256
s3:x-amz-
grant-full-
control
s3:x-amz-
grant-read
s3:x-amz-
grant-read-
acp
s3:x-amz-
grant-write
s3:x-amz-
grant-write-
acp
s3:x-amz-
storage-
class

s3:PutObjectTagging Grants permission to set the supplied Tagging object*

tag-set to an object that exists in a s3:ExistingO
bjectTag/
bucket.
<key>
s3:RequestOb
jectTag/
<key>
s3:RequestOb
jectTagKeys
s3:authType
s3:signature
version
s3:x-amz-
content-
sha256

s3:PutObjectVersionAcl Grants permission to use the ACL Permission object*

subresource to set the access control Management s3:ExistingO
bjectTag/
list (ACL) permissions for an object
<key>
that exists in a bucket. s3:authType
s3:signature
version
s3:versionid
s3:x-amz-acl
s3:x-amz-
content-
sha256
s3:x-amz-
grant-full-
control
s3:x-amz-
grant-read
s3:x-amz-

IAM accounts and account entities 57

Table 16. S3 (continued)
Action Description Access Resource Condition Keys
Level Type (*
required)

grant-read-
acp
s3:x-amz-
grant-write
s3:x-amz-
grant-write-
acp
s3:x-amz-
storage-
class

s3:PutObjectVersionTag Grants permission to set the supplied Tagging object*

ging tag-set for a specific version of an s3:ExistingO
bjectTag/
object.
<key>
s3:RequestOb
jectTag/
<key>
s3:RequestOb
jectTagKeys
s3:authType
s3:signature
version
s3:versionid
s3:x-amz-
content-
sha256

s3:ReplicationInfo Grants permission to retrieve an Read object* -

object replication status.
s3:PutBucketLogging Grants permission to configure bucket Permission bucket* -
logging on a source bucket. Management
s3:GetBucketLogging Grants permission to retrieve bucket Permission bucket* -
logging on a source bucket. Management

Table 17. DCM

Action Description Access Resource Condition Keys
Level Type (*
required)
dcm:GetWebhookConfigur Retrieves a webhook configuration to Read webhook* Dependent actions
ation which notifications can be published.
dcm:CreateWebhookConfi Creates a webhook configuration to Write webhook* Dependent actions
guration which notifications can be published.
dcm:DeleteWebhookConfi Deletes a webhook configuration. Write webhook* Dependent actions
guration
dcm:ListWebhookConfigu Retrieves a list of webhook List webhook* Dependent actions
rations configurations of the requester.
Each call returns a limited list of
configurations, up to 100.

58 IAM accounts and account entities

Principal types in IAM Policies
This section lists all the supported principal types in IAM policies that allow system account users or IAM users to perform
operations.
NOTE: The principal element is not used in IAM identity-based policies. You can use principal elements in trust policies for
IAM roles and in resource-based policies (bucket policy and trust policy).

Table 18. Principal Types in IAM Policies

Principal Type Format
Root user
"Principal": { "AWS": "urn:osc:iam::<account-ID>:root" }

IAM users
"Principal": { "AWS": "urn:osc:iam::<account-ID>:user/<user-
name>" }

Federated users (using

SAML federation) "Principal": { "Federated": "urn:osc:iam::<account-ID>:saml-
provider/<provider-name>" }

IAM roles
"Principal": { "AWS": "urn:osc:iam::<account-ID>:role/<role-
name>" }

Assumed-role sessions
"Principal": { "AWS": "urn:osc:sts::<account-ID>:assumed-role/
<role-name>/<role-session-name>" }

Services
"Principal": { "Service": "<service-name>" }

Anonymous users
"Principal" : { "AWS" : "*" }

Identity Provider
An identity provider (IdP)is a trusted provider that lets you use single sign-on (SSO) to access other websites. With an
identity provider, you can manage the identities of account users outside of ObjectScale and give these external user identities
permissions to use ObjectScale resources in an account.
Security Assertion Markup Language 2.0 (SAML) is an open federation standard that allows an identity provider to authenticate
users and pass identity and security information about them to a service provider (SP), typically an application or service.
Currently, ObjectScale only supports the SAML identity provider.
An identity provider always belongs to an account.

View a list of identity providers in an account

ObjectScale users can view all identity providers configured for an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Identity Provider tab.
The Identity Provider list appears displaying:
● all of identity providers configured for the selected account.
● the following identity provider details:

IAM accounts and account entities 59

 ○ Name - Displays the name of the identity provider.
○ Type - Displays the type of the identity provider.
○ Created On - Displays the date the identity provider was added to the account.

Add a new identity provider to an account

This task describes how to add an identity providers to an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Identity Provider tab.
The Identity Provider list appears.
3. Click New Identity Provider.
The New Identity Provider window opens.
4. Fill all the required fields in the New Identity Provider window.
● Name - Identity Provider name can be a combination of up to 128 letters, digits, and period (.), underscore (_), and
hyphen (-). Consecutive supported special characters are allowed.
● Type - Identity Provider type, supports SAML.
● Metadata Provider - An XML document generated by an identity provider (IdP) that supports SAML 2.0. The document
includes the name of issuer, expiration information and keys that can be used to validate the SAML authentication
response (assertions) that are received from the Identity Provider. You must generate the metadata document using the
identity management software that is used as your organization's Identity Provider.
5. Click SAVE.
A New Identity Provider is created for the account.

Edit the details of an identity providers added to an account

You can edit one or more identity providers in an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Identity Provider tab.
The Identity Provider list appears.
3. Select an identity provider from the list and click Edit Identity Provider.
The Edit Identity Provider window opens.
4. Edit the identity provider using the Edit Identity Provider window.
● Name is not editable.
● Type is not editable.
● Metadata Provider - You can only edit an identity provider metadata provider by uploading new metadata file.
5. Click SAVE.
The identity provider is updated for the account.

Delete an identity provider from an account

You can delete one or more identity providers from an account with the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Identity Provider tab.
The Identity Provider list appears.

60 IAM accounts and account entities

3. Select an identity provider and click Delete Identity Provider.
If desired, you can select multiple identity providers to be deleted at the same time. A confirmation window opens that
displays the selected identity providers to be deleted.
4. Click YES.
The selected identity provider has been deleted from the account.

Root Access Keys

Use the following tasks to manage ObjectScale IAM Account's root access key(s).
Dell strongly recommends that you do not use the root user for everyday tasks. Safeguard the root user credentials and use
them to perform only the tasks that the root user can perform.

Create a Root Access Key

Use the ObjectScale Portal user interface to generate a new key for an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Root Access Key tab.
The Root Access Key list appears.
3. Click ADD KEY.
The Add Secret Key window opens.
4. Click GENERATE.
5. Review the new secret key.
NOTE: This is the only time that the secret access keys can be viewed or downloaded. You cannot recover them later.
However, you can create access keys at any time.

● Access Key ID

● Access Secret Key - Optionally click

● Click DOWNLOAD
6. Click OK and optionally view the new key in the Root Access Key table.

Manage existing Root Access Keys

Use the ObjectScale Portal user interface to activate, deactivate, or remove a key in an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Select an account from the account list and then select the Root Access Key tab.
The Root Access Key list appears.
3. Select the Root Access Keys to activate, deactivate, or remove.
● Click DEACTIVATE and then in the Deactivate access key(s) window and confirm the keys to deactivate and click
Yes.
If these are not the correct keys to deactivate, click No.
● Click ACTIVATE and then in the Activate access key(s) window and confirm the keys to activate and click Yes.
If these are not the correct keys to activate, click No.
● Click REMOVE and then in the Remove access key(s) window and confirm the keys to activate and click Yes.
If these are not the correct keys to remove, click No.

IAM accounts and account entities 61

Notification Destinations
Use the following tasks to manage notification destinations.

Create a notification destination

Use the ObjectScale Portal user interface to add a notification destination to an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Click an account from the account list and then click the Notification Destination tab.
The Notification Destination list appears.
3. Click New Notification Destination.
The New Notification Destination window opens.
4. Complete the required fields to create a notification destination:

Option Description
Name Type the name of the new destination.
Comment Type the description of the destination.
Endpoint Type the valid URL for the destination endpoint.
Authentication Token Type the authentication token value for the endpoint.
Backup Limit Select a backup limit for the first 100 destinations.
5. Click SAVE.

Edit a notification destination

Use the ObjectScale Portal user interface to edit an existing notification destination in an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Click an account from the account list and then click the Notification Destination tab.
The Notification Destination list appears.
3. Select a notification destination from the list and then select the Edit tab.
The Edit window appears.
4. Modify a destination value to modify the notification destination:

Option Description
Name Type the name of the new destination.
Comment Type the description of the destination.
Endpoint Type the valid URL for the destination endpoint.
Authentication Token Type the authentication token value for the endpoint.
Backup Limit Select a backup limit for the first 100 destinations.
5. Click SAVE.

62 IAM accounts and account entities

Delete a notification destination
Use the ObjectScale Portal user interface to delete an existing notification destination in an account.

Steps
1. From the ObjectScale Portal user interface, click Accounts.
The list of Accounts that the user is authorized to view is displayed.
2. Click an account from the account list and then click the Notification Destination tab.
The Notification Destination list appears.
3. Select a notification destination from the list and then select the Delete tab.
The Delete window appears.
4. Confirm that the correct notification destination is selected and click OK.

IAM accounts and account entities 63

 4
Object stores
This chapter contains:
Topics:
• About ObjectScale object stores
• Creating and maintaining object stores
• Managing object stores

About ObjectScale object stores

ObjectScale introduces the new logical construct that is called the object store. Object stores are a discrete storage system
with an individualized lifecycle and are Kubernetes (k8s) applications deployed by ObjectScale.
One ObjectScale instance can contain multiple object stores. In an ObjectScale instance, one object store can be in NVMe.
Object stores are created, updated, and deleted independently from all other object stores managed by ObjectScale. As you
manage the object store through its life cycle, ObjectScale interacts with the underlying Kubernetes infrastructure as needed.
ObjectScale allows Kubernetes to handle the necessary changes to the cluster resources, such as storage, CPU, and other
resources.
You must associate an IAM account with an object store in order to allow users within the IAM account to manage aspects of
the object store. When an IAM account is added to an object store, that account becomes a tenant within that object store. A
tenant is a logical construct resulting from the binding of an IAM account to an object store.
After associating the IAM account with the object store,
● IAM accounts associated with the object store can create buckets in the object store.
● the account that created the bucket then owns that buckets. This is similar to AWS S3.
● the tenants can be used to specify quota restrictions for that account in that object store.
● the user can set specific compliance settings.
● the user can set specific retention policies.
The size of the persistent volumes (PVs) which are bound to the storage server (SS) pods or NVMe engine in an object store
represent the persistent storage capacity allocated for raw user data. An object store with three 200GB SS pods provides
600GB raw disk space. ObjectScale joins the persistent volumes and pods to hosts to protect data using erasure coding. Each
object store has a maximum of one SS pod per k8s worker node.
To simplify the creation of object stores of the correct size and resource profile, ObjectScale uses a workload sizer tool
within the workflows for creating or modifying an object store. Administrators and end users, with the appropriate permissions,
can choose the correct level of resources for the object store. ObjectScale then dynamically calculates the object store
requirements as you enter the required values for the object store. Supply the initial object store capacity to allow the sizing tool
to correctly calculate the capacity requirements.
To support the workload inputs provided, ObjectScale then determines:
● The number of replicas in the object store.
● PV size necessary to meet the storage needs for the life of the object store.
● Any additional capacity must cover overhead, such as metadata and data protection.

64 Object stores
Figure 7. New Object Store

In addition to the number and size of storage server instances that are required for an object store, ObjectScale also determines
the size and quantity of all components that make up the object store. ObjectScale uses performance profiles to size object
store resources. This release of ObjectScale includes the large performance profile.
NOTE: ObjectScale may be unable to create the requested object store if certain pods are unable to start. In these cases,
ObjectScale creates an alert that provides details about the pod and resource that are not available. You can view the alert
in the Alerts section of the ObjectScale Portal user interface. Use the details in the alert to resolve the underlying issue or
cancel the operation and remove any partial pod that was created before the failure.

Object store naming conventions

This topic details the rules that apply to the naming of ObjectScale object stores.

Object store naming

The following rules apply to the naming of object stores in ObjectScale:
● Object store names are required to be between three and 31 characters in length.
● Object store names can consist only of lowercase letters, numbers and hyphens (-).
● Object store names must begin with an alphabetic character, and end with an alphanumeric character.
● Object store names should be unique. Do not use the same name for two or more object stores within different namespaces
in the ObjectScale instance.

Group label naming

Use this value to place a logical grouping construct on your object store. For example, "development" can be used to filter all
your non-production object stores. The following rules apply to the group label in ObjectScale:
● Group can be up to 63 characters long.
● Group can consist only of alphanumeric characters, hypen (-), underscore ( _ ), and dot (.)
● Group must begin and end with an alphanumeric character.

Object stores 65
Creating and maintaining object stores
Creating a new object store
Create a new object store, then add an ObjectScale account to the object store to enable account entities to create buckets
within the object store.

Create an object store

About this task
Each object store is a unique and independent storage system with an individualized lifecycle. One or more object stores are
deployed by each ObjectScale instance. Object stores are created, updated, and deleted independently from all other object
stores managed by the shared ObjectScale instance.
From ObjectScale Portal, use the New Object Store wizard to set the initial object store resource requirements based on the
information collected to satisfy the demands specified by the administrator.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click New Object Store.
The New Object Store wizard appears.
4. In the General configuration page complete the required fields.

Option Description
Name Enter the Name for the object store. You must verify the name by clicking the VERIFY button next to
the name.
Namespace Verify that you are creating the object store in the correct namespace.
Version Select the Version. This logically tags the object store to allow filtering via a group.
Group Enter the Group value(s) to apply to the object store. Use the version to logically group the object
stores and to filter the object stores with the group.
The Group field is optional.

My Templates If previously created, select a template to populate the configuration values for the new object store.
The My Templates field is optional.

Data protection ObjectScale is the only supported data protection type.

type
User Storage Select the storage class to be used to provision the user storage for object store and bucket metadata.
Class
System Storage Select the storage class to be used to provision the management service storage for metadata, service
Class registration, and metrics data.
It is recommended to choose System Disk Logical Volumes for System Storage Class. Select other
options only if there is an extra disk available to hold the System meta data. Please note that one whole
disk of the selected storage type will be utilized for LVG allocation.

5. Click Quick or Advanced to continue with the creation of the new object store.
● Click Quick and set the Requested Raw Capacity for the new object store in the Storage And Review section of the
wizard.
ObjectScale will make workload selections based on this requested value. Optionally, expand and review the selected
values, and set any optional labels or other values, at this time.

66 Object stores
 Afterwards, click Save to create the object store, which will take approximately 15 to 20 minutes to become complete.
● Click Advanced and go through the next steps to manually create the object store.
6. Optional: Complete the Labels page.
● Type the label name in the Name field. The name can be 63 characters or less. It must begin and end with an
alphanumeric character. Names can contain dashes (-), underscores (_), dots (.) and alphanumeric values in between.
● Value is an optional field and can be 63 characters or less. It must begin and end with an alphanumeric character. Values
can contain dashes (-), underscores (_), dots (.) and alphanumeric values in between.
When adding more than one label, click ADD LABEL. You can add up to five labels to an object store.

7. Complete the Topology configuration page.

Review the available resources and select the desired topology scheduling to apply.
To enable node exclusion using topology labels, select Advanced and select the nodes to exclude while creating the object
store. You can filter the nodes by selecting the Source, Key, and Value of the desired nodes to exclude.

8. In the Storage configuration page, set desired Storage values and click Next to continue.
In this step define the following storage values for the new object store:
● Requested Raw Capacity for the object store. The requested raw capacity cannot be greater than the currently
licensed capacity for ObjectScale.
● Storage Server Replica(s) for the object store. The number of storage server replicas should be less than or equal
to maximum storage server replica count and available storage server replicas count. When capacity is incremented, the
storage server replica count may increase.
● Volumes per Storage Server Replica for the object store.
Based on these values, ObjectScale will configure the remaining fields within the Storage tab.

9. Optional: Complete the Connectivity configuration page and click Next.

● Select Automatic Network Configuration to allow ObjectScale to automatically configure networking with Kubernetes
internally signed certifications.
● Select Advanced Network Configuration to customize the network interface types and TLS certificate generation
details.
Complete the required connectivity values for the object store S3, Management, and Replication tabs.
For each object store connectivity service, set the Service type and then the associated connectivity values.
Additionally for the S3 and Management services, define the certificate type that each service will use.
NOTE: An object store will not support Bucket Replication features if ClusterIP is selected as the Service type for
the Replication Service.
10. NOTE: If necessary, click Edit to modify any of the values for the object store that have been incorrectly set.

NOTE: Additionally, if you wish to save a portion of the object store settings as a template for future use, simply add a
name to the Save as template field at the bottom of this page of the wizard.

Figure 8. Creating a template

Finally, use the Review page to review the values to be used for configuring the new object store and click Save.
The object store creation process begins and will take approximately 15 to 20 minutes to become available.
11. Optional: Refresh the UI to observe the various states of Health for the object store during the creation process such as
Initializing, Starting, and Provisioning. The process is complete when the Health of the object store becomes Avaliable.
During object store creation, new pods for the object store become visible in the selected namespace.

Results
The new object store has been created. Before you can create buckets or use this object store, you must associate this object
store with an IAM account. See Associate an account with an object store for more information.

Object stores 67
Associate an account with an object store
This task describes how to associate an account with an object store in the ObjectScale Portal user interface.

About this task

After adding an account to an object store that account becomes a tenant within that object store. A tenant is a logical
construct resulting from the binding of an account to an object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Select the object store to add the tenant account to by clicking the object store name.
4. Select the Accounts tab.
The Accounts list appears displaying the accounts currently associated with the object store.
5. Click Add.
The Add Account to Object Store: <OBJECT_STORE_NAME> wizard is displayed.

Figure 9. Add an account to an object store

6. Complete the Add Account to Object Store wizard and click Save.
This process consists of:
● Select the Account ID, which will be the tenant to this account to be added to the object store.
● Type an Alias for this account.
● Enable/disable Encryption, as desired. By default the encryption status will show the account's encryption status.
● Default Bucket Quota limit for the account in the object store.

68 Object stores
 ● Set the Block writes at Quota limits for which writes must be blocked.
● Set the Notification at Quota. This is the quota at which a notification should be sent out. This can be set by providing
a quota value in the input box or as percentage of block writes at quota by selecting appropriate % from the drop-down.
The selected tenant account is now associated with the object store and is a tenant of the object store.

Edit the settings of an object store

After creating an object store, you can later modify certain settings of that object store if required. These editable object store
settings include labels, storage, and connectivity values.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Select the object store to edit by clicking the checkbox to the left of the object store name.
4. Click Edit.
The Edit Object Store wizard appears.
5. Go to the section in the Edit Object Store wizard containing the value of the object store to modify.
From this wizard, you can:
● Click General to edit the general settings of the object store. The only value that can be edited here is the Group value.
● Click Labels to edit the labels on the object store. A maximum of five labels can be added to an object store.
● Click Topology to view topology of the object store. You cannot modify the topology of an existing object store.
● Click Storage to expand the object store, either horizontally or vertically.
NOTE: See Horizontally expand the capacity of an object store and Vertically expand the capacity of an object store
for more detailed instructions on expanding an object store.
● Click Connectivity to modify the object store connectivity values.
6. Once complete, click Save to save the changes to the object store.

Delete an object store

About this task
Deleting the object store using the ObjectScale Portal user interface automatically deletes the storage (persistent volume claims
[PVCs]) that are associated with it. The volumes and file systems are removed from the back-end storage as a result. Deleting
an object store will not invoke garbage collection.
CAUTION: Deleting an object store deletes all associated user data, and the delete action is irreversible.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Select the object store to delete by clicking the checkbox to the left of the object store name.
4. Click Delete and confirm in the dialog box that appears.
The object store disappears from the table before all the resources associated with it (pods, PVCs, volumes, so on) have
been deleted. The removal operations continue in the background for a few minutes (depends on the size of the store).
Removal can be monitored with kubectl (for example, using get pods).

Object stores 69
Managing object stores
View the Summary of an object store
About this task
The object store Summary page in the ObjectScale Portal user interface displays an at-a-glance view of the details about the
configuration of the selected object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store that you want to review.
The Object Store Summary page is displayed. Here you can review the high-level details about the select object store:
● General details
● Storage Classes
● Storage details
● S3 Service details
● Management Service details
● Replication Service details

View the Dashboard of an Object Store

About this task
The object store Dashboard page in the ObjectScale Portal user interface displays an at-a-glance view of the details about the
performance, health, and capacity usage details of the selected object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store that you want to review.
The object store Summary page is displayed.
4. Click Dashboard.
The object store Dashboard page is displayed. Here you can review the high-level details about the select object store,
including:

Table 19. Object store dashboard sections

Section Description
Performance history graph Displays Latency against Time with options to view the data
for the following actions:
● Read First Byte(p50)
● Write Last Byte(p50)
● Read First Byte(p99)
● Write Last Byte(p99)
You can apply timeframe filters for the performance graph
with the following options:
● Last 24 hours
● Last 7 days
● Last 30 days
● Custom Range

70 Object stores
Table 19. Object store dashboard sections (continued)
Section Description
The following information is displayed right below the graph:
● Bucket Count
● Object Count
● Deleted Object Count(24 hr)
● Replication Object Count
● Compression Ratio
Health Displays the Object store system alerts based on severity:
● Critical
● Error
● Warning
Click the link in the row to view the related alerts.
Capacity Utilization Displays all capacities at the Object store level, which
includes:
● Physical used - Sum of allocated capacity for all
partitions
● Available - Capacity available
● Reserved - Capacity reserved
● Total - Total capacity consumed by all partitions of
Storage Server Replicas
● % Full - Percentage of total used capacity by total
physical capacity
● Days till Full (Est) - Estimated number of days remaining
for object store to use 80% capacity
Physical User Data Displays the physical capacity used for repository chunks
holding data uploaded by Object store users. The data is
available based on the following parameters:
● Local Data - The hard drive capacity consumed to store
the injected data
● Replica Data - The hard drive capacity consumed to
store the injected data
● Offline Capacity Available - Unrecovered offline data
capacity of the object store as a quantity
● Offline Capacity Recovered - Recovered offline data
capacity of the object store as a quantity
System Data Displays the capacity used by the ObjectScale processes
that track and describe the data in the system. The various
categories are:
● Data Protection - Protection overhead
● Metadata - System and user metadata
● Metadata Protection - Protection of overhead metadata
● Data pending for EC - Data in system pending EC
protection
● Rate of EC per second - Rate of data being erasure
coded per second
Data Management Displays reclaimed and reclaimable capacity in the
ObjectScale level, with the following options:
● Data being reclaimed - Data currently processed
● Unreclaimable metadata - Garbage for system data that
cannot be reclaimed
● Unreclaimable user data - Garbage for user data that
cannot be reclaimed
● Reclaimable metadata - Capacity of system data pending
to be reclaimed

Object stores 71
 Table 19. Object store dashboard sections (continued)
Section Description
● Reclaimable user data - Capacity of system data pending
to be reclaimed
● Capacity reclaimed - Capacity reclaimed by removing
garbage data

Understand object store space reclamation

Space reclamation (garbage collection) is an automatic process in ObjectScale.
ObjectScale uses the following thresholds to maximize processing and space efficiency:
● When entire objects are deleted, space reclamation is automatic.
● In an object store that contains all large objects (all objects are equal to or larger than 100KB), then automatic space
reclamation is triggered when garbage data is more than 50% of used capacity.
● In an object store that contains a mix of large and smaller objects, but mostly larger objects, automatic space reclamation
is triggered when garbage data is more than 50% of the used capacity. The smaller objects must consume less than 1/3 *
128MB of space.
● In an object store that contains mostly smaller objects (that is, the smaller objects consume more than 1/3 * 128MB),
automatic space reclamation is triggered when garbage data is more than 66% of the used capacity.
● The thresholds assume that consumed capacity is above 10% total capacity.
If your data usage, object sizes, and delete patterns do not fall into the above categories, then space reclamation is not
automatic. For example, if your object store contains many small objects, space reclamation may not occur.
If you experience capacity problems because space reclamation does not occur automatically, contact customer support.
Customer support may perform manual space reclamation if your system meets certain criteria. The criteria include suitable CPU
and system load availability, throughput capability, and enough free capacity.

Set capacity alerts for an object store

Capacity alerts are triggered once the object store consumption reaches the selected percentage of available capacity allowing
you to make the necessary modifications to the object store contents or sizing before total capacity is reached.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click on the name of the object store.
4. Click Capacity Alerts.
The Capacity Alerts page appears.
5. Enable or disable the Critical Alert setting. If you enable critical alerts, you must set the Critical Threshold percentage at
which to send the alert.
A critical alert is triggered once the object store consumption reaches the selected percentage of available capacity.
6. Enable or disable the Warning Alert setting. If you enable warning alerts, you must set the Warning Threshold percentage
at which to send the alert.
A warning alert is triggered once the object store consumption reaches the selected percentage of available capacity.
7. Once complete, click Save to save the changes to the object store capacity alert settings.
Or, click RESET undo any unsaved changes to the object store capacity alerts.

72 Object stores
Managing accounts associated with object stores

Add additional accounts to an object store

Multiple accounts can be added to an object store. After adding an account to an object store that account becomes a tenant
within that object store. A tenant is a logical construct resulting from the binding of an account to an object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
The Summary tab of the selected object store appears.

4. Select the Accounts tab.

The Accounts list appears displaying the accounts currently associated with the object store.
5. Click Add.
The Add Account to Object Store: <OBJECT_STORE_NAME> wizard is displayed.
6. Complete the Add Account to Object Store wizard using the details of the account you are adding to the object store.
● Select the Account ID, which will be the tenant to this account to be added to the object store.
● Type an Alias for this account.
● Enable/disable Encryption. If enabled all buckets created in this account will be encrypted, if encryption is supported by
your ObjectScale license. If set to Disabled, you can still encrypt a bucket at the bucket level when you create the bucket
in the object store.
NOTE: You cannot change this value after you add the account to the object store.
● Default Bucket Quota limit for the account in the object store.
● Set the Notification at Quota. This is the quota at which a notification should be sent out. This can be set by providing
a quota value in the input box or as percentage of block writes at quota by selecting appropriate % from the drop-down.
● Set the Block writes at Quota limits for which writes must be blocked.
7. Click Save.
The selected account is now associated with the object store and is a tenant of the object store.

Edit an account values for an object store

Using the ObjectScale Portal user interface, you can change the configured alias and quota values for an account that has been
added to an object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
The Summary tab of the selected object store appears.

4. Select the Accounts tab.

The Accounts list appears displaying the accounts currently associated with the object store.
5. Select the account to modify and click Edit.
The Edit Account: <OBJECT_STORE_NAME> wizard is displayed.
6. Modify one or more account value(s).
You can change these values:
● The Alias for this account.
● The Default Bucket Quota limit for the account in the object store.
● Tthe Block writes at Quota limits for which writes must be blocked.

Object stores 73
 ● The Notification at Quota. This is the quota at which a notification should be sent out. This can be set by providing a
quota value in the input box or as percentage of block writes at quota by selecting appropriate % from the drop-down.
7. Click Save.

Remove an account from an object store

Using the ObjectScale Portal user interface, you can remove an account from an object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
The Summary tab of the selected object store appears.

4. Select the Accounts tab.

The Accounts list appears displaying the accounts currently associated with the object store.
5. Select the account(s) to remove and click Remove.
6. In the popup that appears, confirm that you have selected the correct account(s) to remove from the object store and then
click Yes. Otherwise, click No.

View the certificates for an object store

Using the ObjectScale Portal user interface, you can view the certificate details for an existing object store.

About this task

To view the properties of an object store certificate:

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
The Summary of the selected object store appears.

4. Click the Certificates tab.

The Certificates tab appears and consists of S3, Management, and Replication Reciever sections.
Each section of the Certificates tab shows details on the certificate for each of the object store services, including
certificate issuer, signing, and expiration details.

Monitor and manage replication for an object store

Using the ObjectScale Portal user interface, you can monitor and manage ObjectScale Replication for an object store.

Prerequisites
● The object store must not contain any buckets with replication rules configured.
● The user must have access to an object store that contains source buckets with replication configured.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.

74 Object stores
3. Click the name of the object store.
4. Click Replication.
5. To view summary information about replication of this object store, click the down arrow next to Replication Metrics.
The cards that appear show replication metrics aggregated across all destination object stores.
● Data Out (48 hours)
● Data yet to be replicated (24 hours)
● Failed Objects Size (48 hours)
● Failed Objects Count
● Replicated Delete Marker Count
● Delete Marker Failed for Replication
● Delete Marker Pending Replication
6. To manage replication destinations, scroll past the data cards.
The table is a list of object stores that are configured as replication destinations. You can manage these object stores in the
following ways:
● THROTTLE: This operation limits the replication rate from source object store to selected object stores.
● UNTHROTTLE: This operation removes the limit on replication rate from source object store to selected object stores.
● PAUSE: This operation pauses the replication from source object store to selected object stores for a certain duration.
● SUSPEND: This operation suspends the replication from source object store to selected object stores.
● RESUME: This operation recovers from both PAUSE and SUSPEND.
7. Select an object store.
● If the selected object store is not in the paused or suspended state, the THROTTLE, UNTHROTTLE, PAUSE, and
SUSPEND buttons are enabled.
● If the selected object store is in the paused or suspended state, the RESUME buttons are enabled.
8. Click a button.
● THROTTLE, or
● UNTHROTTLE, or
● PAUSE, or
● SUSPEND, or
● RESUME.

Table 20. Working with ObjectScale Replication at Object Store

Action Result
Click THROTTLE > SAVE. ● The user is allowed to throttle the data that is being
replicated to the selected object store by supplying a
single numeric value that represents MB/s.
● CANCEL and SAVE buttons are enabled.
Click UNTHROTTLE > SAVE. ● The user is allowed to unthrottle the data that is being
replicated to the selected object store by supplying a
single numeric value that represents MB/s.
● CANCEL and SAVE buttons are enabled.
a. Click PAUSE, and fill the required fields. a. ● An estimate of the overhead that might be incurred
b. Click APPLY. by the pause operation is displayed.
● The APPLY button is enabled after you click
the checkbox acknowledging the understanding of
overhead incurred.
● The CANCEL button is enabled.
b. ● The replication data flowing to the object store is
paused.
● Objects that are created in the source bucket during
the pause duration are replicated upon resume.
● The status column in the object store row changes to
PAUSED.
Click SUSPEND > YES. ● The replication data that is flowing to the object store is
suspended.

Object stores 75
 Table 20. Working with ObjectScale Replication at Object Store (continued)
Action Result
● Objects that are created in the source bucket during the
suspend duration are not replicated upon resume.
● The status column in the object store row changes to
SUSPEND.
Click RESUME > YES. ● The object stores that are either paused or suspended
resumes being replicated to the destination buckets.
● The status column in the object store changes to
Running.

NOTE: For details on the ObjectScale Replication Control APIs available in this release, see the ObjectScale REST API
ZIP file posted at https://www.dell.com/support/home/product-support/product/objectscale/drivers.

View the health of an object store

The ObjectScale Portal user interface shows health and alerts for an object store.
To view health for a specific object store in the ObjectScale Portal user interface, go to Administration > ObjectScale >
Object Stores > <OBJECT_STORE_NAME> > Health.
The Health page displays the full list of current health alerts and health events for the selected object store.
The Alerts tab displays issues for the selected object store. There are two categories of health alerts: Auto or Manual.
● Auto alerts are generated within the product when a component does not behave as expected. These alerts are cleared
automatically when the problem is resolved.
● Manual alerts are not cleared until a user acknowledges them. You can use the ACKNOWLEDGE or UNACKNOWLEDGE
buttons to manage manual health issues.
The health Logs tab shows the full list of current logged events.
The Health Check tab allows you to perform health checks on the object store.
● To perform a check on the health of an object store, select healthcheck and click Check Health.
● On ObjectScale for Red Hat OpenShift only, you can perform a preupdate health check of the object store before updating
the object store. Select pre-update and click Check Health.
● To perform a health check of the object store following an upgrade, select Check Health.

Metrics
The Metrics tab opens the object store metrics in pre-configured Grafana dashboard(s).
For details about ObjectScale and object store metrics dashboards, see Metrics for ObjectScale and object stores.
Also, see Grafana for basic details of navigation in Grafana dashboards.

76 Object stores
 5
Buckets
This chapter contains:
Topics:
• About ObjectScale buckets
• Creating and managing buckets using ObjectScale
• Configuring bucket entities

About ObjectScale buckets

Buckets are object containers that are used to control access to objects. ObjectScale supports bucket-to-bucket replication of
the objects within a bucket.
In S3, object containers are called buckets and this term has been adopted as a general term in ObjectScale. In ObjectScale,
buckets are limited to S3 only.

Bucket and object naming conventions

This topic details the rules that apply to the naming of ObjectScale buckets and their objects.

Bucket names
The following rules apply to the naming of buckets in ObjectScale:
● Bucket names are required to be between three and 63 characters in length.
● Bucket names can include dots (.), hyphens (-), lowercase letters, and number characters ([a-z, 0-9]).
● Bucket names must begin and end with a number or lowercase letter characters ([a-z, 0-9]).
● Bucket names cannot be formatted as an IP address.

Object names
The following rules apply to the naming of ObjectScale S3 objects:
● Cannot be null or an empty string
● Length range is 1..255 (Unicode character)
● No validation on characters.

Bucket versioning and Object Lock

Bucket versioning controls whether multiple versions of objects are stored, or a single null version exists for an object. Object
Lock protects a version from unintended changes.

ObjectScale extends the standard S3 protocol to support Object Lock for all bucket versioning states. In ObjectScale, when
Versioning is set to ENABLED, Object Lock protects versions. When Versioning is OFF or SUSPENDED, Object Lock operates
at the object level, protecting the current null version.

Buckets 77
S3
The following features are defined in the standard S3 protocol:
● Bucket versioning—Versioning keeps multiple variants of an object in the same bucket. Versioning is used to preserve,
retrieve, and restore every version of every object that is stored in the S3 bucket.
● Object Lock—Object Lock protects versions from unintended overwrites and accidental deletions. In the S3 protocol, Object
Lock requires versioning to be enabled. The locking features apply to individual versions of an object and each version can
have a different Object Lock.
Users can optionally set a default Object Lock policy on a bucket. The default policy applies to all future objects that are created
in the bucket. This default policy can be disabled. Also, each version of each object can have locks applied to them.
S3 defines several lock types. For information about using Object Lock and the types of locks that are supported, see the S3
documentation here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html.

ObjectScale extensions
In S3, Object Lock is supported only on buckets that have Versioning set to ENABLED. ObjectScale extensions support Object
Lock on buckets with Versioning in OFF and SUSPENDED states. When Object Lock is enabled on buckets in these versioning
states, overwrite and delete operations are prevented when a locked null version exists for the object. All other behavior for
Object Lock is consistent between the versioning states and with standard S3 protocol behavior.
To override the S3 default behavior of automatically enabling versioning when Object Lock is enabled, ObjectScale introduces a
new header ("x-emc-retain-versioning-state:true").
ObjectScale does not restrict you from changing the versioning state to SUSPENDED on buckets that have Object Lock set to
ENABLED. It does not require special flags or permissions to do so.

Versioning configuration and effects on bucket operations

Buckets are configured with Versioning OFF, ENABLED, or SUSPENDED.
To understand the configurations, first understand the difference between null and non-null versioning schemes.
● Non-null—When Versioning is ENABLED, ObjectScale creates non-null versions.
● Null—At the most, one null version exists for an object at a time. The creation of a new null version causes an existing null
version to be removed but has no impact on non-null versions.
A bucket has three possible versioning states: OFF, ENABLED, and SUSPENDED. You can change the version state from OFF
to ENABLED. Once ENABLED, you can change the state between ENABLED and SUSPENDED. Objects create non-null versions
only when their state is ENABLED. Objects create null versions only when they are in the OFF or SUSPENDED state. An object
can have both a null version and non-null versions when operations occurred during multiple versioning states.

Table 21. Versioning states and their effects on objects in the bucket
State Description
OFF OFF is the initial state.
● Objects that are created in the bucket have an implicit null version.
● Overwriting objects overwrites that null version.
● Deleting objects removes that null version.
If the state is changed to ENABLED, it can never return to OFF.

ENABLED In this state:

● Each overwrite or delete action creates a new non-null version of the object.
SUSPENDED When you change a bucket to this state, all existing versions for all objects continue to exist.
Once In this state:
● Each overwrite or delete action creates a new null version of the object.
● Only one null version can exist for each object.
● An existing null version for an object is removed when a new null version is created.

78 Buckets
Object Lock configuration and the effects on bucket operations
Object Lock default configuration
You enable Object Lock on a bucket. You can enable it at bucket creation or by editing the bucket configuration later. Once
enabled, Object Lock can never be disabled.
The ObjectScale Portal user interface provides a toggle switch for enabling Object Lock, and a different toggle switch that
controls versioning. These switches let you decide whether to enable or disable versioning when enabling Object Lock.
NOTE: In the API, if you want Object Lock without versioning, you must explicitly include a flag (x-emc-retain-
versioning-state:true) in the API header to keep your wanted versioning configuration unchanged. Otherwise, the
S3 default behavior remains: versioning is automatically enabled when you enable Object Lock. For more information,
see the Dell ObjectScale 1.3 REST API Reference on the ObjectScale 1.3 product page at https://www.dell.com/support/
home/product-support/product/objectscale/overview.
The standard S3 protocol lock types are available:
● Compliance
● Governance
● None
● Retention periods define the duration of locking.
The following table describes Object Lock configuration settings and their effects on operations with the versioning
configurations.

Table 22. Object Lock states and their effects on objects in the bucket
Object Versioning State Description
Lock
State
ENABLED ENABLED The S3 protocol behavior for Object Lock:
● Locks are applied to individual versions, and only the removal of locked versions is
restricted.
● Overwrite and delete operations on the object are successful even if the latest version
is locked because those operations create another (new) version.
● Overwriting and deleting are not allowed on a named version.
ENABLED SUSPENDED or OFF ● Overwriting and deleting are blocked if there exists a locked null version for the object.
● Removing a locked version is prohibited except when all the following are true:
○ The lock is a governance retention lock, and
○ The appropriate bypass header is passed, and
○ The necessary bypass permission exists for the IAM user who is issuing the request.
DISABLED any state Overwriting and deleting are permitted.

About ObjectScale Bucket Logging

Bucket logging records all the requests going into a source bucket to a designated target bucket in a consistent format.
While creating or editing a source bucket, you can configure the target bucket and an optional prefix. Multiple source buckets
can log to the same target bucket, and share the same prefix. Logs are collected in a fixed format on a target bucket in objects
that identify time they were delivered.
NOTE:
● Source and target bucket must reside in the same object store.
● Source and target buckets can have different IAM accounts.
● Multiple accounts can log to the same target.
● A new IAM service principal to control who can log to the target bucket.
● Restrictions can be applied based on prefix, account, or source bucket.
● Only default encryption is supported.
● Bucket Logging can be disabled without a restart.

Buckets 79
ObjectScale Bucket Logging Object Naming Format
Log objects have a consistent naming format: [prefix]YYYY-MM-DD-HH-MM-SS-<UNIQUE-STRING The different parts of
the object naming format are:
● Prefix - This is optional and is configured from the source bucket. Multiple source buckets with the same prefix or target are
grouped together.
● Date & time - The logs collected have times less than or equal to the timestamp of the log object.
● Unique string - The unique string is derived from S3 pod ID and random characters. This prevents collisions.
Example: logs/2023-04-12-21-12-02-7d49cf9f5d-4sf5b--e458 accounts/
2023-04-12-21-11-48-6d193f5fbc-5e42a—2x6j 2023-04-12-21-11-48-5ab752fba1-tr763—8sga

ObjectScale Bucket Logging Request Log Format

Bucket Logging Access Log request format follows the S3 server access log format.
NOTE:
● Missing fields replaced with a dash ('-').
● Fields specific to ObjectScale will be placed after the fields defined for S3 for greater compatibility with existing
applications.

Table 23. Access Log Format for ObjectScale Bucket Logging

Position Field Comment
1 Canonical bucket owner ID -
2 Bucket -
3 Time -
4 Remote ip:port s3 does not include port.
5 User -
6 Request ID -
7 Operation HTTP operation (PUT, GET) Not operation REST.PUT.OBJECT
8 Object name -
9 Request uri Query string
10 HTTP status -
11 - Error code - unavailable
12 Bytes sent Using size content read (PUT or POST).
13 Object size Using size content returned (GET)
14 Total time Using latency value
15 Turn around time Using processing duration
16 Referrer -
17 User agent -
18 Version ID (obtained from query string) -
19 - Host id (x-amz-id2 or s3 extended request id) -
unavailable
20 - Signature version - unavailable
21 Cipher suite -
22 Authentication type -
23 Host header -

80 Buckets
Table 23. Access Log Format for ObjectScale Bucket Logging (continued)
Position Field Comment
24 TLS version -
25 - Access point arn - unavailable
26 - acl required - unavailable
27 "COPY" (begin ObjectScale additional fields) If copy
requested or - if not present.
28 Deep copy size If copy requested or - if not present
29 X-forwarded-for - if not present
30+ Additional headers Any additional headers will be placed after the
x-forwarded-for field.

Examples of requests stored in a log object:

urn:osc:iam::osai7e2b22326ad30da3:root sourcebucket [05/Jul/2023:21:43:48

+0000] 172.17.0.39:9020 172.17.0.1:5260 urn:osc:iam::osai7e2b22326ad30da3:root
ac110027:1892800040e:1a:426 GET QueryHookHandler.class - 404 - - 265 8 5 - curl/7.66.0
- - - - - 10.97.16.142 HTTP/1.1 - - - - -

urn:osc:iam::osai7e2b22326ad30da3:root sourcebucket [05/Jul/2023:21:43:47

+0000] 172.17.0.39:9020 172.17.0.1:14374 urn:osc:iam::osai7e2b22326ad30da3:root
ac110027:1892800040e:17:5bb GET README.txt - 404 - - 279 18 3 - curl/7.66.0 - - - - -
10.97.16.142 HTTP/1.1 - - - - - 'Connection: Keep-Alive’

ObjectScale Bucket Logging - Managing Log File Objects

When Bucket Logging is enabled, additional objects are written to target buckets. The size of these additional objects depends
on the volume of requests against source buckets. The objects are owned by the target bucket and are the responsibility of the
target bucket owner. Lifecycle policies can be used to manage the log file objects being generated.

Table 24. Log file size estimate based on request volume for different time periods
Average Requests for 10 Log size after 10 Log size per hour Log size per 24 hours
requests per minutes of capture minutes
second
100 64,900 23 MB 138 MB 3312 MB
280 168,000 59 MB 354 MB 7776 MB
1,000 600,000 211 MB 1,266 MB 30,384 MB
3,000-5.000 2,400,000 350-400 MB 2,400 MB 57,600 MB
10,000 6,000,000 2,117 MB 12,702 MB 304,848 MB

NOTE:
● The length of a log entry varies depending on the name of the bucket, the name of the object, and additional headers
that may be present in the request.

Buckets 81
Creating and managing buckets using ObjectScale
Create a bucket
This section describes how to set up a new bucket using the ObjectScale Portal user interface.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace and Object store fields at the top of the Buckets page to locate the object store in which you want
to create the bucket.
a. Select the namespace from the namespace drop-down on the top of the Buckets page.
b. Select the name of the object store that contains the bucket.
3. Click New Bucket.
The New Bucket wizard appears.
4. In the General page, complete the required fields and then click Next:

Option Description
Name Type a name for the new bucket. Bucket names can consist only of lowercase letters, numbers, dots
(.), and hyphens (-).
Namespace Select the namespace. Only the namespaces for which the user has edit permissions are listed in the
dropdown.
Object Store Select the object store from the ObjectStore dropdown menu within which to create the bucket.
Name
Bucket Owner Select the bucket owner account from the Bucket Owner Account dropdown menu.
Account
To select any account, first remove the selected account to see all accounts, and click the dropdown
to list all accounts. Select an account from the data list, or you can begin typing the account id to the
Bucket Owner Account data list.

5. In the Policy page, describe the policy to apply to the bucket and then click Next.
For more detailed information about creating a bucket policy statement, see the "About bucket policies" section in the
ObjectScale 1.3 Administration Guide.
a. Switch to the Text view of the policy editor by clicking between the View and Text views of the policy editor.
b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax
for policies is the same as the syntax used for Amazon AWS.
c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your
policy and to expand and contract statements.
6. In the Controls page, complete the required fields and then click Next.

Option Description
Versioning ● To maintain multiple versions of the same object within the bucket, set Versioning to On.
● To maintain a single version of an object, keep Versioning Off.
For more information, see the "Versioning configuration and effects on bucket operations" section in the
ObjectScale 1.3 Administration Guide.
Object Lock Enable Object Lock to protect objects from deletion or overwrite, for a fixed amount of time or indefinitely,
depending on the configuration.
● When Versioning is On, Object Lock protects versions.
● When Versioning is Off (or edited later to Suspended), Object Lock applies to the object. Delete
object and overwrite object operations are blocked for objects in the bucket with a lock that is still in
effect.
Optionally set a default Object Lock retention mode for the objects in this bucket. Objects are automatically
locked when they are added to a bucket that has a configured default retention lock. The default retention

82 Buckets
 Option Description

lock is applied to objects even if the user does not have s3:PutObjectLegalHold and s3:PutObjectRetention
permissions. Available lock types are:
● GOVERNANCE
● COMPLIANCE
● NONE (Object Lock is enabled, but the retention mode is not set.)
If GOVERNANACE or COMPLIANCE is selected, you can optionally set a Retention period. Select either
Days or Years and type the number of days or years for the period. This setting is a default retention
period for the bucket. Objects can have explicit retention periods set for them. An explicit retention period
set on an object supersedes this bucket default retention period.
Object Lock is Off by default.
For more information about Object Lock states and how Object Lock works with the different versioning
states, see the "Object Lock configuration and the effects on bucket operations" section in the ObjectScale
1.3 Administration Guide.

Quotas Quotas is disabled by default.

Enable Quotas for the bucket. When enabled, you can set the storage level or the object count level in
ObjectScale that results in:
● Block writes at Quota—a hard quota
● Notification at Quota—a soft quota
If a Default Bucket Quota is set at the account level when you are adding quotas to an object store, the
same value is set for Block Writes At. You can update quota values or disable the Quotas field in the
bucket.

Encryption Enable encryption to save the bucket data in an encrypted format. If Encryption was enabled at the
account level when add to object store, you cannot disable encryption in the bucket.
Encryption is disabled by default.

NOTE: Encryption cannot be disabled after you create the bucket with encryption enabled.

Bucket Enable Bucket logging to log requests for a specific source bucket to a designated target bucket in a
Logging consistent format. Enter the name of the target bucket in the Target Bucket field. Enter an optional prefix
to prepend to generated log files in the Prefix for Bucket Logging files field.
NOTE:
● The target bucket must be configured to allow bucket logging before allowing a source bucket to
set it as a target.
● The prefix makes it easier to locate log objects.
● Multiple source buckets can share the same prefix.

7. In the Event Rule page, complete the required fields and then click Next.
For more information about editing bucket event notifications, see Setting up bucket event notifications.
You must have at least one notification destination for this account to set up a bucket event rule.

Option Description
Event Rule Name Type a name for the new event.
Events Select one or more event types that trigger this event notification.
Prefix/Suffix Type the object prefix or suffix values that trigger this event notification.
Send To Select the notification destination to be used to send the notifications for the configured events.

NOTE: Click Add Event Rule to add multiple event rules. You cannot create configurations on the same bucket that
share a common event type.

8. Finally, use the Review page to review the values for configuring the new bucket and click Save.
If necessary, click Edit to modify any of the values.

Buckets 83
Results
The system creates the bucket in the object store, and the bucket name appears on the main Buckets page.

Edit a bucket
On the ObjectScale Portal user interface, edit the details of existing buckets.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Select the bucket to modify and click Edit.
The Edit Bucket wizard appears.
4. Select one or more of the sections with the bucket values to modify:
● General (Review only, no edit.)
● Policy
● Controls
● Event Rules
5. In the Policy page, modify the policy to apply to the bucket and then click Next.
For more detailed information about editing a bucket policy statement, see About bucket policies.
a. Switch to the Text view of the policy editor by clicking between the View and Text views of the policy editor.
b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax
for policies is the same as the syntax used for Amazon AWS.
c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your
policy and to expand and contract statements.
6. In the Controls page, modify the required fields and then click Next.

Option Description
Versioning When Versioning is Off, you can change it to On.
When Versioning is On, you can change it to Suspended.
To maintain multiple versions of the same object in a bucket, set Versioning to On. To maintain
a single version of each object, keep Versioning OFF or change it to Suspended. For more
information, see Versioning configuration and effects on bucket operations.

Object Lock Object Lock allows objects to be locked or protected from deletion or overwriting, for a fixed
amount of time or indefinitely, depending on the configuration.
● When Versioning is On, Object Lock protects versions.
● When Versioning is Off or Suspended, Object Lock applies to the object. Delete object and
overwrite object operations are blocked for objects in the bucket with a lock that is still in
effect.
You can change Object Lock from Off to On. If Object Lock is set to On, you cannot change it
to Off.
You can modify the default Object Lock retention mode and retention period.
The Object Lock Retention mode is a default retention mode for new objects that are added
to the bucket. Objects are automatically locked when they are added to a bucket that has a

84 Buckets
 Option Description

configured default retention lock. The default retention lock is applied to objects even if the
user does not have s3:PutObjectLegalHold and s3:PutObjectRetention permissions. Available lock
types are:
● GOVERNANCE
● COMPLIANCE
● NONE (Object Lock is enabled, but a retention mode is not set.)
If the mode is GOVERNANACE or COMPLIANCE, you can set or change the retention period.
Select either Days or Years and type the number of days or years for the period. This setting is a
default retention period for the bucket. Objects can have explicit retention periods set for them.
An explicit retention period set on an object supersedes this bucket default retention period.
NOTE: Setting a default retention on a bucket does not set any retention settings on objects
that are already in the bucket. Changing the bucket default retention period does not change
the existing retention period for any objects in that bucket.
For more information about Object Lock states and how Object Lock works with the different
versioning states, see Object Lock configuration and the effects on bucket operations.

Quotas Enable quotas for the bucket. When quotas are enabled, you can set the storage level or object
count level in ObjectScale that results in:
● Block writes at Quota—a hard quota
● Notification at Quota—a soft quota
Bucket Logging Enable Bucket logging to log requests for a specific source bucket to a designated target bucket
in a consistent format. Enter the name of the target bucket in the Target Bucket field. Enter a
prefix to prepend to generated log files in the Prefix for Bucket Logging files field.
NOTE:
● The target bucket must be configured to allow bucket logging before allowing a source
bucket to set it as a target.
● The prefix makes it easier to locate log objects.

7. In the Event Rule page, modify the required fields and then click Next.
For more detailed information about editing bucket event notifications, see Setting up bucket event notifications.
You must have at least one notification destination for this account to set up a bucket event rule.

Option Description
Event Rule Name Type a name for the new event.
Events Select one or more event types that trigger this event notification.
Prefix/Suffix Type the object prefix or suffix values that trigger this event notification.
Send To Select the notification destination to use to send the notifications for the configured events.
8. After making the necessary changes, click Save.

View the summary of a bucket

Using with the ObjectScale Portal user interface, select a bucket to view a detailed summary view of that bucket.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.

Buckets 85
 c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click on the name of the bucket.
The bucket Summary page appears displaying details on the selected bucket.
● Capacity Statistics
● Bucket Settings
● Quota Statistics
● Policy
● Object Counts
● Object Lock Configuration
● Event Notification Details

Delete a bucket
Using with the ObjectScale Portal user interface, delete a bucket when the object store no longer needs the bucket.

Prerequisites
The bucket must be empty. ObjectScale will only allow the deletion of buckets without any data within them.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Select the bucket to be deleted and click Delete.
ObjectScale prompts Are you sure you want to delete following bucket(s)?
4. In the Delete Bucket confirmation window, confirm that the appropriate bucket will be deleted.

Results
The bucket is deleted from the object store.

Configure Bucket Logging

Bucket logging records all the requests going into a source bucket on a target bucket, which can be set up using the
ObjectScale Portal user interface.

Prerequisites
The source and target buckets must reside in the same Object store.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.

86 Buckets
2. Use the Namespace and Object store fields at the top of the Buckets page to locate the object store where you create
the new bucket.
a. Select the namespace from the namespace drop-down on the top of the Buckets page.
b. Select the name of the object store that contains the bucket.
3. Click New Bucket to create a target bucket.
The New Bucket wizard appears.
4. In the General page, complete the required fields and then click Next:

Option Description
Name Type a name for the new bucket. Bucket names can consist only of lowercase letters, numbers, dots
(.), and hyphens (-).
Namespace Select the namespace. Only the namespaces for which the user has edit permissions will be listed in
the dropdown.
Object Store Select the object store from the ObjectStore dropdown menu within which to create the bucket.
Name
Bucket Owner Select the bucket owner account from the Bucket Owner Account dropdown menu.
Account
To select any account, first remove selected account to see all accounts and click on the dropdown to
list all accounts. Account can be selected from the data list, or you can begin typing the account id to
the Bucket Owner Account data list.

5. In the Policy page, describe the policy that allows bucket logging specifying the arn of source and target buckets, and then
click Next.
See Bucket Logging IAM Principal for more information.
a. Switch to the Text view of the policy editor by clicking between the View and Text views of the policy editor.
b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax
used for policies is the same as that used for Amazon AWS.
c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your
policy and to expand and contract statements.
6. Click New Bucket to create the source bucket.
The New Bucket wizard appears.
7. In the General page, enter the name of the source bucket and then click Next.
8. In the Controls page, complete the required fields and then click Next:

Option Description
Bucket Enter the name of the target bucket in the Target Bucket field, and enter an optional prefix in the
Logging Prefix for bucket logging files. fields.
NOTE: The prefix helps to locate the log objects.

9. Finally, use the Review page to review the values to be used for configuring the new bucket and click Save.
If necessary, click Edit to modify any of the values for the bucket that have been incorrectly set.

Results
Bucket logging is enabled, and you can view source and target buckets at the Buckets section in the ObjectScale Portal user
interface.

About bucket policies

ObjectScale provides a bucket policy editor to enable you to create a bucket policy for a bucket, either new or existing.
Bucket policies provide fine grained control over permissions for bucket operations and for operations on objects within the
bucket. Policy conditions are used to assign permissions for a range of objects that match the condition and are used to
automatically assign permissions to newly uploaded objects.
Policies are defined in JSON format in the Text view of the policy editor. Once defined a policy can be viewed in the Tree view.
The syntax used for policies is the same as that used for Amazon AWS. The operations for which permissions can be assigned
are limited to those operations supported by ObjectScale.

Buckets 87
The bucket policy editor has a code view and a tree view.
● The code view enables you to enter JSON policies from scratch or to paste existing policies into the editor and modified. For
example, if you have existing policies in JSON format, you can paste them into the code view and modify them.
● The tree view provides a mechanism for navigating a policy and is useful where you have a large number of statements in a
policy. You can expand and contract the statements and search them.

Create or edit a bucket policy

You can create or modify a bucket policy for an existing bucket using the Policy editor with the ObjectScale Portal user
interface.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Select the bucket to modify and click Edit.
The Edit Bucket wizard appears.
4. In the Policy page, modify the policy to apply to the bucket and then click Next.
a. Switch to the Text view of the policy editor by toggling between the View and Text views of the policy editor.
b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax
used for policies is the same as that used for Amazon AWS.
c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your
policy and to expand and contract statements.
5. Save.

Bucket Logging IAM Logging Principal

The IAM Logging Principal is used to grant permission to write request-logs in the target bucket.
● The required principal is "logging.s3.objscale.dell.com".
● The principal must be given S3:PutObject permission.
● The principal is added to the bucket policy on the target bucket.
The following example adds the logging principal to a target bucket with a bucket policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AccessLogsPolicy",
"Effect": "Allow",
"Principal": {
"Service": "logging.s3.objscale.dell.com"
},
"Action": [
"s3:PutObject"
],
"Resource":
["arn:aws:s3:osci4e7b81b1fedc9e6e:ostibc6ebb467b46f882:targetbucket/*"],

"Condition": {
"StringEquals": {
"aws:SourceAccount": ["osai3483c631fef0da7a"]

88 Buckets
 },

"ForAnyValue:ArnLike": {
"aws:SourceArn":
["arn:aws:s3:osci4e7b81b1fedc9e6e:ostibc6ebb467b46f882:sourcebucket*"]
}
}
}
]
}

The Statement specifies the following elements:

Resource The resource contains the target bucket ARN which includes the ObjectScale ID, the Object
store ID, and the name of the target bucket. This is followed by a wildcard for the
kinds of objects the logging principal is allowed to create. If a prefix is added, then the
logs would be restricted to sources with the designated prefix. For example, "Resource":
[arn:aws:s3:osci4e7b81b1fedc9e6e:ostibc6ebb467b46f882:targetbucket/
Billing/*] allows source buckets to log to the targetbucket,
only if the source bucket is assigned a prefix Billing/ to its bucket logging configuration.

Condition The condition allows further restrictions on what source buckets are allowed to send request logs to the
target bucket. If a Condition is not set, any source bucket from any account in the same Object store
is allowed to log requests on the target bucket.
NOTE: Bucket logging supports bucket policies that grant or restrict access to the target bucket by
aws:SourceAccount and awsSourceArn.

If a SourceAccount condition is set, only buckets owned by accounts listed are permitted to log
requests to the target bucket.
If a SourceArn condition is set, only source buckets matching the condition are permitted to log
requests to the target bucket.
SourceAccount and SourceArn conditions are not dependent on each other. A combination of one or
the other or both can be used to restrict access to the target bucket.

Bucket policy support

ObjectScale supports the setting of S3 bucket access policies. Unlike ACLs, which either permit all actions or none, access
policies provides specific users, or all users, conditional and granular permissions for specific actions. Policy conditions can be
used to assign permissions for a range of objects that match the condition and can be used to automatically assign permissions
to newly uploaded objects.
How access to resources is managed when using the S3 protocol is described in https://docs.aws.amazon.com/AmazonS3/
latest/dev/s3-access-control.html and you can use the information as the basis for understanding and using S3 bucket policies.
This section provides basic information about the use of bucket policies, and to identify the differences when using bucket
policies.
The following provides an example of a bucket policy:

{
"Version": "2012-10-17",
"Id": "S3PolicyIdNew2",
"Statement":[
{
"Sid":"Granting PutObject permission to user2 ",
"Effect":"Allow",
"Principal": {
"AWS":
"urn:osc:iam::<ACCOUNT_ID>:user/<USERNAME>"},
"Action":["s3:PutObject"],
"Resource": [

"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:<BUCKETNAME>/*",

"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:<BUCKETNAME>"

Buckets 89
 ],
"Condition": {
"StringEquals": {"s3:x-amz-server-side-encryption": [ "AES256"]}
}
}
]
}

NOTE: Alternatively, you can use the following structure for the Principal value in the bucket policy statement:

"Principal": {
"AWS": "*"
},

Each policy is a JavaScript Object Notation (JSON) document that includes a version, an identifier, and one or more statements.

Version The Version field specifies the policy language version and can be either 2012-10-17 or 2008-10-17.
If the version is not specified, 2008-10-17 is automatically inserted.
It is good practice to set the policy language for a new policy to the latest version, 2012-10-17.

Id The Id field is optional.

Each Statement includes the following elements:

SID A statement ID is a string that describes what the statement does.

Resources The bucket or object that is the subject of the statement. The resource can be associated with a
Resource or NotResource statement.
The resource name is the bucket and key name and is specified differently depending on whether you are
using virtual host style addressing or path style addressing, as shown:

Host Style: http://bucketname.ns1.emc.com/objectname

Path Style: http://ns1.emc.com/bucketname/objectname

In either case, the resource name is: bucketname/objectname.

You can use the (*) and (?) wildcard characters, where asterisk (*) represents any combination of
zero or more characters and a question mark (?) represents any single character. For example, you can
represent all objects in bucket that is called bucket name, using:

arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:<BUCKETNAME>/*

Actions The set of operations that you want to assign permissions to (enable or deny). The supported operations
are listed in Supported bucket policy operations.
The operation can be associated with an Action or NotAction statement.

Effect Can be set to Allow or Deny to determine whether you want to enable or deny the specified actions.
Principal The user who is enabled or denied the specified actions.
To grant permissions to everyone, as anonymous access, you can set the principal value to a wildcard,
"*", as shown:

"Principal": "AWS": "*

Conditions The condition under which the policy is in effect. The condition expression is used to match a condition
that is provided in the policy with a condition that is provided in the request.
The following condition operators are not supported: Binary, ARN, IfExists, Check Key Exists. The
supported condition keys are listed in Supported bucket policy conditions.

NOTE: ObjectScale bucket policies do not support federated users, nor do they support Amazon IAM users and roles.

More information about the elements that you can use in a policy are described in the Amazon S3 documentation, https://
docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html.

90 Buckets
Bucket policy scenarios
In general, the bucket owner has full control on a bucket and can grant permissions to other users and can set S3 bucket
policies using an S3 client. Users can also set bucket policies using the bucket policy editor in the New Bucket and Edit Bucket
wizards in the ObjectScale Portal user interface.
You can use bucket policies in the following typical scenarios:
● Grant bucket permissions to a user
● Grant bucket permissions to all users
● Automatically assign permissions to created objects

Grant bucket permissions to a user

To grant permission on a bucket to a user apart from the bucket owner, specify the resource that you want to change the
permissions for. Set the principal attribute to the name of the user, and specify one or more actions that you want to enable.
The following example shows a policy that grants a user who is named user1 the permission to update and read objects in the
bucket that is named mybucket:

{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "Grant permission to user1",
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::<ACCOUNT_ID>:user/user1"},
"Action": [ "s3:PutObject","s3:GetObject" ],
"Resource": [
"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:mybucket/*"
]
}
]
}

You can also add conditions. For example, if you only want the user to read and write object when accessing the bucket from a
specific IP address, add a IpAddress condition as shown in the following policy:

{
"Version": "2012-10-17",
"Id": "S3PolicyId1",
"Statement": [
{
"Sid": "Grant permission ",
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::<ACCOUNT_ID>:user/<USERNAME>"},
"Action": [ "s3:PutObject","s3:GetObject" ],
"Resource": [
"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:mybucket/*"
],
"Condition": {"IpAddress": {"aws:SourceIp": "<IP_ADDRESS>"}
}
]
}

Grant bucket permissions to all users

To grant permission on a bucket to a user apart from the bucket owner, specify the resource that you want to change the
permissions for. Set the principal attribute as anybody (*), and specify one or more actions that you want to enable.
The following example shows a policy that grants anyone permission to read objects in the bucket that is named mybucket:

{
"Version": "2012-10-17",

Buckets 91
 "Id": "S3PolicyId2",
"Statement": [
{
"Sid": "statement2",
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": [ "s3:GetObject" ],
"Resource": [
"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:mybucket/*"
]
}
]
}

Automatically assign permissions to created objects

You can use bucket policies to automatically enable access to ingested object data. In the following example bucket policy,
user1 and user2 can create subresources (that is, objects) in the bucket that is named mybucket and can set object ACLs.
With the ability to set ACLs, the users can then set permissions for other users. If you set the ACL in the same operation,
a condition can be set such that a canned ACL public-read must be specified when the object is created. This ensures that
anybody can read all the created objects.

{
"Version": "2012-10-17",
"Id": "S3PolicyId3",
"Statement": [
{
"Sid": "statement3",
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::<ACCOUNT_ID>:user/user1",
"AWS": "urn:osc:iam::<ACCOUNT_ID>:user/user2"},
"Action": [ "s3:PutObject,
s3:PutObjectAcl" ],
"Resource":[

"arn:aws:s3:<OBJECTSCALE_ID>:<OBJECT_STORE_ID>:mybucket/*"

]
"Condition":{"StringEquals":{"s3:x-amz-acl":["public-read"]}}
}
]
}

Supported bucket policy operations

Table 25. Permissions for Object Operations
Permission keyword Supported S3 operations
s3:GetObject applies to latest GET Object, HEAD Object
version for a version-enabled
bucket
s3:GetObjectVersion GET Object, HEAD Object This permission supports requests that specify a version number
s3:PutObject PUT Object, POST Object, Initiate Multipart Upload, Upload Part, Complete Multipart Upload
PUT Object - Copy
s3:GetObjectAcl GET Object ACL
s3:GetObjectVersionAcl GET ACL (for a Specific Version of the Object)
s3:PutObjectAcl PUT Object ACL
s3:PutObjectVersionAcl PUT Object (for a Specific Version of the Object)
s3:DeleteObject DELETE Object

92 Buckets
Table 25. Permissions for Object Operations (continued)
Permission keyword Supported S3 operations
s3:DeleteObjectVersion DELETE Object (a Specific Version of the Object)
s3:ListMultipartUploadParts List Parts
s3:AbortMultipartUpload Abort Multipart Upload

Table 26. Permissions for Bucket Operations

Permission keyword Supported S3 operations
s3:DeleteBucket DELETE Bucket
s3:ListBucket GET Bucket (List Objects), HEAD Bucket
s3:ListBucketVersions GET Bucket Object versions
s3:GetLifecycleConfiguration GET Bucket lifecycle
s3:PutLifecycleConfiguration PUT Bucket lifecycle

Table 27. Permissions for Bucket Sub-resource Operations

Permission keyword Supported S3 operations
s3:GetBucketAcl GET Bucket acl
s3:PutBucketAcl PUT Bucket acl
s3:GetBucketCORS GET Bucket cors
s3:PutBucketCORS PUT Bucket cors
s3:GetBucketVersioning GET Bucket versioning
s3:PutBucketVersioning PUT Bucket versioning
s3:GetBucketPolicy GET Bucket policy
s3:DeleteBucketPolicy DELETE Bucket policy
s3:PutBucketPolicy PUT Bucket policy
s3:GetBucketLogging GET Bucket logging
s3:PutBucketLogging PUT Bucket logging

Supported bucket policy conditions

The condition element is used to specify conditions that determine when a policy is in effect.
The following tables show the condition keys that are supported and that can be used in condition expressions.

Table 28. Supported generic AWS condition keys

Key name Description Applicable operators
aws:CurrentTime Used to check for date/time conditions Date operator
aws:EpochTime Used to check for date/time conditions using a date in epoch or UNIX time Date operator
(see Date Condition Operators).
aws:principalType Used to check the type of principal (user, account, federated user, etc.) String operator
for the current request.
aws:SourceIp Used to check the requester's IP address. String operator
aws:UserAgent Used to check the requester's client application. String operator
aws:username Used to check the requester's user name. String operator

Buckets 93
Table 29. Supported S3-specific condition keys for object operations
Key name Description Applicable permissions
s3:x-amz-acl Sets a condition to require specific s3:PutObject, s3:PutObjectAcl,
access permissions when the user s3:PutObjectVersionAcl
uploads an object.
s3:x-amz-grant-permission (for explicit Bucket owner can add conditions s3:PutObject, s3:PutObjectAcl,
permissions), where permission can using these keys to require certain s3:PutObjectVersionAcl
be:read, write, read-acp, write-acp, full- permissions.
control
s3:x-amz-server-side-encryption Requires the user to specify this header s3:PutObject, s3:PutObjectAcl
in the request.
s3:VersionId Restrict the user to accessing data only s3:PutObject, s3:PutObjectAcl,
for a specific version of the object s3:DeleteObjectVersion

Table 30. Supported S3-specific condition keys for bucket operations

Key name Description Applicable permissions
s3:x-amz-acl Set a condition to require specific access s3:CreateBucket, s3:PutBucketAcl
permissions when the user uploads an
object
s3:x-amz-grant-permission (for explicit Bucket owner can add conditions using s3:CreateBucket, s3:PutBucketAcl
permissions), where permission can these keys to require certain permissions
be:read, write, read-acp, write-acp, full-
control
s3:prefix Retrieve only the object keys with a s3:ListBucket, s3:ListBucketVersions
specific prefix.
s3:delimiter Require the user to specify the delimiter s3:ListBucket, s3:ListBucketVersions
parameter in the Get Bucket (List
Objects) request.
s3:max-keys Limit the number of keys returned s3:ListBucket, s3:ListBucketVersions
in response to the Get Bucket (List
Objects) request by requiring the user to
specify the max-keys parameter.

Setting up bucket event notifications

ObjectScale supports the configuration of bucket level event notification to allow you to easily monitor when certain
configurable events occur within the bucket, such as when objects are created or deleted within the bucket. Bucket event
notifications can be utilized to build out distributed and decoupled modern applications.
To set up ObjectScale's bucket event notification feature, you must configure two independent components:
● Configure the destination WebHook server
● Configure the Bucket Event Notifications

Configure the destination WebHook server Overview

For ObjectScale the only supported destination is a WebHook. WebHooks are a way to receive information when it happens,
rather than continually polling for that data.
The following is the expected schema for configuring the WebhookConfig element:

<WebhookConfig>
<AuthToken>token</AuthToken>
<BackupLimit>1000</BackupLimit>
<Comment>comment</Comment>

94 Buckets
 <Endpoint>http://10.55.66.77:3000/hook</Endpoint>
<Name>MyWebhook</Name>
</WebhookConfig>

WebHook Config syntax Description

Name Identifier which uniquely identifies this destination among all other WebHook
configured within an account in the object store.
Endpoint Webhook server endpoint.
AuthToken Opaque string or JWT authorization token.
BackupLimit Maximum limit size for undelivered messages.
Comment (Optional) Comment to this setting.

The WebHook name must be unique among WebHooks within an account in the object store. Once configured, ObjectScale will
internally build the <Urn> element, which you can collect with a GET.
ObjectScale supports these Destination Configuration Manager (DCM) APIs for interacting with the destination WebHook
server:
● GetEventDestinationConfiguration (type, name)
● DeleteEventDestinationConfiguration (type, name)
● ListEventDestinationConfigurations (type)
● PutEventDestinationConfiguration (type, name, config)

Configure the Bucket Event Notifications Overview

You can use the ObjectScale UI or the S3 API for PutBucketNotificationConfiguration and/or
GetBucketNotificationConfiguration to configure bucket event notifications by creating an Event Rule that will trigger an event
notification when the event occurs.
Each bucket uses an Event Rule section with the required fields to configure event notifications for that bucket. Using the
ObjectScale UI, you can create a new bucket or modify an existing bucket to add, edit, or delete event rules.

Buckets 95
Figure 10. Edit Bucket - Event Rule

● Event Rule Name - Type a name for the new event.

● Events - Select one or more event types that will trigger this event notification.
● Prefix/Suffix - Type the object name prefix or suffix values that will trigger this event notification.
● Send To - Select the notification destination to be used to send the notifications for the configured event(s).
You must have at least one notification destination for this account to set up a bucket event notification rule.
NOTE: Click on Add Event Rule to add multiple event rules. But, you cannot create configurations on the same bucket
that share a common event type.
When using the API, a notification configuration for a bucket is an XML document describing zero or more topic configurations.
If no configuration is set on a bucket, it will implicitly have an empty NotificationConfiguration element.
For example, shown below is the request.body of the NotificationConfiguration without any topic configurations:

<NotificationConfiguration>
</NotificationConfiguration>

When creating new bucket notifications, use the following syntax for the request.body of the BucketNotificationConfiguration
xml:

<NotificationConfiguration>
<TopicConfiguration>
<Id>Name</Id>

96 Buckets
 <Event>event-type</Event>
...
<Filter>
<S3Key>
<FilterRule>
<Name>(prefix|suffix)</Name>
<Value>string</Value>
</FilterRule>
...
</S3Key>
</Filter>
<Topic>webhook-urn</Topic>
</TopicConfiguration>
...
</NotificationConfiguration>

NotificationConfiguration syntax Description

TopicConfiguration May contain zero or more TopicConfigurations
ID Optional. If unspecified, ObjectScale will generate one
Event Must be one or more from
● s3:ObjectCreated:*
● s3:ObjectCreated:Put
● s3:ObjectCreated:Copy
● s3:ObjectCreated:CompleteMultipartUpload
● s3:ObjectRemoved:*
● s3:ObjectRemoved:Delete
● s3:ObjectRemoved:DeleteMarkerCreated
● s3:Replication:OperationFailedReplication
Topic Webhook URN referring to a webhook configuration in DCM
Filter Optional. May contain 0 or 1 S3Key filters
S3Key Optional. May contain 0-2 FilterRules
FilterRule Optional.
Name Optional. Must be one of prefix | suffix. Only one FilterRule of each type may be
specified in an S3Key filter.
Value Optional.

For example, the following NotificationConfiguration file shows how to configure a notification to a webhook any time an object
is created.

<NotificationConfiguration>
<TopicConfiguration>
<Id>CreateEvents</Id>
<Topic>createWebhook</Topic>
<Event>s3:ObjectCreated:*</Event>
</TopicConfiguration>
</NotificationConfiguration>

About bucket event notifications

At the highest level, an event is a change to the state of an object within a bucket.

Event Types
ObjectScale provides event notifications for the following types of events.

Buckets 97
Supported Event Type Description
s3:ObjectCreated:Put An object is created via an S3 PUT operation
s3:ObjectCreated:Copy An object is created via an S3 COPY operation
s3:ObjectCreated:CompleteMultipartU An object is created via an S3 CompleteMultipartUpload operation
pload
s3:ObjectCreated:* Any time an object is created
s3:ObjectRemoved:Deleted Any time a non-versioned object is deleted or an object version is permanently
deleted
s3:ObjectRemoved:DeleteMarkerCreat Any time a delete marker is created for a versioned object
ed
s3:ObjectRemoved:* Any time an object is deleted
s3:Replication:ObjectFailedReplication Any time an object fails replication

Event Notification Structure

ObjectScale's event notification structure conforms to the S3 event notification structure standard.
Below shows an example notification structure:

{
"Records": [
{
"eventVersion": "2.2",
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "2021-02-12T02:14:48.398Z",
"eventName": "s3:ObjectCreated:Put",
"userIdentity": {
"principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root"
},
"requestParameters": {
"sourceIPAddress": "172.17.0.1"
},
"responseElements": {
"x-amz-request-id": "ac11001b:17793e42a6a:a7:147",
"x-amz-id-2": "87fec1b544f39058bab52f8dec0a0e257a3703454c40e260355f1578bc597406"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "MyConfiguration1",
"bucket": {
"name": "bucket01",
"ownerIdentity": {
"principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root"
},
"arn": ".bucket01"
},
"object": {
"key": "object-for-notification",
"size": "10",
"etag": "c239368c6b3ec9b9dbc5a6b799e3756a",
"versionId": "AAABd5QE804oTME0iiFB2rY0z1_bH-nEK7w",
"sequencer": "100000000000000000000000000000000000000000072e338"
}
}
}
]
}

98 Buckets
Configure Webhook Destination for S3 Notifications
Set up the destination configuration for the WebHooks server that will receive the bucket event notifications.

Prerequisites
You must have configured:
● an IAM user with an Access Key and Secret.
● the destination configuration of the webhooks server. The URN for the destination configuration is a required field in the
bucket notification configuration.

About this task

Using a Linux workstation with kubectl and s3curl.pl, and that has access to ObjectScale on this k8s cluster:

Steps
1. List the objectscale-gateway service endpoint.

kubectl -n <NAMESPACE_NAME> get svc objectscale-gateway

2. Set an environment variable for the DCM endpoint.

DCM_ENDPOINT=<OBJECTSCALE_GATEWAY_IP>

3. Set your user Access Key and Secret that you created during the s3curl setup.

ACCESS_KEY=<ACCESS_KEY>

SECRET=<SECRET_KEY>

4. Calculate the signature for the request to add a new WebHook destination configuration.

NL=$'\n'
RESOURCE=/destconf/webhook/
DATEVAL="`date -u ${adj} +'%a, %d %b %Y %H:%M:%S %z'`"
STRING_TO_SIGN="PUT${NL}${NL}application/xml;charset=utf-8${NL}${DATEVAL}${NL}$
{RESOURCE}"
SIGNATURE=`/bin/echo -n "$STRING_TO_SIGN" | openssl sha1 -hmac
${SECRET} -binary | base64`

5. Add a new WebHook destination configuration to DCM passing the WebhookConfig XML in the request payload.

CREATE_WEBHOOK_OUTPUT=$(curl -v -H "Date: ${DATEVAL}" -H "Authorization:

AWS ${ACCESS_KEY}:${SIGNATURE}" -H "Content-Type: application/xml;charset=utf-8"
-d @<PATH_TO_WEBHOOK_CONFIG_XML_FILE> -X PUT https://${DCM_ENDPOINT}:<GATEWAY_PORT>/
destconf/webhook/)

6. Review the reply from the request.

The reply should look like:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<Webhook>
<Urn>urn:objectscale:webhook::24069e07-2b7a-4dc4-98ef-ee7d4017cf96:MyWebhook2</Urn>
<WebhookConfig>
<AuthToken>token</AuthToken>
<BackupLimit>100000</BackupLimit>
<Comment>optional comment</Comment>
<Endpoint>http://10.247.102.238:3000/hook</Endpoint>
<Name>MyWebhook2</Name>
</WebhookConfig>
</Webhook>

Buckets 99
7. Save the Urn of the WebHook configuration created in 5.
When creating the bucket notification configuration(s) in Create a bucket notification configuration using the ObjectScale
APIs , use the value from $WEBHOOK_URN in the <Topic></Topic> of the desired TopicConfiguration.

WEBHOOK_URN=$(echo $CREATE_WEBHOOK_OUTPUT | xmllint -format - | grep Urn | sed 's/

<Urn>\(.*\)<\/Urn>/\1/g' | sed -e 's/^[ \t]*//')

8. Optional: Review the WebHook destination configuration:

Use the ${DCM_ENDPOINT} value from 2.

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} -- https://$

{DCM_ENDPOINT}:<GATEWAY_PORT>/destconf/webhook/${WEBHOOK_URN}

9. Optional: If/when you need to remove a WebHook destination configuration, delete a webhook configuration by:
Use the ${DCM_ENDPOINT} value from 2.

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} --delete -- https://$

{DCM_ENDPOINT}:<GATEWAY_PORT>/destconf/webhook/${WEBHOOK_URN}

Create a bucket notification configuration using the ObjectScale APIs

Prerequisites
Before you can create bucket notification configuration, you must have:
● Created an object store and a bucket within the object store.
● Set up the WebHook destination configuration.

Steps
1. Create the NotificationConfigurations file with your config xml.
When creating the bucket notification configuration(s), use the value from $WEBHOOK_URN in the <Topic></Topic> of
the desired TopicConfiguration.
ObjectScale provides event notifications for the following types of events.

Supported Event Type Description

s3:ObjectCreated:Put An object is created via an S3 PUT operation
s3:ObjectCreated:Copy An object is created via an S3 COPY operation
s3:ObjectCreated:CompleteMultipart An object is created via an S3 CompleteMultipartUpload operation
Upload
s3:ObjectCreated:* Any time an object is created
s3:ObjectRemoved:Deleted Any time a non-versioned object is deleted or an object version is permanently
deleted
s3:ObjectRemoved:DeleteMarkerCrea Any time a delete marker is created for a versioned object
ted
s3:ObjectRemoved:* Any time an object is deleted

The example below shows a configuration which listens for all ObjectCreated and ObjectRemoved events.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<NotificationConfiguration xmlns = "http://s3.amazonaws.com/doc/2006-03-01/">
<TopicConfiguration>
<Id>MyConfiguration</Id>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>

100 Buckets
 <Topic>urn:objectscale:webhook::722d25f2-9c5b-41fe-82ac-605782945488:MyWebhook</Topic>
</TopicConfiguration>
</NotificationConfiguration>

2. Put the notification configuration to the bucket

s3curl.pl --ord --debug --id=<ACCESS_KEY> --key=<SECRET> --calculateContentMd5 --

put=<PATH_TO_CONFIG> -- http://$(kubectl get svc |awk '/-s3/{print $4}'):80/${BUCKET}?
notification -v

3. Verify the bucket notification configuration was set in the bucket.

s3curl.pl --ord --debug --id=<ACCESS_KEY> --key=<SECRET> http://$(kubectl get svc |

awk '/-s3/{print $4}'):80/{BUCKET}?notification -v | xmllint -format -

Results
You will now receive notification record output like this in your WebHook server as users perform S3 operations in the monitored
bucket. For an example webhook server listening, see Sample setting of simple listener at Webhook server.

Received notification #7 from ::ffff:172.17.0.38

{
"Records": [
{
"eventVersion": "2.2",
"eventSource": "aws:s3",
"awsRegion": "us-west-2",
"eventTime": "2021-02-12T02:14:48.398Z",
"eventName": "s3:ObjectCreated:Put",
"userIdentity": {
"principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root"
},
"requestParameters": {
"sourceIPAddress": "172.17.0.1"
},
"responseElements": {
"x-amz-request-id": "ac11001b:17793e42a6a:a7:147",
"x-amz-id-2": "87fec1b544f39058bab52f8dec0a0e257a3703454c40e260355f1578bc597406"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "MyConfiguration1",
"bucket": {
"name": "bucket01",
"ownerIdentity": {
"principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root"
},
"arn": ".bucket01"
},
"object": {
"key": "object-for-notification",
"size": "10",
"etag": "c239368c6b3ec9b9dbc5a6b799e3756a",
"versionId": "AAABd5QE804oTME0iiFB2rY0z1_bH-nEK7w",
"sequencer": "100000000000000000000000000000000000000000072e338"
}
}
}
]
}

Sample setting of simple listener at Webhook server

NOTE: Requires npm and nodejs installed:

mkdir webhook
cd webhook
npm init -y
npm install express body-parser

Buckets 101
 # create index.js from the linked file
node index.js
Webhook listening on :3000 /hook

Content of index.js

const express = require("express")

const bodyParser = require("body-parser")

const app = express()

const PORT = 3000
const PATH = "/hook"

count = 0

app.use(bodyParser.json())

app.post(PATH, (req, res) => {

count++
console.log("Received notification #"
console.log(JSON.stringify(req.body, null, 2))
res.status(200).end()
})

app.listen(PORT, () => console.log(`Webhook listening on :${PORT} ${PATH}`))

Configuring bucket entities

View the Bucket Summary
Using with the ObjectScale Portal user interface, you can view the summary details of an existing bucket.

About this task

To view the summary details of a bucket:

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click on the name of the bucket to view the bucket Summary tab.
The Summary tab displays details on the bucket, such as:
● Capacity Statistics
● Bucket Settings
● Quota Statistics
● Policy
● Object Counts
● Object Lock Configuration
● Event Notification Details

102 Buckets
Managing Bucket Replication
ObjectScale Replication allows you to manage and monitor replication policies and replicate bucket data. Replication between
object store buckets complies with the S3 protocol of AWS.
ObjectScale replication enables the copying of objects across buckets within ObjectScale instances. For detailed information
about ObjectScale Replication and configuring bucket replication, see ObjectScale Replication.

Buckets 103
 6
Federate ObjectScale Systems
This chapter contains details on creating a federation of multiple ObjectScale systems.
Topics:
• Federating ObjectScale Systems
• Create a federation of ObjectScale systems
• Add additional ObjectScale instances to an existing ObjectScale federation

Federating ObjectScale Systems

When you create a federation of ObjectScale systems, it allows IAM entities to replicate from one ObjectScale system to other
ObjectScale systems in the federation.
The ObjectScale System page contains details on the federation status of an ObjectScale system. It also displays details on any
other ObjectScale systems that are a part of the federation of ObjectScale systems. The ObjectScale Systems tab allows you
to create and join federations.

NOTE: Once an ObjectScale instance joins a federation, it cannot be removed from that federation.

Within a federation, an ObjectScale instance can either be the primary instance or a secondary instance. There can be two or
more secondary instances within a federation, but there can only be a single primary instance. Any instance not in a federation
appears as Not Federated, which is the default state. After you create a federation, a heartbeat will be maintained between
trusted ObjectScale instances to track their connectivity status (Online or Offline).

Figure 11. ObjectScale Systems

ObjectScale creates ObjectScale federations by using the external endpoints of the ObjectScale Gateway service. This
ObjectScale Gateway service itself is made up of two services:
● objectscale-gateway-internal
● objectscale-gateway
ObjectScale Gateway and ObjectScale Gateway Internal services are separate paths for forwarding requests to ObjectScale-
level services. The ObjectScale Gateway service has a certificate that is used as a customer-facing endpoint. The ObjectScale
Gateway Internal service has an internally signed certificate and is used for ObjectScale-to-ObjectScale communication.
Here is an overview of what an Administrator user must do to create a federation of ObjectScale systems:

104 Federate ObjectScale Systems

 NOTE: Access to both the primary and secondary ObjectScale systems is required.

1. From the ObjectScale system that you want to become the primary system in the federation, create the federation. It
becomes the primary ObjectScale instance.
2. From the ObjectScale system that you want to become a secondary system in the federation, join a federation. This
downloads the FederationSigning_<objectscale_system>.xml file that the primary ObjectScale system must
sign.
3. From the primary ObjectScale system, upload FederationSigning_<objectscale_system>.xml request
to the primary ObjectScale. The primary ObjectScale generates a signed certificate that is named
PostFederationSigning_<objectscale_system>.xml and automatically download the file.
4. From the secondary ObjectScale system, add a new remote instance by uploading the signed certificate that was
downloaded from the primary ObjectScale system. The ObjectScale type is Secondary.
5. Finally, from each system you must initiate trust of the remote instances from each ObjectScale system.
Once both the Remote Instances have Trust Initiated Trust Status, the federation service on each side tries
to communicate to the remote instance over the objectscale-gateway-internal endpoint. If this system-to-system
communication is successful, it moves each remote instance to the Trusted Trust Status. Communication is successful
once both ObjectScale instances verify the certificates in the trusted list.

You can add additional ObjectScale instances to this federation by repeating this process.
Use the following tasks to create and maintain a federation of ObjectScale systems.

Create a federation of ObjectScale systems

Creating a federation of ObjectScale systems allows you to replicate IAM entities from one ObjectScale system to other
systems.

About this task

A user with the administrator role on both ObjectScale systems will need to complete these steps to create the federation of
ObjectScale systems.

Steps
Complete these steps on the ObjectScale system you wish to make the primary ObjectScale instance in the federation.
1. From the ObjectScale Portal user interface, click Administration > ObjectScale and then select Federation tab.
The details on the ObjectScale instances in a federation that the user is authorized to view are displayed.
2. Click Create Federation and click YES to create the necessary federation CA and establish this ObjectScale instance as the
primary.

Figure 12. Create Federation

After you click Create Federation this ObjectScale system becomes the primary instance for the federation. You will no
longer be able create additional federation CAs or join this ObjectScale instance to any other federations.
3. Ensure the instance is now listed as Primary in the ObjectScale Systems page.
Complete these steps on the ObjectScale system you wish to make a secondary ObjectScale instance in the federation.
4. From the ObjectScale Portal user interface, click Administration > ObjectScale and then select Federation tab.
The details on the ObjectScale instances in a federation that the user is authorized to view are displayed.

Federate ObjectScale Systems 105

5. Click Join Federation and download the federation signing request file,
FederationSigning_<OBJECTSCALE_NAMESPACE>.xml.
Ensure that this file is accessible by both ObjectScale instances.
6. From the primary ObjectScale instance, click New Remote Instances and upload the federation signing request file from
the other ObjectScale instance. After uploading the xml file, click SAVE.

Figure 13. New Remote Instance

Once you have uploaded this file, the primary ObjectScale instance automatically generates and downloads the signed
request file, PostFederationSigning_<OBJECTSCALE_NAMESPACE>.xml. Additionally, the Secondary ObjectScale
instance will now appear as a Not Trusted remote instance of ObjectScale within the federation managed by the primary
ObjectScale instance.
7. From the secondary ObjectScale instance, click New Remote Instances and upload the signed federation signing request
file from the primary ObjectScale instance.
The primary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation.
8. Finally, to complete the federation process linking these two ObjectScale instances, establish trust between the two
ObjectScale instances.
a. From the Secondary ObjectScale instance, select the checkbox of the primary ObjectScale instance that needs to
establish trust with the Secondary ObjectScale instance and click Initiate Trust.
The Initiate Trust window appears.
b. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the
federation and click Yes.

Figure 14. Initiate trust with the primary ObjectScale

c. From the primary ObjectScale instance, select the checkbox of the Secondary ObjectScale instance that needs to
establish trust with the primary ObjectScale instance and click Initiate Trust.

106 Federate ObjectScale Systems

 d. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the
federation and click Yes.

Figure 15. Initiate trust with the secondary ObjectScale

Each of the remote instances will attempt to establish trust, joining the instances in an ObjectScale federation. Initially, the
ObjectScale instances are listed as Trust Initiated Trust Status as the federation service on each side tries to communicate
to the remote instance over the objectscale-gateway-internal endpoint, and if this system-to-system communication is
successful, it moves each remote instance to the Trusted Trust Status.

Add additional ObjectScale instances to an existing

ObjectScale federation

About this task

A user with the administrator role on both ObjectScale systems will need to complete these steps to create the federation of
ObjectScale systems.

Steps
Complete these steps on the ObjectScale system you wish to make a secondary ObjectScale instance in the federation.
1. From the ObjectScale Portal user interface, click Administration > ObjectScale and then select Federation tab.
The details on the ObjectScale instances in a federation that the user is authorized to view are displayed.
2. Click Join Federation and download the federation signing request file,
FederationSigning_<objectscale_system>.xml.
Ensure that this file is accessible by both ObjectScale instances.
Complete these steps on the primary ObjectScale instance in the federation.
3. From the ObjectScale Portal user interface, click Administration > ObjectScale and then select Remote Instances tab.
The details on the ObjectScale remote instances in a federation that the user is authorized to view are displayed.
4. Click New Remote Instances and upload the federation signing request file from the other ObjectScale instance. After
uploading the xml file, click SAVE.

Federate ObjectScale Systems 107

 Figure 16. New Remote Instance

Once you have uploaded this file, the primary ObjectScale instance automatically generates and downloads the signed
request file, PostFederationSigning_<OBJECTSCALE_NAMESPACE>.xml. Additionally, the Secondary ObjectScale
instance will now appear as a Not Trusted remote instance of ObjectScale within the federation managed by the primary
ObjectScale instance.
5. From the secondary ObjectScale instance, click New Remote Instances and upload the signed federation signing request
file from the Primary ObjectScale instance.
The Primary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation.
6. Finally, to complete the federation process linking these two ObjectScale instances, establish trust between the two
ObjectScale instances.
a. From the Secondary ObjectScale instance, select the checkbox of the primary ObjectScale instance that needs to
establish trust with the Secondary ObjectScale instance and click Initiate Trust.
The Initiate Trust window appears.
b. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the
federation and click Yes.

Figure 17. Initiate trust with the primary ObjectScale

c. From the primary ObjectScale instance, select the checkbox of the Secondary ObjectScale instance that needs to
establish trust with the primary ObjectScale instance and click Initiate Trust.
d. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the
federation and click Yes.

108 Federate ObjectScale Systems

 Figure 18. Initiate trust with the secondary ObjectScale

Each of the remote instances will attempt to establish trust, joining the instances in an ObjectScale federation. Initially, the
ObjectScale instances are listed as Trust Initiated Trust Status as the federation service on each side tries to communicate
to the remote instance over the objectscale-gateway-internal endpoint, and if this system-to-system communication is
successful, it moves each remote instance to the Trusted Trust Status.

Federate ObjectScale Systems 109

 7
ObjectScale Replication
Topics:
• Introduction to ObjectScale Replication
• Bucket Replication Policy
• Bucket Replication to multiple destinations
• Delete marker replication on versioning-enabled buckets
• Manage a Bucket Replication Policy using ObjectScale UI
• Set up ObjectScale Replication using the ObjectScale API
• Replication status

Introduction to ObjectScale Replication

ObjectScale Replication allows you to manage and monitor replication policies and replicate bucket data. Replication between
object stores complies with the S3 protocol of AWS.
ObjectScale replication enables the copying of objects across buckets within ObjectScale instances. The destination, or target,
bucket could be in one of the following object stores:
● The same local object store
● An object store in the same ObjectScale instance as the source bucket
● An object store in another ObjectScale instance in an ObjectScale federation

Up to four unique destinations per replication policy can be configured. At any time, replication can be paused, suspended, or
throttled.
One or more accounts can own buckets that are configured for object replication.
ObjectScale replication across object stores is an eventual consistency process. The benefits of eventual consistency replication
are:
● It is good for high-latency connections.
● It provides eventually consistent object access (asynchronous replication).
● Data reads and writes are always local for the best performance.
● The system can perform management operations during network partitions.
To enable replication, add a replication configuration to your source bucket. The minimum configuration includes:
● The destination bucket or buckets where you want ObjectScale to replicate objects.
● An Identity and Access Management (IAM) role that ObjectScale can use to replicate objects.
An IAM Role is an IAM identity that you can create in your account that has specific permissions.
An IAM Role is similar to an IAM end user. They are both an ObjectScale identity with permission policies that determine
what the identity can and cannot do in ObjectScale. However, instead of being uniquely associated with one person, a role is
intended to be assumable by anyone who needs it.

Each source bucket can be configured to replicate some or all its data to one or more destination buckets. The data that are
replicated from the source bucket can be replicated based on a key prefix, a tag or both to make replication granular. An IAM
role must be selected for the source bucket account to replicate the data. On the source bucket, the rule can target specific
destination buckets that are based on the key prefix and tag.
In ObjectScale:
● Replication is bucket-to-bucket.
● Replication rules can be different for every bucket depending on redundancy or locality needs.
● Replication objects can be part of a bucket (by prefix or by tag set on objects).
● Replication can do network throttling between different object stores.

110 ObjectScale Replication

● Objects uploaded to ObjectScale using multipart upload (MPU) can be managed with replication rules after the upload
completes.
● Replication can support large objects up to 50 TB.
ObjectScale administrators and users with the appropriate permissions can perform ObjectScale Replication management
operations, including:
● Review of the replication metrics on different rules for buckets in an object store.
● Throttle or unthrottle the amount of data that is replicated in order to control the load on the system due to replication
traffic.
● Pause or resume ObjectScale Replication.
○ You can specify a pause ObjectScale Replication end time. Replication will automatically resume after the pause end time.
There is no upper limit for the pause time, but the pause time cannot be indefinite. The default pause time is one hour.
○ You can extend the pause duration while the replication is still under suspension.
○ You can resume ObjectScale Replication when the replication status is either paused or suspended.
● Suspend the replication of an object store.
○ When you suspend a destination object store, it disables all replication rules to that object store. Newly created objects in
that object store do not generate replication status for that object store.
○ When a destination object store is suspended, replication for an existing object is paused. When replication for the object
store is resumed, those objects and any updates are replicated then.
○ There are no limitations for a suspend duration. Suspend remains in place until the destination object store replication is
resumed.
● Permanently remove an object store from ObjectScale, because of a planned or unplanned event. Any unreplicated objects in
those object stores go to FAILreplication status.

ObjectScale Replication overview and configuration details

ObjectScale Replication, at its most basic setting, allows users to replicate objects in a bucket to another bucket. The
destination bucket can be defined in any of the following:
● In the same object store
● In another object store, in the same ObjectScale instance
● In another object store, in another ObjectScale instance, in the ObjectScale federation
More complex ObjectScale Replication configurations can be defined. These complex configurations can allow for a replication
to be single-directional or bi-directional replication between buckets or from a single source bucket to multiple destination
buckets.
ObjectScale Replication can be defined using replication rules or by replicating object versions.
● Since ObjectScale Replication supports multiple source buckets replicating to the same destination bucket, and since
replication occurs asynchronously, it is possible to have object name collisions. To avoid having an object being overwritten in
the destination, it is required to enable versioning on both the source and destination bucket.
● Each object version is replicated separately. When it is replicated to the destination, it keeps the same Create Time and
Version ID. You could use the same version ID to access a specific version of that object in the destination bucket. Versions
keep the same order in the destination bucket. After everything is replicated, the latest version and version history of an
object match the destination bucket.

What does ObjectScale Replication replicate?

ObjectScale Replication replicates the following:
● Object versions that are created after a replication rule is added to the source object store.
Updates to the object User Metadata, Storage class, or Encryption type create new versions of the object. The replication
rules determine the replication of the new object version when the version is created.
● A delete marker that is generated by a user delete action.
● When replicated, the replicated object keeps the same metadata as the source object for:
1. Object Name
2. Version ID
3. Object Create Time
4. User Metadata
5. Attributes (ACL, Tag, or Lock)

ObjectScale Replication 111

What does ObjectScale Replication not replicate?
ObjectScale Replication does not replicate:
● User key encrypted objects
● Updates to bucket-level subresources, such as the life cycle configuration of the bucket
● Actions performed by a life cycle configuration
● Object versions that were created before the replication rule itself was created or enabled. Further,
1. There is no replication status for this version of object.
2. Any attribute (ACL, Tag, or Lock) update on this version of the object does not trigger replication of this new object
version.
● The replicated object does not replicate again.
1. Even if the replication configuration between two buckets is bidirection, attribute updates on a replicated object do not
replicate back to the source object.
2. Replication for attribute updates on the source bucket overwrites all ACL, Tag, or Lock on the replicated object.
● A delete marker that is generated by a life cycle configuration.

ObjectScale Replication Configuration Overview

1. Create an object store and bucket.
2. Configure Bucket Replication on the bucket in the created object store.
● Specify the replication configuration on each bucket to start object replication.
● A user can create, read, update, and delete replication rules on an object store, irrespective of the state of other object
stores or network connectivity to other object stores.
● Create and specify an IAM Role ARN, or use an existing role, in their replication configuration. The IAM role must have
permission to replicate objects.
● Specify replication rules to specify a replication destination and the replication behaviors in the replication configuration.
3. Create the replication configuration.

Required IAM permissions

The IAM role used to perform the object replication must have permissions to replicate objects from the source bucket to the
destination bucket. It must be a service role which allows the s3 service to assume it. The role applies to the source account.
To set or change the IAM role that is associated with a replication, use the Bucket Replication Rule Editor in the ObjectScale
Portal user interface. The IAM role field is in the Destination section of the editor. The editor presents a dropdown list of
existing roles in the source account. For more information, see the steps about the Destination tab in Configure a new bucket
replication rule or Edit an existing bucket replication rule.
If you are creating a new IAM role for replication, the following table shows all required permissions that the IAM role must
possess.

Table 31. Required permissions in the source account IAM role for replication
Action Required allow permissions Target
Configure replication policy s3:ReplicateObject On the destination bucket
and rules.
s3:GetBucketVersioning
s3:ListBucketVersions
s3: ReplicateTags
s3:PutObjectRetention
s3:PutObjectLegalHold
s3:BypassGovernanceRetention
Replicate a delete marker. s3:ReplicateDelete On the destination bucket
Change the object owner. s3: ObjectOwnerOverrideToBucketOwner On the destination bucket

112 ObjectScale Replication

Table 31. Required permissions in the source account IAM role for replication (continued)
Action Required allow permissions Target
Access a source bucket. s3:GetObjectVersionForReplication On the source bucket
s3:GetObjectVersionAcl
s3:GetObjectVersionTagging
s3:GetObjectRetention
s3:GetObjectLegalHold

Bucket Replication Policy

A Bucket Replication Policy is an XML document that the user constructs and sets on a bucket.
You can create the Bucket Replication Policy XML document with the ObjectScale Portal user interface or the S3 API. Follow
the XML format defined by AWS (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketReplication.html).
NOTE: Most of the ObjectScale UI fields are compatible with AWS. Some fields (such as destination arn) are not
compatible with AWS.

Bucket Replication Limits

Policy property Default Maximum Description
Destinations per bucket 4 While configuring replication rules, you can configure up to four unique
destinations.
NOTE: From the ObjectScale Portal, you can only select one
destination per policy.

Rules per policy 1000 The maximum number of rules allowed in a single replication policy
Policy size (bytes) 2 MB The maximum size of a replication policy, in bytes
2 MB allows 1000 rules per destination with about 2 KB of filters and
other configurations per rule.

Configuring replication rules using the ObjectScale Portal UI

To use the ObjectScale Portal user interface to manage bucket replication settings, go to Object Store > Bucket >
Replication. From the Replication tab, you can manage the replication policy of the bucket.

ObjectScale Replication 113

Figure 19. Bucket replication settings

You can create a NEW REPLICATION RULE using the New Replication Rule wizard, which helps in adding a rule to the
replication policy of the bucket.
A replication rule can define only one destination target. It cannot define multiple destinations in a single replication rule. A
replication rule can have only one destination bucket.
From the Replication tab, you can also edit or delete rules, enable or disable rules, and change the priority of rules. The Receive
Objects button configures a bucket as a destination bucket. For more information, see Manage a Bucket Replication Policy
using ObjectScale UI.

Configure replication using the S3 API

You can manage policy documents using the S3 API. An example basic policy XML document follows.

<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Role>urn:osc:iam::osai0018c732362653d5:role/crrRole</Role>
<!-- the IAM role used to perform the object replication -->
<Rule>
<!-- Defines a Rule, to match objects and specify where they need to be sent -->
<ID>rule1</ID>
<Status>Enabled</Status>
<Priority>1</Priority>
<Filter><Prefix>important/</Prefix></Filter>
<!-- A filter to identify which objects to replicate -->
<Destination>
<AccessControlTranslation>
<Owner>Destination</Owner>
</AccessControlTranslation>
<Account>Osai0018c732362653d5</Account>
<Bucket>arn:aws:s3:osci26b00472169e8067:osti5a9f9ecef92d9f85:destination1</Bucket>
</Destination>
<DeleteMarkerReplication>
<Status>Disabled</Status>
</DeleteMarkerReplication>
</Rule>
</ReplicationConfiguration>

ObjectScale supports the standard AWS S3 APIs for getting, setting, and deleting the replication policy on a bucket.

114 ObjectScale Replication

Endpoint API Permissions Needed
PUT /?replication PutBucketReplication s3:PutReplicationConfiguratio
n
GET /?replication GetBucketReplication s3:GetReplicationConfiguratio
n
DELETE /?replication DeleteBucketReplication s3:PutReplicationConfiguratio
n

Amazon defines two versions of Replication Policies (V1 and V2). ObjectScale supports the V2 policy format. ObjectScale
bucket replication policies support most of the V2 tags. There is no support for:
● S3 RTC-related tags (Metrics and ReplicationTime)
● The ExistingObjectReplication tag
● Tags that are related to KMS-encrypted objects
● Tags related to delete marker replication or replica sync.

Replication Rules
A replication policy contains rules.
A single policy supports up to 1000 rules. This value is configurable if users have more resources.
The filter of a replication rule can specify:
● An optional prefix for prefix matching the object name
● An optional set of object tags to match
Each rule can specify only one destination bucket.
● If there is a requirement to specify multiple destination buckets, the user has to configure multiple rules.
● The bucket is specified with the bucket ARN which includes the ObjectScale, object store, and bucket name.
● Up to four destinations for a single bucket replication configuration are allowed. (Four is a default value and is configurable if
users have more resources.) On the ObjectScale Portal, you can only select one destination per policy.
Other rule attributes can specify the following:
● The scope of the objects that the rule matches.
The scope could be an entire bucket or object with a specified prefix or specified tags. Configurations for whether to
replicate KMS-encrypted objects are ignored.
● The replication behavior for the rule, such as whether the REPLICA owner must be changed.
● Whether the rule is disabled or enabled.
● The priority for each rule—When a user creates an object and matches multiple rules for one destination, the rule with the
highest priority takes effect. Rules for different targets do not interfere with each other.
● Replication Behavior sets, such as behavior when the object version is created.
Changes (such as priority, behavior, scope, and so on) on rules after creation of the object version only affect whether a further
attribute update is replicated. It does not impact replication behavior for this object version.

Bucket Replication to multiple destinations

ObjectScale supports replication to multiple destinations. The limitation is that there cannot be more than one destination
bucket in the same object store within a single replication policy.
Shown below is a simple example of an XML document for a bucket replication policy with multiple destinations:

<?xml version="1.0" encoding="UTF-8"?>

<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Role>arn:aws:iam::AcctID:role/jimmy</Role>
<Rule>
<ID>rule1</ID>
<Priority>1</Priority>
<Filter><Prefix>foo</Prefix></Filter>

ObjectScale Replication 115

 <Destination><Bucket>arn:aws:s3:US-E:foo:bucket1</Bucket></Destination>
<DeleteMarkerReplication>
<Status>Disabled</Status>
</DeleteMarkerReplication>
</Rule>
<Rule>
<ID>rule2</ID>
<Priority>2</Priority>
<Filter><Prefix>food</Prefix></Filter>
<Destination>
<Bucket>arn:aws:s3:US-E:foo:bucket1</Bucket>
<StorageClass>MCFREEZE</StorageClass>
</Destination>
<DeleteMarkerReplication>
<Status>Disabled</Status>
</DeleteMarkerReplication>
</Rule>
<Rule>
<ID>rule3</ID>
<Priority>3</Priority>
<Filter><Prefix>foodo</Prefix></Filter>
<Destination><Bucket>arn:aws:s3:US-W:foo:bucket2</Bucket></Destination>
<DeleteMarkerReplication>
<Status>Disabled</Status>
</DeleteMarkerReplication>
</Rule>
</ReplicationConfiguration>

In the following example policy, there are two different destination buckets. The name of the object determines the destination
bucket that the object is replicated to and the parameters for that replication.

Object Name Rules Matched Destination Buckets Other Behavior

bar none no replication none
foo rule1 bucket1 none
food rule1, rule2 bucket1 The storage class
of the object in
bucket1 will be set
to MCFREEZE because
rule2 has the highest
priority of all the
rules that matched
for destination
"bucket1".
foodo rule1, rule2, rule3 bucket1 & bucket2 The storage class
of the object in
bucket1 will be set
to MCFREEZE because
rule2 has the highest
priority of all the
rules that matched
for destination
"bucket1". The
storage class of the
object in bucket2
will still be the
bucket's default
storage class.

116 ObjectScale Replication

Delete marker replication on versioning-enabled
buckets
You can choose whether to enable or disable delete marker replications.
In a versioning-enabled bucket, an S3 Delete request generates a delete marker as the latest version of the object. A delete
marker version does not have data or metadata associated with it. A delete marker has the following effects:
● The s3 GET OBJECT API cannot access an object with a delete marker as the latest version. The API returns a 404 No
Object Found error, which is expected behavior on a deleted object.
● The delete marker allows users to reclaim the space from all object versions using life cycle policy. The reclaimed space
includes the current version and the delete marker itself.
In replication policy configuration, you can choose whether to enable or disable delete marker replication from source bucket to
destination buckets. The default setting is DISABLED.
NOTE: Delete marker replication cannot be enabled with a tag filter.)

Use the following guidelines to decide whether to enable or disable delete marker replication.

DeleteMarkerReplicat Description
ion setting
DISABLED Delete markers are not replicated, which means that:
● The source and destination buckets are not synchronized. The data is available in the destination
bucket even when it was deleted in the source bucket.
● Users cannot use life cycle policy to reclaim space for all versions of an object. (Users may
reclaim space for all object versions in the destination bucket only, using manual delete version
operations.)
Use DISABLED in situations when space is not a concern. It is also useful when you have upgraded
to ObjectScale 1.3 from a previous release. If you edit an older replication policy after installing
ObjectScale 1.3, you must set a value on the DeleteMarkerReplication setting. For consistency with
how replication worked in your prior releases, choose DISABLED.

ENABLED Delete markers are replicated, which means that:

● The source and destination buckets are synchronized.
● Users can reclaim space in both the source and destination buckets by life cycle policy.
Use this value to ensure consistency between source and destination buckets and for space
efficiency.

Not set If a policy was defined in an ObjectScale version earlier than 1.3, the XML files for the older policies
do not include the DeleteMarkerReplication setting. The default behavior in this case matches the
DISABLED setting.

Manage a Bucket Replication Policy using ObjectScale

UI
ObjectScale UI can be used to manage bucket replication policies used for ObjectScale Replication.
Use the following tasks to manage a bucket replication policy using ObjectScale.

Configure a new bucket replication rule

Using the ObjectScale Portal user interface, configure bucket rules for ObjectScale Replication.

Prerequisites
The user must have access to the versioning-enabled bucket configured for ObjectScale Replication.
Ensure that you have the Bucket Name available. You must use this name in the New Replication Rule wizard.

ObjectScale Replication 117

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.
The bucket Summary tab is displayed by default.
4. Click the Replication tab.
● The data grid displays a listing of existing replication rules.
● NEW REPLICATION RULE button is enabled by default.
● EDIT and DELETE are disabled until an existing replication rule is selected.
● ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and
Receive Objects.
5. Click the NEW REPLICATION RULE button.
● The NEW REPLICATION RULE window opens.
● The Rule tab is opened by default.
6. Fill the mandatory fields in the Rule tab.
a. Enable versioning for the source bucket.
NOTE: This option appears only if bucket versioning was not enabled when the bucket was created.

b. Enter Rule Name.

c. Select Highest (default) or Lowest in Priority.
d. Click to enable (default) or disable Rule Status.
e. Click NEXT.
The Source Bucket tab opens.
7. Fill the mandatory fields in the Source Bucket tab.
a. The Source Bucket Name and Source Account fields are prepopulated.
b. Select Rule Scope.
● Select Entire Bucket to apply this rule to the entire bucket, or
● Select Prefix/Tag to choose a subset of objects with a specific key prefix and specific tags to which this rule applies.
Type the prefix value in the Enter Prefix field.
Type the Key and Value combinations in the Tags fields. Each tag is a combination of Key and Value pair.
○ Click ADD TAG to add more tags.
○ Click DELETE to delete a tag. A minimum of one tag should be provided.
NOTE: The DeleteMarkerReplication setting cannot be configured with a tag filter.

c. Click NEXT.
The Destination Bucket tab opens.
8. Fill the mandatory fields in the Destination Bucket tab.
NOTE: You cannot configure more than one destination bucket at a time.

a. Select Set Destination.

● Select Buckets in current ObjectScale instance, or
● Select Buckets in remote ObjectScale instance.
b. Select the namespace from the drop-down menu next to Namespace.
The drop-down menu displays the namespaces available in the selected ObjectScale instance.

118 ObjectScale Replication

 c. Select the object store from the drop-down menu next to Object Store.
The drop-down menu displays the object stores available in the selected ObjectScale instance.
d. Set the Destination Bucket Account as Choose a bucket in this account or Specify a bucket in another account.
e. If you selected Specify a bucket in another account, click the Object Owner checkbox to change object ownership to
the destination Bucket owner.
f. Type the name of the destination bucket in the Destination Bucket field.
g. Select the source account IAM role from the drop-down menu next to IAM Role from source account.
h. Set the Delete Marker Replication toggle.
For information about this setting, see Delete marker replication on versioning-enabled buckets .
i. Click NEXT.
The Review tab opens.
9. Review the fields in the Review tab and click SAVE.

Edit an existing bucket replication rule

Using the ObjectScale Portal user interface, edit the destination bucket rules that are configured for ObjectScale Replication.

Prerequisites
The user must have access to the versioning-enabled bucket configured for ObjectScale Replication.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.
The bucket Summary tab is displayed by default.
4. Click the Replication tab.
● The data grid displays a listing of existing replication rules.
● NEW REPLICATION RULE button is enabled by default.
● EDIT and DELETE buttons are enabled.
● ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and
Receive Objects.
5. Select the replication rule that you want to edit and click the EDIT button.
The EDIT REPLICATION RULE window opens. The Rule tab is opened by default.
6. Click the section in the wizard that you want to edit, and make the changes.
● Go to Rule to edit the Rule Name or Rule Status.
● Go to Source Bucket to edit the Rule Scope.
● Go to Destination Bucket to edit values for Set Destination, namespace, Object Store, Destination Bucket Account,
IAM Role from the source account, or Delete Marker Replication.
7. Click SAVE.

ObjectScale Replication 119

Delete a bucket replication rule
Using with the ObjectScale Portal user interface, delete one or more replication rules that are configured for ObjectScale
Replication.

Prerequisites
The user must have access to the versioning-enable bucket configured for ObjectScale Replication.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.
The bucket Summary tab is displayed by default.
4. Click the Replication tab.
● The data grid displays a listing of existing replication rules.
● NEW REPLICATION RULE button is enabled by default.
● EDIT and DELETE buttons are enabled.
● ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and
Receive Objects.
5. Select the replication rule you want to remove and click the DELETE button.
A confirmation window opens.
6. Click Yes to delete.
The replication rule is deleted from the replication policy.

Working with bucket replication rules

Using with the ObjectScale Portal user interface, configure rules on a destination bucket that is configured for ObjectScale
Replication.

Prerequisites
The user must have access to the versioning-enabled bucket configured for ObjectScale Replication.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.

120 ObjectScale Replication

 The bucket Summary tab is displayed by default.
4. Click the Replication tab.
The data grid displays a listing of existing replication rules.
5. Select one or more rules for a destination bucket.
● The EDIT button is enabled.
● The DELETE button is enabled.
● Click the ACTIONS button to display additional actions:
○ The Enable Rule(s) button is enabled only if all the selected rule is in a disabled state.
○ The Disable Rule(s) button is enabled only if all the selected rule is in an enabled state.
○ The Edit Priority button is enabled when one or more rules are defined.
○ The Receive Objects button is always enabled.
6. Click either of the buttons.
● DELETE - The selected rule is deleted for the destination bucket.
● Enable Rule(s) - The selected rule is enabled for the destination bucket.
● Disable Rule(s) - The selected rule is disabled for the destination bucket.
NOTE: To EDIT the selected rule for the destination bucket, see Edit an existing bucket replication rule. To Edit
Priority of the selected rule, see Change the priority of bucket replication rules. For more information on Receive
Objects, see Configure the destination bucket to receive objects.

A confirmation window opens.

7. Click:
● Yes to proceed with enable, disable or delete the selected rules for the destination bucket.
● No to cancel.

Change the priority of bucket replication rules

Using with the ObjectScale Portal user interface, change the priority of the rules on a destination bucket that are configured for
ObjectScale Replication.

Prerequisites
The user must have access to versioning-enabled source and destination buckets configured for ObjectScale Replication.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.
The bucket Summary tab is displayed by default.
4. Click the Replication tab.
● The data grid displays a listing of existing replication rules.
● ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and
Receive Objects.
5. Click Edit Priority.
The Edit Priority wizard opens.
6. Click the up or down arrow next to the replication rule to change the priority of a particular rule or rule(s).
7. After changing the priority of the desired rules, click SAVE.

ObjectScale Replication 121

 The storage policy is updated.

Check replication rule statuses

Check if a replication rule is enabled or disabled.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of a bucket.
4. Click Replication.
The table shows all replication rules that are defined for the bucket and whether they are enabled or disabled.

Configure the destination bucket to receive objects

Using the ObjectScale Portal user interface, configure a destination bucket to receive objects.

Prerequisites
The user must have access to the versioning-enabled destination bucket configured for ObjectScale Replication.

Steps
1. From the ObjectScale Portal user interface, click Buckets.
The list of Buckets that the user is authorized to view is displayed.
2. Use the Namespace, Object store, Account, and Bucket name fields at the top of the Buckets page to locate the
bucket.
a. Select the appropriate namespace from the Namespace drop-down on the top of the Buckets page.
b. Select the name of the Object Store that contains the bucket.
c. Select the account from the Accounts dropdown.
To select another account, remove the current account to see all the available accounts. Then, select another account,
by either using the dropdown to display all accounts, or by typing the account id into to the Select an account field to
dynamically filter the list of accounts.
d. Optionally, type at least the first three characters of a Bucket name to filter the list of buckets.
3. Click the name of the bucket to be modified.
The bucket Summary tab is displayed by default.
4. Click the Replication tab.
The ACTIONS drop-down menu is enabled by default, and consists of ENABLE RULE/S, DISABLE RULE/S, EDIT
PRIORITY, and RECIEVE OBJECTS.
5. Click RECEIVE OBJECTS.
The Receive Objects window opens.

122 ObjectScale Replication

 Figure 20. Receive objects
6. Enter the Bucket Policy statement into the Bucket Policy field.
a. Click View and change the field to Text entry mode.
b. Type the bucket policy statement into the text field.
NOTE: Set the bucket replication policy on a target bucket only when you must replicate between buckets across
different accounts.
For example,
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetBucketVersioning",
"s3:ObjectOwnerOverrideToBucketOwner",
"s3:ReplicateObject",
"s3:ListBucketVersions",
"s3:ReplicateTags",
"s3:PutObjectRetention",
"s3:PutObjectLegalHold",
"s3:BypassGovernanceRetention"
],
"Resource": [
"arn:aws:s3:${TARGET_SCALE_ID}:${TARGET_STORE_ID}:${TARGET_BUCKET_NAME}",
"arn:aws:s3:${TARGET_SCALE_ID}:${TARGET_STORE_ID}:${TARGET_BUCKET_NAME}/*"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"${OSR_ROLE_ARN}",
"urn:osc:iam::${SOURCE_ACCOUNT_ID}:root"
]
}
}

ObjectScale Replication 123

 ]
}

c. Click Text and change the field to View mode.

d. Use the toggle to enable or disable Bucket Versioing.
Bucket Version should be enabled before setting Replication configurations.

7. Click SAVE.
The destination bucket starts to receive replicated objects from the source bucket.

Set up ObjectScale Replication using the ObjectScale

API
Before you can set up ObjectScale Replication between two ObjectScale instances within a federation using the ObjectScale
API, you must first have completed the following prerequisites:
1. Installed the primary ObjectScale instance and created an object store.
2. Installed the secondary ObjectScale instance and created an object store.
3. Created an ObjectScale federation consisting of these two ObjectScale instances.
After you have completed these prerequisites, you can now set up ObjectScale Replication. To set up ObjectScale Replication
using the ObjectScale API, do the following:
1. Create and configure an account and an IAM role
2. Set up the ObjectScale to ObjectScale Replication

Create and configure an account and an IAM role

About this task
NOTE: The $OSR_ROLE_ARN in replication configuration can take any valid service role ARN. Create an IAM role and give
permission to enable replication.

Steps
1. Create a global account.
a. Set the environment variables and display the Account ID:

IAMSVC_ENDPOINT=$(kubectl get svc | awk '/-iam\s/{print $3}' )

FEDSVC_ENDPOINT=$(kubectl get svc | awk '/fedsvc\s/{print $3}' )

TOKEN=$(curl -ik -u root:ChangeMe http://$FEDSVC_ENDPOINT:9500/mgmt/login | awk '/

X-SDS-AUTH-TOKEN/{print $2; exit}')

TOKEN=${TOKEN//[$'\r\n']}

b. Create the account and display the Account ID.

ACCOUNT_ID=$(curl -X POST http://${IAMSVC_ENDPOINT}:9400/iam?'Action=CreateAccount'

-H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep 'AccountId' | sed 's/
<AccountId>\(.*\)<\/AccountId>/\1/g' |sed -e 's/^[ \t]*//')

# output: ACCOUNT_ID="a7bf6bfe35ac4277a1a8857da98b3226"

c. Add the new account to the object store.

ENDPOINT=$(kubectl get svc | awk '/-management-gateway/{print $3}' )

124 ObjectScale Replication

 cat >> $HOME/tenant_creation.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<tenant_create>
<account_id>$ACCOUNT_ID</account_id>
<is_encryption_enabled>$ENCRYPTION</is_encryption_enabled>
<is_compliance_enabled>$COMPLIANCE</is_compliance_enabled>
<alias>test</alias>
</tenant_create>

EOF
curl -vks \
-X POST \
-H "X-SDS-AUTH-TOKEN:$TOKEN" \
-H "Content-Type: application/xml" \
-H "X-EMC-Override: true" \
-T $HOME/tenant_creation.xml \
https://${ENDPOINT}:4443/object/tenants/tenant |xmllint --format -; echo

2. Create an IAM role under the global account.

a. Set the environment variables for the new role name:

OSR_ROLE_NAME="osrRole2"

b. Create an IAM role under the global account.

OSR_ROLE_ARN=$(curl http://${IAMSVC_ENDPOINT}:9400/
iam?'Action=CreateRole&RoleName='$OSR_ROLE_NAME'&MaxSessionDuration=43200&AssumeRole
PolicyDocument=%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effe
ct%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Service%22%3A%22crr.objscale.dell.com%
22%7D%2C%22Action%22%3A%22sts%3AAssumeRole%22%7D%5D%7D' -H "x-emc-
namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep
'Arn' | sed 's/<Arn>\(.*\)<\/Arn>/\1/g' |sed -e 's/^[ \t]*//')

# output: OSR_ROLE_ARN="urn:osc:iam::a7bf6bfe35ac4277a1a8857da98b3226:role/osrRole2"

The AssumeRolePolicyDocument is URL encoded JSON.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "crr.objscale.dell.com"
},
"Action": "sts:AssumeRole"
}
]
}
This policy allow ObjectScale Replication services to assume this role.

3. Create a policy to attach to a role.

a. Set the environment variables for the new policy name:

OSR_POLICY_NAME="osrPolicy"

b. Create a policy to attach to a role.

OSR_POLICY_ARN=$(curl http://${IAMSVC_ENDPOINT}:9400/
iam?'Action=CreatePolicy&PolicyName='$OSR_POLICY_NAME'&PolicyDocument=%7B%22Version%
22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Acti
on%22%3A%22s3%3A*%22%2C%22Resource%22%3A%5B%22*%22%5D%7D%5D%7D' -H "x-emc-
namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep
'Arn' | sed 's/<Arn>\(.*\)<\/Arn>/\1/g' |sed -e 's/^[ \t]*//')

ObjectScale Replication 125

 The policy allows permissions on all S3 actions:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"*"
]
}
]
}

4. Attach the policy to a role.

NOTE: The role will have the permission only after you have attached the policy to it.

curl http://${IAMSVC_ENDPOINT}:9400/
iam?'Action=AttachRolePolicy&RoleName='$OSR_ROLE_NAME'&PolicyArn='$OSR_POLICY_ARN -H
"x-emc-namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format

The role can now be used as a replication role in replication configuration.

Set up the ObjectScale to ObjectScale Replication

Steps
1. Create source and destination buckets on the ObjectScale instances. You must enable versioning on both of the buckets.
2. Attach the bucket policy to the source and destination buckets to ensure that the proper privileges are provided to the
account and IAM service role.
3. Use the ObjectScale fedsvc API to GET the provisioned object store:
a. Set the environment variables:

IAMSVC_ENDPOINT=$(kubectl get svc | awk '/-iam\s/{print $3}' )

FEDSVC_ENDPOINT=$(kubectl get svc | awk '/fedsvc\s/{print $3}' )

TOKEN=$(curl -ik -u root:ChangeMe http://$FEDSVC_ENDPOINT:9500/mgmt/login | awk '/

X-SDS-AUTH-TOKEN/{print $2; exit}')

TOKEN=${TOKEN//[$'\r\n']}

b. Use curl to issue the GET objectstores call:

curl -k -X GET http://$FEDSVC_ENDPOINT:9500/fedsvc/objectstores -H "Content-Type:

application/xml" -H "X-SDS-AUTH-TOKEN:$TOKEN" | xmllint --format -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ObjectStores>
<ObjectStore>
<objectStoreId>OSTI5C2F43D1FF525835</objectStoreId>
<objectScaleId>OSCI6081476846ED9A56</objectScaleId>
<k8sNamespace>default</k8sNamespace>
<objectStoreName>ecs-cluster</objectStoreName>
<apiVersion>ecs.dellemc.com/v1beta1</apiVersion>
<version>0.71.2</version>
<status>Available</status>
<initialized>false</initialized>
<creationTimestamp>2021-04-22T10:09:06Z</creationTimestamp>
<uid>15d61162-d218-4069-b09f-aaa86213098e</uid>

126 ObjectScale Replication

 </ObjectStore>
</ObjectStores>

4. Use the ObjectScale API to PUT the replication configuration. In this configuration, you must specify the ObjectScale and
object store ID in the target bucket ARN.
For example:

SRC_BUCKET_NAME="source1" # on object scale 1

DEST_BUCKET_NAME="destination1" # on object scale 2
SCALE_ID="OSCIFFFFFFFFEBC3958D" # object scale 2 object scale id
STORE_ID="OSTIE5BDD14DBC63185E" # object scale 2 object store id
OSR_ROLE_ARN="urn:osc:iam::a7bf6bfe35ac4277a1a8857da98b3226:role/crrRole2"

rm -f $HOME/.osr-rep-config
cat >> $HOME/.osr-rep-config << EOF
<?xml version="1.0" encoding="UTF-8"?>
<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Role>$OSR_ROLE_ARN</Role>
<Rule>
<Status>Enabled</Status>
<Priority>1</Priority>
<DeleteMarkerReplication>
<Status>Enabled</Status> # or Disabled
</DeleteMarkerReplication>
<Destination>
<Bucket>arn:aws:s3:$SCALE_ID:$STORE_ID:$DEST_BUCKET_NAME</Bucket>
</Destination>
</Rule>
</ReplicationConfiguration>
EOF

$HOME/s3curl/s3curl.pl --id=ecsflex --calculateContentMd5 --put=$HOME/.osr-rep-config

-- -v http://$(kubectl get svc | awk '/-s3/{print $3}')/$SRC_BUCKET_NAME?replication
$HOME/s3curl/s3curl.pl --id=ecsflex -- -v http://$(kubectl get svc | awk '/-s3/{print
$3}')/$SRC_BUCKET_NAME?replication

Replication status
The following table describes replication status values.

Replication Description
status
null (no status) Replication is not enabled for this version of the object.
PENDING Replication is in progress. The created or latest metadata, ACL, or tag update is not yet replicated to the
destination. If life cycle configuration is enabled on the source bucket, life cycle actions are suspended
until status is COMPLETED or FAILED.
The PENDING status can mean any of the following:
1. Replication is progressing without issues.
2. Objects are not able to replicate due to administrative pause or a temporary outage such as service
down, network separation, or an unavailable node or disk. In these cases, the system retries replication
periodically until replication is successful.
3. The destination object store or bucket is full or exceeds the user specified quota. In these cases, the
system generates an alert and continuously retries the replication every 15 minutes.
Any metadata, ACL, or tag updates that are made while the object replication status is PENDING are also
replicated to the destination. This replication occurs even if the replication rule is disabled for the object.
COMPLETE All data and metadata for the version of the object is replicated to the destination.
FAILED Replication is permanently stopped because a configuration error is preventing replication. The system
generates an alert that describes the reason for failure.
User action is required to:

ObjectScale Replication 127

Replication Description
status
● Fix the configuration problem. For information about the failure reasons, see Get replication failure
reasons, failed destinations, and remediation hints .
● Retrigger the replication. The system does not attempt to retry a failed replication. To retrigger, you
must copy the failed object version to the latest version. To do so, use the standard S3 operation.

Monitor for failed replications on the ObjectScale Portal UI

Use the Alerts page on the ObjectScale Portal to monitor an ObjectScale instance for failed replications.

Steps
1. From the ObjectScale Portal user interface, click Alerts.
2. Click Show All.
3. Click the filter icon in the SymptomID column and type OBJST-12010.
An OBJST-12010 entry appears for each unresolved failed replication.
4. To see more information about a failure, click the expansion arrow (>) at the beginning of an entry line.
5. In the table of details that appears, read the Messages column.
The message shows the following information:
● Failure error code
● Object name of the failed replication rule
● Version id of the failed replication
● Failure time
6. Copy the object name and the version id for use in debugging the problem.
7. To determine if multiple failure occurrences are related to the same configuration issue:
a. Clear the filter in the SymptomID column.
b. Set a new filter in that column for OBJST-1602.
The OBJST-1620 alert shows a consolidated view of all failed replications for the past 5 minutes.
c. Click the expansion arrow (>) at the beginning of the entry line.
d. In the table that appears, read the Message contents.
The message is a consolidated list of all recent failures and their reason codes.

Get replication failure reasons, failed destinations, and remediation

hints
If replication fails, you can get information about the failure and take appropriate action to fix the problem.

Steps
1. Get the object name and version id of the failed replication.
See Monitor for failed replications on the ObjectScale Portal UI.
2. Get the failure reason by issuing the S3 detailedReplicationStatus API request on the object name and version id.
The format is:

# /root/s3curl/s3curl.pl --id=ecsflex -- 'http://{s3 endpoint}/source/<objname>?

detailedReplicationStatus&versionId=<version-id>' |xmllint --format -

An ObjectScale extension to this S3 call shows the following information about the destination replication status.

Element name Description

<DestinationARN> Replication destination bucket location—If the replication rule contains
multiple destinations, all are listed. You can determine which
destinations failed from the <ReplicationStatus> element.

128 ObjectScale Replication

 Element name Description
<ReplicationStatus> Replication status for the named destination—When this value is
<FAILED>, the next two elements are provided.

<FailureErrorCode> Failure code

<FailureReason> Explanation of the failure code

Here is an example of the command output.

# /root/s3curl/s3curl.pl --id=ecsflex -- 'http://{s3 endpoint}/source/testobj2?

detailedReplicationStatus&versionId=AAABiR-vS2ImsARyFp6AZ1qfns75LZ-FTzUA' |xmllint --
format -

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ObjectDetailedReplicationStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<ObjectName>testobj2</ObjectName>
<VersionId>AAABiR-vS2ImsARyFp6AZ1qfns75LZ-FTzUA</VersionId>
<DestinationReplicationStatus>
<DestinationARN>arn:aws:s3:osci26b00472169e8067:osti5a9f9ecef92d9f85:destination</
DestinationARN>
<ReplicationStatus>FAILED</ReplicationStatus>
<FailureErrorCode>DST_BUCKET_VERSIONING_NOT_ENABLED_ERROR</FailureErrorCode>
<FailureReason>Destination bucket is not versioning enabled, refer to knowledge
base for detailed information and recommended actions.</FailureReason>
<FailureTime>2023-07-04T06:55:30.969Z</FailureTime>
</DestinationReplicationStatus>
</ObjectDetailedReplicationStatus>

3. If the failure reason message is not enough information, you can get remediation suggestions as follows: .
a. Log in to the Dell Support site.
b. Search for the knowledge base article titled ObjectScale Debugging Guide.
c. Search the document for the <FailureErrorCode>.
The document contains detailed steps for debugging and resolving each failure error code.
4. Retrigger the failed replication object after fixing the configuration issue.
To retrigger, copy the failed object version to a new version using the standard S3 copy request.

ObjectScale Replication 129

 8
Platform settings
ObjectScale includes features that together implement a systemwide safe mode.
Topics:
• ObjectScale hardening overview
• List of protected actions
• Hardening with Federation
• Privileged actions approval system (PAAS)
• Platform protection mode
• Account Protection Mode

ObjectScale hardening overview

ObjectScale is secure by default. For added protection, you can enable additional hardening features.
ObjectScale secures its underlying platform by default. There are also certain customer configuration responsibilities regarding
a hardened environment. For information about the default, always-on security features in ObjectScale and customer
responsibilities, see the Dell ObjectScale 1.3 Security Configuration Guide.
ObjectScale includes the following optional hardening features.

Table 32. ObjectScale hardening options

Feature Description Supported platforms
Privileged Actions Approval System PAAS is a workflow for submitting requests Appliance
(PAAS) and gaining approval for certain privileged
Software Bundle
actions.
Account Protection Mode This mode protects S3 data from a single Appliance
bad actor by requiring approval through
Software Bundle
PAAS for certain bucket configuration
changes. This mode is implemented on IAM
accounts.
Platform Protection Mode This mode protects platform configuration Appliance
and infrastructure from changes by a single
bad actor. Authorized users must obtain
approval through PAAS to open an SSH
shell.

List of protected actions

When PAAS is enabled, the following predefined actions require approval by an Approver User.

PAAS privileged actions

The following actions require approval when PAAS is enabled, regardless of whether Platform Protection and Account
Protection modes are enabled or disabled.

130 Platform settings

Table 33. Privileged actions related to PAAS
Action Description Resource (for API) Required role of Management
User who submits the request
Create Approval Adding new Approver Users after the paas security_admin
User. initial two Approver Users requires
approval.
NOTE: The first two Approver Users
are created before PAAS is enabled
and do not need approvals.

Delete Approval Deleting any Approver User requires paas security_admin

User. approval.
Enable Platform Enabling Platform Protection Mode paas security_admin
Protection Mode. requires approval. See the next table for
a list of additional actions that require
approval when Platform Protection is
enabled.
Disable Platform Disabling Platform Protection requires paas security_admin
Protection Mode. approval.
Reset an Approver If an Approver User forgets a password, paas security_admin
User password. a Management User with security_admin
role can reset the password. This action
requires approval.
NOTE: Approver Users can change
their own passwords without
approval.

Platform Protection Mode privileged actions

The following additional actions require approval when Platform Protection Mode is enabled.

Table 34. Privileged actions related to Platform Protection Mode

Action Description Resource (for API) Required role of Management
User who submits the request
Escalate The escalate permission command platform operations_admin
permission. requires approval. This command allows
the user to open a shell-like environment
at the operating system level on the
ObjectScale Appliance.

Account Protection Mode privileged actions

Account Protection is enabled individually per account. When Account Protection is enabled on an account, the following actions
require approval.

Table 35. Privileged actions related to Account Protection Mode

Action Description Resource (for API) Required role of Management
User who submits the request
Disable account Enabling protection on an account does iam operations_admin
protection. not require approval, but disabling the
protection after it is enabled does require
approval.

Platform settings 131

Table 35. Privileged actions related to Account Protection Mode (continued)
Action Description Resource (for API) Required role of Management
User who submits the request
Configure bucket Change bucket lock configurations. objControl
operations_admin
locks. storage_admin
storage_operator

Configure Object Change Object Lock configurations. s3

operations_admin
Lock. storage_admin
storage_operator

Configure object Reduce retention periods on Object Locks s3 operations_admin

retention. with a GOVERNANCE mode lock. storage_admin
NOTE: The initial setting and changes storage_operator
to increase the retention period are
not protected actions.

NOTE: It is not possible to change

the configuration on a COMPLIANCE
mode lock. A PAA request for a
COMPLIANCE mode override does
not work.

Delete with Delete objects under retention with a s3

operations_admin
retention override . GOVERNANCE mode lock. storage_admin
NOTE: It is not possible to override storage_operator
a COMPLIANCE mode lock. A PAA
request for a COMPLIANCE mode
override does not work.

Hardening with Federation

Consider the effects of federation on the hardening features.
In a federated ObjectScale:
● PAAS and Platform Protection mode settings apply to the ObjectScale level. They must be set on each ObjectScale in the
federation.
● Account Protection modes are global because the IAM accounts are global throughout the federation.

Privileged actions approval system (PAAS)

The privileged actions approval system (PAAS) prevents a management user from obtaining root-like privileges and
circumventing security controls. To prevent that type of scenario, the PAAS workflow requires an approval from a second
user before certain management or account actions can occur. The actions that require approval are ones that are high risk if
they were performed by bad actors.
PAAS supports the following ObjectScale protection modes. These protection modes require PAAS to be enabled.
● Platform protection mode—Protects the ObjectScale platform from bad actor risks by requiring approval through PAAS for
a set of predefined configuration actions and operating system access.
● Account protection mode—Protects S3 data from bad actor risks by requiring approval through PAAS for certain predefined
data access and account configuration actions. Account protection mode is enabled separately on each account.
The approvers of actions are called Approver Users and they must be defined as such in the system. PAAS relies on a separation
of duties between Management Users (who submit requests to perform actions) and Approver Users. In ObjectScale, they are
separate entities.

132 Platform settings

 WARNING: The intent of PAAS is that the Management User requesting an action and the Approver User
approving the action are different users. While it is technically possible for customers to create a Management
User and an Approver User that are the same person, such a setup defeats the purpose of PAAS.
Only Management Users can request actions. If IAM users need protected access, they must ask a Management User to submit
the request on their behalf.
Here is a summary of the PAAS workflow:
1. A Management User with security_admin role creates two Approver Users.
2. The two Approver Users must log in at least one time and follow the prompts to change their password. When they do, their
status changes to Registered Approver.
3. When there are two Registered Approver Users, a Management User with security_admin role can enable PAAS.
4. The Management User with security_admin role can now add additional Approver Users. These additions require approval.
5. Management Users with appropriate roles can enable Platform Protection Mode (gloabally) or Account Protection Mode (on
individual accounts).
6. When an action requires approval, a Management User creates the request for the approval. If an IAM user needs an
account action that requires approval, the IAM user must ask a Management User to create the request on their behalf. All
requests for approval appear in the ObjectScale Portal under Administration > Requests.
7. An Approver User logs into the ObjectScale Portal and approves or rejects requests on the Administration > Requests
page.
8. When a request is approved, the Management User can complete the approved action.
The ObjectScale API accommodates this workflow. There are APIs for Management Users to create or cancel requests. There
are APIs for Approver Users to approve or reject requests. Management Users can use APIs to get generated request IDs and
the PAATOKEN on an approved request .
The ObjectScale Portal user interface accommodates most steps in the workflow. In the following sections, the user interface
procedures are described whenever they are available.
All CREATE, UPDATE and DELETE PAAS actions are captured by KAHM. You can view log messages on the Logs page in the
ObjectScale Portal user interface.

Enable PAAS
The Privileged Actions Approval System (PAAS) provides a mechanism for requiring and granting approvals for certain
predefined privileged actions.
For the list of actions that require approval, see List of protected actions .
The following steps are required to enable PAAS. These steps are the responsibility of a Security Administrator.
1. Add two initial Approver Users to the ObjectScale system.
2. Inform the two initial Approver Users that they must log in to the system and change their password, which changes their
status to REGISTERED.
3. Change the PAAS setting to enabled.
NOTE: You cannot disable PAAS after it is enabled. Instead, you can disable the protection modes that depend on PAAS.
They are Platform Protection Mode and Account Protection Mode.

Add the first two Approver Users

At least two registered Approver Users are required to enable PAAS. This procedure describes how to add the first two
Approver Users before PAAS is enabled.

Prerequisites
You must have the security_admin role.

About this task

Use the following procedure to add Approver Users when PAAS is disabled.

Platform settings 133

Steps
1. From the ObjectScale Portal user interface, go to Administration > Security Settings > Approvers.
2. Click Add Approver.
3. Type the email, username, and an initial password for the Approver User that you want to add.
You must remember and communicate the initial password to the Approver User.
4. Click Save.
The new user appears in the table on the Approvers page with a status of UNREGISTERED.
5. Repeat the previous steps for the second Approver User.
6. Provide the initial passwords to the two users and ask them to follow the steps in Approver User registration.
NOTE: The security_admin user cannot reset an Approver User password while PAAS is disabled. Instead, delete the
user and then add another user.

Approver User registration

Approver Users register themselves by logging in and changing their password.

Prerequisites
● You must be the Approver User with the status of UNREGISTERED.
● You must know the initial password that the Management User assigned when creating the Approver entity.

Steps
1. On the ObjectScale Portal user interface Login screen, log in as follows:
a. User name is the new Approver User email address.
They must log in using the configured email address.
b. Password is the initial password that the Management User assigned when adding the Approver User.
c. User type is Approver User.
d. Click Login.
The system displays a message asking the user to reset the password.
2. Follow the prompts to create a password and log in.
The password complexity rules for this password are the same as the rules that are imposed on Management Users. The
rules are viewable on the ObjectScale Portal at Administration > Security Settings > User settings.

Results
This Approver User status is changed to REGISTERED. A security admin can verify this status change on Administration >
Security Settings > Approvers.

Change PAAS status to enabled

This step enables the PAAS controls.

Prerequisites
● At least two Approver Users with REGISTERED status must be defined in the ObjectScale system.
● You must have the security_admin role.

Steps
1. On the ObjectScale Portal user interface, go to Administration > Platform Settings.
2. In the Privileged Actions Approval System section, click the Disabled/Enabled toggle.

134 Platform settings

Manage Approver Users
Managing Approver User entities is the responsibility of a Management User with the security_admin role.

Add more Approver Users after enabling PAAS

After you enable PAAS, the action to add more Approver Users requires approval.

Prerequisites
● You must have the security_admin role.
● PAAS must be enabled.

About this task

The UI offers two ways to add new Approver Users. The first method creates one request for adding one user. The second
method is convenient for creating multiple requests for several users.

Steps
To add one Approver User:
1. On the ObjectScale Portal user interface, go to Administration > Security Settings > Approvers.
2. Click Add Approver.
3. In Is PAA request available?, choose Not Available.
4. Type the email for the new Approver User.
5. Click Send Request
6. Go to Administration > Requests.
The new request to add an approval entity appears with a status of Pending Approval.
7. Monitor the request until the status changes to Approved.
8. When the status is Approved, return to Administration > Security Settings > Approvers > Create.
9. In Is PAA request available?, click Available.
10. Select the requestid.
11. Type a username and initial password for the new Approver User.
You must remember and communicate this password to the Approver User.
12. Click Save.
13. Verify that the new user appears in the table with a status of UNREGISTERED.
14. Contact the user, provide the new Approver User name and initial password, and ask them to follow the steps in Approver
User registration.

Create multiple Approver User requests

Steps
1. On the ObjectScale Portal user interface, go to Administration > Requests .
2. Click Create.
3. Select Create Approval Entity.
4. Type the email address of a user who you want to make Approver.
5. To add another email, click Add Entity.
Continue adding entities until all potential Approvers are listed.
6. Click Create Request.
7. Go to Administration > Requests.
New requests exist for each new email, each with a status of Pending Approval.
8. When their status is Approved, go to Administration > Security Settings > Approvers.
9. Click Create.

Platform settings 135

10. In Is PAA request available?, click Available.
The existing requestids are listed.
11. Click a requestid.
12. Complete the form, supplying the username and initial password, and click Save.
13. Repeat the above actions for the remaining approved requestids.
14. Verify that the new users appear in the table with a status of UNREGISTERED.
15. Contact each user, provide their new Approver User name and initial password, and ask them to follow the steps in Approver
User registration.

Delete an Approver User

You can delete an Approver.

Prerequisites
You must have security_admin role.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Security Settings > Approvers.
2. Click the checkbox at the beginning of the line for the Approver User line that you want to delete.
3. Click Delete Approver.
4. Complete the dialog that appears as follows:

Option Description
PAAS is not Click to confirm that you want to delete the Approver.
enabled.
PAAS is An approval is required.
enabled. ● To submit a request for approval:
a. For Is PAA request available?, answer Not Available.
b. Confirm the email of the Approver that you are requesting to delete.
c. Click Create Request.
d. Monitor the request on Administration > Requests. When the request status is Approved,
return to this page.
● To act on an approved request:
a. For Is PAA request available?, answer Available.
b. Select the PAArequestid from the resulting list.
c. Click Delete.
5. When the system redisplays the Approvers page, refresh the screen to confirm that the entry is removed.

Reset Approver User password

This procedure describes how to recover from a forgotten password for an Approver User.

Prerequisites
● You must be a Management User with security_admin role.
● PAAS must be enabled.
NOTE: You cannot reset an Approver User password when PAAS is not enabled. In that case, to manage a forgotten
password, you can delete the Approver and add it again.

About this task

NOTE: Approver Users can reset their own passwords at any time from the ObjectScale Portal. To do so, click the User
icon in the upper right corner of the main Portal window, and choose Change Password.

136 Platform settings

Steps
1. On the ObjectScale Portal user interface, go to Administration > Security Settings > Approvers.
2. Click the box at the beginning of the entry to select the username.
3. Click Reset Approver Password.
4. Complete the dialog that appears as follows:
● To submit a request for approval:
a. For Is PAA request available?, answer Not Available.
b. Confirm the email of the Approver whose password you are requesting to reset.
c. Click Create Request.
d. Monitor the request on Administration > Requests. When the request status is Approved, return to this page.
● To act on an approved request:
a. For Is PAA request available?, answer Available.
b. Select the PAArequestid from the resulting list.
c. Type a new password and confirm the password.
d. Click Save.
5. Provide the new password to the Approver User with instructions for changing it again on the first login.

Create and manage requests

An ObjectScale request is asking for approval to perform a protected action.
Requests have the following status values:
● Pending Approval—The request is waiting for an Approver User to act on it.
● Approved—An Approver User approved the request.
● Rejected—An Approver User rejected the request.
● Completed—The requester completed the action after approval was obtained.
● Canceled—The requester cancelled the request.
● Expired—The request had a status of Pending_Approval or Approved for some days (the default is 7).
Two users are involved with a request.
1. A Management User creates a request. The Management User must have a role that aligns with the action being requested.
For S3 account actions, the IAM user asks a Management User to submit the request. The IAM user ID is part of the request
payload.
2. An Approver User approves or rejects the request.
3. If the request is approved, the Management User completes the requested action. For S3 account actions, the IAM user
whose ID is in the request must complete the action.
4. Management Users can cancel their own requests that are in Pending Approval status.

Create requests
Only Management Users can create requests. Approver Users and IAM users cannot submit requests.
When Account Protection is enabled, IAM users must perform S3 protected actions. In this case, the IAM user must engage
with a Management User to submit a request on behalf of the IAM user. The request includes the user ID of the requesting IAM
user. That IAM user is the only user who can complete the action after it is approved.
The ObjectScale Portal user interface offers two ways to create requests.
● Go to Administration > Requests > Create. A dropdown menu lists all the commands that require approval. You can select
a command, complete the resulting screen, and submit the request for approval.
● Go to the context-specific UI page that you would use to perform the action if PAAS was not enabled. When you attempt to
perform the action, the UI provides a button that creates the request for approval for the action.
ObjectScale APIs also support creating requests.

Platform settings 137

View requests
You can view and manage requests on the ObjectScale Portal user interface.

Prerequisites
You must be a Management User or Approver User.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Requests.
2. Requests appear as follows:
● If you are a Management User, you see only the requests that you have submitted.
● If you are an Approver User, you see all requests.
3. Use the Portal filter and column sort features to find requests.

Cancel a request
You can cancel one or more requests that you have submitted. Cancelling a request does not require approval.

Prerequisites
You must be the Management User who entered the request to cancel a request.

About this task

You can cancel requests whose status is Pending Approval.

Steps
1. On the ObjectScale Portal user interface, go to Administration > Requests .
2. Select the requests that you want to cancel by clicking the box at the beginning of each entry.
3. Click Cancel.
4. On the dialog that appears, review the list of requests that you are about to cancel and optionally enter text in the
Comments box.
5. Click Cancel Request(s) to proceed with the cancel action, or click No to back out of the cancel action.

Approve or reject requests

An Approver User approves or rejects requests.

Prerequisites
You must be a registered Approver User.

Steps
1. Login in to the ObjectScale Portal as an Approver User:
.
● Open a connection to the Portal so that the Login window is displayed.
● For User Name, type your Approver User name.
● For Password, type your Approver User password.
● For User Type, select Approver.
● Click Login.
2. Go to Administration > Requests.
3. Look for requests with a status of Pending Approval.
You can optionally filter on the Status column.
4. To select a request to approve or reject, click the checkbox at the beginning of the entry.

138 Platform settings

5. Click Approve or Reject.
6. Optionally type a comment to the requester.
7. Click Save.
The request status changes to Approved or Rejected.

Approval tokens
To complete approved requests using the API, you must first get the approval token that is associated with the request. Then
you submit the request using the approval token in the header of the request.
PAAS issues the approval tokens, also known as the PAA token. There are two types of PAA tokens.

Table 36. PAA tokens

Token Type User Description
For Management service Obtained and used by the ObjectScale A JWT token—The token expires in 15 minutes. Users
actions Management User who submitted the can get new tokens multiple times for requests with
request. Approval status. Tokens are not available for requests
with Completed status.
Use the getPAAToken API to get this token.

For S3 actions Obtained and used by the IAM user A temporary token—The token expires in 12 hours. The
whose username is referenced in an S3 IAM user can use this token to perform the same request
request. multiple times. Multiple executions are not harmful.
ObjectScale checks that the IAM user id, the resource
id, and the request payload exactly match the values in
the approved request.
Use the GetFederationToken API to get this token.

Complete approved management service requests

An approved request grants permission to perform the requested action. After approval, a user must complete the request to
finish the workflow.

Prerequisites
To complete a request after approval, you must be logged in as the appropriate user:
● For S3 actions, you must be the IAM user whose user id was entered in the request. The Management User who submitted
the request on your behalf cannot complete an S3 action.
NOTE: This task does not apply to S3 actions. For descriptions and examples for completing approved S3 actions, see
Complete approved S3 requests on protected accounts.
● For other (non-S3) actions, you must be the Management User who submitted the request. This task describes how
Management Users can complete all other action types.

About this task

Management Users can complete some requests on the ObjectScale Portal user interface. This method is quick and convenient.
If the Portal method is not available for the requested action, use the API. The Portal is not available in the following cases:
● The approval occurred more than 15 minutes ago, and the approval token is expired on the ObjectScale Portal. The approval
token is always available through the API for requests with Approved status.
● Some actions require the API method and cannot be completed on the ObjectScale Portal.

Steps
1. To use the ObjectScale Portal user interface to complete a request:
a. Go to Administration > Requests.
b. Click the Request ID of the request that you want to complete.

Platform settings 139

 c. On the Request Details screen that appears, click an action button in the lower right corner indicating that you can
complete the requested action.
d. If there is no action button, click Cancel and use the API to complete the request.
2. To use the API to complete a request, get the approval token. Then submit the request using the token in the header of the
request.
a. Go to Administration > Requests and copy the requestid.
b. Authenticate on the command line.
c. Get the PAATOKEN for the requestid.
Example request to get a token for a management service action:

curl --location 'https://$OSC_GATEWAY_ENDPOINT/paas/requests/{paa-request-id}/paa-

token' \
--header 'Accept: application/json' \
--header 'Authorization: $OSTOKEN'

d. Complete the action by calling the service API, adding the token obtained in the previous step to the header.
Example request for completing an ObjControlsvc action:

curl --location --request PUT \

'https://$OBJECTSTORE_MANAGMENT_GATEWAY:4443/object/bucket/$BUCKET_NAME/object-lock-
config?namespace=$IAM_ACCOUNT' \
--header 'Content-Type: application/xml' \
--header 'Authorization:$PAA_TOKEN' \
--data '<?xml version="1.0" encoding="UTF-8"?>
<ObjectLockConfiguration>
<ObjectLockEnabled>Enabled</ObjectLockEnabled>
</ObjectLockConfiguration>'

Results
The request status changes to Completed in a best-effort manner. The status may not change to Completed even when the
request runs successfully. In those cases, the status moves to the next life cycle status (Expired) after 7 days.

Platform protection mode

Platform protection mode locks the ObjectScale underlying platform, blocking unintended access. This mode prevents a bad
actor from using the underlying platform to gain inappropriate root privileges or bypass data retention policies.
ObjectScale secures its underlying platform by default. Platform protection mode adds another layer of protection on operating
system access. When platform protection mode is enabled, access to the operating system and service pod through SSH (or
other shell sessions) requires approval.
NOTE: SSH access always requires a special OSTOKEN token. The difference in platform protection mode is that the token
request requires approval through PAAS. See the "Hardening" section in the Dell ObjectScale Security Configuration Guide
for information about gaining access to a shell-like environment with and without platform protection mode.
For a list of actions that require approval when Platform Protection Mode is enabled, see List of protected actions .

Enable platform protection mode

You can enable platform protection mode on the ObjectScale Portal user interface. This protected action requires approval.

Prerequisites
● PAAS must be enabled.
● You must have the security_admin role.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Platform Settings.
2. If the mode is disabled, click Approval Request.
3. On the dialog that appears, click Send Request.

140 Platform settings

4. Go to Administration > Requests to view the request and monitor its status.
5. When the request status changes to Approved, go to Administration > Requests, click the requestid, and then click
Enable Platform Protection.

Disable platform protection mode

You can disable platform protection mode on the ObjectScale Portal user interface. This protected action requires approval.

Prerequisites
● You must have the security admin role.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Platform Settings.
2. If the mode is enabled, click Approval Request.
3. On the dialog that appears, click Send Request.
4. Go to Administration > Requests to view the request and monitor its status.
5. When the request status changes to Approved, go to Administration > Requests, click the requestid, and then click
Disable Platform Protection.

Escalated request for operating system access in Platform

Protection Mode
In protected mode, users must gain approval through the PAAS to open a shell-like environment on a node in the ObjectScale
cluster.
Management Users with admin or operations_admin role can request approval for escalated access. For instructions, see
the Escalated request for operating system access in the Platform Protection Mode section in the chapter "Miscellaneous
Configuration and Management" chapter of the Dell ObjectScale 1.3 Security Configuration Guide.

Account Protection Mode

Account Protection Mode protects S3 data from unauthorized activity. It offers an extra layer of protection on data with object
locks in governance mode.
Account Protection Mode is a configuration setting on an IAM account. It is applied individually to each account, making it
possible to set the protection on some accounts and leave it disabled on others. When Account Protection Mode is enabled,
certain actions on buckets in the account must get approval from a second user. The Privileged Actions Approval System
(PAAS) is used to gain approval.
For a list of actions that require approval when account protection mode is enabled, see Privileged actions related to Account
Protection Mode .

Workflow to perform protected actions

The following steps describe the workflow for performing protected actions when Account Protection Mode is enabled.
1. A Management User submits the request to PAAS. For S3 protected actions that IAM users perform, the IAM user is
identified in the payload of the request. For information about various ways that Management Users can submit a request to
PAAS, see Create requests.
2. An Approver User approves the request.
3. Depending on the type of request, users complete the actions as follows:
● Management Users complete approved management service actions. See Complete approved management service
requests.
● IAM Users complete approved S3 actions. (Management Users cannot complete S3 actions.) See Complete approved S3
requests on protected accounts.

Platform settings 141

Account Protection Mode and ObjectScale federation
When Account Protection Mode is set on an account in a federated ObjectScale system, the setting is automatically replicated
to other ObjectScale systems in the federation.
Each ObjectScale performs an account protection check according to data in the local ObjectScale. It is possible that an IAM
account with Account Protection Mode enabled is replicated to an ObjectScale that does not have PAAS enabled. In that case,
the following actions are required for S3 protected actions:
1. The IAM User must contact an Admin in the primary ObjectScale for the IAM account. The primary ObjectScale is the
ObjectScale in which Account Protection Mode was set on the account.
2. The IAM User must ask the admin to create the PAA request in the primary ObjectScale for the protected action.
3. When the request is approved, the IAM User must get the temporary federated PAA token from the primary ObjectScale.
4. The IAM User can use the temporary PAA token in any ObjectScale in the federation.

Account Protection Mode and object locks in GOVERNANCE mode

Object lock GOVERNANCE mode lets users with the correct permissions bypass an object lock and proceed with actions on
data, such as overwrites and deletes.
When Account Protection Mode is set on the bucket account, GOVERNANCE mode does not work as stated above. In Account
Protection Mode, the IAM user must first gain approval through the PAAS before they can bypass the object lock. Because only
Management Users can submit approval requests to PAAS, the IAM user must ask a Management User to submit the request.

Enable Account Protection Mode on an account

Account protection is enabled per account. Enabling this mode does not require approval, although the PAAS must be enabled.
After protection is enabled, certain actions on buckets in the account are protected and require approval.

Prerequisites
● PAAS must be enabled on the ObjectScale platform.
● You must be a Management User with one of the following roles:
○ Admin
○ Storage admin
○ Operations admin

Steps
1. On the ObjectScale Portal user interface, go to Accounts.
2. Select an account by clicking the checkbox at the beginning of the account line.
3. Click Actions > Enable Account Protection.

Disable account protection on an account

Disabling account protection mode on an account requires approval in PAAS.

Prerequisites
● The Privileged Actions Approval System (PAAS) must be enabled on the ObjectScale platform.
● You must be a Management User with one of the following roles:
○ Admin
○ Storage admin
○ Operations admin

Steps
1. On the ObjectScale Portal user interface, go to Accounts.
2. Select the account by clicking the checkbox at the beginning of the account line.

142 Platform settings

3. Click Actions > Disable Account Protection.
4. On the dialog that appears, click Send Request.
5. Go to Administration > Requests to view the request and monitor its status.
6. When the request status changes to Approved, click the request ID, and then click Disable Account Protection.
7. To verify that the protection mode is disabled, return to Accounts and check the value in the Protection column.

Create account requests

Management Users create requests for account actions on behalf of IAM users. The ARN of the requesting IAM user is in the
request.

Prerequisites
● You must be a Management User with an appropriate role.
● You must have all information for completing the action when you submit the request. Coordinate with the IAM user who
owns the account and who would be the person asking for the action.

About this task

For a list of actions that require approval when Account Protection Mode is enabled on an account, see Privileged actions
related to Account Protection Mode . That reference also shows the roles that can perform each protected action.
Each request must include all information that is required to complete the request (the request payload). The ObjectScale Portal
user interface includes customized forms that collect the required payload for each request type. For example, if the request is
to change the lock configuration for a bucket, the payload in the request collects the configuration changes. The payload would
contain the ARN for the requesting IAM user, the object or bucket ARN, the new lock type value, and an optional retention
period.

Steps
1. On the ObjectScale Portal user interface, go to Administration > Requests.
2. Click Create.
3. Select the Request type.
4. Complete the form that appears. Obtain all information from the IAM user who is requesting the action.
5. Click Create Request.
The request appears on the Requests page, with a status of Pending Approval.
6. Monitor the request until the status changes to Approved.
7. When the status changes to Approved, click the Request ID.
The Request Details screen shows all details of the request.
● If the request was for an ObjControl action, the Request Details screen includes action buttons to either cancel or
complete the requested action. The Management User can complete the action using the buttons.
● If the request was for an S3 action, the Request Details screen does not include action buttons. The IAM user whose
ARN is in the approved request must use APIs to get the approval token and complete the action. See the next task for
instructions.

Complete approved S3 requests on protected accounts

An IAM user completes an approved S3 request.

Prerequisites
● The request must have a status of Approved.
● You must be the IAM user whose user id was entered in the request.

Steps
To complete S3 actions, the IAM user must do the following:
a. Get the PAA requestID. One way is to copy it from the Request screen in the ObjectScale Portal user interface. There are
also APIs that list requests.

Platform settings 143

b. Get the PAA approval token associated with the PAA requestID. Use the using the getFederationToken API . Then use
the token to perform S3 requests using the S3 client API.
Example request using the AWS CLI to get a PAA token for an S3 action:

aws sts get-federation-token --name PAA_REQUEST_SESSION --endpoint-url=https://

$OSC_GATEWAY_ENDPOINT/sts \
--tags Key=OSCPAARequestId,Value=$PAA_REQUEST_ID

c. Use the token with the appropriate S3 request to perform the approved action.
Example request for completing an S3 PutObjectRetention request:

export AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID_IN_PAA_TEMPORARY_CRED
export AWS_SECRET_ACCESS_KEY=$SECRET_KEY_IN_PAA_TEMPORARY_CRED
export AWS_SESSION_TOKEN=$SESSION_TOKEN_IN_PAA_TEMPORARY_CRED
aws s3api put-object-retention \
--bucket my-bucket-with-object-lock \
--key object1\
--retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2024-01-01T00:00:00" }' \
--bypass-governance-retention \
--endpoint-url=$OBJECTSCALE_S3_ENDPOINT

Results
The request status changes to Completed in a best-effort manner. Sometimes, the status may not change to Completed
even when the request runs successfully. In those cases, the status moves to the next life cycle status (Expired) after 7 days.

144 Platform settings

 9
Management Users and Roles in ObjectScale
Management Users manage the ObjectScale instance.
NOTE: IAM users are separate entities from Management Users. IAM users are associated with IAM accounts. These users
have permissions to use the S3 protocol to read and write data into buckets in an ObjectScale account. The IAM users are
created and managed in the Manage S3 > Accounts > <account-name> section of the ObjectScale Portal or using the
ObjectScale Management REST API.

NOTE: For ObjectScale for OpenShift deployments, Management Users are limited to management tasks using the
ObjectScale Management REST API. See the ObjectScale Management REST API for usage details.

ObjectScale for OpenShift uses the Kubernetes layer authentication and role assignments to manage access to the
ObjectScale Portal user interface. These users are known as Administrative Users and are created within the OpenShift
cluster.

Topics:
• Management Users in ObjectScale Software Bundle
• Approver users

Management Users in ObjectScale Software Bundle

In the ObjectScale Software Bundle, Management Users can authenticate into the ObjectScale Portal, the ObjectScale
Management REST API, and the Kubernetes command line.
Assigned roles determine the actions that a Management User is authorized to perform. Roles are predefined in ObjectScale and
are specific to various storage use cases. A Management User can have multiple roles.
Roles give Management Users authorization to create and manage users, configure ObjectScale, create and manage object
stores, and view object stores.

Creating and managing Management Users

Management Users can be local users or users in an external authentication provider.
● Local Users are created and assigned roles in ObjectScale, using the ObjectScale Portal or the API.
● External provider users are created in the external provider system. Roles for those external users are assigned in
ObjectScale, using the Portal or the API.
The installation process creates an initial Management User. Management Users with the appropriate roles create other
management users, configure external authentication providers, and map ObjectScale roles to users in external provider
systems.

Auditing of Management Users

ObjectScale maintains audit logs that capture Management User authentications, including which external provider
authenticated the user. Auditing captures all actions that Management Users perform on the ObjectScale Portal or using the
ObjectScale Management REST API.
NOTE: In ObjectScale for OpenShift deployments, auditing of Management Users is limited to actions performed with the
ObjectScale Management REST API. The ObjectScale Portal user interface for OpenShift deployments does not support
Management Users.

Management Users and Roles in ObjectScale 145

Limitations on Management Users
Management Users support in ObjectScale 1.3.0.0 has the following limitations:
1. Management Users in the ObjectScale Portal user interface are relevant only to ObjectScale Software Bundle deployments.
2. Custom-defined roles are not supported.
3. External authentication provider support is limited to Active Directory or LDAP.
4. Audit log support is limited to actions on resources for ObjectScale and object stores using the ObjectScale Management
REST API or UI.

Roles for Management Users

Roles control the actions that are permitted on the ObjectScale Portal, on the Kubernetes command line, and in the API.

Role overview
ObjectScale roles are predefined. Custom roles are not supported.
Management Users with the Admin or Security Admin role can create other users and assign roles or edit roles. A Management
User can have more than one role.
Roles control access to the ObjectScale instance and to individual object stores.
● For ObjectScale, roles grant permissions to configure and monitor the system, configure users, and create and manage
object stores.
● For object stores, roles grant permissions to manage (Edit) or View specific stores.

Roles affect what the user sees on the ObjectScale Portal

The ObjectScale Portal is customized according to the roles of the logged in user. Users only see features that are available to
them. For example:
● If a user does not have access to any administrative actions, the Portal filters out administrative sections when that user
logs in.
● If a user has View access to two object stores, the Portal shows information about only those two object stores.

How to view your assigned roles

In the ObjectScale Portal, you can view the roles that are granted to the username that you used to log in. On any page, click
the username in the upper right corner, and select View Permissions.
Users with appropriate roles can view the role assignments of other users. See View information about Management Users.

Summary of roles
The following table summarizes the available roles in ObjectScale. The roles correspond to typical user personas in a security-
conscious organization.

Role name in the ObjectScale Role name in K8S and the Description
UI ObjectScale API
Admin admin This role grants full control over all management
operations.
Operations Admin operations_admin This role grants full control over all management
operations except for security operations. It includes
Read access to user and public certificates.

146 Management Users and Roles in ObjectScale

Role name in the ObjectScale Role name in K8S and the Description
UI ObjectScale API
ReadOnly readonly This role grants read-only access to everything except
for security information. It includes Read access to user
and public certificates.
Security Admin security_admin This role grants full control over security operations
only. It includes read access for other operations.
Storage Admin storage_admin This role grants full control over storage management,
including creating and deleting object stores.
Storage Operator storage_operator This role grants full control over storage management,
except for creating and deleting object stores.
<namespace>/ <namespace>.<objectstorename This role grants read-only access to the named object
<objectstorename> View >.<ObjectStoreID>.view store.
<namespace>/ <namespace>.<objectstorename This role grants full control over the named object store,
<objectstorename> Edit >.<ObjectStoreID>.edit except for deleting the object store.

Default user
The ObjectScale installation process creates an initial Management User.
The initial user is a Local User with username root and the Admin role. This username is immutable, nor can the username and
role assignment be changed.
The root account cannot be deleted.
The password for this username was set during installation. If the installer did not specifically provide a password, it defaulted to
ChangeMe. On first login, the user is required to change the password.

Password management for local users

Users can change their own passwords. Administrators can change passwords on user accounts and can require a password
change on the next login. Administrators can set and change password policies for local Management Users.
NOTE: This section describes controls that apply to local users. To manage passwords for usernames in external
authentication providers, use the interface for the external provider.

Password complexity
A Management User with Security Admin role can configure password complexity rules. Password complexity rules specify
values for the following password characteristics:
● Minimum password length
● Maximum password length
● Whether at least one of the following character types are required:
○ Lower-case alphabetic character
○ Upper-case alphabetic character
○ Digit
○ Special characters: _-.@!#$%^&* and space
These settings are enforced when a password is set or changed. Contact your Security Administrator for the settings in your
deployment.

Management Users and Roles in ObjectScale 147

Change password
Local Management Users can change their password at any time after they are successfully logged in. On any UI page, click
the down arrow next to the username in the upper right. Then select Change password. The user must know the current
password.

Required password change

When a Security Admin creates or edits a user, they can enable the Required password change option. That option
requires the user to change the password. On the next attempt to log in, the system displays the Password Change screen
and a message states that the password must be changed before proceeding. The user must know the last password value.

Expired password
The Maximum Age of Password security setting specifies how many days a password remains valid. If that duration elapses
without a password change, the system expires the password. When the user attempts to log in, the system displays the
Password Change screen. The user must know the last password value.

Forgotten password
If a user forgets their password, a Management User with Security Admin role can edit the user account and assign a new
password. The Security Admin is recommended to also enable the Required password change option. The Security Admin
must provide the new password value to the user.

Managing Local Management Users

A Management User with Admin or Security Admin role can create more management users and manage those users.
The following actions are available on the ObjectScale Portal at Administration > Management Users:
● View username details—The main page lists all local usernames. You can expand each entry to see details about each
username.
● Create—Create a Management User, assign roles, assign a password, and enable the user.
● Edit—Change user roles, change the password, and enable or disable the user.
● Delete—Delete a user.

View information about Management Users

Using the ObjectScale Portal user interface, you can view information and usage metrics about Management Users.

Prerequisites
You must be a Management User with Admin, Security Admin, Operations Admin, or ReadOnly role.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2.
3. Click Management Users.
The page lists all Management Users that are defined as local users and the following information about each user.

Column Description
Username Management User name
Status Status of the user account:
● Enabled—The user can log in and perform actions according to their assigned roles.

148 Management Users and Roles in ObjectScale

 Column Description
● Disabled—The user cannot log in and has no access to the system. The user information remains in
the system.
First and Last Identifying information about the user, if available.
Name
ObjectScale The number of ObjectScale roles that are assigned to the user.
Roles
Object Store The number of object stores that the user can edit or view. If assigned roles permit access to all object
Roles stores, this number is a count of object stores in the system.
Locked Whether the user is locked out of logins for security reasons. User lockout is a temporary situation. The
rules that define when to lock out a user and the duration of the lockout are configurable. For more
information, see Security settings .

4. To see more details about a user, click the right arrow next to the username.
The user entry expands to show the following information:
● When the account was created and who created it.
● Whether a password change is required
● User email if available
● Last login time
● Last password change date
● Password expiration date
● ObjectScale roles that are assigned to the user
● Object store names that the user has access to and whether that access is View or Edit.

Create a Management User

Using the ObjectScale Portal user interface, you can create a Management User, assign roles, provide an initial password, and
enable or disable the user.

Prerequisites
You must be a Management User with Admin or Security Admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Management Users.
3. Click Create.
4. Configure the new user.

Username Specify the new login username. The information tip on the UI describes the rules for username.
First and Last Name Optional.
Email Optional.
Password Specify an initial password for the user. You can force the user to change the password by
checking the Password Change Required field below. The password must meet the password
complexity requirements that are configured at Administration > Security Configuration.
Enabled Leave Enabled checked to allow the user to log in. Clear the box to save the user account
information but not permit the user to log in.
You can enable and disable a user later on the Edit screen.

Password Change Check this field to force the user to change a password at the first login. The user receives a
Required Change Password form when they try to log in.

Management Users and Roles in ObjectScale 149

 Any user can change their password after logging in by clicking the username in the right corner of
the UI.

ObjectScale Assign permissions to the user by checking the boxes. For more details about the permissions in a
permissions role, see Roles for Management Users.
Object store Assign access permissions to specific object stores as follows:
permissions
NOTE: Some ObjectScale roles give access to all object stores by default. The Portal does not
offer the ability to assign object store permissions to users who already have that permission.
a. Click Add Object store Permissions.
b. Select an object store from the drop-down list.
c. Select the type of access to grant this user by clicking Edit or View.
d. Repeat this process to add access to another object store.

5. Click Save.

Edit a user
Using the ObjectScale Portal user interface, you can edit roles, edit object store access, disable or enable users, and reset
passwords for Management Users.

Prerequisites
You must be a Management User with Admin or Security admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Management Users.
3. Select a user by clicking the box next to the username.
4. Click Edit.
5. Change fields as needed, including:
● Personal information
● Reset the password.
● Disable or enable the user.
● Require a password change.
● Change permissions.
NOTE: You cannot change the username.

6. Click Save.

Delete a user
Using the ObjectScale Portal user interface, remove a Management User account from the system.

Prerequisites
You must be a Management User with Admin or Security Admin role.

About this task

NOTE: This action deletes the user from the system. You can temporarily disable an account without deleting it on the Edit
tab.

150 Management Users and Roles in ObjectScale

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Management Users.
3. Select a user by clicking the box next to the username.
4. Click Delete.
5. Confirm the delete action.

Management User audit logs and alerts

ObjectScale creates log entries for all Management User access to the ObjectScale system. There is one alert type that is
related to Management Users.

The log entries and alerts are Kubernetes events that KAHM can collect. These events are available for viewing on the
ObjectScale Portal or the Kubernetes command line.
Use the following information to filter and view logs and alerts that are related to Management Users.

Application name objectscale-manager

Component Either Management Service or Federation Service

Audit events
The following activities related to Management Users are captured.

Audit Message Type Reason

Management User Login Login
Management User Refresh Token RefreshToken
Management User Password Change PasswordChanged
Management User Password Change Token PasswordChangeTokenGenerated
Management User Logout Logout
Management User Login Failed LoginFailed
Management User Disabled LoginFailed
Management User Locked LoginFailed
Management User Created Created
Management User Updated Updated
Management User Deleted Deleted
Management LDAP Provider Created Created
Management LDAP Provider Updated Updated
Management LDAP Provider Deleted Deleted
Management Local Accounts Config Updated Updated

Alerts
The following alerts are related to Management Users:
● OBJSC-FED-1001—ConnectionStatusChanged

Management Users and Roles in ObjectScale 151

View audit entries in the ObjectScale Portal
Using the ObjectScale Portal user interface, you can view the Management User audit events.

Prerequisites
You must be a Management User with Admin, Operations Admin, or ReadOnly role.

About this task

The ObjectScale Portal shows all events from KAHM. To display only the audit events that are related to Management Users,
you can filter or sort the entries.

Steps
1. From the ObjectScale Portal user interface, click Monitoring.
The ObjectScale Monitoring section is displayed, with the Alerts and Logs that the user is authorized to view.
2. Click Logs.
By default, the page shows all events that were collected over the last 24 hours. You can select another timeframe in the
timeframe drop-down menu.
3. To filter for Management User events:
a. Click the filter symbol in the Component column.
b. Enter Management Service.
The following example shows several Management Service entries.

Get Management User audit entries using the KAHM API

You can get the details of an audit event after it is passed to KAHM using the KAHM API.

Prerequisites
You must be a Management User with Admin, Operations Admin, or ReadOnly role.

Steps
1. Obtain KAHM IP and port number.

# kubectl -n <OBJECTSCALE_NAMESPACE> get svc | grep kahm-restapi

kahm-restapi ClusterIP 10.105.34.45 <none> 17999/TCP 7m59s

152 Management Users and Roles in ObjectScale

2. Set a variable.

KAHMIP=10.105.34.45

3. Obtain KAHM REST API credentials.

$ kubectl -n <OBJECTSCALE_NAMESPACE> get secrets kahm-restapi-secrets -o yaml

apiVersion: v1
data:
credentials.conf:
cmVhbG06IGthaG0tcmVzdGFwaQp1c2VybmFtZToga2FobQpwYXNzd29yZDogQ2hhbmdlTW

4. Using base64, decode the value of credentials.conf to get credentials.

5. Send the request.
Use either Management Service or Federation Service in the filter parameter to identify relevant events.

NOTE: This API does not support OR logic.

curl -G --data-urlencode "filter=component eq Management Service" -k -X GET -u

kahm:ChangeMe http://$KAHMIP:17999/v1/api/events | python -m json.tool | less
{
"id": "objectscale-manager-default-fedsvc-98p22",
"type": "Normal",
"component": "Management Service",
"message": "Management user root logged out of sessionId d3bf6ad3-
ded1-4489-9996-e0333a9f9921",
"reason": "LOGOUT",
"appname": "objectscale-manager",
"namespace": "default",
"createdon": "2023-02-20T21:41:01Z",
"updatedon": "2023-02-20T21:41:01Z",
"resourceID": "d3bf6ad3-ded1-4489-9996-e0333a9f9921",
"count": 1
},

View Management User audit entries on the Kubernetes command line

You can see the details of the original event on the Kubernetes command line.

Prerequisites
You must be a Management User with Admin, Operations Admin, or ReadOnly role.

About this task

The following steps show how to find a Management User event with kubectl commands.

Steps
1. Start a Kubernetes session and log in as a Management User.
2. List application names.

$ kubectl get app

NAME TYPE VERSION OWNER READY AGE
decks decks 2.2.0 6m18s
kahm kahm 2.2.0 6m21s
objectscale-manager objectscale-manager 1.2.0 18h

3. Set a variable for application name.

APPNAME=objectscale-manager

4. Get an event.

Management Users and Roles in ObjectScale 153

 The event in this example shows that there was a failed login attempt for the root user.

$ kubectl get event --field-selector involvedObject.name=$APPNAME,type=Normal -o yaml

- apiVersion: v1
count: 1
eventTime: null
firstTimestamp: "2023-02-21T18:00:21Z"
involvedObject:
apiVersion: app.k8s.io/v1beta1
kind: Application
name: objectscale-manager
namespace: default
kind: Event
lastTimestamp: "2023-02-21T18:00:21Z"
message: Management user root login failed
metadata:
annotations:
eventId: urn:fedsvc:Audit:b48a3cb0-d2ac-4577-9eb9-db25211202c1
creationTimestamp: "2023-02-21T18:00:21Z"
generateName: fedsvc-
labels:
kahm/enabled: "true"
resourceID: root
name: fedsvc-4n5x4
namespace: default
resourceVersion: "5385"
uid: d860fe1d-ed58-4b5c-a8ab-4313c046ad30
reason: LoginFailed
reportingComponent: ""
reportingInstance: ""
source:
component: Management Service
type: Normal

Approver users
Approver users are part of the Privileged Actions Approval System (PAAS). Approver users can only approve or reject PAAS
requests.
For information about the capabilities of approver users and how to add approver users, see Privileged actions approval system
(PAAS).

154 Management Users and Roles in ObjectScale

 10
Authentication Providers in ObjectScale
Software Bundle
You can add authentication providers in ObjectScale Software Bundle if you want users to be authenticated by systems external
to ObjectScale.
An authentication provider is a system that is external to ObjectScale in the ObjectScale Software Bundle that can authenticate
users on behalf of ObjectScale. ObjectScale stores the information that allows it to connect to the authentication provider so
that ObjectScale can request authentication of a user.
In ObjectScale, the following types of authentication provider are available:
● Active Directory (AD) authentication
● Lightweight Directory Access Protocol (LDAP) authentication
NOTE: ObjectScale for Red Hat OpenShift cannot add AD/LDAP authentication providers for external authentication of
users.
Authentication providers can be created from the ObjectScale Portal or by using the ObjectScale Management REST API. You
can use the following procedures to create AD/LDAP authentication providers.

Topics:
• Configuring external authentication providers

Configuring external authentication providers

An external authentication provider authenticates users on behalf of ObjectScale.

ObjectScale supports the following types of external authentication provider:

● Active Directory (AD)
● Lightweight Directory Access Protocol (LDAP)
You can define multiple external providers. User accounts are defined and maintained in the external provider systems. For
permissions control, it is useful if users are added to groups on the external system. As part of configuration, you will map the
predefined ObjectScale roles to users and groups in the external system.
When users log in, they select either Local User or the name of the external provider that should authenticate them.
ObjectScale sends the authentication request to the specified provider. .
The ObjectScale Portal supports all required configurations for integrating an external provider. Actions include:
● Creating external providers and configuring connection information.
● Specifying the users and groups in the external provider that are allowed access to ObjectScale and mapping the
ObjectScale predefined roles to those entities.
● Managing the providers (edit and delete providers).

Create an external authentication provider

Using the ObjectScale Portal user interface, you can add multiple external authentication providers.

Prerequisites
● You must know the connection information for the provider.

Authentication Providers in ObjectScale Software Bundle 155

● The external provider must be running and available on the network. ObjectScale attempts to connect to the provider using
the information that you provide. If the connection is unsuccessful, you cannot create the provider.
● You must know the path names of folders on the provider where the users and groups that need access to ObjectScale are
defined.
● You must be a Management User with the Admin or Security Admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration > Security Settings.
2. Click Authentication Providers.
The screen lists all configured external providers.
3. Click New Authentication Provider.
4. Complete the General screen.

Directory Type LDAP or AD.

Description Optional description of this provider.

Domain Name Domain name of the authentication server. The value must be alphanumeric characters.
For example: myserver.example.com.

NOTE: An IP address such as 10.10.10.1 is not valid in this field.

Base Distinguished Name The starting point for searches on the server. Provide the components in comma-separated
format without spaces. For example:

dc=myserver,dc=example,dc=com

Server Addresses One or more IP addresses for connecting to the provider. For example, 10.10.10.1.
Server Port The port on the provider that receives authentication requests.
If secure is true, then it defaults to port 636. If not provided, it defaults to 389.

Authentication Type This field is preconfigured. ObjectScale supports only username and password
authentication.
Bind User The distinguished name for the user account to use when connecting to the provider. This
user account must have permission to access the users and groups that you intend to add
to ObjectScale. For example:

CN=Administrator,CN=Users,dc=myserver,dc=example,dc=com

Bind Password The password for the bind user account.

Status Specify whether to enable communication when this provider is successfully created in
ObjectScale. The option is set to No by default.

Secure Specify whether SSL communication is required to connect to the provider. The option is
set to Yes by default.

CA Certificate Required if Secure is set to Yes. Provide the contents of the .pem file that holds the
certificate for SSL connection to the provider. You can either:
● Paste the contents of the .pem file into the text box.
● Click Select to browse to the .pem file on your system. ObjectScale copies the
contents of the file and pastes it into the text box.
Network Timeout (sec) Optional but recommended. Specify how long, in seconds, that ObjectScale waits for a
connection to the authentication provider.
Search Timeout (sec) Optional but recommended. Specify how long, in seconds, ObjectScale waits for the
authentication provider to respond to a request.

5. Click Next.

156 Authentication Providers in ObjectScale Software Bundle

6. Complete the Users screen.

User Search Path The distinguished name that describes the folder on the external provider that contains the
users who need ObjectScale access. For example:

CN=users,dc=myserver,dc=example,dc=com

User Name Attribute The attribute name used in the external provider for username values. For example:
sAMAccountName

User Object Class The object class name used by the external provider for users. For example: user

User Inherited Groups Level Optional. Levels of parent groups that are associated with users. For example, 2 would
indicate two levels in the group hierarchy.

7. Click Next.
8. Complete the Groups screen.

Group Search Path The distinguished name that describes the folder on the external provider that contains the
groups whose members need ObjectScale access. For example:

CN=users,dc=myserver,dc=example,dc=com

Group Name Attribute The attribute name used in the external provider for group name values. For example: cn.

Groups Attribute Optional. The attribute name that contains the groups for a user on the AD or LDAP user
entry. For example: memberOf

Group Object Class The object class name used by the external provider for groups. For example: group.

9. Click Next.
10. Review the information and then click Save.
ObjectScale verifies the contents of all fields. If values are rejected, error messages appear on the screen. ObjectScale
attempts to connect to the authentication server using the connection information that you provided. If the connection is
not successful, ObjectScale displays an appropriate error message on the Portal screen. You must correct all errors before
you can save the new provider.
11. To correct errors:
a. Click Back to return to the appropriate screen.
b. Make corrections.
c. Click Next to return to the last screen.
d. Click Save.

Results
A success message appears when the provider is created.

Next steps
Continue to Map ObjectScale roles to external users to assign ObjectScale permissions to users and groups that are defined
in the external provider. Without role mappings, external users in the configured User and Group Search Path fields can log
in but they are immediately logged out. In that case, the ObjectScale Portal displays a message stating that the user has no
ObjectScale permissions.

Authentication Providers in ObjectScale Software Bundle 157

Manage external authentication providers
Using the ObjectScale Portal user interface, you can view, edit, and delete external providers. Editing an external provider lets
you change any configuration information. You can also enable and disable the service.

Prerequisites
To view the list of providers and their configurations, you must be a Management User with Admin, Security Admin, Operations
Admin, or ReadOnly role.
To edit or delete providers, you must be a Management User with the Admin or Security Admin role.

About this task

NOTE: For information about role mapping, see Map ObjectScale roles to external users.

Steps
1. From the ObjectScale Portal user interface, click Administration > Security Settings.
2. Click Authentication Providers.
The screen lists all configured external providers.
3. To view more details about a provider, click the provider domain name.
The domain name is a link to another screen that is specific to that provider.
4. To return to the list of all providers, click Authentication Providers at the top of the screen.
5. To edit attributes for a provider:
a. Click the checkbox next to the provider name.
b. Click Edit.
c. Change one or more values on any of the screens .
d. Click Save. You may click Save after each change or after multiple changes.
6. To delete a provider:
a. Click the checkbox next to the provider name.
b. Click Delete.
c. Confirm that you want to delete the provider.

Map ObjectScale roles to external users

Using the ObjectScale Portal user interface, you can assign ObjectScale roles to external users and groups.

Prerequisites
You must be a Management User with the Admin or Security Admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration > Security Settings.
2. Click Authentication Providers.
The screen lists all configured external providers.
3. Click the domain name of the provider whose users and groups need role mapping.
The domain name is a link to another screen that is specific to that provider.
4. Click Role Mappings > New Role Mapping.
5. Complete the New Role Mapping screen.

Entity Type Select USER or GROUP.

Entity Name Type the entity name. The name must be in the directory that was specified when the
provider was configured.

158 Authentication Providers in ObjectScale Software Bundle

 ObjectScale Permissions Select one or more predefined ObjectScale roles. For more information about roles and the
permissions that are associated with them, see Roles for Management Users.

6. If the user needs permissions to specific object stores, click Add Object Store Permissions. Otherwise, skip this step.
a. Select an object store from the list.
b. Select View or Edit.
c. Repeat these steps multiple times to give the user or group permissions to more object stores.
7. Click Save.

Manage role mappings

Using the ObjectScale Portal user interface, you can view, edit, and delete role mappings.

Steps
1. From the ObjectScale Portal user interface, click Administration > Security Settings.
2. Click Authentication Providers.
The screen lists all configured external providers.
3. Click the domain name of the provider whose role mappings you want to manage.
The domain name is a link to another screen that is specific to that provider.
4. Click Role Mappings.
The screen lists the existing role mappings for users and groups in the provider.
5. To view more details about a mapping, such as assigned roles, click the right arrow next to the user or group name.
This action expands the entry and shows all configured information for the user or group.
6. To edit the role mappings for a user or group:
a. Click the checkbox next to the user or group name.
b. Click Edit.
c. Change the role selections.
d. Click Save.
7. To delete a role mapping:
a. Click the checkbox next to the user or group name.
b. Click Delete.
c. Confirm that you want to delete the role mapping.

Authentication Providers in ObjectScale Software Bundle 159

 11
ObjectScale Administration
Topics:
• About the Administration section of the ObjectScale Portal
• Installing and maintaining the health of ObjectScale
• Licensing ObjectScale
• System support
• SAML Service Provider Metadata
• Manage ObjectScale Certificates
• Security settings
• Active sessions
• ObjectScale Upgrades

About the Administration section of the ObjectScale

Portal
The Administration tab consists of the ObjectScale settings options that you can view and configure.
The Administration tab includes the following sections.

Table 37. Administration section in the ObjectScale Portal user interface

Name Description
ObjectScale Administers the ObjectScale entity at the top level. The ObjectScale screen contains the following
tabs:
● Federation
● Remote Instances
● Object Stores
● ObjectScale Certificates
Licensing Supports ObjectScale licensing requirements. The screen includes the following tabs:
● Apply
● Delete
● Generate Activation XML
SupportAssist Configures SupportAssist.
SAML Metadata Configures SAML.
Requests Supports approval and rejection activities for the Privileged Actions Approval System (PAAS).
Platform Settings Contains platform security hardening settings.
Security Settings Assists Security Administrators in creating and managing users and configuring security-related system
settings. The Security Settings screen contains the following tabs:
● Management Users
● Authentication Providers
● Approvers
● User Settings
Active Sessions Permits Administrators to view all active UI sessions and cancel sessions if needed.
Upgrades Assists in upgrading ObjectScale to newer software bundles.

160 ObjectScale Administration

 NOTE: Some UI sections do not appear when ObjectScale is deployed on Red Hat OpenShift clusters.

Installing and maintaining the health of ObjectScale

In order for ObjectScale to function as efficiently as possible, you should deploy and maintain it according to the recommended
guidelines.

Installing and configuring ObjectScale

For detailed instructions on installing and configuring ObjectScale, see the ObjectScale Installation Guide for your deployment
type.
● Dell ObjectScale Software Bundle Installation Guide
● Dell ObjectScale Application Installation Guide for Red Hat OpenShift
● ObjectScale XF960 Installation Guide

Licensing ObjectScale
ObjectScale can be licensed in several different ways. This section describes the different types of available licenses and how to
install a license.

About ObjectScale Licensing

ObjectScale provides various deployment models with different licensing options.
ObjectScale requires a valid license in order to create an object store or to configure SupportAssist. The types of license
available for ObjectScale are:
● Permanent
● Subscription
● Evaluation
● Community Edition
ObjectScale licenses are available in two forms: one with Data at Rest Encryption (D@RE) and one without. Dell Technologies
recommends that all users have the ObjectScale software with D@RE except where D@RE is not lawful. For customers who
should have access to D@RE, the ObjectScale license file includes D@RE. When this license is applied to ObjectScale, the
feature is initialized and available.

Permanent license
ObjectScale supports a permanent license. Customers using a permanent license have full access to all ObjectScale features
and capabilities and the license does not expire. After purchasing the license, you must activate the license at https://
licensing.emc.com/.
Customers with valid site ids can configure SupportAssist and rely on all its features.
You can apply a new Permanent license to expand capacity as necessary. This action retains the object stores, buckets, and
other settings and configurations that were made to ObjectScale while the previous license was applied.

Subscription license
ObjectScale supports a subscription license. Customers using a subscription license have access to all ObjectScale features and
capabilities, up to the subscribed capacity, until the subscription is no longer active. After purchasing the license, you must
activate the license at https://licensing.emc.com/.
Customers with valid site ids can configure SupportAssist and rely on all its features.

ObjectScale Administration 161

You can apply an expanded or updated Subscription or Permanent license. This action retains the object stores, buckets, and
other settings and configurations that were made to ObjectScale while the previous license was applied.

Evaluation license
ObjectScale supports an evaluation license. The evaluation license acts as a short-term license for trials or evaluation of
ObjectScale. The evaluation license does not require activation before using in ObjectScale.
Evaluation licenses can carry restrictions for how ObjectScale and object stores can be configured, and the period that the
license is valid. These attributes are described within the license file and are enforced by ObjectScale, until the license is no
longer valid.
To extend an evaluation license to a longer time or change the licensed capacity, request a new evaluation license and apply it to
the ObjectScale instance.
You can apply a Subscription or Permanent license. This action retains the object stores, buckets, and other settings and
configurations that were made to ObjectScale while the previous license was applied.

Community Edition capacity-limited license

Dell provides a Community Edition, capacity-limited license on the ObjectScale product page in Dell Support (https://
www.dell.com/support/home/en-us/product-support/product/objectscale/drivers). This Community Edition license does not
expire, but is limited to a maximum overall capacity of 30 TiB. The Community Edition license does not require activation before
using in ObjectScale.
ObjectScale instances using the Community Edition license must meet the minimum hardware and software requirements that
are found in the ObjectScale installation guides.
Customers with valid site ids can configure SupportAssist and allow it to send telemetry dial homes back to Dell Support.
SupportAssist on ObjectScale instances using the Community Edition license does not allow you to send issues or allow remote
dial-in by Dell Support.
You can apply an Evaluation, Subscription, or Permanent license. This action retains the object stores, buckets, and other
settings and configurations that were made to ObjectScale while the Community license was applied.

More details about activating purchased ObjectScale licenses

After purchasing an ObjectScale license, Dell sends License Activation Code (LAC) letter to the customer-provided email
address associated with the Dell Support account. This email contains the necessary information and steps that you must follow
to activate the ObjectScale license.
Go to https://licensing.emc.com/ and follow the online process to generate license files or keys from their entitlements. License
activation occurs after Customers or Partners receive a License Authorization Code (LAC) letter and redeem the LAC number in
their LAC letter.
● Dell issues new license entitlements to a customer based on a purchase, evaluation, or other event. The entitlements are
associated to a unique License Authorization Code (LAC).
● A LAC can have one or more entitlements that are associated to it. A LAC is the primary identifier of the entitlements, which
the customer or partner uses to locate and activate licenses.
● When a LAC is generated, an email is sent to the customer. Evaluation licenses or purchases of certain products may not
generate a LAC.
When your software order is fulfilled, you receive an email or letter that includes the LAC for your order and instructions for
activating entitlements online.
If you have any questions about your Dell order, contact your Dell Sales Account Representative or your Authorized Reseller.
If you have any questions about Dell software licensing, contact the Licensing Support team.
When the license is activated, Dell Licensing sends a Dell software license activation notification email. This email notifies you of
the software licenses that are associated with your Dell account that were activated.
Review the details within the email and contact Dell support if you think this license activation is erroneous or unintended.
The activated ObjectScale license .xml file is attached to this email notification. Use this license .xml file within the
ObjectScale Portal UI to activate the product. For more information, see Apply an ObjectScale License.

162 ObjectScale Administration

Manage ObjectScale licenses
An ObjectScale instance requires a valid license in order to create an object store or to configure SupportAssist.

Apply an ObjectScale License

Use the ObjectScale Portal user interface to activate an ObjectScale license and apply the license file to the ObjectScale
instance.

Prerequisites
To obtain the XML license file from the Dell license management website, you must have the License Authorization Code (LAC),
which is emailed from Dell. If you have not received the LAC, contact your Customer Support representative.

About this task

Activating ObjectScale with a valid license allows you to create object stores.
● Subscription and Permanent licenses allow you to create object stores with an overall capacity greater than 30 TiB, within
the licensed capacity.
● The Community Edition license allows you to create object stores up to an overall capacity no larger than 30 TiB, and limits
SupportAssist features.
To add a license:

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Licensing.
3. Click Apply.
The Apply License box appears.
4. Click Select to browse and upload the ObjectScale license file. Once uploaded, click Apply.
5. Expand the license in the Licensing table to display details about the ObjectScale license and its enabled features and
capacities.

Delete an ObjectScale license

Use the ObjectScale Portal user interface to remove a license, either when removing ObjectScale or to replace the existing
license.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Select Licensing.
3. Click Delete.
The Delete License box appears.
4. Select the license to remove and click Apply.

System support
You can use the ObjectScale Portal user interface to manage and modify the SupportAssit settings that are typically configured
during installation.

ObjectScale Administration 163

About SupportAssist
SupportAssist provides a network based connection to Dell Support. SupportAssist enables Dell Support to receive telemetry
and issues, events, and alerts from your ObjectScale instance, and to perform remote troubleshooting, resulting in a fast and
efficient time to resolution.
NOTE: Dell strongly recommends that you enable the SupportAssist feature to accelerate problem diagnosis, perform
troubleshooting, and help speed time to resolution. If you do not enable the SupportAssist feature, you may need to collect
appliance information manually to assist Dell Support with troubleshooting and resolving problems with your appliance.
The SupportAssist feature employs multiple security layers throughout each step in the remote connectivity process to ensure
that you and Dell can use the solution with confidence:
● All notifications to Dell originate from your site - never from an outside source - and are kept secure through the use of
Advanced Encryption Standard (AES)-256 bit encryption.
● IP-based architecture integrates with your existing infrastructure and maintains the security of your environment.
● Communications between your site and Dell are bilaterally authenticated using RSA® digital certificates.
● Only authorized Dell Customer Service professionals verified through two-factor authentication can download the digital
certificates needed to view a notification from your site.
● The optional SupportAssist v3 Policy Manager application enables you to grant or restrict Dell Support access based on your
own unique guidelines and requirements, and includes a detailed audit log

Connect to Dell support services through SupportAssist

Use the ObjectScale Portal user interface to establish a connection through SupportAssist to ensure access to Customer
Support. SupportAssist enables ObjectScale to connect to Dell support services directly or through a gateway server.

Prerequisites
1. For SupportAssist connectivity you can connect directly with access to the Dell-maintained FQDN: esrs3-
core.emc.com:443.
2. You have applied a valid license to the ObjectScale instance.
3. You are an active Dell customer with login access to https://www.dell.com/support/home/.
4. You must obtain an access key and pin from Dell in order to configure SupportAssist for the first time. This access key and
pin ensure the accuracy of contact and other customer values and access to Dell Support. To obtain an access key and
pin, go to https://www.dell.com/support/home/en-us/product-support/product/objectscale/overview and click Generate
Access key. After completing the required form, Dell sends an email to the email address they have set up for the Dell portal
login. The email is from the "Dell | ServicesConnectivity Team" and contains the site ID, access key, and pin for the selected
customer.
NOTE: The generated access key is valid for seven days.
5. See "SupportAssist port requirements" listed in the ObjectScale Administration Guide and validate that the required ports
are configured properly before configuring SupportAssist.
6. If you are planning on connecting using a Gateway server, ensure that a Dell Secure Connect Gateway (SCG 5.1x) server is
configured on site.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click the SupportAssist tab.
3. Accept the Connect to SupportAssist End User License Agreement.
4. On the Connect to SupportAssist page, select the Select Connection Type to determine which SupportAssist connectivity
type to deploy:
● Select Connect Directly to Dell Support services to connect ObjectScale directly to Dell.
NOTE: Dell highly recommends deploying ObjectScale SupportAssist using a Secure Connect Gateway.
● Select Connect via Gateway Server to connect ObjectScale to Dell through a Secure Connect Gateway (SCG) server.
You must add the Priority, Gateway IP/Host, and Gateway Port values for the gateway server in the Connect via
Gateway Server option.

5. On the Access Key Portal page, select the Model and Software Instance.

164 ObjectScale Administration

6. Configure the Access Key value for Site ID in the Party Number field, and then click Submit. The Access Key is generated,
and the details of the Access Key is emailed to the registered email address.
7. On the Access Key & PIN SupportAssist page, enter the Access Key and PIN SupportAssist values for SiteID, Access Key,
and PIN, and then click Next.
8. Select the Support Contacts tab to add existing Primary or Secondary contacts.
a. Provide the listed values for the Primary contact.
● First Name
● Last Name
● Email address
● Phone number
● Preferred Language
b. Click Add Secondary Contact and provide the required values to configure the contact.
9. Click Apply.
10. In the Connection tab verify that the SupportAssist connection was successful and matches the example.

Figure 21. Successfully configure SupportAssist

If a Failed Status is shown, view the Status Message to determine the failure details.
11. Click Test Connectivity to validate the SupportAssist connection. When successful, the value that is shown in the Last
Connected column is updated with a newer date and time.
As required, use the panel refresh icon to update the screen before automatic updates.
12. Remote support allows authorized Remote Support engineers to troubleshoot your ObjectScale instance. Enable Remote
Support on the SupportAssist page, Connection tab. You can also click the EDIT button and enable Remote Support on the
Connect to SupportAssist page.

SupportAssist port requirements

Dedicated SupportAssist ports required for ObjectScale SupportAssist and other network traffic.

Port Protocol Direction Description

22 TCP Inbound from SRS Gateway to ObjectScale ● SSH
● Secure Copy (SCP)
● Secure File Transfer
Protocol (SFTP)
9443 TCP Outbound from ObjectScale to Secure Connect Gateway Secure Connect Gateway
5.14 or greater
443 TCP Outbound from ObjectScale to Direct Connect Dell Connectivity Direct
Connect
8443 TCP Outbound from ObjectScale to Direct Connect Dell Connectivity Direct
Connect

8443 TCP Inbound from SRS Gateway to ObjectScale Secure Connect Gateway
5.14 or greater

ObjectScale Administration 165

Update or configure SupportAssist contact details
Use the ObjectScale Portal user interface to provide contact information for the person that Customer Support will contact
with diagnostic reports. You can add or update contact details for SupportAssist at any time.

Prerequisites
You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

About this task

Add, modify, or remove SupportAssist contact details:

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SupportAssist.
3. Click Support Contacts tab to modify or delete primary and/or secondary contacts using the EDIT and DELETE buttons.
To manage a primary contact, complete the following steps:
a. Enter the following information:
● First Name
● Last Name
● Email address
● Phone number
b. Select the Preferred Language from the list.
c. To add a secondary contact, click Add Secondary Contact and enter the required information.
4. Click Apply.

Change SupportAssist connection settings

Use the ObjectScale Portal user interface to change SupportAssist connection settings.

Prerequisites
You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

About this task

To change SupportAssist connection settings:

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SupportAssist.
3. On the Connection tab, click the EDIT button, and then select the Select Connection Type option to set the
SupportAssist connectivity type:
● Select Connect Directly to Dell Support services to connect ObjectScale directly to Dell.
NOTE: Dell highly recommends deploying ObjectScale SupportAssist using a Secure Connect Gateway.
● Select Connect via Gateway Server to connect ObjectScale to Dell through a gateway server.
You must add the Priority, Gateway IP/Host, and Gateway Port values for the gateway server in the Connect via
Gateway Server option.

4. Remote support allows authorized Remote Support engineers to troubleshoot your ObjectScale instance. Enable Remote
Support on the SupportAssist page, Connection tab. You can also click the EDIT button and enable Remote Support on the
Connect to SupportAssist page.
5. Enter the Access Key & PIN SupportAssist values for SiteID, Access Key, and PIN.

166 ObjectScale Administration

6. Click Apply.

Modify advanced SupportAssist settings

Use the ObjectScale Portal user interface to configure SupportAssist advanced settings to allow ObjectScale to generate
Automatic Support Requests, change the system mode, or to re-authenticate SupportAssist.

Prerequisites
You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SupportAssist.
3. Click Advanced to modify the system mode, enable Automatic Support Requests, or to re-authenticate SupportAssist.
When modifying the system mode, you are able to set the ObjectScale instance to PreProduction, Normal, or
Maintenance.

Disable SupportAssist
Use the ObjectScale Portal user interface to disable SupportAssist.

Prerequisites
You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SupportAssist.
3. Click Disable, and then click Yes to disable SupportAssit services for ObjectScale.

Remove SupportAssist
Use the ObjectScale Portal user interface to remove the SupportAssist software on ObjectScale.

Prerequisites

NOTE: After you remove SupportAssist, you have to create a PIN and obtain a new access key.

You are logged in to ObjectScale, and SupportAssist has been previously configured to run on ObjectScale.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SupportAssist.
3. Select Delete to remove SupportAssist services from ObjectScale.
4. Select the I wish to proceed with the delete checkbox, and then click Yes.
5. Click Apply.

ObjectScale Administration 167

Monitoring system health
In addition to the summary system health view provided in the ObjectScale Dashboard, the ObjectScale Portal user interface
provides detailed health information for the various object stores and the ObjectScale system overall.
Throughout ObjectScale there are processes that are constantly monitoring and collecting information on the ObjectScale
instance and object stores. When the status of a component or operation changes, the change is captured and noted in the
following places in ObjectScale:
● Monitoring > Alerts
● Monitoring > Logs
● Manage S3 > ObjectScale > Object Stores > <OBJECT_STORE_NAME> > Health
For more information on these portions of the ObjectScale Portal user interface, as well as detailed information on ObjectScale
alerts, see Alerts within this guide.

SAML Service Provider Metadata

Security Assertion Markup Language (SAML) is an open standard for exchanging authentication and authorization data between
parties, in particular, between an identity provider and a service provider.

Java Key Store The Java Key Store containing keys required to log in to SAML Provider
Key Alias The Key Alias for the key for SAML Service Provider
Key Password Password for Key Store for SAML Service Provider
DNS Base URL The DNS Base URL required to connect to the SAML Provider Server

Generate SAML Service Provider Metadata

Use the ObjectScale Portal user interface to generate ObjectScale metadata XML to configure ObjectScale trust relationship
with the identity provider. The generation requires a java key store and a DNS-domain-name which will be used as the entity
Base URL to set the Location in the Assertion Consumer Service.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click SAML Service Provider Metadata.
3. Click Choose to select a Java Key Store.
4. Enter the details in the Key Alias, Key Password, DNS Base URL fields.
5. Click GENERATE.

Next steps
If you need to delete the SAML Service Provider Metadata, click DELETE METADATA.
If you need to download this SAML Service Provider Metadata, click DOWNLOAD METADATA.

Manage ObjectScale Certificates

An ObjectScale instance is protected by a system-wide Secure Sockets Layer (SSL) protocol certificate.
The following three services are protected using this system SSL certificate:
● S3: Used to create, update, and delete S3 objects and buckets using the S3 protocol.
● Management-gateway: Used to manage the object stores.
● ObjectScale-gateway: Used to expose the customer-facing ObjectScale services such as Identity and Access Management
(IAM), Federation, and DCM.

168 ObjectScale Administration

You can specify the type of certificate used by ObjectScale. ObjectScale supports either a Self-Signed Certificate, which is the
default certificate type when you install the ObjectScale, or you can provide your own certificate.
The Administrator and Security Administrator roles can review or manage certificates for the ObjectScale instance on the
Administration > ObjectScale Certificates page in the ObjectScale Portal user interface.
For more information on managing ObjectScale certificates, see "Certificate management and rotation" in the ObjectScale
Security Configuration Guide.

Security settings
User security settings are related to local user accounts. The settings include password complexity and account lockout rules.

Set password complexity rules

Use the ObjectScale Portal user interface to configure the password complexity rules for Local Users.

Prerequisites
You must be a Management User with Administrator or System Admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Security Configuration.
3. Under Password Rules, make changes as needed to meet your password complexity goals.

Field Description Default Allowed Range

value
Minimum Number of The minimum password length for local accounts. 9 8-31
Characters
Maximum Number of The maximum password length local accounts. 63 31-127
Characters
Maximum Age of Password The number of days that a password remains valid. 30 7-365
(days) If this duration elapses without a password change,
the system expires the password. The user is
required to change the password on the next login.
Special Character Required Whether at least one special character is required True
in a password.
Upper Case Required Whether at least one upper case letter is required True
in a password.
Lower Case Required Whether at least one lower case letter is required in True
a password.
Digit Required Whether at least one digit is required in a True
password.

4. To save your changes, click Save.

To discard your changes and redisplay the current settings, click Reset.

ObjectScale Administration 169

Set lockout rules
Use the ObjectScale Portal user interface to configure the session handling and user lockout policies for Local Users. User
lockout refers to login constraints after failed login attempts.

Prerequisites
You must be a Management User with Administrator or System Admin role.

Steps
1. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
2. Click Security Configuration.
3. Under Account Lockout Settings, make changes as needed to meet your session handling and user lockout goals.

Field Description Default value Allowed Range

Idle Timeout (sec) The length of time, in seconds, that a UI session 300 0-3600
can be idle. When this time elapses, the session
ends. The user can log in again.
Lockout Threshold The number of failed login attempts that 10 0-30
triggers a lock out. The failed login attempts
counter is reset to zero as specified in the
Lockout Counter Reset After setting. The
duration of the lockout is specified in the
Lockout Duration setting.
If this value is 0, failed login attempts do not
cause user lockouts.

Lockout Duration The length of time, in seconds, that an account 60 0-3600

(sec) is locked after the number of failed login
attempts triggers a lockout. After the duration
time elapses, the user may attempt to log in.
If this value is 0, no lockout occurs.

Lockout Counter The elapsed time, in seconds, that causes a 50 0-3600

Reset After (sec) reset of the failed attempts counter. This is the
time between the last failed login attempt and
the current login attempt The value must be
less than or equal to the Lockout Duration
value. A blank input resets the counter to zero.

4. To save the changes, click Save.

To discard changes and redisplay the current settings, click Reset.

Active sessions
The Active Sessions screens list all active sessions running on the ObjectScale system. The screens provide authorized users
with the capability to immediately cancel one or multiple active sessions.
You can view and cancel actively running sessions that were started by the following types of users:
● Management Users
● Authentication Provider users—This tab identifies the Authentication Provider and the specific user logged in under the
provider.
● Approver Users

170 ObjectScale Administration

View active sessions
You can list all active sessions and view detailed information about each session.

Prerequisites
You must have admin or security_admin role to view active sessions.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Active Sessions.
2. Click the tab for the type of user that you want to view.
Choose Management Users, Authentication Providers, or Approvers.
The tables show the following information about user sessions:
● Session ID
● User ID
● Information about the authentication tokens
● Token Expires indicates the length of time until the session is set to expire.
● Token Last Updated indicates the length of time since the session was created or refreshed.
● Refresh Count indicates the number of times that the session has been refreshed.
The label Current Session indicates the viewer's session.

Cancel active sessions

You can cancel one or multiple active sessions. The system does not permit you to cancel your own active session.

Prerequisites
You must have admin or security_admin role to cancel sessions.

Steps
1. From the ObjectScale Portal user interface, go to Administration > Active Sessions.
2. Click the tab for the type of user that you want to cancel.
Choose Management Users, Authentication Providers, or Approvers.
3. Click the selection box at the beginning of the entries for the sessions that you want to cancel.
You may select one or multiple lines. You cannot select your own session, which is identified with the Current Session
label.
4. Click Close.
5. Confirm that you want to end the selected sessions.
The system immediately closes and logs out the selected sessions.

ObjectScale Upgrades
Use this section to understand and complete upgrades to newer versions of ObjectScale, its components, and object stores.

ObjectScale Administration 171

ObjectScale 1.3 upgrade considerations
If you upgrade to ObjectScale 1.3 from an earlier release, be aware of the following effects of new features on existing
configurations. Contact Dell Support for ObjectScale upgrade from version 1.2.x to 1.3.0.

Table 38. Upgrade considerations for ObjectScale 1.3

ObjectScale 1.3 new feature Upgrade considerations
Replication of S3 delete markers In previous releases, the S3 delete marker in versioned buckets was not replicated. In this
release, the destination bucket includes the DeleteMarkerReplication setting. You
can configure this setting to ENABLE or DISABLE delete marker replication.
Be aware of the following effects on existing policies:
● In upgrade situations when a policy was defined in an ObjectScale version
earlier than 1.3, the XML files for the older policy do not include the
DeleteMarkerReplication setting. The default behavior in this case matches the
DISABLED setting.
● If you edit an older replication policy after installing ObjectScale 1.3, you must add
and set a value on the DeleteMarkerReplication setting as part of the editorial
change. You can set it to ENABLED or DISABLED. For consistency with how replication
worked in your prior releases, choose DISABLED.
Support for larger multipart ObjectScale 1.3 supports MPUs for objects up to 50TiB in size and up to 50,000 parts.
uploads (MPUs) Upgrade considerations include:
● ObjectScale 1.3 significantly increases efficiency, reduces timeouts, and reduces
memory usage in MPU processing.
● It is acceptable to start uploading parts in ObjectScale 1.2 and continue uploading parts
for the same object after an upgrade to ObjectScale 1.3.
● The maximum size of replicated objects also increases to 50TiB when both the source
and destination are upgraded to ObjectScale 1.3.

Upgrade ObjectScale on OpenShift

Upgrade ObjectScale on OpenShift

Steps for upgrading ObjectScale version 1.3.0 to 1.3.x on OpenShift deployment.

Prerequisites
See Dell ObjectScale Application 1.3.x Installation Guide for Red Hat OpenShift for prerequisites and preparatory steps.

Steps
1. Upgrade CSI Components.
a. Create environment variables.

export CSI_VERSION=1.3.0-648.59a295a
export CSI_OPERATOR_VERSION=1.3.0-121.2e006fb
export CSI_CHARTS_DIR=<CHART_DIRECTORY>/1.3.0-121.2e006fb

NOTE: <CHART_DIRECTORY> is the directory that you created and where you downloaded the ObjectScale
charts.tgz files.

b. Get existing installation values from helm.

NOTE: <CSI_NAMESPACE>

172 ObjectScale Administration

 is the namespace where CSI components are installed.

helm -n <CSI_NAMESPACE> get values csi-baremetal -o yaml > csi-baremetal.yaml

helm -n <CSI_NAMESPACE> get values csi-baremetal-operator -o yaml > csi-baremetal-
operator.yaml
helm -n <CSI_NAMESPACE> get values csi-baremetal-alerts -o yaml > csi-baremetal-
alerts.yaml
helm -n openshift-secondary-scheduler-operator get values secondary-scheduler-
operator -o yaml > secondary-scheduler-operator.yaml
helm -n objectscale get values objs -o yaml > objectscale-portal-values.yaml

c. Upgrade CSI Secondary Scheduler.

helm upgrade -n openshift-secondary-scheduler-operator secondary-scheduler-

operator $CSI_CHARTS_PATH/secondaryscheduleroperator-1.3.0-121.2e006fb.tgz -f
secondary-scheduler-operator.yaml --set global.registry=$REGISTRY --set
csv.version=secondaryscheduleroperator.v1.1.2 --set image.tag=1.3.0-121.2e006fb

d. Upgrade CSI Operator.

helm upgrade -n csi-ns csi-baremetal-operator $CSI_CHARTS_PATH/csi-

baremetal-operator-1.3.0-121.2e006fb.tgz -f csi-baremetal-operator --set
global.registry=$REGISTRY --set image.tag=$1.3.0-121.2e006fb

e. Upgrade CSI deployment.

helm upgrade -n csi-ns csi-baremetal $CSI_CHARTS_PATH/csi-

baremetal-deployment-1.3.0-121.2e006fb.tgz -f csi-baremetal.yaml --set
global.registry=$REGISTRY --set image.tag=1.3.0-648.59a295a

f. Upgrade CSI Bare-Metal alerts.

Download the latest version of CSI Bare-metal alerts file from the Dell Support Site.

helm upgrade -n csi-ns csi-baremetal-alerts <CHART_DIRECTORY>/csi-baremetal-

alerts-1.3.0.tgz csi-bm-alerts.yaml

NOTE: After upgrading CSI, ensure all the pods in the CSI namespace are up and running.

2. Upgrade Postgres.

helm -n objectscale get values postgres -o yaml > postgres-values.yamlhelm

upgrade postgres -n <OBJECTSCALE_NAMESPACE> ./postgres-ha-1.3.0.tgz --set
global.registry=$REGISTRY -f postgres-values.yaml

3. Upgrade ObjectScale Portal.

Run the below command for objectscale-portal upgrade.

# ObjectScale 1.3.0 to 1.3.x upgrade

helm -n objectscale upgrade <objectscale_portal_name> ./objectscale-portal-1.3.0.tgz
--version 1.3.0 -f objectscale-portal-values.yaml

NOTE: After the portal upgrade, the ObjectScale portal, graphql, and install-controller pods should be restarted. Ensure
they are running before moving to the next step.

4. Log in to the ObjectScale Portal using the admin account to upgrade ObjectScale components.
5. From the ObjectScale Portal user interface, click Administration.
The Administration sections that the user is authorized to view are displayed.
6. Click Upgrades.
The Components tab is displayed and lists ObjectScale Components.
7. Select all the ObjectScale components and click Pre-Upgrade Health Check to check the components for upgrade
readiness.
The Last Health Check column provides a pass or fail status. If any of the component health checks failed, you must
resolve any issues with the component prior to performing the upgrade.

ObjectScale Administration 173

8. After successfully checking the components for upgrade readiness, clear all components, and then select the kahm
ObjectScale component. Click Upgrade to begin the upgrade of the component.
For kahm ObjectScale component, Up to date is displayed in the Available Upgrades column.
9. Select the objectscale-manager ObjectScale component. Click Upgrade to begin the upgrade of the component.
NOTE: Before going to the next step, ensure that all related pods are ready. Use the below command to find and list all
the pods.

kubectl get pod -n objectscale -l release=objectscale-manager

For objectscale-manager ObjectScale component, Up to date is displayed in the Available Upgrades column.
10. Select the decks ObjectScale component. Click Upgrade to begin the upgrade of the component.
NOTE: Before going to the next step, ensure that all related pods are ready.

For decks ObjectScale component, Up to date is displayed in the Available Upgrades column.
11. Select the SupportAssist if it is configured previously. Click Upgrade to begin the upgrade of the component.
NOTE: Before going to the next step, ensure that all related pods are ready.

For SupportAssist ObjectScale component, Up to date is displayed in the Available Upgrades column.
12. Upgrade snmp-notifier if it is configured previously.

helm upgrade snmp-notifier snmp-notifier-2.3.0-<VERSION>.tgz --set

product=objectscale --set snmpServer.host=myhost --set global.registry=$REGISTRY -n
objectscale

NOTE: Before going to the next step, ensure that all related pods are ready.

All the ObjectScale components are upgraded to the latest version.

13. Click the Object Stores tab to upgrade Object Stores.
The upgrade package is displayed for available versions from which you can choose the version for the target release.
14. Click Precheck and confirm by clicking Start precheck.
The precheck process starts.
15. When the Precheck Passed message is displayed on the top of the screen, click Upgrade
A dialogue box is displayed with the upgrade version details.
16. Click Start Upgrade
The upgrade starts, and the various stages of progress are displayed.

Results
You have successfully upgraded ObjectScale to the latest version.

Upgrade Considerations for OpenShift Container Platform from 4.12 to 4.13

This section describes some aspects to consider while upgrading the OpenShift Container Platform (OCP) from 4.12 to 4.13.
OpenShift Container Platform cluster can be upgraded with a single operation on the OpenShift web console.
Before upgrading the platform, ensure you have upgraded ObjectScale to the latest <ObjectScale_Version>.
During Red Hat OpenShift upgrade, the Machine Config Operator (MCO) reboots primary and secondary nodes to apply
the new configuration across a cluster. It cordons the number of nodes that is specified by the maxUnavailable field on
the machine configuration pool and marks them as unavailable. By default, this value is set to "1". It then applies the new
configuration and reboots each node.
As the nodes are updated by the Red Hat OpenShift Container Platform, the pods are restarted and returned to a schedulable
state.
There is a risk that the rolling reboot driven by the MCO can bring ObjectScale offline by taking down too many nodes. This risk
can be mitigated in the following ways:
● Ensure the MachineConfigPools are configured so that only one node is rebooted at a time, if the number of the nodes
less than 10. The default maxUnavailable value is "1"

174 ObjectScale Administration

● Do not upgrade OCP from 4.12 to 4.13 in a four-node cluster.

Prerequisites for OpenShift Container Platform Upgrade from 4.12 to 4.13

This topic describes the prerequisites for upgrading the OpenShift Container Platform (OCP) from 4.12 to 4.13.
Below are the prerequisites to consider before upgrading the OpenShift Container Platform.
● Ensure you have access to the cluster as a user with admin privileges to perform the upgrade.
● Take a recent etcd data backup. This helps in a scenario where the upgrade fails, and you must restore your cluster to a
previous state. Ensure to save the backup from a single primary node. See "Backing up etcd" and "Restoring to a previous
cluster state" sections in the Red Hat OpenShift Container Platform Documentation for steps.
● Ensure that the cluster is healthy enough to perform the upgrade by performing the following checks.
○ Check that all nodes are in the Ready status and using the same version of Kubernetes in all nodes using oc get
nodes.
○ Check that no node is in a degraded state using oc get mcp.
○ Check that all pods are up and running using oc get pods -A | egrep -v 'Running|Completed|
Terminated|Succeeded'.
○ Verify that the current status version is available and there is no upgrade in progress using oc get clusterversion.
○ Ensure that all operators are available, and none of them are degraded using oc get co.
○ Ensure that all machine config pools (MCPs) are running and not paused using oc get mcp. Nodes associated with a
paused MCP are skipped during the update process.
○ Confirm that the latest update is available for the cluster using oc adm upgrade.

Upgrade Procedure for OpenShift Container Platform from 4.12 to 4.13

This section provides information about the procedure for upgrading the OpenShift Container Platform.

Prerequisites

Steps
1. Ensure ObjectScale is upgraded to 1.3.0.
You must contact Dell Support to upgrade ObjectScale from version 1.2.x to 1.3.0.

2. See the "Updating a cluster by using the web console" section in the Red Hat OpenShift documentation for the upgrade
procedure.
NOTE: If you encounter any issues during the Red Hat OpenShift upgrade, contact Red Hat OpenShift support after
gathering cluster information. See "Gathering data about your cluster" section in the Red Hat OpenShift documentation
for more information.

3. After successfully upgrading to the latest version, remove enforce label from the namespace where ObjectScale is
deployed.

kubectl label namespace <NAMESPACE> pod-security.kubernetes.io/enforce-

Results
OpenShift Container Platform is upgraded to version 4.13.

ObjectScale Administration 175

Upgrade ObjectScale on Software Bundle
This section provides the steps to upgrade ObjectScale from version 1.3.0 to 1.3.x in Software Bundle deployment.

Prerequisites
Any Linux system that has connectivity to the target cluster can be used for the upgrade. This can be one of the running nodes
or an external system on the same network. It is recommended to use an external system to upgrade the cluster.
The system should have 100 GB free space for package download and extraction on the partition where the bundle is to be
downloaded. This space is in addition to the space required for the ObjectScale System.
If you are downloading the package on one of the Kubernetes nodes,
● Ensure it is not downloaded on the root partition. Preferably, use /tmp partition for the download.
● Ensure you have the kubeconfig to access the cluster.
● Obtain the current running version of the cluster using kubectl version.
● Ensure all the cluster nodes are on SLES 15 SP4.
● Ensure that all nodes are in Ready status using kubectl get nodes.
The system should have these software versions installed:
● Kubectl 1.26 or above.
● Helm 3.38 or above
● Curl

Steps
1. Download the software-bundle upgrade package from the Dell Support Site and copy the package to one of the cluster
nodes or one of the external service nodes which has access to the Kubernetes cluster.
To determine the upgrade path, you can check Release Notes of the latest patch release to understand the compatible
target upgrade versions of ObjectScale and/or RKE2.
NOTE: If the upgrade bundle is downloaded to the control-plane node, which also hosts the http-share pod, there could
be space constraints that may lead to an upgrade failure. This is because there is no free space available in the /var
partition for extracting the bundle. It is recommended to download the target upgrade bundle on a node other than
the one where the http-share pod is hosted. To identify the node that hosts the http-share pod, use the following
command:

kubectl get pods -A -owide | grep -i http

cmo http-share-0 2/2 Running 0 19h 192.168.35.207 dhcp-10-236-65-131.cluster.local

<none> <none>

In the above example, http-share is hosted on 10.236.65.131.

cmo http-share-0 2/2 Running 0 19h 192.168.35.207 dhcp-10-236-65-131.cluster.local

<none> <none>

2. Extract the package.

tar -xvf objectscale-<version>-software-bundle.tar

The artifacts folder will have the required manifest files for the next step.
3. Upload the required lcm_manifest.json, objectscale-lcm-manifest.json, and platform.tgz to the http-
share service.
NOTE:
● The upload location is fixed and must have the following path: https://$INGRESS_IP/httpshare/upload/
bundle/objectscale/upgrade/<DIR>/, where <DIR> is usually the ObjectScale target upgrade version.
● Before uploading, ensure that the http-share location is clear by deleting any existing files.

176 ObjectScale Administration

 ● If you are retrying upgrade after a failed upgrade, ensure that the necessary files are present in the http-share
location.

# Target upgrade version

VERSION=1.3.0

# Get http-share ingress IP address

INGRESS_IP=$(kubectl get svc rke2-ingress-nginx-controller -n kube-system --output
jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Clean/remove the http-share location

curl -i -k -X DELETE https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade

# Upload the required artifacts

cd objectscale-$VERSION-software-bundle/artifacts
curl -i -k --create-dirs https://$INGRESS_IP/httpshare/upload/bundle/objectscale/
upgrade/VERSION=1.3.0/ -F "[email protected]"
curl -i -k https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade/
VERSION=1.3.0/ -F "file=@lcm_manifest.json"
curl -i -k https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade/
VERSION=1.3.0 -F "[email protected]"

4. Log in to the ObjectScale Portal user interface using the admin account.
5. From the ObjectScale Portal user interface, click Administration > Upgrades.
The uploaded upgrade package is displayed.
6. Click Select Upgrade.
The Precheck option is displayed.
7. Click Precheck and confirm by clicking Start precheck.
The precheck process starts.
8. When the Precheck Passed message is displayed on the top of the screen, click Upgrade
A dialogue box is displayed with the upgrade version details.
9. Click Start Upgrade
The upgrade starts, and the various stages of progress is displayed.

Results
You have successfully upgraded ObjectScale to the target version.

Next steps
After upgrading ObjectScale to the target version, you must upgrade Kubernetes if there are any patch fixes available. See
Infrastructure Upgrade for ObjectScale Appliance - RKE2 Upgrade for steps.

Upgrade ObjectScale on Appliance

This section details the steps to upgrade ObjectScale from version 1.3.0 to 1.3.x.

Prerequisites
Any Linux system that has connectivity to the target cluster can be used for the upgrade. This can be one of the running nodes
or an external system on the same network. It is recommended to use an external system to upgrade the cluster.
The system should have 100 GB free space for package download and extraction on the partition where the bundle is
downloaded. This space is in addition to the space required for the ObjectScale System. If you are downloading the package on
one of the Kubernetes nodes,
● Ensure it is not downloaded on the root partition. Preferably, use /tmp partition for the download.
● Ensure you have the kubeconfig to access the cluster.
● Obtain the current running version of the cluster using kubectl version.
● Ensure all the cluster nodes are on SLES 15 SP4.
● Ensure that all nodes are in Ready status using kubectl get nodes.
The system should have these software versions installed:
● Kubectl 1.26 or above.

ObjectScale Administration 177

● Helm 3.38 or above
● Curl

Steps
1. Download the appliance-bundle upgrade package from the Dell Support Site and copy the package to one of the cluster
nodes or one of the external service nodes which has access to the Kubernetes cluster.
To determine the upgrade path, you can Check Release Notes of the latest patch release to understand the compatible
target upgrade versions of ObjectScale, and/or operating system, and/or RKE2.
NOTE: If the upgrade bundle is downloaded to the control-plane node, which also hosts the http-share pod, there could
be space constraints that may lead to an upgrade failure. This is because there is no free space available in the /var
partition for extracting the bundle. It is recommended to download the target upgrade bundle on a node other than
the node where the http-share pod is hosted. To identify the node that hosts the http-share pod, use the following
command:

kubectl get pods -A -owide | grep -i http

cmo http-share-0 2/2 Running 0 19h 192.168.35.207 dhcp-10-236-65-131.cluster.local

<none> <none>

In the above example, http-share is hosted on 10.236.65.131.

2. Extract the package.

tar -xvf objectscale-<version>-appliance-bundle.tar

The artifacts folder will have the required manifest files for the next step.
3. Upload the required lcm_manifest.json, objectscale-lcm-manifest.json, and platform.tgz to the http-
share service.
NOTE:
● The upload location is fixed and must have the following path: https://$INGRESS_IP/httpshare/upload/
bundle/objectscale/upgrade/<DIR>/, where <DIR> is usually the ObjectScale target upgrade version.
● Before uploading, ensure that the http-share location is clear by deleting any existing files.
● If you are retrying upgrade after a failed upgrade, ensure that the necessary files are present in the http-share
location.

# Target upgrade version

VERSION=1.3.0

# Get http-share ingress IP address

INGRESS_IP=$(kubectl get svc rke2-ingress-nginx-controller -n kube-system --output
jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Clean/remove the http-share location

curl -i -k -X DELETE https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade

# Upload the required artifacts

cd objectscale-$VERSION-software-bundle/artifacts
curl -i -k --create-dirs https://$INGRESS_IP/httpshare/upload/bundle/objectscale/
upgrade/VERSION=1.3.0/ -F "[email protected]"
curl -i -k https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade/
VERSION=1.3.0/ -F "file=@lcm_manifest.json"
curl -i -k https://$INGRESS_IP/httpshare/upload/bundle/objectscale/upgrade/
VERSION=1.3.0 -F "[email protected]"

4. Log in to the ObjectScale Portal user interface using the admin account.
5. From the ObjectScale Portal user interface, click Administration > Upgrades.
The uploaded upgrade package is displayed.
6. Click Select Upgrade.
The Precheck option is displayed.
7. Click Precheck and confirm by clicking Start precheck.

178 ObjectScale Administration

 The precheck process starts.
8. When the Precheck Passed message is displayed on the top of the screen, click Upgrade
A dialogue box is displayed with the upgrade version details.
9. Click Start Upgrade
The upgrade starts, and the various stages of progress is displayed.

Results
You have successfully upgraded ObjectScale to the target version.

Next steps
After upgrading ObjectScale to the target version, you can upgrade operating system and RKE2 using CLI if any operating
system or RKE2 fixes are required. See below sections for steps.

Infrastructure Upgrade for ObjectScale Appliance - Operating System

Upgrade
These steps are applicable only for ObjectScale Appliance deployment.

Prerequisites
● All applications (CMO and ObjectScale) on the target version are already upgraded.
● The software package for upgrade is already downloaded and extracted.
NOTE: The artifacts folder has the linux-sles-15.4.tar.gz file if any operating system upgrade is required.
● Kubernetes cluster admin privileges.
● There should be a minimum of five nodes to run the infrastructure update.

Steps
1. Upload the required lcm_manifest.json and linux-sles-15.4.tar.gz to the http-share service.

# get http-share ingress-ip address

# get IP address to upload/download files from http-share
kubectl get svc rke2-ingress-nginx-controller -n kube-system --output
jsonpath='{.status.loadBalancer.ingress[0].ip}'

# upload the required artifacts

cd objectscale-1.3.0-appliance-bundle/artifacts
curl -i -k --create-dirs https://<ingress-ip>/httpshare/upload/bundle/objectscale/
upgrade/1.3.0/ -F "file=@lcm_manifest.json"
# upload the OS bundle as well for the appliance upgrade
curl -i -k https://<ingress-ip>/httpshare/upload/bundle/objectscale/upgrade/1.3.0 -F
"[email protected]"

2. Upgrade the operating system using CLI.

a. Ensure all the nodes are in Ready state.

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP
EXTERNAL-IP OS-IMAGE KERNEL-VERSION
CONTAINER-RUNTIME
cmo42 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.42
<none> SUSE Linux Enterprise Server 15 SP4 5.3.18-24.107-default
containerd://1.6.8-k3s1
cmo43 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.43
<none> SUSE Linux Enterprise Server 15 SP4 5.3.18-24.107-default
containerd://1.6.8-k3s1

b. Ensure all the pods within the system are in Running phase.
c. Create the following os-update.yaml file for updating the operating system.

NOTE:

ObjectScale Administration 179

 ● You can assign a unique name for <lcmupdate-name>.
● All the nodes are updated through the management (host/IP) names.
● Identify the namespace of the CSI using the command helm list -A| grep -i csi-baremetal-
operator. In the sample output below, the second column denotes the CSI namespace value, which is csi.

csi-baremetal-operator csi 2
2023-09-29 07:27:11.013043982 +0000 UTC deployed csi-baremetal-
operator-1.3.0-121.2e006fb 1.3.0admin@dhcp-10-236-65-131:~>

You can identify the namespace of the ObjectScale component using the command helm list -A| grep -i
objectscale-manager. In the sample output below, the second column denotes the ObjectScale namespace
value, which is objectscale.

objectscale-manager objectscale 2
2023-09-29 07:31:45.231404122 +0000 UTC deployed
objectscale-manager-1.3.0-4184
1.3.0-4184admin@dhcp-10-236-65-131:~>

You can identify the namespace of the object store component using the command helm list -A| grep
-i ecs-cluster. In the sample output below, the second column denotes the object store namespace value,
which is objectscale.

wilson-str1 objectscale 2
2023-09-29 07:34:50.143272032 +0000 UTC deployed ecs-
cluster-1.3.0-4184 1.3.0-4184

● In the below example, ObjectScale component is installed in the <objectscale-namespace>, CSI in

<csi-namespace>, and object store in <object-store-namespace>. If both ObjectScale and object store
components are installed in the <objectscale-namespace>, the additionalParamValue section changes
to additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>"}'.

apiVersion: lcm-cluster.dell.com/v1
kind: LCMUpdate
metadata:
name: <lcmupdate-name>
namespace: kube-system
spec:
bundlePathBaseDirectory: "https://http-share.cmo:443/download/bundle/
objectscale/upgrade/<OBJECTSCALE_VERSION>"
infrastructureConfig:
componentsToUpgrade :
- componentType: "os"
componentName: "sles"
componentVersion: "15-SP4"
customConfig:
taint:
key: "node.dell.com/drain"
value: "planned-downtime"
effect: "NoSchedule"
nodeHookConfig:
customPostUpgradeNodeHook:
- env:
mode: sequential
profileName: lcm-cmo-post-maintenance
additionalParams:
- additionalParamName: optional_json

additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>, <object-store-namespace>"}'
customPreUpgradeNodeHook:
- env:
additionalParams:
- additionalParamName: optional_json
additionalParamValue: '{"lcm_nodehook_taint_key":"node.dell.com/

180 ObjectScale Administration

 drain","lcm_nodehook_taint_value":"planned-
downtime","lcm_nodehook_taint_effect":"NoSchedule"}'
mode: sequential
profileName: lcm-cmo-pre-maintenance
skipNodeHooks: false
nodeList: ["<comma_separated_cluster_node_ssh_IP_addresses>"]

Below is a sample nodeList:

nodeList:
["10.236.126.118","10.236.126.119","10.236.126.120","10.236.126.121","10.236.126.122
","10.236.126.123","10.236.126.124","10.236.126.125"]

d. Apply the resource to the target cluster and monitor the status.

kubectl apply -f os-update.yaml

kubectl get lcmupdate <lcmupdate-name> -o yaml

NOTE:
● This may take a few hours as this is a rolling update for each node.
● If operating system LCM update fails abruptly, before retrying the upgrade on other nodes delete the existing
operating system LCM Update and proceed further:

kubectl delete lcmupdate <lcmupdate-name>

Results
You have successfully upgraded the operating system.

Infrastructure Upgrade for ObjectScale Appliance - RKE2 Upgrade

These steps are applicable for ObjectScale Appliance and Software Bundle deployment.

Prerequisites
● All applications (CMO and ObjectScale) on the target version are already upgraded.
● The software package for upgrade is already downloaded.
NOTE: The artifacts folder has the k8s-distribution-<kubernetes_version>.tgz file (if RKE2 upgrade
is required). Check Release Notes of the latest patch upgrade to find the target <kubernetes_version>.
● Kubernetes cluster admin privileges.
● There should be a minimum of five nodes to run the infrastructure update.

Steps
1. Upload the required lcm_manifest.json and platform.tgz to the http-share service.

# get http-share ingress-ip address

# get IP address to upload/download files from http-share
kubectl get svc rke2-ingress-nginx-controller -n kube-system --output
jsonpath='{.status.loadBalancer.ingress[0].ip}'

# upload the required artifacts

cd objectscale-1.3.0-software-bundle/artifacts
curl -i -k --create-dirs https://<ingress-ip>/httpshare/upload/bundle/objectscale/
upgrade/1.3.0/ -F "file=@lcm_manifest.json"
# upload the RKE2 bundle as well
curl -i -k https://<ingress-ip>/httpshare/upload/bundle/objectscale/upgrade/1.3.0 -F
"[email protected]"

2. Upgrade RKE2 using CLI.

ObjectScale Administration 181

 a. Ensure all the nodes are in Ready state.

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP
EXTERNAL-IP OS-IMAGE KERNEL-VERSION
CONTAINER-RUNTIME
cmo42 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.42
<none> SUSE Linux Enterprise Server 15 SP2 5.3.18-24.107-default
containerd://1.6.8-k3s1
cmo43 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.43
<none> SUSE Linux Enterprise Server 15 SP2 5.3.18-24.107-default
containerd://1.6.8-k3s1

b. Ensure all the pods within the system are in Running phase.
c. Create the following rke-update.yaml file for updating the Kubernetes distribution.
NOTE:
● You can assign a unique name for <lcmupdate-name>.
● All the nodes are updated through the management (host/IP) names.
● Identify the namespace of the CSI using the command helm list -A| grep -i csi-baremetal-
operator. In the sample output below, the second column denotes the CSI namespace value, which is csi.

csi-baremetal-operator csi 2
2023-09-29 07:27:11.013043982 +0000 UTC deployed csi-baremetal-
operator-1.3.0-121.2e006fb 1.3.0admin@dhcp-10-236-65-131:~>

You can identify the namespace of the ObjectScale component using the command helm list -A| grep -i
objectscale-manager. In the sample output below, the second column denotes the ObjectScale namespace
value, which is objectscale.

objectscale-manager objectscale 2
2023-09-29 07:31:45.231404122 +0000 UTC deployed
objectscale-manager-1.3.0-4184
1.3.0-4184admin@dhcp-10-236-65-131:~>

You can identify the namespace of the object store component using the command helm list -A| grep
-i ecs-cluster. In the sample output below, the second column denotes the object store namespace value,
which is objectscale.

wilson-str1 objectscale 2
2023-09-29 07:34:50.143272032 +0000 UTC deployed ecs-
cluster-1.3.0-4184 1.3.0-4184

● In the below example, ObjectScale component is installed in the <objectscale-namespace>, CSI in

<csi-namespace>, and object store in <object-store-namespace>. If both ObjectScale and object store
components are installed in the <objectscale-namespace>, the additionalParamValue section changes
to additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>"}'.

apiVersion: lcm-cluster.dell.com/v1
kind: LCMUpdate
metadata:
name: <lcmupdate-name>
namespace: cmo #provide the namespace name on which cmo components are installed
spec:
bundlePathBaseDirectory: "https://http-share.cmo:443/download/bundle/
objectscale/upgrade/<OBJECTSCALE_VERSION>"
infrastructureConfig:
componentsToUpgrade :
- componentType: "kubernetes"
componentName: "rke2"
componentVersion: "1.26.4"
customConfig:
taint:

182 ObjectScale Administration

 key: "node.dell.com/drain"
value: "planned-downtime"
effect: "NoSchedule"
nodeHookConfig:
customPostUpgradeNodeHook:
- env:
mode: sequential
profileName: lcm-cmo-post-maintenance
additionalParams:
- additionalParamName: optional_json

additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>, <object-store-namespace>"}'
customPreUpgradeNodeHook:
- env:
additionalParams:
- additionalParamName: optional_json
additionalParamValue: '{"lcm_nodehook_taint_key":"node.dell.com/
drain","lcm_nodehook_taint_value":"planned-
downtime","lcm_nodehook_taint_effect":"NoSchedule"}'
mode: sequential
profileName: lcm-cmo-pre-maintenance
skipNodeHooks: false
nodeList: ["<comma_separated_cluster_node_ssh_IP_addresses>"]

Below is a sample nodeList:

nodeList:
["10.236.126.118","10.236.126.119","10.236.126.120","10.236.126.121","10.236.126.122
","10.236.126.123","10.236.126.124","10.236.126.125"]

d. Apply the resource to the target cluster and monitor the status.

kubectl apply -f rke-update.yaml

kubectl get lcmupdate <lcmupdate-name> -o yaml

NOTE:
● This may take a few hours as this is a rolling update for each node.
● If the RKE2 LCM update fails abruptly, before retrying the upgrade on other nodes delete the existing RKE2 LCM
Update and proceed further.

kubectl delete lcmupdate <lcmupdate-name>

Results
You have successfully updated RKE2.

Infrastructure Upgrade for ObjectScale Appliance - operating system and

RKE2 Upgrade
These steps are applicable only for ObjectScale Appliance deployment.

Prerequisites
● All applications (CMO and ObjectScale) on the target version are already upgraded.
● The software package for upgrade is already downloaded.
NOTE: The artifacts folder has the k8s-distribution-1.26.4.tgz file and OS tar.gz file if an upgrade
is required. Check Release Notes of the latest patch upgrade to find the target operating system upgrade version and
<kubernetes_version>.
● Kubernetes cluster admin privileges.
● There should be a minimum of five nodes to run the infrastructure update.

ObjectScale Administration 183

Steps
1. Upload the required lcm_manifest.json, k8s-distribution-1.26.4.tgz and OS tar.gz files to the http-share
service.

# get http-share ingress-ip address

# get IP address to upload/download files from http-share
kubectl get svc rke2-ingress-nginx-controller -n kube-system --output
jsonpath='{.status.loadBalancer.ingress[0].ip}'

# upload the required artifacts

cd objectscale-1.3.0-software-bundle/artifacts
curl -i -k --create-dirs https://<ingress-ip>/httpshare/upload/bundle/objectscale/
upgrade/1.3.0/ -F "file=@lcm_manifest.json"
# upload OS and RKE2 files
curl -i -k https://<ingress-ip>/httpshare/upload/bundle/objectscale/upgrade/1.3.0 -F
"[email protected]"
curl -i -k https://<ingress-ip>/httpshare/upload/bundle/objectscale/upgrade/1.3.0 -F
"[email protected]"

2. Upgrade the operating system and RKE2 using CLI.

a. Ensure all the nodes are in Ready state.

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP
EXTERNAL-IP OS-IMAGE KERNEL-VERSION
CONTAINER-RUNTIME
cmo42 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.42
<none> SUSE Linux Enterprise Server 15 SP2 5.3.18-24.107-default
containerd://1.6.8-k3s1
cmo43 Ready control-plane,etcd,master 25h v1.24.7+rke2r1 10.249.253.43
<none> SUSE Linux Enterprise Server 15 SP2 5.3.18-24.107-default
containerd://1.6.8-k3s1

b. Ensure all the pods within the system are in Running phase.
c. Create the following os-rke-update.yaml file for updating the operating system and RKE2 distribution.
NOTE:
● You can assign a unique name for <lcmupdate-name>.
● All the nodes are updated through the management (host/IP) names.
● Identify the namespace of the CSI using the command helm list -A| grep -i csi-baremetal-
operator. In the sample output below, the second column denotes the CSI namespace value, which is csi.

csi-baremetal-operator csi 2
2023-09-29 07:27:11.013043982 +0000 UTC deployed csi-baremetal-
operator-1.3.0-121.2e006fb 1.3.0admin@dhcp-10-236-65-131:~>

You can identify the namespace of the ObjectScale component using the command helm list -A| grep -i
objectscale-manager. In the sample output below, the second column denotes the ObjectScale namespace
value, which is objectscale.

objectscale-manager objectscale 2
2023-09-29 07:31:45.231404122 +0000 UTC deployed
objectscale-manager-1.3.0-4184
1.3.0-4184admin@dhcp-10-236-65-131:~>

You can identify the namespace of the object store component using the command helm list -A| grep
-i ecs-cluster. In the sample output below, the second column denotes the object store namespace value,
which is objectscale.

wilson-str1 objectscale 2
2023-09-29 07:34:50.143272032 +0000 UTC deployed ecs-
cluster-1.3.0-4184 1.3.0-4184

184 ObjectScale Administration

 ● In the below example, ObjectScale component is installed in the <objectscale-namespace>, CSI in
<csi-namespace>, and object store in <object-store-namespace>. If both ObjectScale and object store
components are installed in the <objectscale-namespace>, the additionalParamValue section changes
to additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>"}'.

apiVersion: lcm-cluster.dell.com/v1
kind: LCMUpdate
metadata:
name: <lcmupdate-name>
namespace: kube-system
spec:
bundlePathBaseDirectory: "https://http-share.cmo:443/download/bundle/
objectscale/upgrade/<OBJECTSCALE_VERSION>"
infrastructureConfig:
componentsToUpgrade :
- componentType: "kubernetes"
componentName: "rke2"
componentVersion: "1.26.4"

- componentType: "os"
componentName: "sles"
componentVersion: "15-SP4"
customConfig:
taint:
key: "node.dell.com/drain"
value: "planned-downtime"
effect: "NoSchedule"
nodeHookConfig:
customPostUpgradeNodeHook:
- env:
mode: sequential
profileName: lcm-cmo-post-maintenance
additionalParams:
- additionalParamName: optional_json

additionalParamValue: '{"lcm_nodehook_untaint_custom_namespaces":"<objectscale-
namespace>, <csi-namespace>, <object-store-namespace>"}'
customPreUpgradeNodeHook:
- env:
additionalParams:
- additionalParamName: optional_json
additionalParamValue: '{"lcm_nodehook_taint_key":"node.dell.com/
drain","lcm_nodehook_taint_value":"planned-
downtime","lcm_nodehook_taint_effect":"NoSchedule"}'
mode: sequential
profileName: lcm-cmo-pre-maintenance
skipNodeHooks: false
nodeList: ["<comma_separated_cluster_node_ssh_IP_addresses>"]

Below is a sample nodeList:

nodeList:
["10.236.126.118","10.236.126.119","10.236.126.120","10.236.126.121","10.236.126.122
","10.236.126.123","10.236.126.124","10.236.126.125"]

d. Apply the resource to the target cluster and monitor the status.

kubectl apply -f os-rke-update.yaml

kubectl get lcmupdate <lcmupdate-name> -o yaml

NOTE:
● This may take a few hours as this is a rolling update for each node.

ObjectScale Administration 185

 ● If the operating system and the RKE2 LCM update fail abruptly, before retrying the upgrade on other nodes,
delete the existing operating system and the RKE2 LCM Update and proceed further.

kubectl delete lcmupdate <lcmupdate-name>

Results
You have successfully updated operating system and RKE2.

Troubleshooting Object store upgrade status in progress after

failed node repair
When there is a node failure during the object store upgrade, although the failed node is repaired, the object store upgrade is
stuck with an in progress status.
If you check ECS CR Upgrading condition, the output is as below:

upgrading:
lastTransitionTime: "2023-09-07T09:29:04Z"
lastUpdateTime: "2023-09-07T09:29:04Z"
message: cluster upgrade is in progress
reason: Pod image is changed
status: "True"
type: in progress

The reason for object store upgrade being stuck in progress is the failure in the preupgrade job. The ObjectScale operator tries
to turn off the DT load balance during the object store upgrade, but meets an unexpected error because there is a failed node.
In order to resolve this issue, first ensure that the failed node has been repaired. Then, delete the failed jobs using kubectl
delete job <job-name> -n <job-ns>. The ObjectScale operator tries to create a new preupgrade job, and if it is
successful, the issue is resolved.
If the new job fails, capture kubectl get job <job-name> -n <job-ns> -o yaml, kubectl logs job.batch/
<job-name> -n <job-ns>, and operator's logs; and get help from Dell Support.

186 ObjectScale Administration

 12
ObjectScale Management REST API
This section describes information about accessing and authenticating with the ObjectScale Management REST API and
provides a summary of the API .

Topics:
• ObjectScale Management REST API introduction
• ObjectScale Management REST API summary
• Authenticate with the ObjectScale Management REST API

ObjectScale Management REST API introduction

You can configure and manage certain aspects of ObjectScale using the ObjectScale REST Management API.
For more information about the ObjectScale Management REST API, see these topics:
● ObjectScale Management REST API summary
● Authenticate with the ObjectScale Management REST API
In addition, review the ObjectScale REST API Reference Guide which is auto-generated from the source code and provides a
reference for the methods available in the API.

ObjectScale Management REST API summary

The ObjectScale Management REST API manages and configures ObjectScale and object stores.

Table 39. ObjectScale-level, Object Store-level, and Object Service APIs - methods summary
API Area Description
ObjectScale Management APIs
Management APIs create and manage ObjectScale-level management users.
Local User APIs (non-RKE) /mgmt/users
APIs (non-RKE platform only) for creating and managing ObjectScale level management users.

Local Account APIs (RKE) /mgmt/local-accounts

APIs (RKE platform only) for creating and managing ObjectScale level management account.

Authentication and Login APIs /mgmt/auth/login

APIs for getting and managing management users token.

Local Account Config APIs /mgmt/local-accounts-config

(RKE)
APIs (RKE platform only) for creating and managing ObjectScale level management account
security config.

Role APIs (non-RKE) /mgmt/roles

APIs (non-rke platform only) for creating and managing ObjectScale level management users.

Role APIs (RKE) /mgmt/roles

This API is for viewing management roles only (not for creating and managing).

Role-Mapping APIs (RKE) /mgmt/ldap-role-mappings

ObjectScale Management REST API 187

Table 39. ObjectScale-level, Object Store-level, and Object Service APIs - methods summary (continued)
API Area Description

APIs (RKE platform only) for creating and managing ObjectScale level LDAP and management
role mappings.

LDAP Auth Provider APIs /mgmt/ldaps

(RKE)
APIs (RKE platform only) for creating and managing ObjectScale level management LDAP
provider.

Federation APIs create and manage ObjectScale-level federations.

Federation APIs /fedsvc
API interface for Federation Operations.

Feature APIs /fedsvc/feature

API interface for Federation Feature Operations.

IAM APIs manage IAM users, roles, and policies in an account. The APIs assume a role and obtain temporary access
credentials.
IAM Actions /iam
API interface for Identity And Access Management Operations.

STS Actions /sts

API interface for Secure Token Service Operations.

Object Store Management APIs

The Metering API gets metrics on object stores, including bucket information, replication status, and performance.
Objmt /object/mt
API for getting object store metrics and failed replication errors. Errors are classified by error
code.

The Recovery API retrieves recovery status for a device or partition.

Recovery /vdc/recovery-status/
API for getting recovery status.

Replication APIs control ObjectScale replication. This API allows you to pause, suspend, resume, and throttle replication.
Replication Control /replication/control
API for ObjectScale replication control. It allows you to pause, suspend, resume, and throttle
replication.

Replication Info replication/info

API to get ObjectScale replication info.

Data Transition Control /datatransition/control/

API for ObjectScale replication Data Transition Controls.

The Monitoring API retrieves audit events for a specified namespace.

Events /vdc/events
API for fetching audit alerts.

The Multitenancy API provisions and manages tenants.

Account Tenant /object/tenants
API for provisioning and managing tenant.

188 ObjectScale Management REST API

Table 39. ObjectScale-level, Object Store-level, and Object Service APIs - methods summary (continued)
API Area Description
The Provisioning API provisions and manages buckets.
Bucket /object/bucket
API for provisioning and managing buckets and view into replication failures by error code.

Object Store Object Service API

Amazon S3 APIs API to manage Amazon S3.
● S3 Bucket
● S3 Bucket ACL
● S3 Bucket Cors
● S3 Bucket Is Stale Allowed
● S3 Bucket Lifecycle
● S3 Bucket List Uploads
● S3 Bucket Location
● S3 Bucket Lock Configuration Operations
● S3 Bucket Notification Operations
● S3 Bucket Object Lock Operations
● S3 Bucket Policy
● S3 Bucket Replication Operations
● S3 Bucket Versioning
● S3 Bucket Versions
● S3 Data Node
● S3 Metadata Key List
● S3 Metadata Key System List
● S3 Metadata Search
● S3 Multi Object Delete
● S3 Object
● S3 Object ACL
● S3 Object Detailed Replication Status Operations
● S3 Object Init Uploads
● S3 Object Legal Hold Operations
● S3 Object Replication Info Operations
● S3 Object Retention Operations
● S3 Object Tagging Operations
● S3 Object Uploads
● S3 Options
● S3 Ping
● S3 Retention Update Operation
● S3 Select Operations

Authenticate with the ObjectScale Management REST

API
ObjectScale uses a token-based authentication system for REST API calls. This section provides examples of authenticating with
the ObjectScale Management API.
When you are authenticated by ObjectScale, the login API returns an authentication token. You can use this token for
authentication in subsequent calls.
For the scoping of the ObjectScale-level and object store-level APIs, see ObjectScale Management REST API summary.

ObjectScale Management REST API 189

Download and set up CURL
Steps
1. Download and install CURL:
● CURL for Linux: https://curl.haxx.se
● CURL for Windows: https://github.com/curl/curl-for-win
2. Extract contents and append PATH to include location to curl_directory_name/bin.
3. Verify curl functionality with curl -h , for help, or curl -V, for version.

Log in and obtain the Access Token for the ObjectScale-level APIs
Use CURL for Windows or Linux to log in to ObjectScale and use ObjectScale-level APIs.

About this task

ObjectScale uses a token-based authentication system for REST API calls.
This section provides examples of authenticating with the ObjectScale Management APIs. When you are authenticated by
ObjectScale, the login API returns an authentication token. You can use this token for authentication in subsequent calls.

Steps
1. Assign the namespace where ObjectScale is installed to objectscaleNamespace

objectscaleNamespace=<OBJECTSCALE_NAMESPACE>

2. Assign the ObjectScale Gateway endpoint IP to OBJECTSCALE_GATEWAY_ENDPOINT.

All ObjectScale-level login and API request need to be made to the ObjectScale gateway loadbalancer endpoint.

OBJECTSCALE_GATEWAY_ENDPOINT=$(kubectl get svc -n $objectscaleNamespace| awk '/

objectscale-gateway[^-]/{print $4}')

3. If you are login into ObjectScale installed on the ObjectScale Software Bundle for the first time, you must change the root
user's default password.
a. Login to ObjectScale using the with root user default password.

curl -vk -X POST -d '{"username":"root", "password":"ChangeMe"}' -H 'Content-Type:

application/json' https://<OBJECTSCALE_GATEWAY_ENDPOINT>:443/mgmt/auth/login

ObjectScale will return the token you can use to update the password.

< HTTP/1.1 205 Reset Content

< Date: Wed, 01 Mar 2023 15:48:09 GMT
< Content-Type: application/xml
< Content-Length: 0
< Connection: keep-alive
< Location: https://172.17.0.241:4443/mgmt/local-accounts/local-account/password
< Method: PUT
< Password-Change-Token: OSTOKEN-
eyJraWQiOiJzeW1mNjhiMDhjZmRmNWY4ZmM3IiwiYWxnIjoiSFMyNTYifQ.eyJzdWIiOiJyb290Iiwib3Nfc
m9sIjpbXSwiaXNzIjoidXJuOm9zYzpvc2Np

YmQ3ZTM1ZmZkZjVmNzQ2OTo6c2VydmljZS9vYmplY3RzY2FsZS1mZWRlcmF0aW9uLTc4NmM3ZjU1YzQtbXQ1
cjciLCJvc19wYXNzY2huZ29ubHkiOmZhbHNlLCJhdWQiOiJvc2NpYmQ3ZTM1ZmZkZjVm

NzQ2OSIsInR5cCI6IkJlYXJlciIsInNpZCI6IjBiMjhjY2RiLWNiYmUtNDgxNC1hM2JhLTM5Nzc4ZDYzZjhm
MyIsImlhdCI6MTY3NzY4NTY4OSwiZXhwIjoxNjc3Njg1OTg5LCJqdGkiOiJvc2F0YjJm

N2QyN2ZlMThhZjJkOCJ9.DvbPEkulFpQ-1e3ob2LRfVRtBPuib6AGvNax03Qp0rA

190 ObjectScale Management REST API

 < Token-Type: Bearer
< Token-Expiry: 300

b. Update the password for the root account using the returned Password-Change-Token as the required auth token.

curl -vk -X PUT -d '{"old_password":"ChangeMe", "password":"<NEW_PASSWORD>"}' -H

'Content-Type: application/json' -H "Authorization:<Password-Change-Token>" https://
<OBJECTSCALE_GATEWAY_ENDPOINT>:443/mgmt/local-accounts/local-account/password

Once the password update is successfully, you have updated the root user account with a new password and can now log in
to ObjectScale.
4. Log in with the username and password of an ObjectScale Management User.

curl -vk -X POST -d '{"username":"root", "password":"<ROOT_ACCOUNT_PASSWORD>"}' -H

'Content-Type: application/json' https://<OBJECTSCALE_GATEWAY_ENDPOINT>:443/mgmt/auth/
login

The response will contain the access_token for authentication, refresh_token can be used to get a new access_token after
the current token expires.

< HTTP/1.1 200 OK

< Date: Wed, 01 Mar 2023 16:59:25 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
<
{ [1114 bytes data]
100 1199 0 1107 100 92 406 33 0:00:02 0:00:02 --:--:-- 440
* Connection #0 to host 172.17.100.1 left intact
{
"access_token" : "OSTOKEN-
eyJraWQiOiJzeW01Y2E1MDA4ZjRhMjMzYTJhIiwiYWxnIjoiSFMyNTYifQ.eyJzdWIiOiJvc2xkYXA3NTkwNDl
iZTg5MmIyMWJhL21nbXRfaXNndXNlcjEiLCJvc19yb2wiOlsicmVhZG9ubHkiXSwiaXNzIjoidXJuOm9zYzpvc
2NpNGNmYmEwZmYyNGViNGQ2Mjo6c2VydmljZS9vYmplY3RzY2FsZS1mZWRlcmF0aW9uLWNmNjQ0NGQtdHBuYzQ
iLCJvc19wYXNzY2huZ29ubHkiOmZhbHNlLCJhdWQiOiJvc2NpNGNmYmEwZmYyNGViNGQ2MiIsInR5cCI6IkJlY
XJlciIsInNpZCI6ImI0ZTVmYTFjLWYxYjktNDE5NC1iM2I1LTIzZTY3MjE4Y2E4MCIsImlhdCI6MTY3NTIyNzU
2NCwiZXhwIjoxNjc1MjI4NDY0LCJqdGkiOiJvc2F0MWQzYWE1NWY0ZjA2ZjIwYSJ9.N8O7Z4fcjNRTbh6dmMw5
UCSEpmYYt4jRQ3uVfiyMrJ8",
"expires_in" : 900,
"refresh_token" :
"eyJraWQiOiJzeW01Y2E1MDA4ZjRhMjMzYTJhIiwiYWxnIjoiSFMyNTYifQ.eyJzdWIiOiJvc2xkYXA3NTkwND
liZTg5MmIyMWJhL21nbXRfaXNndXNlcjEiLCJpc3MiOiJ1cm46b3NjOm9zY2k0Y2ZiYTBmZjI0ZWI0ZDYyOjpz
ZXJ2aWNlL29iamVjdHNjYWxlLWZlZGVyYXRpb24tY2Y2NDQ0ZC10cG5jNCIsImF1ZCI6Im9zY2k0Y2ZiYTBmZj
I0ZWI0ZDYyIiwidHlwIjoiUmVmcmVzaCIsInNpZCI6ImI0ZTVmYTFjLWYxYjktNDE5NC1iM2I1LTIzZTY3MjE4
Y2E4MCIsImlhdCI6MTY3NTIyNzU2NCwiZXhwIjoxNjc1MjI5MzY0LCJqdGkiOiJvc3J0NTRiYjYwMWY0ZjA2Zj
IwYSJ9.NyfzEP7VJK08wutsD7nvQxQSiYkvCp4dOy4ZFpygfOg",
"refresh_expires_in" : 1800
}

5. Copy the generated token so you can add it to each command using the -H "Authorization:$access_token"
syntax.
You can use the following command to set the Access Token as the "$token" environment variable.

access_token=$(curl -k -X POST
-d '{"username":"root","password":"<ROOT_ACCOUNT_PASSWORD>"}' https://
$OBJECTSCALE_GATEWAY_ENDPOINT:443/mgmt/auth/login -H 'Content-Type: application/json'
-H 'Accept: application/json' | grep access_token | cut -f4 -d'"')

Log in and obtain the Access Token for the object store-level APIs
Use CURL for Windows or Linux to log in to ObjectScale and use object store-level APIs.

About this task

ObjectScale uses a token-based authentication system for REST API calls.

ObjectScale Management REST API 191

This section provides examples of authenticating with the ObjectScale Management APIs. When you are authenticated by
ObjectScale, the login API returns an authentication token. You can use this token for authentication in subsequent calls.

Steps
1. Assign the namespace where ObjectScale is installed to objectscaleNamespace

objectscaleNamespace=<OBJECTSCALE_NAMESPACE>

2. Assign the ObjectScale Gateway endpoint IP to OBJECTSCALE_GATEWAY_ENDPOINT.

All ObjectScale-level login and API request need to be made to the ObjectScale gateway loadbalancer endpoint.

OBJECTSCALE_GATEWAY_ENDPOINT=$(kubectl get svc -n $objectscaleNamespace| awk '/

objectscale-gateway[^-]/{print $4}')

3. Log in with the username and password of an ObjectScale Management User.

curl -vk -X POST -d '{"username":"root", "password":"<ROOT_ACCOUNT_PASSWORD>"}' -H

'Content-Type: application/json' https://<OBJECTSCALE_GATEWAY_ENDPOINT>:443/mgmt/auth/
login

The response will contain the access_token for authentication, refresh_token can be used to get a new access_token after
the current token expires.

< HTTP/1.1 200 OK

< Date: Wed, 01 Mar 2023 16:59:25 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
<
{ [1114 bytes data]
100 1199 0 1107 100 92 406 33 0:00:02 0:00:02 --:--:-- 440
* Connection #0 to host 172.17.100.1 left intact
{
"access_token" : "OSTOKEN-
eyJraWQiOiJzeW01Y2E1MDA4ZjRhMjMzYTJhIiwiYWxnIjoiSFMyNTYifQ.eyJzdWIiOiJvc2xkYXA3NTkwNDl
iZTg5MmIyMWJhL21nbXRfaXNndXNlcjEiLCJvc19yb2wiOlsicmVhZG9ubHkiXSwiaXNzIjoidXJuOm9zYzpvc
2NpNGNmYmEwZmYyNGViNGQ2Mjo6c2VydmljZS9vYmplY3RzY2FsZS1mZWRlcmF0aW9uLWNmNjQ0NGQtdHBuYzQ
iLCJvc19wYXNzY2huZ29ubHkiOmZhbHNlLCJhdWQiOiJvc2NpNGNmYmEwZmYyNGViNGQ2MiIsInR5cCI6IkJlY
XJlciIsInNpZCI6ImI0ZTVmYTFjLWYxYjktNDE5NC1iM2I1LTIzZTY3MjE4Y2E4MCIsImlhdCI6MTY3NTIyNzU
2NCwiZXhwIjoxNjc1MjI4NDY0LCJqdGkiOiJvc2F0MWQzYWE1NWY0ZjA2ZjIwYSJ9.N8O7Z4fcjNRTbh6dmMw5
UCSEpmYYt4jRQ3uVfiyMrJ8",
"expires_in" : 900,
"refresh_token" :
"eyJraWQiOiJzeW01Y2E1MDA4ZjRhMjMzYTJhIiwiYWxnIjoiSFMyNTYifQ.eyJzdWIiOiJvc2xkYXA3NTkwND
liZTg5MmIyMWJhL21nbXRfaXNndXNlcjEiLCJpc3MiOiJ1cm46b3NjOm9zY2k0Y2ZiYTBmZjI0ZWI0ZDYyOjpz
ZXJ2aWNlL29iamVjdHNjYWxlLWZlZGVyYXRpb24tY2Y2NDQ0ZC10cG5jNCIsImF1ZCI6Im9zY2k0Y2ZiYTBmZj
I0ZWI0ZDYyIiwidHlwIjoiUmVmcmVzaCIsInNpZCI6ImI0ZTVmYTFjLWYxYjktNDE5NC1iM2I1LTIzZTY3MjE4
Y2E4MCIsImlhdCI6MTY3NTIyNzU2NCwiZXhwIjoxNjc1MjI5MzY0LCJqdGkiOiJvc3J0NTRiYjYwMWY0ZjA2Zj
IwYSJ9.NyfzEP7VJK08wutsD7nvQxQSiYkvCp4dOy4ZFpygfOg",
"refresh_expires_in" : 1800
}

4. Copy the generated token so you can add it to each command using the -H "Authorization:$access_token"
syntax.
You can use the following command to set the Access Token as the "$token" environment variable.

access_token=$(curl -k -X POST
-d '{"username":"root","password":"<ROOT_ACCOUNT_PASSWORD>"}' https://
$OBJECTSCALE_GATEWAY_ENDPOINT:443/mgmt/auth/login -H 'Content-Type: application/json'
-H 'Accept: application/json' | grep access_token | cut -f4 -d'"')

Obtain the endpoint IP for the object store Management Gateway and log in to issue object store-level API calls

192 ObjectScale Management REST API

5. Assign the object store's name to the variable objectstoreName and assign the object store's namespace to the variable
objectstoreNamespace .

OBJECTSTORE_MANAGEMENT_GATEWAY=$(kubectl get svc -n $objectstoreNamespace | grep

$objectstoreName | awk '/-management-gateway/{print $4}')

Below is an example of how to use ObjectScale-level token to make and object store API call.

curl -ks -X GET -H "Authorization:$access_token" -H "Accept:application/xml" -H "X-

EMC-Override: true" https://${OBJECTSTORE_MANAGEMENT_GATEWAY}:4443/object/tenants

ObjectScale Management Service

The ObjectScale Management Service manages management users and roles and is used for establishing trust with other
external identity providers. It provides an API for authentication/authorization that allows for secure token generation which will
be accepted by other ObjectScale services.
The ObjectScale management service provides the following functionality:
● Defines roles for management users.
● Supplies the /mgmt APIs.
● Provides method to process Access Token correctly for the IAM, Federation Service, and Object Control Service in an object
store.
● Modifies the IAM and Federation Service client to transparently handle Access Token interactions.
1. A user first logs in to the /mgmt/auth/login endpoint. The returned Access Token will have the roles associated with the
user.
2. The user can then present this Access Token to request services from IAM, Federation Service, and Object Control in an
object store.
3. These ObjectScale services will first authenticate the Access Token with Management service and based on the roles
available in the token a determination is made whether the user is authorized to access the requested resource.

Objectscale Access Token (OSTOKEN) format

Access Token, also known as an OSTOKEN, is based on JSON Web Token (JWT) and is used as the auth token for system
resource access.
You can refresh an Access Token using the /mgmt/auth/token API.
The default expiration for /mgmt/auth/login is 900 seconds (15 minutes).
NOTE: All Access Tokens are opaque and are intended to be used as is. ObjectScale exposes some APIs to determine the
expiry time of the Access Token.

ObjectScale Management User Roles

A management user in the ObjectScale Management API must be assigned one or more roles.

Table 40. Management user roles for ObjectScale on ObjectScale Software Bundle
Role name Role description Role ID
admin Full control over all management operations. admin
operations_admin Full control over all management operation except for security operations_admin
operations. Read access to user and public certs.
readonly Read access only, to everything other than security information. Read readonly
access to user and public certs.
security_admin Full control over security operations only, read access for others. security_admin
storage_admin Full control over storage management, including the ability to create and storage_admin
delete object stores.

ObjectScale Management REST API 193

Table 40. Management user roles for ObjectScale on ObjectScale Software Bundle (continued)
Role name Role description Role ID
storage_operator Full control over storage management, except the ability to create and storage_operator
delete object stores.

Table 41. Management user roles for ObjectScale on Red Hat OpenShift
Role name Role description Role ID
Security Manages certificates, administering other management users, and the SECURITY_ADMIN
Administrator federation of other ObjectScale instances.
System Manages IAM accounts, ObjectScale licensing, object stores, and SYSTEM_ADMIN
Administrator monitoring (alerts and auditing).
System Monitor Read-only access. Manages monitoring (alerts, audits). SYSTEM_MONITOR
Account Manages IAM accounts. STORE_ADMIN
Administrator

Logout
The logout API logs out a user's authentication token provided in Authorization header.
The following example shows a logout request, where <ACCESS_TOKEN> is your access token value or variable. You pass in the
authentication token from header or cookie to log out.

curl -ik -X POST https://$OBJECTSCALE_GATEWAY_ENDPOINT:443/mgmt/auth/logout -H 'Content-

Type: application/json' -H "Authorization:<ACCESS_TOKEN>" -v

The response should be HTTP 204.

194 ObjectScale Management REST API

 13
Accessing data with IAM and S3
This section details the protocols supported by ObjectScale for end-user access to ObjectScale object storage.
Topics:
• ObjectScale IAM overview
• Amazon S3 API support in ObjectScale

ObjectScale IAM overview

Identity and Access Management (IAM) enables you to have fine-grained access to the ObjectScale S3 resources securely. This
functionality ensures that each access request to an ObjectScale resource is identified, authenticated, and authorized.
ObjectScale IAM allows you to add users, roles, and groups.
You can also grant and restrict the access by adding policies to the ObjectScale IAM entities.

IAM Account Management

Account Management enables you to manage IAM identities within each account such as users, groups, and roles.
All IAM entities have a unique ID associated with it. Deleting and re-creating an entity with the same name creates a unique ID
for the new entity.
An IAM Account contains other IAM entities like Users, Groups, Roles, Policies, and Service Providers associated with it. You
cannot create or modify an Account to have another Account associated with it. Each account consists of replicated IAM
entities and local IAM entities. Local IAM entities remain local within the ObjectScale instance and are not replicated. Global
entities are replicated to other ObjectScale instances. Replicated IAM entities and local IAM entities have separate APIs.

IAM Identities
Table 42. Identities
Field Description
Account root user ● Account root user is an admin user in the account.
● Only the account root user can access the ObjectScale
Portal user interface.
● Account root user is the owner of the buckets and any
objects within created by its IAM entities.
IAM user ● An IAM user is a person or an application in the account
that can interact with ObjectScale resources.
● An IAM user can belong to one or more IAM groups.
● It is possible to create, view, modify, delete, and list IAM
users in ObjectScale using both API and the ObjectScale
Portal user interface.
● IAM users cannot access the ObjectScale Portal user
interface.
IAM group ● An IAM group is a collection of IAM users.
● IAM groups do not nest and contain only IAM users.
● IAM groups let you specify permissions for all the users in
the group making management easier.

Accessing data with IAM and S3 195

Table 42. Identities (continued)
Field Description
● Creating and managing groups can be done from both the
ObjectScale Portal user interface and API.
● Tagging on groups is not supported.
IAM role ● An IAM role is similar to a user, in that it is an identity with
permission policies that determine what the identity can
and cannot do.
● An IAM role does not have any credentials that are
associated with it.
● An entity assumes a role by calling an API that provides it
with temporary credentials to access a resource.
● A federated user can assume an IAM role by authenticating
with external identity provider.
● An IAM user can assume a role in the same or different
account (cross-account access).

NOTE: IAM and account root users access S3 and IAM APIs using Access Keys. Access Keys are long-term credentials
which consist of an access key ID and secret access key. A user can have at most two Access Keys associated with it at
any time.

Tagging IAM Entities (Users and Role)

A tag is a label that you assign to a resource. Each tag consists of a key and an optional value, both of which you define. Custom
attributes are added to users and roles using a tag key-value pair. These tags can be used to control the access of an entity to
resources or to control what tags can be attached to an entity. Groups and policies cannot be tagged. You can apply the same
tag to multiple entities. But multiple tags on one entity cannot have the same key. Fifty tags per IAM entity are allowed.

IAM error codes

Table 43. IAM error codes
Error type HTTP status code Description
AccessDeniedException 403 Indicates that you do not have the required access to
perform the action.
ConcurrentModification 409 Indicates that multiple requests are submitted simultaneously
to modify the object. You need to wait for a few minutes and
submit the request again.
DeleteConflict 409 Indicates that the request is raised to delete a resource that
is attached with another entity.
EntityAlreadyExists 409 Indicates that the request is raised to create a resource that
already exists.
ExpiredToken 400 Indicates that the Web identity token that is used to perform
the action is expired or not valid.
IDPRejectedClaim 403 Indicates that the identity provider (IdP) reported that
authentication failed.
InternalFailure 500 Indicates that the request failed due to an unknown error,
exception, or failure.
InvalidAction 400 Indicates that the requested action is not valid.
InvalidInput 400 Indicates that an invalid or an out-of-range value is provided
for an input.

196 Accessing data with IAM and S3

Table 43. IAM error codes (continued)
Error type HTTP status code Description
InvalidParameterValue 400 Indicates that an invalid or an out-of-range value is provided
for an input parameter.
LimitExceeded 409 Indicates that the request is rejected because an attempt is
made to create resources beyond the current account limits.
MalformedPolicyDocument 400 Indicates that the provided policy document is malformed.
MissingAction 400 Indicates that the action or a required parameter is missed in
the request.
MissingParameter 400 Indicates that the required parameter is missed in the
request.
NoSuchEntity 404 Indicates that the referenced entity does not exist.
NotImplemented 501 Indicates that the mentioned functionality is not implemented
yet.
PackedPolicyTooLarge 400 Indicates that the total packed size of the session policies
and session tags combined is too large.
PermissionDenied 403 Indicates that the principal does not have the required
permission to perform the action.
ServiceFailure 500 Indicates that the request is failed because of an unknown
error, exception, or failure.
ServiceUnavailable 503 Indicates that the request is failed due to a temporary failure
of the server.
ValidationError 400 Indicates that the input fails to satisfy the constraints
specified by the specific API.

IAM supported condition keys

IAM supports the following condition keys:

Global condition keys Type Description

aws:CurrentTime Date To check for date and time conditions
aws:EpochTime Date To check for date and time conditions using a date in
epoch or UNIX time
aws:PrincipalArn ARN Checks the ARN of the IAM user or role to whom the
permissions are allocated.
aws:UserAgent String To check the client application of the requestor.
aws:PrincipalTag/ tag-key String Checks that the tag attached to the principal making
the request matches the specified key name and
value.
aws:RequestTag/ tag-key String Checks that the tag key-value pair is present in an
AWS request.
aws:ResourceTag/ tag-key String Checks that the tag key-value pair is attached to the
resource.
aws:SourceIp IpAddr To check the IP address of the requester
aws:TagKeys String, This context key is a list of tag keys without values
ForAllValues:String
ForAnyValue: String

Accessing data with IAM and S3 197

Global condition keys Type Description
aws:TokenIssueTime Date Checks the date and time that temporary security
credentials were issued.
aws:principaltype String Indicates the type of principal making the request.
● Root user is Account.
● IAM user is User.
● SAML or Assumed role user is AssumedRole.
aws:userid String Based on authorized user access is set to the
following:
● Root user ARN if root user is requester.
● IAM user unique id IAM user is requester.
● If SAML federated user is requester, it is set to the
role-id:caller-specified-role-name
● If assumed role user is requester, it is set to the
role-id:caller-specified-role-name
role-id: is the unique id of role
caller-specified-role-name: is the
RoleSessionName in AssumeRole request or the
name attribute value in SAML assertion passed to
AssumeRoleWithSAML request.
aws:username String Based on authorized user access, if requester is an
IAM user, it is set to the IAM username otherwise it is
not set.

IAM condition keys Type Description

iam:PermissionsBoundary String Checks that the specified policy is attached as
permissions boundary on the IAM principal resource.
iam:PolicyARN ARN Checks the ARN of a managed policy in requests that
involve a managed policy.
iam:ResourceTag/ key-name String Checks that the tag attached to the IAM entity (user or
role) matches the specified key name and value.

STS and SAML condition keys Type Description

saml:aud String An endpoint URL to which SAML assertions are
presented. The value for this key comes from the SAML
Recipient field in the assertion, not the Audience
field.
saml:edupersonorgdn String This is an eduPerson attribute in SAML assertion.
saml:iss String The issuer, which is represented by a URN.
saml:namequalifier String This contains a hash value that represents the
combination of the saml:doc and saml:iss values.
It is used as a account qualifier; the combination
of saml:namequalifier and saml:sub uniquely
identifies a user.
saml:sub String This is the subject of the claim, which includes a
value that uniquely identifies an individual user within an
organization.
saml:sub_type String This key can have the value persistent , transient ,
or consist of the full Format URI from the Subject and
NameID elements used in your SAML assertion. A value
of persistent indicates that the value in saml:sub
is the same for a user between sessions. If the value is

198 Accessing data with IAM and S3

STS and SAML condition keys Type Description
transient , the user has a different saml:sub value for
each session.

S3 condition keys Description

s3:x-amz-acl Specifies the canned ACL in the request.
s3:x-amz-grant- permission Specifies permission for the following access.
● read
● write
● read-acp
● write-acp
● full-control
s3:x-amz-copy-source Enables restricting copy source to a specific bucket, folder, or object.
s3:x-amz-metadata-directive Specifies certain behavior to be enforced during object uploads (COPY vs
REPLACE).
s3:x-amz-server-side-encryption Specifies that the request should contain this header to ensure that the uploads
are stored encrypted.
s3:VersionId Limits access to specific versions of object.
s3:LocationConstraint Using this condition key, you can restrict a user to create a bucket in a specific
AWS Region.
s3:delimiter Used to require the requester to specify delimiter parameter.
s3:max-keys Limits ListBucket requests to the set s3:max-keys value.
s3:prefix Limits ListBucket and ListBucketVersions to specific prefix.
s3:ExistingObjectTag/ <tag-key> Using this condition key, you can limit the permission for the
s3:PutObjectAcl action to only on objects that have a specific tag key and
value.
s3:RequestObjectTagKeys Using this condition key, you can limit permission for the s3:PutObject
action by restricting the object tags allowed in the request.
s3:RequestObjectTag/ <tag-key> Using this condition key, you can limit permission for the s3:PutObject
action by restricting the object tags allowed in the request.

IAM limitations on entities and objects

IAM has certain limitations on its resources such as naming the entities, characters to be used for the identities, number of
policies to be attached to an entity, and the number of resources that can be linked to an entity.
NOTE: Paths are not supported for IAM entities.

IAM entity name limits

Resource Limits
Names of users, groups, roles, and ● Must be unique within the account.
managed policies ● Must be alphanumeric and it may include any of these special characters: Plus (+),
equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-).
NOTE: These names are case insensitive.

Inline policy names ● Must be unique to the user, group, or to the role that they are embedded in.
● Can contain any Basic Latin (ASCII) characters except these special characters:
Backward slash (\), forward slash (/), asterisk (*), question mark (?), and space.

Accessing data with IAM and S3 199

Resource Limits
These characters are reserved according to the RFC (Request for Comments)
3986 Internet standard.
Policy documents Can contain these Unicode characters: horizontal tab (U+0009), linefeed (U+000A),
carriage return (U+000D), and characters in the range from U+0020 to U+00FF.

IAM entity object limits

Resource Limit
Customer managed policies in an account 500
Groups in an account 100
Roles in an account 200
Managed policies that are attached to an IAM group 10
Managed policies that are attached to an IAM role 10
Managed policies that are attached to an IAM user 10
IAM users in a group Equal to user quota in an account
Users in an account 500

IAM entities limits

Resource Limit
Access keys that are assigned to an IAM user 2
Access keys that are assigned to the account root user 2
Groups an IAM user can be a member of 10
Identity providers (IdPs) associated with an IAM SAML 10
provider object
Keys per SAML provider 1
Managed policies attached to an IAM group 10
Permissions boundaries for an IAM user 1
Permissions boundaries for an IAM role 1
SAML providers in an AWS account 10
Tags that can be attached to an IAM user 50
Tags that can be attached to an IAM role 50
Versions of a managed policy that can be stored 5

IAM entity character limits

Description Limit
Path 512 characters
User name 64 characters
Group name 128 characters
Role name 64 characters

200 Accessing data with IAM and S3

Description Limit
Tag key 128 characters
Tag value 256 characters
NOTE: Tag values can be empty. That is, tag values can
have a length of 0 characters.

Unique IDs created by IAM 128 characters

Policy name 128 characters
Role trust policy JSON text (the policy that determines who is 2,048 characters
allowed to assume the role)
Role session name 64 characters
Max role session duration 24 hours
For inline policies You can add as many inline policies as you want to an IAM
user, role, or group. But the total aggregate policy size (the
sum size of all inline policies) per entity cannot exceed the
following limits:
● User policy size cannot exceed 2,048 characters.
● Role policy size cannot exceed 10,240 characters.
● Group policy size cannot exceed 5,120 characters.
NOTE: IAM does not count white space when calculating
the size of a policy against these limitations.

For managed policies ● You can add up to 10 managed policies to an IAM user,
role, or group.
● The size of each managed policy cannot exceed 6,144
characters.
NOTE: IAM does not count white space when calculating
the size of a policy against these limitations.

For session policies ● You can pass only one JSON policy as a parameter when
you programmatically create a temporary session for a role
or federated user.
● The size of each session policy cannot exceed 2,048
characters.

Access Management
Access is managed by creating policies and attaching them to IAM identities or resources.

ObjectScale IAM protects the following resources:

● Object Head API
○ S3 (buckets and objects). See Accessing data with IAM and S3 for details on the supported IAM S3 APIs.
● STS APIs
○ AssumeRole (Provides temporary credentials for cross account access)
○ AssumeRoleWithSAML (Provides temporary credentials for SAML authenticated users)
● IAM API

Accessing data with IAM and S3 201

IAM Policies and ACLs
A policy is an object that when associated with an identity or resource defines their permissions. Permissions in the policies
determine if the request is permitted or denied.

IAM Policies
ObjectScale IAM enables creation, modification, listing, assigning, and deletion of policies on an identity or resource. IAM policies
are stored in JSON format.
Using policies you can:
● Specify actions on a resource.
● Identify resources.
● Identify principals that are applicable for the policies.
● Specify conditions that are applicable
IAM policies define permissions for an action regardless of the method that you use to perform the operation. The following
policy types, are designed for use in ObjectScale:

Table 44. Policy Types

Policies Description
Identity-based policies Policies that are assigned to users, groups, and roles which grant permissions to an identity.
● Inline Policies
● Managed Policies (Both ObjectScale and Customer managed)
Resource-based policies Resource-based policies are inline policies that are assigned to an ObjectScale resource that
grants specified principal permission to perform specific action on the resource.
● Bucket Policy - Tweaked existing support for bucket policies to support IAM use cases.
● Trust Policy - Is a resource-based policy that is attached to an IAM role. Trust policies
identify the principal entities that can assume the role.
Permission Boundaries Use a managed policy as the permissions boundary for an IAM entity (user or role). That policy
defines the maximum permissions that the identity-based policies can grant to an entity, but
does not grant permissions. Permissions boundaries do not define the maximum permissions
that a resource-based policy can grant to an entity.
Session policies Session policies are used with AssumeRole and AssumeRoleWithSAML APIs. Session policies
limit the permissions that the identity-based policies of a role or user grants to the session.
Session policies limit permissions for a created session, but do not grant permissions.
Access Control Lists (ACLs) ACLs are cross-account permissions policies that grant permissions to the specified principal.
ACLs cannot grant permissions to entities within the same account.

NOTE: If there is an explicit deny in any policy, then the request is denied otherwise there must be a policy that explicitly
allows the request. If neither then by default the request is denied.

Policy Basics
Policy is made up of one or list of statements. A statement is contained within a series of elements.

Version Specify the version of the policy language that you want to use. As a best practice, use the latest
2012-10-17 version.
Statement Use this main policy element as a container for the following elements. You can include more than one
statement in a policy.
Sid (Optional) Include an optional statement ID to differentiate between your statements.
Effect Use Allow or Deny to indicate whether the policy allows or denies access.
Principal (Required in only some circumstances) If you create a resource-based policy, you must indicate the
account, user, role, or federated user to which you would like to allow or deny access. If you are creating

202 Accessing data with IAM and S3

 an IAM permissions policy to attach to a user or role, you cannot include this element. The principal is
implied as that user or role.
Action Include a list of actions that the policy allows or denies.
Resource ARN of resources the permission applied on. * apply to all resources.
Condition (Optional) Specify the circumstances under which the policy grants permission.

ACLs
Access control lists allow you to manage access to objects and buckets. An ACL is attached to all objects and buckets. With S3
ObjectScale IAM access:
● Buckets are owned by the account to which they belong and objects are owned by the account to which the user that
created the object belongs.
● Buckets and object owners can never be changed.
● Only an account can be a non-group grantee in an ACL.

S3 request authorization
During the S3 request authorization process, ObjectScale evaluates permission using user, bucket, and object contexts as
needed.

Context Description
User In this context, if the requester is an ObjectScale IAM principal, the principal must have permission from
the parent account to which it belongs. In this step, the subset of policies that are owned by the parent
account (also referred as the context authority) is evaluated. This subset of policies includes the user policy
that the parent attaches to the principal. If the parent also owns the resource in the request (bucket,
object), then the corresponding resource policies (bucket policy, bucket ACL, and object ACL) are also
evaluated at the same time.
Bucket In this context, ObjectScale evaluates policies that are owned by the account that owns the bucket. If the
account that owns the object in the request is not same as the bucket owner, in the bucket context the
policies are checked to verify that the bucket owner has not explicitly denied access to the object. If there
is an explicit deny set on the object, then the request is not authorized.
Object In this context, the requester must have permissions from the object owner to perform a specific object
operation. In this step, the object ACL is evaluated if required.

Bucket authorization
In the S3 bucket operation authorization process, at first the system evaluates whether the requester is an IAM user. If yes,
then the request is evaluated against the user context and the bucket contexts. If both verifications are authorized, the access
is granted. Else, it is denied.
The below table describes the summary of access details for the same and cross account bucket operation:

Bucket owner Requestor Comments

(account) (account, user)
A1 U1 The user or the bucket policy determines the access. There is no bucket ACL
check.
A1 U2 U2 needs IAM policy from A2, if A1 bucket policy does not a make a
determination, then the system checks the bucket ACL.
A1 R1 IAM policy is not relevant for root user (R1). If A1 bucket policy does not a
make a determination, then the system checks the bucket ACL.
A1 R2 IAM policy is not relevant for root user (R2). If A1 bucket policy does not a
make a determination, then the system checks the bucket ACL.

Accessing data with IAM and S3 203

Bucket owner Requestor Comments
(account) (account, user)

NOTE: In this table, the following legends are used:

A1 = first account, A2 = second account, U1 = user from the first account, U2 = user from the second account, R1 = root
user from the first account, and R2 = root user from the second account.

Object authorization
In the S3 object operation authorization process, at first the system evaluates whether the requester is an IAM user. If yes, then
the request is evaluated against the user, bucket, and object contexts. If these three contexts verifications are authorized, the
access is granted. Else, it is denied.
The below table describes the summary of access details for the same and cross account bucket operation:

Bucket owner Object owner Requestor Comments

(account) (account)
A1 A1 U1 Access is determined by the user and/or by the bucket
policy. No object ACL check
A1 A1 U2 U2 needs IAM policy from A2 and if A1 bucket policy does
not a make a determination, then the system checks the
object ACL
A1 A1 R1 IAM policy not relevant for R1. If A1 bucket policy does not
a make a determination, then the system checks the object
ACL
A1 A1 R2 IAM policy not relevant for R2. If A1 bucket policy does not
a make a determination, then the system checks the object
ACL
A1 A2 U1 U1 needs IAM policy or bucket policy allow. Object ACL must
allow A1 access.
A1 A2 U2 U2 needs IAM policy allow. Bucket policy should not deny.
NOTE: Bucket policy cannot allow access.

A1 A2 U3 U3 needs IAM policy allow. Bucket policy should not deny.

Object ACL must allow A3 access.
NOTE: Bucket policy cannot allow access.

A1 A2 R1 IAM policy not relevant. Bucket policy should not deny.

Object ACL needs to allow A1 access.
NOTE: Bucket policy cannot allow access.

A1 A2 R2 IAM policy not relevant. Bucket policy should not deny.

Object ACL must allow A2 access.
NOTE: Bucket policy cannot allow access.

A1 A2 R3 IAM policy not relevant. Bucket policy should not deny.

Object ACL must allow A3 access.
NOTE: Bucket policy cannot allow access.

NOTE: In this table, the following legends are used:

A1 = first account, A2 = second account, A3 = third account, U1 = user from the first account, U2 = user from the second
account, U3 = user from the third account, R1 = root user from the first account, R2 = root user from the second account,
and R3 = root user from the third account.

204 Accessing data with IAM and S3

IAM and STS resource requests
ObjectScale evaluates the authorization requests on ECS IAM and STS resources within one account in the following manner.
1. Deny evaluation - By default, all requests are denied (implicit deny). PEM evaluates all policies within the account that
apply to the request. These include resource-based policies, permissions boundaries, role session policies, and identity-based
policies. In all these policies, enforcement code looks for a Deny statement that applies to the request (explicit deny). If the
code finds even one explicit deny that applies, the code returns a final decision of Deny. If there is no explicit deny, the
evaluation continues.
2. Resource-based policies - If the requested resource has a resource-based policy that allows the principal entity to perform
the requested action, then the code returns a final decision of Allow. If there is no resource-based policy, or if the policy
does not include an Allow statement, then the code continues. This logic can behave differently if you specify the ARN of
an IAM role or user as the principal of the resource-based policy. Someone can use session policies to create a temporary
credential session for that role or federated user. In that case, the effective permissions for the session might not exceed
those allowed by the identity-based policy of the user or role.
3. IAM permissions boundaries - The enforcement code then checks whether the IAM entity that is used by the principal has
a permissions boundary. If the policy that is used to set the permissions boundary does not allow the requested action, then
the request is implicitly denied. The code returns a final decision of Deny. If there is no permissions boundary, or if the
permissions boundary allows the requested action, the evaluation continues.
4. Session policies - The code then checks whether the principal entity is using a session that was assumed by passing a
session policy. You can pass a session policy while using temporary credentials for a role or federated user. If the session
policy is present and does not allow the requested action, then the request is implicitly denied. The code returns a final
decision of Deny. If there is no session policy, or if the policy allows the requested action, the code continues.
5. Identity-based policies - The code then checks the identity-based policies for the principal entity. For an IAM user, these
include user policies and policies from groups to which the user belongs. If any statement in any applicable identity-based
policies allows the requested action, then the PEM evaluation returns a final decision of Allow. If there are no statements
that allow the requested action, then the request is implicitly denied, and the code returns a final decision of DenyErrors
that is any errors that are encountered by PEM during the evaluation will throw an exception and stops evaluation.

Security Token Service

The Security Token Service (STS) enables you to request temporary credentials, for IAM users or for other users that are
externally authenticated (SAML).
ObjectScale IAM supports these two STS APIs:
● AssumeRole (Provides temporary credentials for cross account access)
● AssumeRoleWithSAML (Provides temporary credentials for SAML authenticated users)
NOTE: The temporary credentials from the AssumeRole and AssumeRoleWithSAML APIs consist of an access key ID,
secret access key, and a session token. These temporary credentials cannot be revoked.

Accessing accounts using AssumeRole

AssumeRole returns a set of temporary security credentials that you can use to access IAM and S3 resources.
NOTE: The role trust relationship should grant permission to an entity to assume the role.

Same account access with AssumeRole

You can access the same account using AssumeRole by attaching a policy to the user (identical to the previous user in a
different account) or by adding the user as a principal directly in the role trust policy.

Method Example
Attaching a policy to the user 1. Trust policy for Role assumeRoleSameAccount in ns1:

{
"Version": "2012-10-17",
"Statement": [
{

Accessing data with IAM and S3 205

Method Example
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::ns1:root"
},
"Action": "sts:AssumeRole"
}
]
}

2. Policy is attached to the user1 in ns1 to AssumeRole:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Resource": "urn:osc:iam::ns1:role/
assumeRoleSameAccount",
"Effect": "Allow",
"Sid": "VisualEditor0"
}
]
}

Adding the user to the role trust Trust policy for Role in ns1 with an ObjectScale IAM user:
policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::ns1:user/user1"
},
"Action": "sts:AssumeRole"
}
]
}

Cross account access with AssumeRole

By default, an IAM user in one account has no access to buckets in another account. However, you can access different
accounts using the role trust policy through AssumeRole.
Your account must be trusted by the role to assume a role from a different account. The trust relationship is defined in the role
trust policy when the role is created. That trust policy states which accounts are allowed to delegate that access to users in the
account. Also, ensure that you have permissions that are delegated from the user account administrator. The administrator must
attach a policy that allows you to call AssumeRole for the Amazon Resource Name (ARN) of the role in the other account.
For example, your organization has multiple account. From which, you segregate a staging environment from a production
environment. Certain users such as developers from the staging account may also want to access the production account when
you move the staging environment to the production.
● For this scenario, the admin creates two groups for the staging account namely Dev and QE, and each group has its own
policy.
● In the production account, the administrator performs the following:
○ Specifies a trust policy to the role to state that the staging account as a Principal. So that the authorized users from the
staging account can use that role.
○ Specifies which role users have read and write permissions to the productionsys bucket through a permissions policy.
○ Shares the account and role information with the users who need to assume the role.
● In the staging account, the administrator grants permission to the Dev group to assume the UpdateSys role. By doing this,
the Dev group members can switch their role to the required and permitted role. For example, the Dev group members can

206 Accessing data with IAM and S3

 switch their role to the UpdateSys role in the production account. Other users such as QE group members cannot switch
their role. Hence, they cannot access the productionsys bucket.
In this process, STS verifies whether the requester is a trusted entity. After verifying, it returns temporary credentials to the
authorized users to perform the required actions.

Example
1. Trust policy for Role in ns1:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "urn:osc:iam::ns2:root"
},
"Action": "sts:AssumeRole"
}
]
}

2. Policy that is attached to the user in ns2 to AssumeRole:

{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sts:AssumeRole"
],
"Resource": "urn:osc:iam::ns1:role/assumeRoleCrossAccount",
"Effect": "Allow",
"Sid": "VisualEditor0"
}
]
}

IAM SAML support

Security Assertion Markup Language (SAML) is an open standard for exchanging authentication and authorization data between
parties, in particular, between an identity provider (IdP) and a service provider.
ObjectScale currently supports only SAML integration with Microsoft Active Directory Federation Services (ADFS) version
6.3.0.0. This establishment enables the federated users to access ObjectScale resources.

Setting up a SAML-compliant provider

ObjectScale currently supports only ADFS as a SAML-compliant Identity Provider. Perform the following steps to use ADFS as a
SAML-compliant Identity Provider. ObjectScale will be the service provider.

About this task

You can use this interface to generate ObjectScale metadata XML to configure ObjectScale trust relationship with the identity
provider. The generation requires a java key store and a DNS-domain-name which will be used as the entity Base URL to set the
Location in the Assertion Consumer Service.

Steps
1. Download the Identity Provider (ADFS) metadata file. The default URL to download ADFS metadata is https://[server-
name]/FederationMetadata/2007-06/FederationMetadata.xml.
2. Upload the downloaded metadata xml file when creating Identity provider.
3. To create the Identity Provider in ObjectScale Portal UI, follow to Add a new identity provider to an account.
In order to establish trust relationship between ObjectScale and ADFS, ObjectScale metadata xml file is required.

Accessing data with IAM and S3 207

4. To create or download the ObjectScale SAML Service Provider Metadata file, refer Generate SAML Service Provider
Metadata.
5. Establish trust relationship between ObjectScale and ADFS using the downloaded ObjectScale SAML Service Provider
Metadata file.
6. Add claim rules in ADFS to add the required elements such as NameId, RoleSessionName, and Roles to the SAML
authentication process.
NOTE: If required, contact Dell remote support for configuring claim rules in ADFS.

NOTE: Only one IdP is supported in the federation metadata from ADFS.

AssumeRoleWithSAML
In order to use AssumeRoleWithSAML, you must configure your SAML identity provider (IdP) like ADFS to issue the claims
required by ObjectScale.
● IAM role must be created that specifies this SAML Provider in the trust policy.
● In order to use AssumeRoleWithSAML from each ObjectScale instance, you must first setup Relying Party Trust with that
ObjectScale service provider metadata and get the SAML Token from that specific relying party trust.
● AssumeRoleWithSAML returns a set of temporary security credentials for users who have been authenticated through a
SAML authentication response.
● This operation provides a mechanism for tying an enterprise identity store or directory to role-based access without
user-specific credentials or configuration.
● Calling AssumeRoleWithSAML does not require the use of ObjectScale security credentials. The identity of the caller is
validated by the claims that are provided in the SAML Assertions by the identity provider.
● Temporary credentials consist of an access key ID, a secret access key, and a security token.
● Following condition keys are supported in the AssumeRolePolicyDocument.
○ saml:aud
○ saml:iss
○ saml:sub
○ saml:sub_type
○ saml:edupersonorgdn
○ saml:namequalifier

Example role trust policy

{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"Federated":"urn:aws:iam::s3:saml-provider/provider1"
},
"Action":"sts:AssumeRoleWithSAML",
"Condition":{
"StringEquals":{
"SAML:sub":"ADFS\\Bob",
"SAML:aud":"https://10.247.179.105/saml",
"SAML:eduPersonOrgDN":[
"ObjectScale"
],
"SAML:iss":"http://AD.adfs.emc.com/adfs/services/trust"
}
}
}
]
}

208 Accessing data with IAM and S3

Attributes in SAML assertion
The following attributes are required in SAML assertion.
● https://aws.amazon.com/SAML/Attributes/RoleSessionName
● https://aws.amazon.com/SAML/Attributes/Role
NOTE:
● The Role attribute must be of the format SAML Provider URN, Role URN to be used from ObjectScale for an AD
Group.
● If you must use saml:edupersonorgdn, then oid attribute must also be present in the SAML assertion as
urn:oid:1.3.6.1.4.1.5923.1.1.1.3. However, it is optional to use this attribute.

For example:

<AttributeStatement>
<Attribute Name="https://aws.amazon.com/SAML/Attributes/RoleSessionName">
<AttributeValue>[email protected]</AttributeValue>
</Attribute>
<Attribute Name="https://aws.amazon.com/SAML/Attributes/Role">
<AttributeValue>urn:osc:iam::s3:saml-provider/provider1,urn:osc:iam::s3:role/
ADFS-Dev</AttributeValue>
<AttributeValue>urn:osc:iam::s3:saml-provider/provider1,urn:osc:iam::s3:role/
ADFS-Production</AttributeValue>
</Attribute>
<Attribute Name="urn:oid:1.3.6.1.4.1.5923.1.1.1.3">
<AttributeValue>ObjectScale</AttributeValue>
</Attribute>
</AttributeStatement>

User-specific access using SAML keys

Dell recommends that you specify permissions based on the users identity when creating access policies in IAM.
As to create policies that contain user-specific information, the user identity should be available in SAML keys. The following
SAML keys can be used in policy conditions to create unique user identifiers.

SAML keys Description

saml:namequalifier A hash value based on the concatenation of the Issuer response value (saml:iss) and a
string with the ObjectScale account (account ID) and the friendly name (the last part of the
ARN) of the SAML provider in IAM. The account (account ID) and provider name must be
separated by a '/' as in "123456789012/provider_name".
The combination of NameQualifier and Subject can be used to uniquely identify
a federated user. The following pseudocode shows how this value is calculated. In this
pseudocode, "+" indicates concatenation, SHA1 represents a function that produces a message
digest using SHA-1, and Base64 represents a function that produces Base-64 encoded version
of the hash output.
Base64 ( SHA1 ( "https://example.com/saml" + "ObjectScaleAccount" + "/SamlProvider" ) )

saml:sub This is the subject of the claim, which includes a value that
uniquely identifies an individual user within an organization. For example,
_3e52ef03414f3464d2461c00ebae0152c25fb88bbc.

saml:sub_type This key can be persistent, transient, or the full Format URI from the Subject and NameID
elements used in your SAML assertion. A value of persistent indicates that the value in
saml:sub is the same for a user across all sessions. If the value is transient, the user has
a different saml:sub value for each session.

Accessing data with IAM and S3 209

IAM Policy
The following example shows a permission policy that uses the preceding keys to grant permissions to a user-specific
folder in Amazon S3. The policy assumes that the Amazon S3 objects are identified using a prefix that includes
both saml:namequalifier and saml:sub. Notice that the Condition element includes a test to be sure that
saml:sub_type is set to persistent. If it is set to transient, the saml:sub value for the user can be different for each
session, and the combination of values should not be used to identify user-specific folders.

{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::exampleObjectScaleBucket/backup/${saml:namequalifier}/${saml:sub}",
"arn:aws:s3:::exampleObjectScaleBucket/backup/${saml:namequalifier}/${saml:sub}/*"
],
"Condition": {"StringEquals": {"saml:sub_type": "persistent"}}
}
}

Example with sample values

● Create a role using AssumeRoleWithSAML.
● Attach an IAM policy to this role as below.

{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::exampleObjectScleBucket/backup/${saml:namequalifier}/${saml:sub}",
"arn:aws:s3:::exampleObjectScleBucket/backup/${saml:namequalifier}/$
{saml:sub}/*"
],
"Condition": {"StringEquals": {"saml:sub_type": "persistent"}}
}
}

The values in the above example are as follows:

● saml:iss = http://AD.adfs.emc.com/adfs/services/trust. See IAM supported condition keys for the list of SAML condition
keys.
● account = s3
● providername = provider1
● saml:sub = ADFS\Bob
● Base64 SHA1 ("http://AD.adfs.emc.com/adfs/services/trust " + "s3" + "/provider1")
● SHA1 = BB9445BB2D9C57D519ACEBD08EFD428076522D5B
● Base64 of BB9445BB2D9C57D519ACEBD08EFD428076522D5B is u5RFuy2cV9UZrOvQjv1CgHZSLVs=.

210 Accessing data with IAM and S3

IAM Resource ARNs
Resource ARN formats and Unique ID Prefixes that are supported by ObjectScale IAM are described here.

Table 45. Global Replicated Entities

Entity ARN Unique ID Prefix
User urn:osc:iam:<accountid>:user<path><usern OIDA
ame>

Group urn:osc:iam:<accountid>:group<path><user OGPA

name>

Role urn:osc:iam:<accountid>:role<path><usern OROA

ame>

Customer-managed policy urn:osc:iam:<accountid>:policy<path><use ONPA, ONVA(versioned

rname> policy)
ObjectScale system-managed urn:osc:iam::policy<path><username> ONPA
policy
SAML provider urn:osc:iam:<accountid>:saml-provider/ -
<samlprovidername>

Federated user urn:osc:sts:<accountid>:federated-user/ -

<federatedusername>

Active session with assume role urn:osc:sts:<accountid>:assumed- -

role<path><rolename>/<rolesessionname>

Access key - OKIA

Temporary (STS) keys - OSI

Table 46. Bucket resources

Resource ARN Scope Operations
ObjectScale Bucket arn:aws:s3:<object-scale- ObjectScale CRUD, List
name-id>:<object-store-name-
id>:<bucket-name>

Amazon S3 API support in ObjectScale

ObjectScale supports the Amazon Simple Storage Service (Amazon S3) Application Programming Interface (API).

Table 47. S3 Object Service

Protocol Ports
HTTP 80
HTTPS 443

S3 support in ObjectScale
S3 APIs are supported in this release of ObjectScale.

S3 API
See the ObjectScale Rest API .zip file on Dell Support at https://www.dell.com/support/home/en-us/product-support/
product/objectscale/docs for a complete list of the supported S3 APIs.

Accessing data with IAM and S3 211

S3 Select
ObjectScale supports S3 Select. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-
reference-select.html for details on the S3 Select usages supported on ObjectScale.
The S3 Select API allows applications to retrieve a subset of data of an object by using SQL expressions. S3 Select saves on
both the bandwidth and client-side processing. For example, without S3 Select applications must download the entire object and
then do the processing on that data. With S3 Select, an application instead issues a SQL select command and gets only a small
subset of the object data. Further, an application does not have to do any additional processing.
S3 Select can be used for objects in the following formats:
● CVS
● Json
● Parquet
S3 Select supports querying gzip and bzip2 compressed objects of the above file types.
You can use S3 Select API on its own or with a query engine, like presto. A connector in presto can determine if a particular
query can be sent directly to the storage, for example s3 select pushdown.

Multipart upload (MPU) support for huge objects

ObjectScale supports uploading large objects, using the multipart upload (MPU) process in the S3 protocol. ObjectScale
supports objects up to 50 TiB in size.
MPU allows you to upload huge objects by constructing a single, large object from uploaded smaller parts. You can upload these
smaller parts at any time and in any order. Dell Technologies recommends that you verify that your request to complete MPU is
successful. After ObjectScale receives the completeMPU request, ObjectScale assembles the parts and creates the object.
If the upload of one or more of the parts fails, you reupload only the failed part or parts.
The three steps to complete an MPU are:
1. Initiate the upload.
2. Upload the object parts.
3. Complete the multipart upload, which cleans up the uploaded parts and a new combined version is created.
After an MPU, you can access this huge object as you would any other object in your bucket. Replication of these huge objects
within ObjectScale is also supported.
NOTE: Replication of objects larger than 50 TiB is not supported.

For the best results, the part name numbering (partNumber) should be contiguous, beginning with 1 for the first object part.
Each subsequent part number for the parts of the object should increment by whole numbers with no gaps in the numbering.
Also, all parts should be the same size, with an exception for the last part, which can be smaller than the other parts. Deviations
from these recommendations increase the metadata overhead and worsen performance.
MPU in ObjectScale has the following limits:
● The maximum size for a part is 5 GiB. There is no minimum size.
● The maximum supported size for an object is 50 TiB. There is no minimum size.
● The maximum number of parts for an MPU is 50,000.
Object part size and number of parts impact the time that it takes to upload and complete the creation of the combined object.
Dell Technologies recommends that the part sizes be greater than 2.13 MiB and less than 2 GiB, for optimal processing.
See the ObjectScale Management Rest API .zip file on Dell Support at https://www.dell.com/support/home/en-us/product-
support/product/objectscale/docs for the MPU APIs.

212 Accessing data with IAM and S3

ObjectScale S3 error codes
The error codes that can be generated by the ObjectScale S3 are listed in the following table.

Table 48. Error Codes

Error Code HTTP Generic Error Code Description Error
Status
Code
AccessDenied 403 AccessDenied Access Denied
BadDigest 400 BadDigest The Content-MD5 you specified did
not match that received.
BucketAlreadyExists 409 BucketAlreadyExists The requested bucket name is not
available. The bucket namespace is
shared by all users of the system.
Please select a different name and
try again.
BucketNotEmpty 409 BucketNotEmpty The bucket you tried to delete is not
empty.
ContentMD5Empty 400 InvalidDigest The Content-MD5 you specified was
invalid.
ContentMD5Missing 400 InvalidRequest The required Content-MD5 header
for this request is missing.
EntityTooSmall 400 EntityTooSmall The proposed upload is smaller than
the minimum allowed object size.
EntityTooLarge 400 EntityTooLarge The proposed upload exceeds the
maximum allowed object size.
IncompleteBody 400 IncompleteBody The number of bytes specified by
the Content-Length HTTP header
were not provided.
InternalError 500 InternalError An internal error was encountered.
Please try again.
ServerTimeout 500 ServerTimeout An internal timeout error was
encountered. Please try again.
InvalidAccessKeyId 403 InvalidAccessKeyId The Access Key Id you provided does
not exist.
InvalidArgument 400 InvalidArgument Invalid Argument.
NoNamespaceForAnonymousRequest 403 AccessDenied ObjectScale could not determine
the namespace from the anonymous
request. Please use a namespace
BaseURL or include an x-emc-
namespace header.

InvalidBucketName 400 InvalidBucketName The specified bucket is not valid.

InvalidDigestBadMD5 400 InvalidDigest The Content-MD5 you specified was
invalid.
InvalidDigest 403 SignatureDoesNotMatch The Content-MD5 you specified was
an invalid.
InvalidRequest 400 InvalidRequest Invalid Request.
InvalidPart 400 InvalidPart One or more of the specified parts
could not be found. The part might
not have been uploaded.

Accessing data with IAM and S3 213

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
InvalidPartOrder 400 InvalidPartOrder The list of parts was not in ascending
order. Parts list must specified in
order by part number.
InvalidPartSizeZero 400 InvalidPartSizeZero The upload part size cannot be zero.
MissingEncryption 400 InvalidRequest The multipart upload initiate
requested encryption. Subsequent
part requests must include the
appropriate encryption parameters.
NoEncryptionNeed 400 InvalidRequest The multipart initiate request did not
request encryption. Please resend the
request without sending encryption
parameters.
BadMD5 400 InvalidRequest The calculated MD5 hash of the key
did not match the hash that was
provided.
BadEncryptKey 400 InvalidRequest The provided encryption parameters
did not match the ones used
originally.
InvalidRange 416 InvalidRange The requested range cannot be
satisfied.
KeyTooLong 400 KeyTooLong The specified key is too long.
MalformedACLError 400 MalformedACLError The XML provided was not well-
formed or did not validate against the
ObjectScale published schema.
MalformedXML 400 MalformedXML Malformed xml (that does not
conform to the published xsd) for the
configuration was sent.
MaxMessageLengthExceeded 400 MaxMessageLengthExceeded The request was too big.
MetadataTooLarge 400 MetadataTooLarge The metadata headers exceed the
maximum allowed metadata size. *
InvalidProject 400 InvalidProject The specified project is Invalid.
InvalidVPool 400 InvalidVPool The specified vPool (Replication
Group) is Invalid.
InvalidNamespace 400 InvalidNamespace The specified namespace is Invalid.
MethodNotAllowed 405 MethodNotAllowed The specified method is not allowed
against this resource.
MissingContentLength 411 MissingContentLength The Content-Length HTTP header
must be provided.
MissingRequestBodyError 400 MissingRequestBodyError An empty XML document was sent.
The error message is: Request body
is empty.
MissingSecurityHeader 400 MissingSecurityHeader The equest was missing a required
header.
IncompleteLifecycleConfig 400 IncompleteLifecycleConfig At least one action needs to be
specified in a rule.

214 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
MalformedLifecycleConfig 400 MalformedLifecycleConfig The XML provided was not well-
formed or did not validate against the
published schema.
MalformedDateLifecycleConfig 400 MalformedDateLifecycleConfig The XML provided was not well-
formed or did not validate against
the published schema. Invalid Date or
Days.
NoSuchBucket 404 NoSuchBucket The specified bucket does not exist.
NoSuchBucketPolicy 404 NoSuchBucketPolicy The bucket policy does not exist.
NoSuchKey 404 NoSuchKey The specified key does not exist.
NoSuchRetention 404 NoSuchRetention The specified retention does not
exist.
ObjectUnderRetention 409 ObjectUnderRetention The object is under retention and
cannot be deleted or modified.
NoSuchUpload 404 NoSuchUpload The specified multipart upload does
not exist. The upload ID might be
invalid.
NotImplemented 501 NotImplemented The requested functionality is not
implemented.
OperationAborted 409 OperationAborted A conflicting conditional operation
is currently in progress against this
resource. Please try again.
PermanentRedirect 301 PermanentRedirect The bucket you are attempting to
access must be addressed using the
specified endpoint. Please send all
future requests to this endpoint.
PreconditionFailed 412 PreconditionFailed At least one of the preconditions you
specified did not hold.
RequestIsNotMultiPartContent 400 RequestIsNotMultiPartContent Bucket POST must be of the
enclosure type multipart/form-
data.

RequestTimeout 400 RequestTimeout The socket connection to the server

was not read from or written to
within the timeout period.
RequestTimeTooSkewed 403 RequestTimeTooSkewed The difference between the request
time and the server's time is too
large.
DateIsRequired 403 AccessDenied A valid Date or x-amz-date header
is required.
SignatureDoesNotMatch 403 SignatureDoesNotMatch The request signature calculated
does not match the signature
provided. Check the Secret Access
Key and signing method.
ZeroAmzExpires 403 Forbidden Zero value specified for x-amz-
expires.

InvalidAmzExpires 400 Bad Request Invalid value specified for x-amz-

expires.

Accessing data with IAM and S3 215

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
ServiceUnavailable 503 ServiceUnavailable Please reduce your request rate.
TemporaryRedirect 307 TemporaryRedirect Requests are being redirected to the
bucket while DNS updates.
TooManyBuckets 400 TooManyBuckets The request attempted to create
more buckets than allowed.
UnexpectedContent 400 UnexpectedContent The request does not support this
content.
UnresolvableGrantByEmailAddress 400 UnresolvableGrantByEmailAddress The email address you provided does
not match any account on record.
InvalidBucketState 409 InvalidBucketState The request is not valid with the
current state of the bucket.
SlowDown 503 SlowDown Please reduce your request rate.
AccountProblem 403 AccountProblem There is a problem with the specified
account that prevents the operation
from completing successfully.
CrossLocationLoggingProhibited 403 CrossLocationLoggingProhibited Cross location logging is not allowed.
Buckets in one geographic location
cannot log information to a bucket in
another location.
ExpiredToken 400 ExpiredToken The provided token has expired.
IllegalVersioningConfigurationExcepti 400 IllegalVersioningConfigurationExcepti The Versioning configuration
on on specified in the request is invalid.
IncorrectNumberOfFilesInPostReques 400 IncorrectNumberOfFilesInPostReques POST requires exactly one file upload
t t per request.
InvalidAddressingHeader 500 InvalidAddressingHeader The specified role must be
Anonymous role.
InvalidLocationConstraint 400 InvalidLocationConstraint The specified location constraint is
not valid.
InvalidPolicyDocument 400 InvalidPolicyDocument The content of the form does not
meet the conditions specified in the
policy document.
InvalidStorageClass 400 InvalidStorageClass The storage class you specified is not
valid.
InvalidTargetBucketForLogging 400 InvalidTargetBucketForLogging The target bucket for logging does
not exist, is not owned by you, or
does not have the appropriate grants
for the log delivery group.
InvalidToken 400 InvalidToken The provided token is malformed or
otherwise invalid.
InvalidURI 400 InvalidURI Unable to parse the specified URI.
MalformedPOSTRequest 400 MalformedPOSTRequest The body of the POST request is
not well-formed multipart/form-
data.

MaxPostPreDataLengthExceededErr 400 MaxPostPreDataLengthExceededErr The POST request fields preceding

or or the upload file were too large.

216 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
NoLoggingStatusForKey 400 NoLoggingStatusForKey There is no such thing as a logging
status subresource for a key.
NoSuchLifecycleConfiguration 404 NoSuchLifecycleConfiguration The lifecycle configuration does not
exist.
NoSuchVersion 404 NoSuchVersion Indicates that the version ID specified
in the request does not match an
existing version.
RequestTorrentOfBucketError 400 RequestTorrentOfBucketError Requesting the torrent file of a
bucket is not permitted.
UserKeyMustBeSpecified 400 UserKeyMustBeSpecified The bucket POST must contain the
specified field name. If it is specified
please check the order of the fields.
AmbiguousGrantByEmailAddress 400 AmbiguousGrantByEmailAddress The email address you provided
is associated with more than one
account.
BucketAlreadyOwnedByYou 409 BucketAlreadyOwnedByYou The previous request to create the
named bucket succeeded and you
already own it.
CredentialsNotSupported 400 CredentialsNotSupported The request does not support
credentials.
InlineDataTooLarge 400 InlineDataTooLarge The inline data exceeds the maximum
allowed size.
InvalidPayer 403 InvalidPayer All access to this object has been
disabled.
TokenRefreshRequired 400 TokenRefreshRequired The provided token must be
refreshed.
AccessModeNotSupported 409 AccessModeNotSupported The bucket does not support file
access or the requested access mode
is not allowed.
AccessModeInvalidToken 409 AccessModeInvalidToken The token for the file access switch
request is invalid.
NoSuchBaseUrl 400 NoSuchBaseUrl The specified BaseUrl does not exist.
NoDataStoreForVirtualPool 404 NoDataStoreForVirtualPool No Data Store found for Replication
Group of the bucket.
VpoolAccessNotAllowed 400 Cannot AccessVpool Bucket is hosted on a Replication
Group that is not accessible from S3.
InvalidCorsRequest 403 InvalidCorsRequest Invalid CORS request.
InvalidCorsRule 400 InvalidCorsRule Invalid CORS rule.
NoSuchCORSConfiguration 404 NoSuchCORSConfiguration The CORS configuration does not
exist.
InvalidAclRequest 404 NoACLFound The ACL does not exist.
InsufficientStorage 507 InsufficientStorage The server cannot process the
request because there is not enough
space on disk.
BadMaxParts 400 InvalidArgument Argument max-parts must be an
integer between 0 and 2147483647.

Accessing data with IAM and S3 217

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
BucketNotFound 404 NoSuchBucket The specified bucket does not exist.
NotSupported 400 Not Supported The bucket may be locked.
InvalidContentLength 400 Invalid content length The content length has invalid value.
InvalidVersioningRequest 403 Invalid request for version control The bucket is in compliance mode.
InvalidLifeCycleRequest 403 Invalid request for life cycle The bucket is in compliance mode.
RetentionPeriodRequired 400 Invalid request for bucket with The bucket requires a retention
compliance period.
Conflict 409 Conflict The bucket may be locked.
MethodForbidden 403 Forbidden Check if quota has been exceeded.
NotAcceptable 406 Content encoding not acceptable The object Content-Encoding
does not match requested Accept-
Content.

InvalidEncoding 400 Invalid URL enconding The URL encoding used is invalid.
InvalidMetadataQuery 400 Invalid metadata query entered The metadata query entered does not
conform to valid syntax
InvalidMetadataSearchList 400 Invalid metadata search list entered A keyname on the request is not a
valid indexable key, or the format of
the request list is incorrect.
MetadataSearchNotEnabled 405 Metadata search not enabled Metadata search is not enabled for
this bucket.
MetadataSearchBadParameter 400 Metadata search invalid parameter Invalid search index key name, sort
used in query key name or attribute name value.
MetadataSearchInvalidArgument 400 Metadata search invalid parameter Invalid search index value format or
used in query operator used.
MetadataSearchInvalidValuefor 400 Metadata search key indexing found Object operation failed because a
Datatype invalid input value user metadata value cannot be
converted to its defined datatype.
MetadataOperationNotSupported 405 Metadata search operation not Metadata query with both AND and
supported OR logical operators not supported.
MetadataSearchBadSortParameter 400 Metadata search invalid sort The sort parameter has to be present
parameter in the query as a search parameter.
MetadataSearchRestriction 400 Buckets that are encrypted or within Metadata search is mutually exclusive
an encrypted namespace cannot have with bucket/namespace encryption.
metadata search enabled
MetadataSearchTooManyIndexKeys 400 The number of Index keys exceeds The number of keys to be indexed
the maximum allowed exceeds the maximum number
allowed, try with fewer keys.
InvalidOrNoCustomerProvided 400 Invalid or no customer provided No encryption key, or an encryption
EncryptionKey encryption key key that did not match the one in the
system, was provided.
DareUnavailable 403 Server side encryption (D@RE) is not D@RE JAR/license is unavailable
supported hence server side encryption
requests are not supported.
SelfCopyInvalidRequest 400 InvalidRequest The copy request is illegal because
it is trying to copy an object to

218 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
itself without changing the object's
metadata or encryption attributes.
OverLappingPrefixes 400 Invalid Request Found overlapping prefixes.
SamePrefix 400 Invalid Request Found two rules with same prefix.
XAmzContentSHA256Mismatch 400 XAmzContentSHA256Mismatch The Content-SHA256 you specified
did not match what we received
InvalidJSON 400 InvalidJSON Policies must be valid JSON and the
first byte must be {.

InvalidBucketPolicy 400 InvalidBucketPolicy Invalid Bucket Policy.

MalformedPolicy 400 MalformedPolicy Malformed Policy.
MaxIDLengthExceeded 400 InvalidArgument ID length should not exceed allowed
limit of 255.
CrossHeadAccessBeforeUpgrade 400 InvalidRequest Cross head access is not supported.
InvalidDate 400 InvalidArgument Date must be no earlier than
1970-01-01T00:00:00.000Z.
BadContentLengthRequest 400 RequestTimeout Content-Length specified is not
matching with Length of the Content
in the body.
IncompatibleNode 500 InternalError The I/O request was sent to the
wrong node, please configure client
or Load-balancer correctly to route
request to the correct node.
InvalidFileNameArgument 400 InvalidArgument Header value cannot be represented
using ISO-8859-1.
InvalidPartNumber 400 InvalidPartNumber Part number must be an integer
between 1 and 10000, inclusive
InvalidTenant 400 TenantNotFound Specified Tenant is Invalid.
Redirect 307 Redirect Temporary redirect.
UrlAmzExpires 403 Forbidden Request has expired
VersioningCannotChange 409 InvalidBucketState An Object Lock configuration is
present on this bucket, so the
versioning state cannot be changed.
ObjectLockNotEnabled 409 InvalidBucketState Object Lock configuration cannot be
enabled on existing buckets
ObjectLockConfigurationNotFoundEr 404 ObjectLockConfigurationNotFoundEr Object Lock configuration does not
ror ror exist for this bucket
ObjectLockConfigMalformedXML 400 MalformedXML The XML you provided was not well-
formed or did not validate against our
published schema.
ObjectLockConfigInvalidArgument 400 InvalidArgument Default retention period must be a
positive integer value.
ObjectLockConfigInvalidArgumentTo 400 InvalidArgument Default retention period too large.
oLarge

Accessing data with IAM and S3 219

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
ObjectLockMalformedXML 400 MalformedXML The XML you provided was not well-
formed or did not validate against our
published schema.
ObjectLockConfigurationMissing 400 InvalidRequest Bucket is missing Object Lock
Configuration
ObjectLockNoSuchObjectLockConfig 404 NoSuchObjectLockConfiguration The specified object does not have a
uration ObjectLock configuration
ObjectLockRetailUntilMustBeInFuture 400 InvalidArgument The retain until date must be in the
future!
ObjectLockAccessDenied 403 AccessDenied Access Denied
ObjectLockAndAdo 400 InvalidRequest Object Lock enabled bucket is not
compatible with ADO.
ObjectLockAndFsa 400 InvalidRequest Object Lock enabled bucket is not
compatible with File System Access.
ObjectLockAccessDeniedNonIAM 403 AccessDenied Only IAM users are supported with
object lock enabled buckets.
ObjectLockMissingHeader 400 InvalidArgument x-amz-object-lock-retain-until-date
and x-amz-object-lock-mode must
both be supplied
ObjectLockUnknownModeDirective 400 InvalidArgument Unknown Mode directive.
ObjectLockBadDateFormat 400 InvalidArgument The retain until date must be
provided in ISO 8601 format
InvalidVersionId 400 InvalidArgument Invalid version id specified.
AccessModeInvalidToken 409 AccessModeInvalidToken The token for the file access switch
request is invalid
UnmodifiedSince 304 Not modified it has not been modified since the
specified time
Match 304 Not modified its entity tags (ETag) are not
different from the one specified
ObjectRetentionPeriodRequired 400 RetentionPeriodRequired The retention period value is required
ObjectRetentionCannotBeDecreased 400 RetentionCannotBeDecreased The new retention period value must
be greater than current
MetadataSearchInvalidQueryMarker 400 InvalidArgument The marker provided is incorrect
InvalidMetadataSearchKeys 400 Invalid metadata key Duplicate metadata key with different
type in search list entered
NoContent 204 NoContent
INVALID_CONTINUATION_TOKEN 400 InvalidArgument The continuation token provided is
incorrect
ContentTypeMissing 400 InvalidRequest Missing required header for this
request:Content-Type
ContentTypeArgMissing 400 InvalidArgument Content-Type missing for object in
CopyRangeRequest
CopyModeMissing 400 Bad Request Invalid x-emc-copy-mode value

220 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
InvalidCopyPath 400 InvalidCopySource The path of source segment is invalid
or not found
InvalidETag 400 InvalidArgument The etag of source segment miss
matching
EmptyRequestBody 400 InvalidRequest Empty request body is not allow for
copy range API
Invalid_Copy_Range 400 InvalidCopyRange Invalid source object range provide
ACCESS_DENIED_SOURCE_OBJEC 400 InvalidArgument Access denied reading one or more
T source objects
Max_Copy_Ranges_Exceed 400 MaxMessageLengthExceeded Your request exceeded the maximum
number of segments (250).
Invalid_LifeCycle_Version_Config 400 InvalidRequest can't support mixed version lifecycle
config
Metadata_Not_Allowed 400 InvalidArgument Metadata cannot be specified in this
context
Invalid_Part_Number 416 InvalidPartNumber The requested partnumber is not
satisfiable
UnSupportedV2ListingParams 400 InvalidArgument Unsupported query parameter with
GET.BUCKET in list-type=2
UnSupportedV1ListingParams 400 InvalidArgument One or more query parameters only
supported in GET.BUCKET with list-
type=2
InvalidArgumentVersion 400 InvalidArgument Version is not supported in this
request
InvalidLifecycleDays 400 InvalidArgument Days for lifecycle action must be a
positive integer
InvalidLifecycleRuleId 400 InvalidArgument Rule ID must be unique. Found same
ID for more than one rule
Invalid_Index_Granularity 400 InvalidIndexGranularity Invalid value specified for x-emc-
index-granularity
MaxLifecycleRulesLimitExceed 400 MalformedXML Number of rules should not exceed
allowed limit of 1000
MisMatchDare 400 InvalidRequest Missing Encryption parameters or the
one provided does not match the
original.
MetadataPrefixSearchBadParameter 400 invalid parameter used in query query on ObjectName not supported
with prefix
UnSupportedCopyRangeRequest 403 Forbidden IAM user is not supported for Copy
Range API
CastFailed 400 CastFailed Attempt to convert from one data
type to another using CAST failed in
the SQL expression.
ColumnTooLong 400 ColumnTooLong The length of a column in the result is
greater than maxCharsPerColumn of
1 MB.

Accessing data with IAM and S3 221

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
CSVEscapingRecordDelimiter 400 CSVEscapingRecordDelimiter Quoted record delimiter found
in the file. To allow quoted
record delimiters, please set
AllowQuotedRecordDelimiter to
'TRUE'.
CSVParsingError 400 CSVParsingError Encountered an error parsing the
CSV file. Check the file and try again.
CSVUnescapedQuote 400 CSVUnescapedQuote Unescaped quote found while
parsing the .csv file. Ensure that
AllowQuotedRecordDelimiter is set to
'TRUE' if quoted record delimiters are
present.
ExpressionTooLong 400 ExpressionTooLong The SQL expression is too long: The
maximum byte-length for the SQL
expression is 256 KB.
EvaluatorBindingDoesNotExist 400 EvaluatorBindingDoesNotExist A column name or a path provided
does not exist in the SQL expression.
EvaluatorInvalidArguments 400 EvaluatorInvalidArguments Incorrect number of arguments in the
function call in the SQL expression.
EvaluatorInvalidTimestampFormatPat 400 EvaluatorInvalidTimestampFormatPat Invalid timestamp format string in the
tern tern SQL expression.
EvaluatorTimestampFormatPatternD 400 EvaluatorTimestampFormatPatternD Timestamp format pattern
uplicateFields uplicateFields contains multiple format specifiers
representing the timestamp field in
the SQL expression.
EvaluatorTimestampFormatPatternH 400 EvaluatorTimestampFormatPatternH Timestamp format pattern contains a
ourClockAmPmMismatch ourClockAmPmMismatch 12-hour hour of day format symbol
but doesn't also contain an AM/PM
field, or it contains a 24-hour hour of
day format specifier and contains an
AM/PM field in the SQL expression.
EvaluatorInvalidTimestampFormatPat 400 EvaluatorInvalidTimestampFormatPat Timestamp format pattern contains a
ternSymbolForParsing ternSymbolForParsing valid format symbol that cannot be
applied to timestamp parsing in the
SQL expression.
EvaluatorUnterminatedTimestampFor 400 EvaluatorUnterminatedTimestampFor Timestamp format pattern contains
matPatternToken matPatternToken unterminated token in the SQL
expression.
EvaluatorInvalidTimestampFormatPat 400 EvaluatorInvalidTimestampFormatPat Timestamp format pattern contains
ternToken ternToken an invalid token in the SQL
expression.
EvaluatorInvalidTimestampFormatPat 400 EvaluatorInvalidTimestampFormatPat Timestamp format pattern contains
ternSymbol ternSymbol an invalid symbol in the SQL
expression.
IllegalSqlFunctionArgument 400 IllegalSqlFunctionArgument Illegal argument was used in the SQL
function.
InvalidColumnIndex 400 InvalidColumnIndex Column index in the SQL expression
is invalid.

222 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
InvalidCompressionFormat 400 InvalidCompressionFormat The file is not in a supported
compression format. Only GZIP and
BZIP2 are supported.
InvalidExpressionType 400 InvalidExpressionType The ExpressionType is invalid. Only
SQL expressions are supported.
InvalidFileHeaderInfo 400 InvalidFileHeaderInfo The FileHeaderInfo is invalid. Only
NONE, USE, and IGNORE are
supported.
InvalidKeyPath 400 InvalidKeyPath Key path in the SQL expression is
invalid.
InvalidJsonType 400 InvalidJsonType The JsonType is invalid. Only
DOCUMENT and LINES are
supported.
InvalidQuoteFields 400 InvalidQuoteFields The QuoteFields is invalid. Only
ALWAYS and ASNEEDED are
supported.
InvalidRequestParameter 400 InvalidRequestParameter The value of a parameter in
SelectRequest element is invalid.
Check the service API documentation
and try again.
OverMaxColumn 400 OverMaxColumn The number of columns in the
result is greater than the maximum
allowable number of columns.
OverMaxRecordSize 400 OverMaxRecordSize The length of a record in the
input or result is greater than
maxCharsPerRecord of 1 MB.
TruncatedInput 400 TruncatedInput Object decompression failed. Check
that the object is properly
compressed using the format
specified in the request.
UnauthorizedAccess 401 UnauthorizedAccess You are not authorized to perform
this operation.
ExternalEvalException 400 ExternalEvalException The query cannot be evaluated.
Check the file and try again.
InvalidDataSource 400 InvalidDataSource Invalid data source type. Only CSV,
JSON, and Parquet are supported.
InvalidDataType 400 InvalidDataType The SQL expression contains an
invalid data type.
InvalidTableAlias 400 InvalidTableAlias The SQL expression contains an
invalid table alias.
InvalidTextEncoding 400 InvalidTextEncoding Invalid encoding type. Only UTF-8
encoding is supported.
JSONParsingError 400 JSONParsingError Encountered an error parsing the
JSON file. Check the file and try
again.
UnrecognizedFormatException 400 UnrecognizedFormatException Encountered an invalid record type.

Accessing data with IAM and S3 223

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
MissingRequiredParameter 400 MissingRequiredParameter The SelectRequest entity is missing
a required parameter. Check the
service documentation and try again.
S3SelectNoMemory 503 S3SelectNoMemory Not enough memory available for the
SelectRequest.
MultipleDataSourcesUnsupported 400 MultipleDataSourcesUnsupported Multiple data sources are not
supported.
ObjectSerializationConflict 400 ObjectSerializationConflict InputSerialization specifies more
than one format (CSV, JSON,
or Parquet), or OutputSerialization
specifies more than one format (CSV
or JSON). InputSerialization and
OutputSerialization can only specify
one format each.
UnsupportedFunction 400 UnsupportedFunction Encountered an unsupported SQL
function.
UnsupportedSqlOperation 400 UnsupportedSqlOperation Encountered an unsupported SQL
operation.
UnsupportedSqlStructure 400 UnsupportedSqlStructure Encountered an unsupported SQL
structure. Check the SQL Reference.
UnsupportedStorageClass 400 UnsupportedStorageClass Encountered an invalid storage class.
Only STANDARD, STANDARD_IA,
and ONEZONE_IA storage classes
are supported.
UnsupportedSyntax 400 UnsupportedSyntax Encountered invalid syntax.
UnsupportedRangeHeader 400 UnsupportedRangeHeader Range header is not supported for
this operation.
LexerInvalidChar 400 LexerInvalidChar The SQL expression contains an
invalid character.
LexerInvalidOperator 400 LexerInvalidOperator The SQL expression contains an
invalid literal.
LexerInvalidLiteral 400 LexerInvalidLiteral The SQL expression contains an
invalid operator.
LexerInvalidIONLiteral 400 LexerInvalidIONLiteral The SQL expression contains an
invalid operator.
ParseExpectedDatePart 400 ParseExpectedDatePart Did not find the expected date part in
the SQL expression.
ParseExpectedKeyword 400 ParseExpectedKeyword Did not find the expected keyword in
the SQL expression.
ParseExpectedTokenType 400 ParseExpectedTokenType Did not find the expected token in
the SQL expression.
ParseExpected2TokenTypes 400 ParseExpected2TokenTypes Did not find the expected token in
the SQL expression.
ParseExpectedNumber 400 ParseExpectedNumber Did not find the expected number in
the SQL expression.

224 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
ParseExpectedRightParenBuiltinFunc 400 ParseExpectedRightParenBuiltinFunc Did not find the expected right
tionCall tionCall parenthesis character in the SQL
expression.
ParseExpectedTypeName 400 ParseExpectedTypeName Did not find the expected type name
in the SQL expression.
ParseExpectedWhenClause 400 ParseExpectedWhenClause Did not find the expected WHEN
clause in the SQL expression. CASE
is not supported.
ParseUnsupportedToken 400 ParseUnsupportedToken The SQL expression contains an
unsupported token.
ParseUnsupportedLiteralsGroupBy 400 ParseUnsupportedLiteralsGroupBy The SQL expression contains an
unsupported use of GROUP BY.
ParseExpectedMember 400 ParseExpectedMember The SQL expression contains an
unsupported use of MEMBER.
ParseUnsupportedSelect 400 ParseUnsupportedSelect The SQL expression contains an
unsupported use of SELECT.
ParseUnsupportedCase 400 ParseUnsupportedCase The SQL expression contains an
unsupported use of CASE.
ParseUnsupportedCaseClause 400 ParseUnsupportedCaseClause The SQL expression contains an
unsupported use of CASE.
ParseUnsupportedAlias 400 ParseUnsupportedAlias The SQL expression contains an
unsupported use of ALIAS.
ParseUnsupportedSyntax 400 ParseUnsupportedSyntax The SQL expression contains
unsupported syntax.
ParseUnknownOperator 400 ParseUnknownOperator The SQL expression contains an
invalid operator.
ParseInvalidPathComponent 400 ParseInvalidPathComponent The SQL expression contains an
invalid path component.
ParseMissingIdentAfterAt 400 ParseMissingIdentAfterAt Did not find the expected identifier
after the @ symbol in the SQL
expression.
ParseUnexpectedOperator 400 ParseUnexpectedOperator The SQL expression contains an
unexpected operator.
ParseUnexpectedTerm 400 ParseUnexpectedTerm The SQL expression contains an
unexpected term.
ParseUnexpectedToken 400 ParseUnexpectedToken The SQL expression contains an
unexpected token.
ParseUnExpectedKeyword 400 ParseUnExpectedKeyword The SQL expression contains an
unexpected keyword.
ParseExpectedExpression 400 ParseExpectedExpression Did not find the expected SQL
expression.
ParseExpectedLeftParenAfterCast 400 ParseExpectedLeftParenAfterCast Did not find the expected left
parenthesis after CAST in the SQL
expression.
ParseExpectedLeftParenValueConstr 400 ParseExpectedLeftParenValueConstr Did not find expected the left
uctor uctor parenthesis in the SQL expression.

Accessing data with IAM and S3 225

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
ParseExpectedLeftParenBuiltinFuncti 400 ParseExpectedLeftParenBuiltinFuncti Did not find the expected left
onCall onCall parenthesis in the SQL expression.
ParseExpectedArgumentDelimiter 400 ParseExpectedArgumentDelimiter Did not find the expected argument
delimiter in the SQL expression.
ParseCastArity 400 ParseCastArity The SQL expression CAST has
incorrect arity.
ParseInvalidTypeParam 400 ParseInvalidTypeParam The SQL expression contains an
invalid parameter value.
ParseEmptySelect 400 ParseEmptySelect The SQL expression contains an
empty SELECT.
ParseSelectMissingFrom 400 ParseSelectMissingFrom The SQL expression contains a
missing FROM after SELECT list.
ParseExpectedIdentForGroupName 400 ParseExpectedIdentForGroupName GROUP is not supported in the SQL
expression.
ParseExpectedIdentForAlias 400 ParseExpectedIdentForAlias Did not find the expected identifier
for the alias in the SQL expression.
ParseUnsupportedCallWithStar 400 ParseUnsupportedCallWithStar Only COUNT with (*) as a parameter
is supported in the SQL expression.
ParseNonUnaryAgregateFunctionCall 400 ParseNonUnaryAgregateFunctionCall Only one argument is supported
for aggregate functions in the SQL
expression.
ParseMalformedJoin 400 ParseMalformedJoin JOIN is not supported in the SQL
expression.
ParseExpectedIdentForAt 400 ParseExpectedIdentForAt Did not find the expected identifier
for AT name in the SQL expression.
ParseAsteriskIsNotAloneInSelectList 400 ParseAsteriskIsNotAloneInSelectList Other expressions are not allowed in
the SELECT list when '*' is used
without dot notation in the SQL
expression.
ParseCannotMixSqbAndWildcardInSel 400 ParseCannotMixSqbAndWildcardInSel Cannot mix [] and * in the same
ectList ectList expression in a SELECT list in SQL
expression.
ParseInvalidContextForWildcardInSel 400 ParseInvalidContextForWildcardInSel Invalid use of * in SELECT list in the
ectList ectList SQL expression.
ValueParseFailure 400 ValueParseFailure Timestamp parse failure in the SQL
expression.
IncorrectSqlFunctionArgumentType 400 IncorrectSqlFunctionArgumentType Incorrect type of arguments in
function call in the SQL expression.
AmbiguousFieldName 400 AmbiguousFieldName Field name matches to multiple fields
in the file. Check the SQL expression
and the file, and try again.
MissingHeaderName 400 MissingHeaderName Some headers in the query are
missing from the file.
IntegerOverflow 400 IntegerOverflow Integer overflow or underflow in the
SQL expression.

226 Accessing data with IAM and S3

Table 48. Error Codes (continued)
Error Code HTTP Generic Error Code Description Error
Status
Code
LikeInvalidInputs 400 LikeInvalidInputs Invalid argument given to the LIKE
clause in the SQL expression.
InvalidCast 400 InvalidCast Attempt to convert from one data
type to another using CAST failed in
the SQL expression.
ParquetNotEnabled 400 ParquetNotEnabled Functionality for parsing Parquet
format is not enabled.
ParquetParsingError 400 ParquetParsingError Error parsing Parquet file. Please
check the file and try again.
NumberFormatError 400 NumberFormatError Error parsing a number. This can
be caused by under/over flow of
integers.
EvaluatorLikePatternInvalidEscapeSe 400 EvaluatorLikePatternInvalidEscapeSe Invalid argument given to LIKE
quence quence expression.
EvaluatorNegativeLimit 400 EvaluatorNegativeLimit LIMIT must not be negative.
OverMaxParquetBlockSize 400 OverMaxParquetBlockSize Parquet file is above the max row
group size.
UnsupportedParquetType 400 UnsupportedParquetType Unsupported Parquet type.
ParquetUnsupportedCompressionCod 400 ParquetUnsupportedCompressionCod Unsupported Parquet compression
ec ec codec.
UnsupportedScanRangeInput 400 UnsupportedScanRangeInput Scan range queries are not supported
on this type of object.
ErrorWritingRow 400 ErrorWritingRow Cannot format output for your query.
Please check the file and query, and
try again
ReplicationConfigurationNotFoundErr 404 ReplicationConfigurationNotFoundErr The replication configuration was not
or or found.
ReplicationStatusNotFoundError 404 ReplicationStatusNotFoundError Detailed replication status not found.
S3SelectOptionNotYetImplemented 400 S3SelectOptionNotYetImplemented The option specified not yet
implemented.
RANGE_UPDATE_NOT_SUPPORT 400 UnsupportedFeature Range update is not supported in
current release
BucketNotificationMalformedArn 400 InvalidArgument The ARN is not well formed
BucketNotificationInvalidTopic 400 InvalidArgument Unable to validate the following
destination configurations
BucketNotificationIdMaxLengthExcee 400 InvalidArgument ID length exceeded allowed limit of
ded 255
BucketNotificationDuplicateId 400 InvalidArgument Same ID used for multiple
configurations. IDs must be unique.
BucketNotificationUnsupportedEvent 400 InvalidArgument The event is not supported for
notifications
BucketNotificationFilterPrefixLimitEx 400 InvalidArgument Cannot specify more than one prefix
ceeded rule in a filter.

NOTE:

Accessing data with IAM and S3 227

 ● The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited
to 2 KB in size. User-defined metadata is a set of key-value pairs. The size of user-defined metadata is measured by
taking the sum of the number of bytes in each key and value plus four: a colon and space to separate the name and
value and two bytes for carriage return-linefeed.
● When the system throws a 500 error, it allows the user to retry the request. In such cases, it is recommended to
use a backoff algorithm which waits progressively longer between retries for consecutive error responses. For more
information about guidance on 500 error rate response in ObjectScale, see https://dell.com/support/objectscale.

Authenticating with the S3 service

The ObjectScale S3 service enables authentication using Signature Version 4. This topic identifies any ObjectScale-specific
aspects of the authentication process.
Amazon S3 uses an authorization header that must be present in all requests to identify the user and provide a signature for the
request.
In order to create an authorization header, you need an AWS Access Key Id and a Secret Access Key. In ObjectScale, the AWS
Access Key Id is shown in the table located at Account > <USER_NAME> > Secret Key for each user. The Access Key ID is
listed in the table.
The following notes apply:
● When users add or change the secret key, they should wait two minutes for ObjectScale to refresh with the new secret key
before using the new secret key.

Authenticating using Signature V4

The Authorization header when using Signature V4 looks like this:

Authorization: AWS4-HMAC-SHA256
Credential=OKIA60819103813C1F40/20130524/us/s3/aws4_request,
SignedHeaders=host;range;x-amz-date,
Signature=fe5f80f77d5fa3beca038a248ff027d0445342fe2855ddc963176630326f1024

The Credential component comprises your Access Key Id followed by the Credential Scope. The Credential Scope comprises
Date/Region/Service Name/Termination String. For ObjectScale, the Service Name is always s3 and the Region can be any
string. When computing the signature, ObjectScale uses the Region string passed by the client.
Authentication using Signature V4 is described in:
● http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html , and
● http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
An example of a PUT replication request using Signature V4 is provided below:

PUT https://<objectscale>/testbucket?replication
Authorization: AWS4-HMAC-SHA256 Credential=OKIA60819103813C1F40/20160726/us/s3/
aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=e75a150daa28a2b2f7ca24f6fd0e161cb58648a25121d3108f0af5c9451b09ce
Content-MD5: x0ns_8TT8w5fB2woe72A==
Host: 10.247.195.130:9021
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20160726T033659Z

Response:

200 OK
Date: Tue, 26 Jan 2022 03:37:00 GMT
Server: ViPR/1.0
x-amz-request-id: 0af7c382:156123ab861:4192:896
x-amz-id-2: 3e2b2280876d444d6c7215091692fb43b87d6ad95b970f48911d635729a8f7ff

228 Accessing data with IAM and S3

Use SDKs to access the S3 service
When developing applications that talk to the ObjectScale S3 service, there are a number of SDKs that support your
development activity.
The following topics describe the use of the Amazon S3 SDK and the use of the ObjectScale Java S3 client SDK.
The ObjectScale Java S3 SDK is available at: https://github.com/EMCECS/objectscale-s3-client-java
The ObjectScale Community provides information about the various clients that are available and provides guidance on their use:
https://www.dell.com/community/ObjectScale/bd-p/ObjectScale.

Using the AWS SDK for Java

You can access ObjectScale object storage using the AWS SDK for Java.
By default the AmazonS3Client client object is coded to work directly against amazon.com. This section shows how to set up
the AmazonS3Client to work against ObjectScale.
In order to create an instance of the AmazonS3Client object, you need to pass it credentials. This is achieved through creating
an AWSCredentials object and passing it the AWS Access Key (your ObjectScale user name) and your generated secret key for
ObjectScale.
The following code snippet shows how to set this up.

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

By default the Amazon client attempts to contact Amazon WebServices. In order to override this behavior and contact
ObjectScale you need to set a specific endpoint.
You can set the endpoint using the setEndpoint method. The protocol specified on the endpoint dictates whether the client
should be directed at either be a load balancer in Kubernetes or a NodePort, which would be an IP and a random high-level port,
depending on your network configuration.
NOTE: If you intend to use the HTTPS port, the JDK of your application must be set up to validate the ObjectScale
certificate successfully; otherwise the client will throw SSL verification errors and fail to connect.
In the snippet below, the client is being used to access ObjectScale over HTTP:

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

client.setEndpoint("http://<objs.yourco.com>:<PORT>");

When using path-style addressing (objs1.dell.com/mybucket), you will need to set the setPathStyleAccess option, as shown
below:

S3ClientOptions options = new S3ClientOptions();

options.setPathStyleAccess(true);

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

client.setEndpoint("http://<objs.yourco.com>:<PORT>");
client.setS3ClientOptions(options);

The following code shows how to list objects in a bucket.

ObjectListing objects = client.listObjects("mybucket");

for (S3ObjectSummary summary : objects.getObjectSummaries()) {
System.out.println(summary.getKey()+ " "+summary.getOwner());
}

The CreateBucket operation differs from other operations in that it expects a region to be specified. Against S3 this would
indicate the data center in which the bucket should be created. However, ObjectScale does not support regions. For this reason,
when calling the CreateBucket operation, we specify the standard region, which stops the AWS client from downloading the
Amazon Region configuration file from Amazon CloudFront.

client.createBucket("mybucket", "Standard");

Accessing data with IAM and S3 229

The complete example for communicating with the ObjectScale S3 data service, creating a bucket, and then manipulating an
object is provided below:

public class Test {

public static String uid = "root";
public static String secret = "KHBkaH0Xd7YKF43ZPFbWMBT9OP0vIcFAMkD/9dwj";
public static String s3Endpoint = "http://<objs.yourco.com>:<PORT>";

public static String bucketName = "myBucket";

public static File objectFile = new File("/photos/cat1.jpg");

public static void main(String[] args) throws Exception {

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

S3ClientOptions options = new S3ClientOptions();

options.setPathStyleAccess(true);

AmazonS3Client client = new AmazonS3Client(credentials);

client.setEndpoint(s3Endpoint);
client.setS3ClientOptions(options);

client.createBucket(bucketName, "Standard");
listObjects(client);

client.putObject(bucketName, objectFile.getName(), objectFile);

listObjects(client);

client.copyObject(bucketName,objectFile.getName(),bucketName, "copy-" +
objectFile.getName());
listObjects(client);
}

public static void listObjects(AmazonS3Client client) {

ObjectListing objects = client.listObjects(bucketName);
for (S3ObjectSummary summary : objects.getObjectSummaries()) {
System.out.println(summary.getKey()+ " "+summary.getOwner());
}
}
}

ObjectScale Java S3 client SDK

The ObjectScale Java S3 client SDK is a library to assist users of the ObjectScale platform. This includes an API to interact with
ObjectScale's own API..
Requirements:
● Java 11 or higher
● ObjectScale 1.0.0 or higher.
An example of using this SDK (S3Client) is shown below for metadatasearch.

package main.java.metadatasearch;

import com.dellemc.objectscale.s3.ObjectScaleS3Client;
import com.dellemc.objectscale.s3.model.*;
import com.dellemc.objectscale.s3.ObjectScaleS3ClientBuilder;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.Bucket;
import software.amazon.awssdk.services.s3.model.ListBucketsRequest;
import software.amazon.awssdk.services.s3.model.ListBucketsResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;

import java.net.URI;

// This is an example of a program which creates 10 buckets with

// ObjectScale's metadata search enabled, and then queries all buckets in a store

230 Accessing data with IAM and S3

// for all objects created after Jan 1 2015.
public abstract class SearchMetadata {

// You can adjust these according to your setup.

static final String S3_IP = "127.0.0.1";
static final String S3_PORT = "80";
static final String ACCESS_KEY = "OKIA----------------";
static final String SECRET = "----------------------------------------";
static final String BUCKET = "bucket-metadata-search-example";

static ObjectScaleS3Client client;

// This is an example of how one can create buckets with ObjectScale's metadata
search feature
// enable, get a list of all buckets, and search for metadata using the query
objects endpoint
// with selectors.
public static void main( String[] args ) {
ObjectScaleS3ClientBuilder builder = ObjectScaleS3Client.builder()
.endpointOverride(URI.create("http://"+S3_IP+":"+S3_PORT))
.credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials
.create(ACCESS_KEY, SECRET)));
client = builder.build();

// Create a set of buckets

for ( int i = 1; i <= 10; i++ ) {
createTestBuckets(client, BUCKET+"-"+i);
}

// Get a list of the current buckets

ListBucketsRequest listRequest = ListBucketsRequest.builder().build();
ListBucketsResponse listResponse = client.listBuckets(listRequest);

for ( Bucket b : listResponse.buckets() ){

System.out.println("===== Bucket " + b.name() + " =====");
try {
// Query the objects in the bucket for anything modified after january
1st, 2015.
QueryObjectsRequest qo =
QueryObjectsRequest.builder().bucket(b.name()).query("LastModified>2015-01-01T00:00:00Z")
.build();
QueryObjectsResponse resp = client.queryObjects(qo);
// For every Object...
for ( QueryObject o : resp.objects() ) {
// For every queried metadata set
for ( QueryMetadata m : o.queryMetadata() ) {
// For every key in the metadata map
for ( String s : m.metadataMap().keySet() ) {
// Print out the info.
System.out.println(o.objectName() + ": " + m.typeAsString()
+ ": " + s + " " + m.metadataMap().get(s));
}
}
}
} catch ( S3Exception e ){
if( e.getLocalizedMessage().startsWith("Invalid search index key name") )
{}
System.out.println("metadata search not enabled on this bucket, or key
not searchable");
}
}
}

// Create a bucket with a given name and client where one can query / filter based
// on the LastModified field.
public static void createTestBuckets( S3Client client, String name ){
CreateBucketRequest createBucketRequest = CreateBucketRequest.builder()
.metadataSearchKeys("LastModified;datetime").bucket(name).build();
// Use toStandardRequest to use this as a CreateBucketRequest
client.createBucket(createBucketRequest.toStandardRequest());
}
}

Accessing data with IAM and S3 231

Working with S3 workloads in ObjectScale
After setting up accounts and users as well as an object store and bucket, you can perform S3 workloads using the ObjectScale
instance object storage.

Record S3 endpoint values

About this task
Use this table to record the values used to create the bucket. These three S3 values, which you will collect duing this task, are
required for application access to the endpoint.

Object store name

Bucket name

ObjectScale Name S3 Browser Name Your Value

ACCESS KEY Access Key ID
SECRET_KEY Secret Access Key
EXTERNAL_ENDPOINT REST Endpoint

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Collect the S3 endpoint value from the object store Summary tab.
Record this value in the EXTERNAL_ENDPOINT value in the table above.
4. Locate the Secret Key .csv file previously saved locally for the user that owns the bucket in the object store.
This user should be a part of the IAM account that is a tenant within the selected object store.
Record this value as the SECRET_KEY in the table above.
5. Finally, collect the Access Key ID for the user.
a. Go to the object store Accounts tab and click on the name of the IAM account that manages the user account.
b. Select the Users tab and click on the name of the user account that will be used for S3.
c. Click on Secret Key and record the Access Key ID value displayed in the Secret Key table.
NOTE: If you do not have the previously created Secret Key for this user or wish to change it for any reason, you
can DEACTIVATE or REMOVE old Secret Keys/Access Key pairs and click ADD KEY to generate a new key for the
user.

View the certificates for an object store

Using the ObjectScale Portal user interface, you can view the certificate details for an existing object store.

About this task

To view the properties of an object store certificate:

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
The Summary of the selected object store appears.

232 Accessing data with IAM and S3

4. Click the Certificates tab.
The Certificates tab appears and consists of S3, Management, and Replication Reciever sections.
Each section of the Certificates tab shows details on the certificate for each of the object store services, including
certificate issuer, signing, and expiration details.

Perform an S3 workload using the S3 Browser

About this task
Use the freely available S3 Browser or similar application to verify connectivity to the object store.

Steps
1. In S3 browser, create an account with the details of the object store bucket.
Use the S3 values you recorded in the last task to complete this step.
a. Type a Name for the account.
b. Select S3 Compatible Storage from the Account Type dropdown.
c. Enter the EXTERNAL_ENDPOINT value into the REST Endpoint field.
For example:
Rest Endpoint: 10.55.66.77:443

d. Enter the ACCESS KEY value into the Access Key ID field.
For example:
Access Key: AKIA5F587FA0E4E4FF81

e. Enter the SECRET_KEY value into the Secret Access Key field.
For example:
Secret Key: KqM5xHvaG7Bv9SH0lGoMrGYDWJrUoZsVvZ71JBeY

2. Connectivity is verified by creating new buckets and uploading objects via S3 Browser or a similar S3 Compatible application.
3. Optional: Create additional buckets using S3 browser or within the object store.
4. Use the S3 Browser to place one or move objects in this new bucket.
You should see the object(s) uploaded in the within the bucket.

Accessing data with IAM and S3 233

 14
Alerts
Monitoring alerts and logs provides information about the monitoring messages in the ObjectScale Portal user interface.
Topics:
• About ObjectScale instance event and issue monitoring
• Monitoring Events, Audits, and Alerts

About ObjectScale instance event and issue

monitoring
Throughout ObjectScale are processes that are constantly monitoring and collecting information about the ObjectScale instance
and object stores. When the status of a component or operation changes, the change is captured and noted in the following
places in ObjectScale:
● Alerts
● Logs
● Administration > ObjectScale > Object Stores > <OBJECT_STORE_NAME> > Health
ObjectScale alert and log data are available to be filtered for the last 24 hours, last seven days, last one month, or all.
NOTE: Logs with the description pods are not scheduled due to insufficient resources may appear in
the Health tab even though all pods are up and running. The Kubernetes FailedScheduling event gets generated during
successful object store creations as well sometimes as the creation is in progress. These issues can be ignored and will get
immediately cleared after the pod is scheduled.

About alert severities

Alert and log event message severity labels have the following meanings:
● Critical: Messages about conditions that require immediate attention.
● Error: Messages about error conditions that report either a physical failure or a software failure.
● Warning: Messages about less than optimal conditions.
● Normal: Routine status messages.
● Audit: Audit messages for events only.

View ObjectScale health issues and events

The ObjectScale Portal user interface shows health summaries and details.
The Dashboard section of the Portal shows a Health summary.
To review health issues, select the Alerts section of the Portal. This section shows the full list of current alerts describing
potential issues with the ObjectScale system. You can filter the alerts using the dropdown above the table. You can filter issues
from the last 24 hours, last 7 days, last month, or all.
You can also filter each individual column using the filter icon.
● Select an issue listed in the table.
● If the issue type is Manual, use Acknowledged or Unacknowledged to modify an issue once reviewed.
To review logged health, select Logs to display the full list of current logged events. You can filter the logs listed using the
dropdown above the table. You can filter events from the last 24 hours, last 7 days, last month, or all.

234 Alerts
View the health of an object store
The ObjectScale Portal user interface shows health and alerts for an object store.
To view health for a specific object store in the ObjectScale Portal user interface, go to Administration > ObjectScale >
Object Stores > <OBJECT_STORE_NAME> > Health.
The Health page displays the full list of current health alerts and health events for the selected object store.
The Alerts tab displays issues for the selected object store. There are two categories of health alerts: Auto or Manual.
● Auto alerts are generated within the product when a component does not behave as expected. These alerts are cleared
automatically when the problem is resolved.
● Manual alerts are not cleared until a user acknowledges them. You can use the ACKNOWLEDGE or UNACKNOWLEDGE
buttons to manage manual health issues.
The health Logs tab shows the full list of current logged events.
The Health Check tab allows you to perform health checks on the object store.
● To perform a check on the health of an object store, select healthcheck and click Check Health.
● On ObjectScale for Red Hat OpenShift only, you can perform a preupdate health check of the object store before updating
the object store. Select pre-update and click Check Health.
● To perform a health check of the object store following an upgrade, select Check Health.

Hardware Alerts
Hardware alerts for ObjectScale appliance on Dell server assets support proactive or reactive engagement by customers or
services to resolve issues.
Hardware alerts help in monitoring the health and performance of the system, and in notifying hardware failures in a timely
manner.
Some of the considerations related to hardware alerts are:
● Hardware alerts are enabled by default.
● Customers can enable or disable hardware alerts from the UI.
● The customer can see hardware alerts on the ObjectScale UI Portal within 60 s from the event.
● Alerts have a detailed description of the issue with:
○ Severity type
○ Symptom code
○ Reason
○ Impacted resource
○ Timestamp of the issue

View ObjectScale Hardware Alerts

Use the ObjectScale Portal user interface to view hardware alerts.

About this task

The Alerts section within the ObjectScale Portal user interface displays the hardware alerts.

Steps
1. From the ObjectScale portal user interface, click Alerts.
By default, alerts with Normal severity are hidden.
2. Click Show All to view all alerts.
All alerts, including alerts that are hidden, are displayed. The following descriptions are shown for each alert.
● Severity
● Message
● ResourceID
● SymptomID
● Reason

Alerts 235
 ● Component
You can filter alerts using the drop-down with the following options:
● Last 24 hours
● Last 7 days
● Last 30 days
● Show All

Enable ObjectScale Hardware Alerts

Use the ObjectScale Portal user interface to enable hardware alerts.

About this task

The Event Settings section in the ObjectScale Portal user interface allows you to change the hardware alert settings for the
cluster.
NOTE: Hardware alerts are available only on an ObjectScale appliance, and they are enabled by default.

Steps
1. From the ObjectScale portal user interface, click Event Settings and go to the Hardware Alerts tab.
The status of alerts is displayed with the available nodes.
2. Select a node with Disabled status, and click Enable to enable alerts for that node.
Alerts are enabled for that specific node.
3. Click Enable All to enable alerts on all available nodes.
Alerts are enabled for all available nodes.

Disable ObjectScale Hardware Alerts

Use the ObjectScale Portal user interface to disable hardware alerts.

About this task

The Event Settings section in the ObjectScale Portal user interface allows changing the hardware alert settings for the cluster.

Steps
1. From the ObjectScale portal user interface, click Event Settings and go to the Hardware Alerts tab.
The status of alerts is displayed with the available nodes.
2. Select a node with Enabled status, and click Disable to disable alerts for that node.
Alerts are disabled for that specific node.
3. Click Disable All to disable alerts on all available nodes.
Alerts are disabled for all available nodes.

Acknowledge ObjectScale Hardware Alerts

Use the ObjectScale Portal user interface to acknowledge or clear hardware alerts.

About this task

The Alerts section within the ObjectScale Portal user interface displays the hardware alerts.

Steps
1. From the ObjectScale portal user interface, click Alerts
Alerts are displayed with regular alerts hidden.
2. Click Show All to view all alerts.
3. Click the box to the left of the alert that you want to acknowledge, and click Acknowledge.
When you acknowledge the alert, it clears the alert from the Alerts pane.

236 Alerts
 A message appears asking you to confirm Acknowledge Alert.
4. Click Yes if you want to acknowledge and clear the alert from the Alerts pane. Click No if you do not want to acknowledge
and clear the alert from the Alerts pane.
If you clicked Yes, a message appears in the upper right indicating that the Alert or alerts were acknowledged successfully.
5. To view acknowledged alerts, click Show All in the upper left of the Alerts pane and select the wanted time period.
Alerts do not display once the selected time period is exceeded. For example, alerts created 25 hours ago display in the Last
7 Days filter, not in the Last 24 Hours filter.
You must manually acknowledge alerts to change the severity level of them from Critical to Normal.

Configure ObjectScale Hardware Alerts Using SNMPv2

Use the ObjectScale Portal user interface to configure hardware alerts using SNMP.

About this task

Within the ObjectScale Portal user interface, configure Hardware Alerts to an external source using Simple Network
Management Protocol (SNMP).

Steps
1. From the ObjectScale portal user interface, click Event Settings
SNMP alerts page is displayed with the current configuration details.
2. Select the checkbox for configuration, and click Edit.
A dialogue box is displayed with more configuration options.
3. In the Edit SNMP Server window, complete the required fields and click Save

Option Description
FQDN IP Enter the IP name.
Port Enter the Port name.
Version Choose SNMPv2.
Community A community string that identifies a collection of SNMP Managers and agents. Choose between public or
private.

The SNMP server is added successfully.

4. Click Send a Test Trap to test the configuration.
A message is displayed indicating a successful test trap request, and the SNMP Sever status is changed to Success.

Configure ObjectScale Hardware Alerts Using SNMPv3

Use the ObjectScale Portal user interface to configure hardware alerts using SNMP.

About this task

Within the ObjectScale Portal user interface, configure Hardware Alerts to an external source using SNMP (Simple Network
Management Protocol).

Steps
1. From the ObjectScale portal user interface, click Event Settings
SNMP alerts page is displayed with the current configuration details.
2. Select the checkbox for configuration, and click Edit.
A dialogue box is displayed with more configuration options.
3. In the Edit SNMP Server window, complete the required fields and click Save

Option Description
FQDN IP Enter the IP name.

Alerts 237
 Option Description
Port Enter the Port name.
Version Choose SNMPv3.
Security Choose whether to include Authentication and Privacy in the communication:
● NONE
● AUTH ONLY
● AUTH and PRIVACY
Engine ID This allows communication between known SNMP entries in the administrative domain, and should be
configured on the corresponding NMS for access to incoming SNMP traps.
Auth Protocol Choose between MD5 (128 bit) or SHA-1 (160 bit).
Auth Password Enter the password for Auth Protocol.
Privacy Protocol Choose between DES (56 bit) or AES (128, 192 or 256 bit) to encrypt all SNMPv3 transmissions.

The SNMP server is added successfully.

4. Click Send a Test Trap to test the configuration.
A message is displayed indicating a successful test trap request, and the SNMP Sever status is changed to Success.

Monitoring Events, Audits, and Alerts

CSI-01
Name DiskHealthIssue
Description Disk Health Issue.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Refer to www.dell.com/support/objectscale disk replacement procedure in the ObjectScale
Administrators Guide to perform disk replacement.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article

CSI-03
Name DiskMissing
Description Disk Missing Issue.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article

CSI-05
Name VolumeFakeAttach
Description Volume Fake Attach Issue.
Issue Category Auto
Notifiers objectscale-snmp-notifier

238 Alerts
Remedies ● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article

DECKS-HC-1000
Name Pre-Update
Description Preupdate health check for application.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies --

DECKS-LIC-1002
Name ExpiringLicense
Description License is expiring or expired.
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies --

DECKS-LIC-1005
Name ExpiringLicense
Description License is expiring or expired.
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check the end date of the ObjectScale license.

● Go to the Dell Software Licensing Center (SLC) to renew or extend the ObjectScale license.
● Contact your Dell sales representative to renew or extend the ObjectScale license.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

DECKS-LIC-1006
Name ExpiringLicense
Description License is expiring or expired.
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check the end date of the ObjectScale license.

● Go to the Dell Software Licensing Center (SLC) to renew or extend the ObjectScale license.
● Contact your Dell sales representative to renew or extend the ObjectScale license.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

Alerts 239
DECKS-LIC-1008
Name InvalidLicense
Description License is invalid.
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Verify that the ObjectScale license is obtained from the Dell Software Licensing Center.
● Verify that the ObjectScale license is not modified before applying it to the cluster.
● Verify that the PRODUCTSHORTNAME is defined in the ObjectScale license.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

DECKS-LIC-1011
Name -
Description License features are no longer tracked.
Issue Category -
Notifiers -
Remedies ● Verify that the ObjectScale license is correct and the feature was intended to be removed.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

DECKS-SA-1023
Name SupportAssistConfiguration
Description SupportAssist configuration issues.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Verify that the supportassist-objectscale-0 pod is running.
● Verify that SupportAssist is enabled.
● Verify connectivity of configured gateways.
● Verify that a valid AccessKey and PIN are used.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

DECKS-SA-1024
Name ESECallBackTransactions
Description SupportAssist ESE callback transaction issues.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Verify the supportassist-objectscale-0 pod is Running
● Verify connectivity of configured gateways.
● Check network connectivity of the k8s cluster

240 Alerts
 ● Check the log of the supportassist-objectscale-0 pod
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

KAHM-HC-1000
Name Pre-Update
Description Pre-Update health check for application.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Check the health status in the <component>-app-configmap to find which checks failed.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-CO-0000
Name COOperatorEnterMM
Description Node EnterMaintenanceMode cluster operation handling by ClusterOperation Operator.
Issue Category Auto, 1440
Notifiers objectscale-snmp-notifier, objectscale-supportassist-ese
Remedies ● Check EnterMM cluster operation CR in the K8s and CO operator logs for details.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSC-CO-0001
Name COOperatorExitMM
Description Node ExitMaintenanceMode cluster operation handling by ClusterOperation Operator.
Issue Category Auto, 1440
Notifiers objectscale-snmp-notifier, objectscale-supportassist-ese
Remedies ● Check ExitMM cluster operation CR in the K8s and CO operator logs for details.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSC-CO-0002
Name COOperatorNodeAddition
Description NodeAddition cluster operation handling by ClusterOperation Operator.
Issue Category Auto, 1440
Notifiers objectscale-snmp-notifier, objectscale-supportassist-ese
Remedies ● Check NodeAddition cluster operation CR in the K8s and CO operator logs for details.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

Alerts 241
OBJSC-CO-0003
Name COOperatorNodeRemoval
Description NodeRemoval cluster operation handling by ClusterOperation Operator.
Issue Category Auto, 1440
Notifiers objectscale-snmp-notifier, objectscale-supportassist-ese
Remedies ● Check NodeRemoval cluster operation CR in the K8s and CO operator logs for details.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSC-CO-0004
Name COOperatorDiskRemoval
Description DiskRemoval cluster operation handling by ClusterOperation Operator.
Issue Category Auto, 1440
Notifiers objectscale-snmp-notifier, objectscale-supportassist-ese
Remedies ● Check DiskRemoval cluster operation CR in the K8s and CO operator logs for details.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

NVMF-1389
Name NVME_BAD_MEMORY_ERROR
Description No memory to allocate to buffer for nvmfengine.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check the memory usage via "top" command in the reported nvmfengine pod.
● Check the logs of reported nvmfengine pod for details.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

NVMF-1390
Name NVME_DEVICE_INIT_FAILED_ERROR
Description Nvme device init failed in nvmfengine

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check the logs of reported nvmfengine pod for details.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

NVMF-1393
Name NVME_PRIVATE_IP_UNAVAILABLE_ERROR

242 Alerts
Description Private network is unavailable in nvmfengine.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check private network and macvlan log.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

NVMF-1395
Name NVME_BIND_FAILED_ERROR
Description Restserver in nvmfengine failed to bind the port

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check macvlan configuration via "kubectl get network-attachment-definitions.k8s.cni.cncf.io macvlan-
conf-1 -n cmo -o yaml".
● Check the logs of reported nvmfengine pod for detail errno.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

NVMF-1396
Name NVMe Drive Removed
Description NVMe drive is removed.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese
Remedies ● Check drive resource status using kubectl get drive -o yaml to identify the removed drive
with the annotation remove-timestamp.
● Check /var/log/messages log for udev entries on the Kubernetes node containing the removed drive
to determine the root cause.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2000
Name NodesReadiness
Description Upgrade precheck for node readiness.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get nodes -A and ensure that the number of nodes is greater than 4.
● Verify if the total number of nodes is less than equal to 9, then the number of ready nodes is greater
than or equal to totalNodes-1.
● Verify if the total nodes are greater than 9, then the number of ready nodes is greater than or equal to
4*TotalNodes/5.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

Alerts 243
OBJPRECHK-2001
Name DeploymentAndStatefulsetsReadiness
Description Upgrade precheck for deployment and stateful sets readiness.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get deployments -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas, that is, check the READY status.
● Run kubectl get statefulsets -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas, that is, check the READY status.
● Verify all the containers of all the replicas are in the Running state.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2002
Name ServicePodReadiness
Description Upgrade precheck for service pod readiness.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get pods -A grep objectscale-manager-service-pod and verify that the
pod is in the Running state.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2003
Name ObjectStoresReadiness
Description Upgrade precheck for object stores' readiness.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2004
Name ObjectStoreMicroServicesReadiness
Description Upgrade precheck for object store microservices readiness.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● Run kubectl get pods -n <objectstore-namespace> and verify all the pods are in the
Running state.

244 Alerts
 ● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2005
Name ObjectStorePVReplacingAndMaintenancePhaseStatus
Description Upgrade precheck for object store PV phase status.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2006
Name ObjectStoreRecoveryCheck
Description Upgrade precheck for object store recovery phase status.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2007
Name ObjectStorePDBStatus
Description Upgrade precheck for object store PDB status.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● Run kubectl get pdb -A. For each of the PDBs, run kubectl describe pdb <pdb-name>
-n <namespace>, and ensure that Current is greater than or equal to Desired.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2008
Name ProductCompatibilityStatus
Description Upgrade precheck for product compatibility status.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● View the objectscale-lcm-manifest.json that is uploaded at http-share and ensure that the
current version of the product is a part of the product compatibility matrix in the manifest file.

Alerts 245
 ● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2009
Name OSCompatibilityStatus
Description Upgrade precheck for OS compatibility status.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● View the objectscale-lcm-manifest.json uploaded at http-share and ensure that the current
OS version is a part of the OS compatibility matrix in the manifest file.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPRECHK-2010
Name UpgradePrerequisiteStatus
Description Upgrade precheck for prerequisite status.

Issue Category Manual

Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl describe Upgrade <cr-name> -n <objectscale namespace> and verify
it bundlePathBaseDirectory is not empty.
● View the objectscale-lcm-manifest.json platform.tgz, and lcm_manifest.json files
are uploaded at http-share.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPSTCHK-3000
Name NodesReadinessPostCheck
Description Upgrade postcheck for nodes' readiness.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get nodes -A and ensure that the number of nodes in greater than 4.
● Verify if the total number of nodes is less than equal to 9, then the number of ready nodes is greater
than or equal to totalNodes-1.
● Verify if the total nodes are greater than 9, then the number of ready nodes is greater than or equal to
4*TotalNodes/5
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPSTCHK-3001
Name DeploymentAndStatefulsetsReadinessPostCheck
Description Upgrade postcheck for deployment and statefulsets readiness.
Issue Category Manual

246 Alerts
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get deployments -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas, that is, check the READY status.
● Run kubectl get statefulsets -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas, that is, check the READY status.
● Verify that all the containers of all the replicas are in the Running state.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPSTCHK-3002
Name ServicePodReadinessPostCheck
Description Upgrade postcheck for service pod readiness.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get pods -A grep objectscale-manager-service-pod and verify that the
pod is in the Running state.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJPSTCHK-3003
Name ObjectStoreReadinessPostCheck
Description Upgrade postcheck for object store readiness.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Status Phase is Available for all.
● For additional information about this event, go to "http://www.dell.com/support/objectscale" and use
the SymptomID to search support for the knowledge base article.

OBJSC-FED-1001
Name Remote instance connection status changed
Description Remote instance connection status changed.

Issue Category Auto

Notifiers objectscale-snmp-notifier
Remedies ● If connection status is offline, check the corresponding ObjectScale instance is reachable
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

OBJSC-IAM-1004
Name IAM Account Entity Limit Reached
Description IAM account entity limit reached.

Issue Category Auto

Alerts 247
Notifiers objectscale-snmp-notifier
Remedies ● IAM Account has reached its MAX allocated limit for the IAM entities.
● In order to add new entities, some existing entities needs to be deleted.

OBJSC-LIC-0004
Name ObjectScale Licensing
Description ObjectScale cumulative object store usage.
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● If near or above capacity, remove stale object stores.

● If near or above capacity, contact Dell Technologies for an updated license.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MGR-3000
Name Update
Description Update for application.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Check the health status in the <component>-app-configmap to find which checks failed.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MGR-HC-1000
Name Pre-Update
Description Pre-Update health check for application.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Check the health status in the <component>-app-configmap to find which checks failed.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-1111
Name Objectscale Capacity
Description Percent of used ObjectScale capacity crosses threshold.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale
maximum capacity usage.

248 Alerts
 ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-1112
Name Objectscale Capacity
Description Percent of used ObjectScale capacity crosses threshold.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale
maximum capacity usage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-1113
Name Objectscale Capacity
Description Percent of used ObjectScale capacity crosses threshold.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale
maximum capacity usage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-3002
Name Directory Table failure
Description Directory Table failure detected
Issue Category Auto, 10
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Contact DellEMC Support for additional information

● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-3003
Name Directory Table failure
Description Directory Table failure detected
Issue Category Auto, 10
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check if user application is fully available

● Contact DellEMC Support for additional information

Alerts 249
 ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-4019
Name Objectscale Monitoring Health
Description No data is moved to the ObjectScale monitoring framework for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check monitoring components health (telegraf, influxdb, fluxd)

● For more information about this event, go to https://www.dell.com/support/kbdoc/en-us/
000195833 and use the SymptomID to search for the knowledge base article.

OBJSC-MON-4020
Name MonitoringFluxd
Description Fluxd has not responded for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Fluxd service status.

● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-4021
Name InfluxDB PVC
Description InfluxDB PVC has a bad state for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check InfluxDB PVC status and fix it.

● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-4022
Name Rsyslog PVC
Description Rsyslog PVC has a bad state for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

250 Alerts
Remedies ● Check Rsyslog PVC status and fix it.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-4025
Name InfluxDB low disk space
Description Percent of used InfluxDB capacity crosses threshold.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Based on capacity usage, InfluxDB may be in read-only mode. Verify InfluxDB capacity usage and
take required actions to free up or increase capacity.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-MON-4028
Name Rsyslog low disk space
Description Percent of used Rsyslog capacity crosses threshold.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Verify Rsyslog capacity usage and take required actions to free up capacity.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-SP-0000
Name RecoverySP
Description Recovery after removing recoverable pod during any SP.
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check recovery service procedure CR in the K8s and SP operator logs for details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-SP-0001
Name DiskReplacementError
Description Recovery after removing recoverable pod during any SP.
Issue Category Auto
Notifiers objectscale-supportassist-ese

Alerts 251
 objectscale-snmp-notifier

Remedies ● Check disk replacement service procedure CR in the K8s and SP operator logs for details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-SP-0002
Name PMMError
Description Permanent Maintenance Mode service procedure handling by SP Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check PMM service procedure CR in the K8s and SP operator logs for details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-SP-0003
Name TMMError
Description Temporary Maintenance Mode service procedure handling by SP Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check TMM service procedure CR in the K8s and SP operator logs for details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-SP-0004
Name NFError
Description Node failure service procedure handling by SP Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Node Failure service procedure CR in the K8s and SP operator logs for details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSC-TARGET-01
Name NvmfTargetConfigIssue
Description ObjectScale NVMF Target Configuration Failure
Issue Category Auto
Notifiers objectscale-snmp-notifier

252 Alerts
Remedies ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1000
Name OperatorDR
Description Disk Replacement service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1001
Name OperatorPMM
Description Permanent Maintenance Mode service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1002
Name OperatorTMM
Description Temporary Maintenance Mode service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1003
Name OperatorUpgrade
Description Upgrade service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details

Alerts 253
 ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1004
Name OperatorHorizontalExpand
Description Horizontal Expand SS service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1005
Name OperatorVerticalExpand
Description Vertical Expand SS service procedure handling by Operator
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Check Object Store Status, Operator logs and platform logs (if applicable) for details
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-1006
Name OperatorObjectStoreCreation
Description Object Store Creation
Issue Category Auto
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Please check Object Store Status, Object Scale Operator logs and platform logs (if applicable) for
details
● Please check Object Store pods that remain in Pending state
● Please try to eliminate errors that block pods scheduling
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-2001
Name CSRRequestAndApprovals
Description ObjectScale CSR issue and approval notifications
Issue Category Auto

254 Alerts
Notifiers objectscale-snmp-notifier

Remedies ● Approve the certificate signing request if the CSR is pending

● Run the command "kubectl get csr" and it will show which CSR(s) are pending
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSOP-2002
Name TLSCertExpire
Description ObjectScale TLS certificate about to expire notification or expired notifications
Issue Category Auto
Notifiers objectscale-snmp-notifier

Remedies ● Check the expiration dates of the certificates

● Renew the certificate(s) before they expire
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1006
Name BUCKET_HARD_QUOTA_EXCEEDED
Description Hard quota on total object count or size that is exceeded for one bucket.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Increase hard count or size quota for this bucket or delete objects in this bucket.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1008
Name BUCKET_SOFT_QUOTA_EXCEEDED
Description Soft quota on total object count or size that is exceeded for one bucket.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Increase soft count or size quota for this bucket or delete objects in this bucket.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-12001
Name REPLICATION_DESTINATION_PAUSED_WITH_BACKLOG
Description ObjectScale Replication is paused and there are pending objects waiting for replication.
Issue Category Auto, 120
Notifiers objectscale-snmp-notifier
Remedies ● Confirm if replication paused setting for reported destination is still required.

Alerts 255
 ● Resume replication if pause is no longer required.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-12003
Name REPLICATION_DESTINATION_REMOVED_FROM_SYSTEM
Description A destination object store for ObjectScale Replication has been removed from federation.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Remove ObjectScale Replication configuration targeting reported destination object store.
● Suspend ObjectScale Replication to reported destination object store.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-12004
Name REPLICATION_DESTINATION_BUCKET_QUOTA_EXCEEDED
Description Destination bucket exceeds user-configured quota.
Issue Category Auto, 30
Notifiers objectscale-snmp-notifier
Remedies ● Modify the destination bucket quota size.
● Clean up some of the data in destination bucket to free up space for replication to continue.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-12005
Name REPLICATION_DESTINATION_OBJECT_STORE_OUT_OF_CAPACITY
Description Destination object store is out of capacity.
Issue Category Auto, 30
Notifiers objectscale-snmp-notifier
Remedies ● Add more space to the destination object store.
● Clean up some of the data in destination object store to free up space for replication to continue.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-12006
Name REPLICATION_CERTIFICATE_ERROR
Description Connection to remote replication endpoint cannot be established due to tls problem
Issue Category Auto, 10
Notifiers objectscale-snmp-notifier
Remedies ● Verify if internal certificate and CA used for geo connection has expired.
● Contact Dell EMC technical support for assistance

256 Alerts
 ● For additional information on this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-12007
Name REPLICATION_EVENT_UNABLE_TO_HANDLE
Description Unable to Handle ObjectScale Replication Event
Issue Category Manual
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● Contact Dell EMC technical support for assistance

● For additional information on this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-12008
Name REPLICATION_DESTINATION_OBJECT_STORE_VERSION_OUTDATED
Description One of the following:
● Large object replication blocked due to destination object store version not compatible.
● Object version or delete marker replication blocked due to destination object store version not
compatible.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Check and upgrade the destination object store version to an equal or higher version than the source
object store.
● For additional information on this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-12010
Name REPLICATION_FAILURE_DETAIL
Description Details of an object scale replication failure, including:
● Object name
● Object version
● Failure reason
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Use information in the message to determine specific failure reason.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-12011
Name REPLICATION_OTHER_FAILURE_DETAIL
Description Unexpected failure happened in ObjectScale replication.
Issue Category Manual

Alerts 257
Notifiers objectscale-snmp-notifier
Remedies ● Contact Dell Technology for support.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-13000
Name STORAGE_TIER_UNAVAILABLE
Description Storage tier is not available or degraded.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Check if all the nodes are in maintenance mode.
● Check if one or more of the nodes have been powered off.
● Check if one or more pods are not in the Running state.
● Verify that the disks attached to each of the nodes are all mounted.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13001
Name FAULT_DOMAIN_UNAVAILABLE
Description Fault domain is not available or degraded.
Issue Category Auto, 3
Notifiers objectscale-snmp-notifier
Remedies ● Check if all the nodes are in maintenance mode.
● Check if one or more of the nodes have been powered off.
● Check if one or more pods are not in the Running state.
● Verify that the disks attached to each of the nodes are all mounted.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13002
Name PRAVEGA_THRESHOLD_EXCEEDED
Description Pravega capacity threshold exceeded.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata
or data values are big due to possible delays in space reclamation.
● Verify if used capacity has exceeded defined thresholds.
● Verify if any hardware (nodes or disks) is down that could be causing less usable storage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13003
Name CAPACITY_USAGE_FOR_OBJECT_STORE

258 Alerts
Description Capacity usage for object store.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata
or data values are big due to possible delays in space reclamation.
● Verify if used capacity has exceeded defined thresholds.
● Verify if any hardware (nodes, disks) is down that could be causing less usable storage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13004
Name CAPACITY_USAGE_FOR_OBJECT_STORE_ERROR
Description Capacity usage for object store error.
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata
or data values are big due to possible delays in space reclamation.
● Verify if used capacity has exceeded defined thresholds.
● Verify if any hardware (nodes or disks) is down that could be causing less usable storage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13005
Name CAPACITY_USAGE_FOR_OBJECT_STORE
Description Capacity usage for object store.
Issue Category Auto, 60
Notifiers objectscale-snmp-notifier
Remedies ● Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata
or data values are big due to possible delays in space reclamation.
● Verify if used capacity has exceeded defined thresholds.
● Verify if any hardware (nodes or disks) is down that could be causing less usable storage.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-13006
Name CAPACITY_USAGE_FOR_OBJECT_STORE_ERROR
Description Capacity usage for object store error.
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies -

Alerts 259
OBJST-13007
Name STORAGE_TIER_UNAVAILABLE
Description Storage tier is degraded or possibly unavailable
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies -

OBJST-13008
Name FAULT_DOMAIN_UNAVAILABLE
Description Fault domain is degraded or possibly unavailable
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies -

OBJST-13009
Name PRAVEGA_CAPACITY_THRESHOLD_EXCEEDED
Description Pravega capacity threshold exceeded.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-13010
Name CHUNKSTREAM_CAPACITY_THRESHOLD_EXCEEDED
Description Stream chunk used space reached 3TB, exceeding the threshold. This means the Tier1 storage GC is not
working well, it will eventually cause the disks space exhaustion.
Issue Category Auto, 3
Notifiers objectscale-snmp-notifier
Remedies ● Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata
or data values are big due to possible delays in space reclamation.
● Verify used capacity has exceeded defined thresholds.
● Verify if any hardware (nodes/disks) are down that could be causing less usable storage.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

OBJST-13011
Name CHUNKSTREAM_CAPACITY_THRESHOLD_EXCEEDED
Description Stream chunk capacity threshold exceeded.
Issue Category Auto

260 Alerts
Notifiers objectscale-snmp-notifier
Remedies ● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

OBJSTEPUPD-4000
Name GetManifest
Description Step: Get Manifest
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Verify the objectscale-lcm-manifest.json file has been uploaded at http-share using
kubectl exec -it <http-share-pod-name> -n <namespace> - /bin/bash and going
to the bundle location.
● Verify the manifest file is not corrupted.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledgebase article.

OBJSTEPUPD-4001
Name Precheck
Description Step: Precheck
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run the pre-checks before upgrading the product by clicking PreUpgrade on the upgrade screen.
● Check the report of the pre-checks on the screen and do the remedies mentioned for the failed
pre-checks.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD-4002
Name BaseAppsUpgrade
Description Step: Base Apps Upgrade
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● If the pre-requisite step is failed, then the images and the charts are not pushed successfully, try
running the upgrade again.
● If the application upgrade failed, the application payload might have issue.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD-4003
Name ObjectscaleManagerUpgrade
Description Step: ObjectScale Manager Upgrade
Issue Category Manual

Alerts 261
Notifiers objectscale-snmp-notifier
Remedies ● Run helm list -A command on cluster to check if the App version of Objectscale manager is same
as the target version.
● Run kubectl describe app objectscale-manager -n <objectscale namespace>
command and check if Assembly phase is set as Succeeded.
● Run kubectl get pods -n <objectscale namespace> and verify that all the pods are in the
running state.
● Run kubectl get deployments -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas i.e. check the READY status.
● Run kubectl get statefulsets -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas i.e. check the READY status.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD-4004
Name SupportAssistUpgrade
Description Step: Support Assist Upgrade.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get supportassist -n <namespace> -o yaml.
● Verify that State status is set as Connected and Phase status is set as Available.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD-4005
Name LicenseUpgrade
Description Step: License Upgrade
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get license -A -o yaml and verify the phase status is set as Available.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD–4006
Name ObjectstoreUpgrade
Description Step: Object store Upgrade
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get ecs -A and verify that Phase status is Available for all.
● Run kubectl get pods -n <objectstore-namespace> and verify that all the pods are in the
Running state.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

262 Alerts
OBJSTEPUPD–4007
Name PostCheck
Description Step: PostCheck
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Run kubectl get deployments -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas i.e. check the READY status.
● Run kubectl get statefulsets -n <objectscale namespace> and verify the Ready
replicas are equal to the Total replicas i.e. check the READY status.
● Verify all the containers of all the replicas are in running state.
● Run kubectl get nodes -A and ensure that the number of nodes in greater than 4.
● Verify if the total number of nodes is less than or equal to 9, then number of ready nodes is greater
than or equal to totalNodes-1.
● Verify if the total nodes is greater than 9, then the number of ready nodes is greater than or equal to
4*TotalNodes/5.
● Run kubectl get pods -A grep objectscale-manager-service-pod and verify the pod
is in the Running state.
● Run kubectl get ecs -A and verify the Status Phase is Available for all.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJSTEPUPD–4008
Name ManifestVersionUpgrade
Description Step: Manifest Version Upgrade
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● As a Kubernetes administrator run kubectl get pods -A and verify the life-cycle-management-
operator and http-share pods are up and running.
● Run kubectl get LCMUpdate -A and verify there is only one LCMUpdate CR created for
manifest version upgrade step.
● Verify the logs of life-cycle-management-operator pod for any errors.
● For additional information on this event, go to www.dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1320
Name BTREE_CHUNK_SR_THRESHOLD_EXCEEDED
Description System metadata space reclamation throughput is too slow to catch up with garbage detection.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1321
Name BTREE_CHUNK_SR_THRESHOLD_EXCEEDED

Alerts 263
Description System metadata space reclamation throughput is too slow to catch up with garbage detection.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1324
Name BTREE_DISK_SR_THRESHOLD_EXCEEDED
Description Capacity free-up throughput is too slow to catch up with system metadata space reclamation.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1325
Name BTREE_DISK_SR_THRESHOLD_EXCEEDED
Description Capacity free-up throughput is too slow to catch up with system metadata space reclamation.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1328
Name BTREE_PARTIAL_SR_THRESHOLD_EXCEEDED
Description Partial space reclamation for system metadata is too slow.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1329
Name BTREE_PARTIAL_SR_THRESHOLD_EXCEEDED
Description Partial space reclamation for system metadata is too slow.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

264 Alerts
OBJST-1332
Name REPO_CHUNK_SR_THRESHOLD_EXCEEDED
Description User space reclamation throughput is too slow to catch up with garbage detection.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1333
Name REPO_CHUNK_SR_THRESHOLD_EXCEEDED
Description User space reclamation throughput is too slow to catch up with garbage detection.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1336
Name REPO_DISK_SR_THRESHOLD_EXCEEDED
Description Capacity free-up throughput is too slow to catch up with user space reclamation.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1337
Name REPO_DISK_SR_THRESHOLD_EXCEEDED
Description Capacity free-up throughput is too slow to catch up with user space reclamation.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1340
Name REPO_PARTIAL_SR_THRESHOLD_EXCEEDED
Description Partial space reclamation for user garbage is too slow.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier

Alerts 265
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1341
Name REPO_PARTIAL_SR_THRESHOLD_EXCEEDED
Description Partial space reclamation for user garbage is too slow.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC technical support for assistance.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1344
Name SR_STATUS_THRESHOLD_EXCEEDED
Description Space reclamation for user data or system metadata is disabled.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1345
Name SR_STATUS_THRESHOLD_EXCEEDED
Description Space reclamation for user data or system metadata is disabled.
Issue Category Auto, 1440
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC technical support for assistance.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1352
Name MEMORY_TABLE_FREE_SPACE_PERCENT
Description Directory Table memory tension detected.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1354
Name MEMORY_TABLE_FREE_SPACE_PERCENT

266 Alerts
Description Directory Table memory tension detected.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check if user application is fully available and throttle load if application is reporting errors for storage
system.
● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1364
Name LISTING_CONVERSION_THRESHOLD_EXCEEDED
Description Listing Conversion speed is slow.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1365
Name LISTING_CONVERSION_THRESHOLD_EXCEEDED
Description Listing Conversion speed is slow.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1366
Name LISTING_CONVERSION_THRESHOLD_EXCEEDED
Description Listing Conversion speed is slow.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1389
Name NVME_BAD_MEMORY_ERROR
Description No memory to allocate to buffer for pod={MY_POD_NAME}, failedCount={count}
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

Alerts 267
OBJST-1390
Name SSD_READ_CACHE_CAPACITY_FAILURE
Description SSD read cache auto cleanup fails when capacity full and fallback to memory cache.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1392
Name SSD_READ_CACHE_CAPACITY_FAILURE
Description SSD read cache auto cleanup fails when capacity full and fallback to memory cache.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese
objectscale-snmp-notifier

Remedies ● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1600
Name OBJMT_EVENT_PROCESSOR_FAILURE
Description Object Store metering event processing lag limit exceeded.
Issue Category Manual
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1601
Name OBJMT_DELTA_LAG
Description Object Store metering event processing lag limit exceeded.
Issue Category Manual
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Contact Dell EMC Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1602
Name OBJMT_REPLICATION_FAILURE
Description Object Scale replication failure detected.
Issue Category Auto, 60

268 Alerts
Notifiers objectscale-snmp-notifier
Remedies ● Check replication rule on source bucket.
● Check IAM role privileges on both source and destination bucket.
● Check replication policy settings on both source and destination buckets.
● Check target bucket for versioning and lock enabled settings.
● Check if destination object store supports encryption.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1603
Name OBJMT_REPLICATION_NOT_PROGRESSING
Description Object Scale replication is not progressing.
Issue Category Auto
Notifiers objectscale-snmp-notifier
objectscale-supportassist-ese

Remedies ● Check the network status between source and destination Object Store.
● Check if replication to destination object store has been paused.
● Check the destination object store capacity and quota.
● Check the destination bucket quota.
● Contact Dell Technologies Support for additional information.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1604
Name ACCOUNT_HARD_QUOTA_EXCEEDED
Description Hard quota on total object count or size that is exceeded for one account.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Increase hard count or size quota for this account or delete objects in buckets that are owned by this
account.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search support for the knowledge base article.

OBJST-1605
Name ACCOUNT_SOFT_QUOTA_EXCEEDED
Description Soft quota on total object count or size that is exceeded for one account.
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Increase soft count or size quota for this account or delete objects in buckets that are owned by this
account.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

Alerts 269
OBJST-1700
Name PRAVEGA_CONNECT_STATUS
Description Pravega connection failed for at least 1 hour(default configuration).
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check that all Pravega pods are ready.
● Use DT tool listing streams or events to confirm Pravega service status.
● Check Pravega service logs for more details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-1701
Name PRAVEGA_SERVICE_STATUS
Description Pravega service is unavailable.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Ensure that the process of object store provisioning has completed.
● Check that all Pravega pods are ready.
● Check Pravega service logs for more details.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-2100
Name CHUNKSTREAM_METADATA_CORRUPTION
Description Chunk stream metadata is invalid, need manual intervention
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledgebase article.

OBJST-2101
Name CHUNKSTREAM_DATA_CORRUPTION
Description Chunk stream data is corrupted in all 3 copies and can not be recovered, need manual intervention.
Issue Category Auto
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledgebase article.

OBJST-MON-4016
Name MonitoringHealth

270 Alerts
Description No data is pushed to the monitoring framework for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-MON-4019
Name MonitoringHealth
Description No data has been pushed to the monitoring framework for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check monitoring components health (telegraf, influxdb, fluxd).
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJST-MON-4020
Name MonitoringFluxd
Description Fluxd has not responded for the last 30 minutes.
Issue Category Auto, 60
Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier
Remedies ● Check Fluxd service status.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJSTORE-HC-1000
Name Pre-Update
Description Preupdate health check for application.
Issue Category Auto
Notifiers objectscale-snmp-notifier
Remedies ● Check the health status in the <component>-app-configmap to find which checks failed.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

OBJUPD-1000
Name ObjectscaleProductUpgrade
Description ObjectScale product upgrade
Issue Category Manual
Notifiers objectscale-snmp-notifier
Remedies ● Identify the step where the upgrade failed by looking into the report.
● Perform the remedies for the failed step.

Alerts 271
 ● For additional information on this event, go to "www.dell.com/support/objectscale" and use the
SymptomID to search support for the knowledge base article.

SNMPNOTI-1000
Name SNMPConnection
Description SNMP connection issue.
Issue Category Auto
Notifiers -
Remedies ● Verify that the SNMP credentials are configured with the correct authentication values.
● Verify that the engineID matches with the engineID that is configured for the product in the
SNMP server.
● Verify that the product SNMP notifier is configured with the correct host or IP address and port.
● Verify the connectivity to the SNMP server, and check firewall, network routing.
● Verify that the SNMP v2c configuration has the correct community string.
● For more information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

TEST TRAP
Name Test SNMP TRAP
Description Test SNMP TRAP
Issue Category -
Notifiers objectscale-snmp-notifier
Remedies ● Verify that the SNMP credentials are configured with the correct authentication values.
● Verify that the engineID matches with the engineID that is configured for the product in the
SNMP server.
● Verify that the product SNMP notifier is configured with the correct host or IP address and port.
● Verify the connectivity to the SNMP server, and check firewall, network routing.
● Verify that the SNMP v2c configuration has the correct community string.
● For additional information about this event, go to https://dell.com/support/objectscale and use the
SymptomID to search for the knowledge base article.

272 Alerts
 15
Metrics for ObjectScale and object stores
Topics:
• ObjectScale metrics
• ObjectScale metrics in Grafana

ObjectScale metrics
Metering details within an ObjectScale instance
Various metering information is available for users on the ObjectScale instance and its object stores and other features.

ObjectScale-level metrics
The ObjectScale instance Dashboard page shows current metric values for the ObjectScale instance.
Object Store Performance is selectable for each object store in the ObjectScale instance.
● Name
● State
● Read First Byte (p50)
● Write Last Byte (p50)
● Read First Byte (p99)
● Write Last Byte (p99)
● Compression ratio
ObjectScale Summary shows the following types of metering data for the ObjectScale instance.
● Health (Critical, Error, and Warning)
● System Data (Data Protection, Metadata, Metadata Protection, Data pending for EC, and Rate of EC per second in both
Base-2 and Base-10)
● Capacity Utilization (Physical Used, Available, Reserved, Total, % Full, and Days till Full (Est) in both Base-2 and Base-10)
● Data Management (Data Being Reclaimed, Unreclaimable Metadata, Unreclaimable User Data, Reclaimable Metadata,
Reclaimable User Data, and Capacity Reclaimed in both Base-2 and Base-10)

Object store-level metrics

The Object Store Dashboard page shows current metric values for the selected object store.
● Latency over time for Read First Byte(p50), Write Last Byte(p50), Read First Byte(p99), and Write Last Byte(p99)
● Health (Critical, Error, and Warning)
● Physical User Data (Local Data, Replica Data, Offline Capacity Available, and Offline Capacity Recovered values in both
Base-2 and Base-10)
● Capacity Utilization (Physical Used, Available, Reserved, Total, % Full, and Days till Full (Est) in both Base-2 and Base-10)
● System Data (Data Protection, Metadata, Metadata Protection, Data pending for EC, and Rate of EC in both Base-2 and
Base-10)
● Logical User Data (Local Data, Replica Data, Compression Ratio, Deleted Data (24 hr), and Deleted Object Count (24 hr) in
both Base-2 and Base-10)
● Data Management (Data Being Reclaimed, Unreclaimable Metadata, Unreclaimable User Data, Reclaimable Metadata,
Reclaimable User Data, and Capacity Reclaimed in both Base-2 and Base-10)

Metrics for ObjectScale and object stores 273

Bucket-level metrics
The Bucket Dashboard page shows metrics for the selected bucket.
● Capacity Statistics (Used Capacity Local, Used Capacity Replication, Deleted Data, TPS, Read Latency, Write Latency )
● Bucket Settings (Versioning, Object Lock, Encryption)
● Quota Statistics (Block Size At (GB), Notification At (GB))
● Policy
● Object Counts (Objects < 10K, Objects 10K > 100 K, Objects 100 K > 1 M, Objects > 1 M)
● Object Lock Configuration (Object Lock Mode, Object Duration Range, Object Duration)
● Event Notification Details (Name, Topic ARN, Events, Filter)
● Bucket Logging (Target Bucket, Prefix for Bucket Logging files)

Account-level metrics
The Account Summary page shows metrics for the selected account.
● Alias
● Account ID
● Enabled
● Protection Mode (enabled or disabled)
● Created On
● Description
● Groups
● Users
● Roles
Account Data
Aggregate Metrics shows total values for the selected account. These values are shown in four formats: Logical - Base-2,
Logical - Base-10, Physical - Base-2, and Physical - Base-10.
● Total Replica Data
● Total User Object Data
Hourly Metrics shows values for the selected account measured hourly. These values are shown in four formats: Logical -
Base-2, Logical - Base-10, Physical - Base-2, and Physical - Base-10.
● Created Object Data
● Deleted Object Data
● Created Replica Data
● Deleted Replica Data

ObjectScale Replication metrics

The Replication page of an object store shows metrics that are related to replication for the object store:
● Data Out
● Data Yet To Be Replicated
● Failed Objects Size
● Failed Objects Count
● Replicated Delete Marker Count, by source bucket and object store
● Delete Marker Failed for Replication, by source bucket and object store
● Delete Marker Pending Replication, by source bucket and object store
The following replication metrics are available on source buckets:
● Data Pending Replication
● Object Pending Replication
● Replicated Data
● Replicated Object Count
● Objects Failed During Replication
● Data Failed During Replication

274 Metrics for ObjectScale and object stores

● Replicated Delete Marker Count
● Delete Marker Failed Replication
● Delete Markers Pending Replication
To see information about the destination bucket, select the destination bucket on the metrics page.

ObjectScale metrics in Grafana

Grafana dashboards overview
The Grafana dashboards show metrics about the operation and efficiency of ObjectScale.
ObjectScale contains predefined dashboards that visualize the collected metrics. Some of the metrics are shown in the UI,
on the main ObjectScale Dashboard and the object store Dashboard pages. Administrators can inspect the reported data in
more detail on the Grafana dashboards. Administrators can identify developing storage and memory problems by monitoring the
dashboards. The dashboards also help identify inefficiencies, and provide a way to diagnose problems.

ObjectScale metrics dashboards

NOTE: Cluster-Admin privilege is required to access ObjectScale metrics.

The predefined dashboards for an ObjectScale instance are:

Table 49. ObjectScale instance dashboards

Dashboard Description

Capacity - Overview Monitors the current and past capacity of each object store in the ObjectScale
instance.

Capacity Utilization: Space Reclamation Monitors the total garbage collection and capacity reclaimed from garbage
collection, current and historical.

Capacity Utilization: Used Capacity Monitors the total, used, and offline capacity data, current and historical.

Capacity Utilization: User Data Monitors the user data, current and historical.

Data Access Performance - Overview Detailed overview of the data access performance for the ObjectScale instance with
views on the transaction summary, successful requests drill down, and failures drill
down.

Garbage Collection: Capacity Reclaimed Monitors the amount of capacity reclaimed from garbage collection and provides a
history of past capacity reclamation.

Garbage Collection: Garbage Detected Monitors the amount of garbage detected within the instance.

IAM Telemetry Details the IAM entities for each IAM account within the ObjectScale instance.

Node Rebalancing Provides an overview of any node rebalancing that has occurred, with details on
the amount of data rebalanced, pending rebalancing. and the rate of rebalance (per
day).

ObjectScale Overview (Default view) Monitors the ObjectScale instance.

Recovery Status Monitors the recovery status, with details on the amount of data to be recovered,
recovery rate (per second), and the time to completion.

Storage Efficiency Monitors the storage efficiency.

Top Buckets Lists the top buckets in the ObjectScale instance by size and by object count.

You can access these dashboards by clicking METRICS on the Dashboard and Accounts pages within the ObjectScale instance.

Metrics for ObjectScale and object stores 275

Object store metrics dashboards
ObjectScale also provides metrics details for each individual object store. The predefined dashboards for an object store are:

Table 50. Object store dashboards

Dashboard Description

Capacity - Overview Monitors the current and past capacity the object store.

Capacity - Overview by Disks Monitors the current and past capacity the disks.

Capacity - Overview by Nodes Monitors the current and past capacity the nodes.

Capacity Utilization: Space Reclamation Monitors the total garbage collection and capacity reclaimed from garbage
collection, current and historical.

Capacity Utilization: Used Capacity Monitors the total, used, and offline capacity data, current and historical.

Capacity Utilization: User Data Monitors the user data, current and historical.

Data Access Performance - by Instance Detailed overview of the data access performance for the instance.

Data Access Performance - Overview Detailed overview of the data access performance for the ObjectScale instance with
views on the transaction summary, successful requests drill down, and failures drill
down.

Data Access Performance Realtime - by Provides a real time view of the data access performance by instance.
Instance

Disk Bandwidth Monitors the overall disk read/write bandwidth for the object store over the
selected period of time, current and historical.

Hardware Health: All Nodes and Disks - Monitors the individual status of each node and their disks. Use the node_id
by Nodes dropdown to view details on a node.

Hardware Health: All Nodes and Disks - Monitors the status of all nodes and disks in the object store.
Overview

Hardware Health: Offline Disks Monitors the status of all offline disks in the object store.

Hardware Health: Offline Nodes Monitors the status offline nodes in the object store.

Node Rebalancing Provides an overview of any node rebalancing that has occurred, with details on
the amount of data rebalanced, pending rebalancing. and the rate of rebalance (per
day).

ObjectStore Overview (Default view) Monitors the object store.

Recovery - Disk Recovery Progress Monitors the current and past disk recover status.

Recovery Status Monitors the recovery status, with details on the amount of data to be recovered,
recovery rate (per second), and the time to completion.

Storage Efficiency Monitors the storage efficiency.

Top Buckets Lists the top buckets in the object store by size and object count.

You can access these dashboards by clicking METRICS link of each object store.

276 Metrics for ObjectScale and object stores

Navigating Grafana
A Grafana dashboard is a set of one or more panels organized and arranged into one or more rows. Our ObjectScale instance
ships with many pre-configured dashboards for monitoring an ObjectScale instance.
These dashboards makes it easy to quickly display properties about the functionality of the ObjectScale instance.

Figure 22. Navigation details

1. Time picker dropdown.

2. Zoom out time range
3. Manual refresh button. Use to refresh all panels of the displayed dashboard.
4. Star Dashboard. Star or unstar the current Dashboard. Starred Dashboards will show up on your own Home Dashboard by
default
5. Related Dashboards. Displays the dashboards related to the currently displayed dashboard.
6. Available Dashboards. Return to the home dashboard or displays the available Dashboards.
7. Dashboard dropdown. This dropdown shows you name of the Dashboard currently displayed, and allows you to switch to a
different Dashboard.
8. Dashboard panels displaying relevant data for the dashboard.

View the Metrics dashboards for the ObjectScale instance

Steps
1. From the ObjectScale Portal user interface, click Dashboard.
The ObjectScale Dashboard page is displayed showing details on object store performance in the selected namespace and a
summary of the ObjectScale instance.
2. Click METRICS.
The ObjectScale Overview dashboard appears.

Metrics for ObjectScale and object stores 277

 Figure 23. ObjectScale Overview dashboard
3. Use the drop-down menu in the upper left of the Metrics page to navigate between the various pre-configured ObjectScale
dashboards.

View the Metrics dashboards for an object store

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Click the name of the object store.
4. Click ... if the Metrics link is not visible and then select Metrics.
The ObjectStore Overview dashboard appears.

278 Metrics for ObjectScale and object stores

 Figure 24. ObjectStore Overview Dashboard
5. Use the drop-down menu in the upper left of the Metrics page to navigate between the various preconfigured object store
dashboards.

Metrics for ObjectScale and object stores 279

 16
Maintain ObjectScale
Topics:
• About ObjectScale service procedures

About ObjectScale service procedures

ObjectScale integrates into very different environments. This requires ObjectScale to support different service procedures for
each of the different platforms. The ObjectScale Operator within the ObjectScale contains an API to manage the different
service procedures for these two environments. In addition to the API, the ObjectScale operator also:
● Keeps history of service procedures
● Keeps information for debug purposes
● Keeps information about different service procedures
The ObjectScale Operator API is a complete solution to perform service procedures. Applications watch and modify the CRD to
communicate with platforms.
Depending on your ObjectScale environment, you will interact with the ObjectScale Operator and the underlying service
procedures in one of the following ways:
● Use kubectl for Objectscale instances on a Red Hat OpenShift cluster
● Use kubectl for Objectscale instances on an ObjectScale software bundle cluster .

Service procedures
In this release, ObjectScale supports the following service procedures for customers to use to maintain their ObjectScale
deployment.
● Horizontal Expansion
● Vertical Expansion
● Temporary Maintenance Mode
● Permanent Maintenance Mode
● Disk Replacement
● Node Replacement
● Add node to ObjectScale deployed on the ObjectScale software bundle
● Retire or remove a node from ObjectScale deployed on the ObjectScale software bundle

About ObjectScale capacity expansion procedures

There are two ways to increase the capacity of an object store in ObjectScale: horizontal expansions and vertical expansions.
Horizontal expansions increase the number of Storage Server Replicas, thereby increasing the number of nodes with data disks
in an object store using the ObjectScale Portal user interface. During the horizontal expansion ObjectScale will confirm there
are enough nodes and resources on the Kubernetes environment to schedule the newly added pods. The ObjectScale Edit
Object Store wizard ensures that all component replica counts are increased as appropriate, according to the object store's new
Storage Server Replica count.
Vertical expansions increase the number of Persistent Volumes per Storage Server Replica, thereby increasing the number of
data disks per node in an object store using the ObjectScale Portal UI. During the vertical expansion ObjectScale will confirm
that there is enough storage available to allocate to the newly added persistent volumes.

280 Maintain ObjectScale

General guidelines or limitations of the expansion service procedures
● Only one expansion operation may run at a time - either horizontal or vertical.
WARNING: Parallel volume expansions of different object stores at the same time can cause node affinity
issues. If you need to expand multiple object stores, begin and complete the expansion of one object store
before beginning the expansion of the next object store.
● Expansions will not change the Erasure Coding (EC) scheme used by the object store.
○ For example, if the object store was deployed with 12+4 EC, it will remain so for the life of the object store, regardless of
the number of nodes added later.
○ Increasing the number of nodes will provide better I/O performance and increase data fragment dispersal.
When performing horizontal expansions, the new expansion node hardware must already be installed into the cluster and that
it meets the necessary hardware and software requirements prior to initiating the horizontal expansion.
● During an object store expansion, the expansion will fail if certain pods are unable to start. In these cases, ObjectScale will
create an alert that provides the specific details on which pod and what resource was not available. Use the details in the
issue alert to resolve the underlying issue.
CAUTION: Ensure that there are sufficient Storage Capacity, Memory and CPU resources available before
you initiate an expansion procedure, otherwise the expansion may get stuck.

Ensure there is no parallel workload occurring which may consume these resources while the expansion
process is in progress. If a vertical expand procedure is initiated with fewer resources than it requires, there
is a chance for a Data Unavailability until enough resources are added and expansion process completes.

Horizontally expand the capacity of an object store

You can expand the capacity of a previously created object store by increasing the number of Storage Server replicas in the
object store.

Prerequisites
Before beginning this expansion, ensure that:
● The object store is in a healthy or available status.
● All prior capacity expansions are complete.

About this task

Use these steps to expand the capacity of a previously created object store by increasing the number of Storage Server replicas
in the object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Record the information about the current capacity of the object store by:
NOTE: Each SS in the object store has the identical number of volumes. New SS replicas have the same number of
volumes as any other existing SS in the current object store.

a. In the table of object stores, click the name of object store whose capacity you want to expand.
b. Click the Summary tab and locate the SS Replica Counts value in the Storage details table.

c. Click to close this view.

4. To horizontally expand the capacity of the object store.
a. In the table of object stores, reselect the object store using the checkbox to the left of the object store.
b. Click EDIT to modify the object store.
c. In the Edit Object Stores section, select the Storage portion of the wizard.
d. Increase the number of Storage Server Replicas and then click Save.
It takes a few minutes for ObjectScale to expand the object store. Once the expansion is completed, the object store
Health returns to Available.

Maintain ObjectScale 281

 In the object store list view, you can verify that the Total Utilized Capacity increases after the horizontal expand procedure
is complete.
5. Return to the Summary tab and locate the SS Replica Counts in the Storage details table for the object store.
The number of Storage Server replicas has increased.

6. Click to close this view.

Vertically expand the capacity of an object store

You can expand the capacity of each storage server replica by increasing the number of Volumes per Storage Server Replica
within the existing object store.

Prerequisites
Before beginning this expansion, ensure that:
● The object store is in a healthy or available state.
● All prior object store expansions are complete.
● There is available unused capacity for use in this expansion.

About this task

Use these steps to expand the number of volumes per replica of a previously created object store.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Record the information about the current capacity of the object store by:
a. In the table of object stores, click the name of object store whose volumes you want to increase.
b. Click the object store Summary tab and locate the Volumes per Storage Server Replica value in the Storage details
table.

c. Click to close this view.

Total Utilized Capacity of object stores can be recorded, and it increases after vertical expansion is complete.
4. To vertically expand the object store:
a. In the table of object stores, reselect the object store using the checkbox to the left of the object store.
b. Click EDIT to modify the object store.
c. In the Edit Object Stores wizard, click to the Storage portion of the wizard.
d. Expand CUSTOMIZE(OPTIONAL) and then increase the number of Volumes per Storage Server Replica and then
click Save.
It takes a few minutes for ObjectScale to expand the object store. Once the expansion is completed, the object store
Health returns to Available.
5. Return to the object store Summary tab and locate the Volumes per Storage Server Replica in the Storage details table for
the object store.
The number of PVCs has increased to accommodate the new capacity added.
NOTE: During vertical expansion, each SS replica is restarted in turn. This may cause a drop in overall I/O throughput
while the pods restart and the cluster stabilizes.

6. Click to close this view.

282 Maintain ObjectScale

ObjectScale Maintenance Modes

About ObjectScale maintenance modes

ObjectScale maintenance mode overview

Taints and toleration are used to coordinate maintenance modes. A user can take a single node down for maintenance in order
to repair a faulted hardware component. Or, a user might do a sequential rolling maintenance mode to perform a software
upgrade on all nodes. In the rolling MM case one node enters MM, finishes the maintenance operation, and is exited from MM
and the next node is put into MM.
ObjectScale on OpenShift supports two types of maintenance modes; Temporary Maintenance Mode (TMM) and Permanent
Maintenance Mode (PMM). The ObjectScale Software Bundle supports Temporary Maintenance Mode (TMM) and Node
Remove maintenance modes.
Two different types of taints trigger these maintenance modes. ObjectScale Operator reacts to these taints to handle the pods
on the node. ObjectScale Operator only handles the pods with a special label, and only ObjectScale pods have the label.
In TMM, all ObjectScale and object store stateful sets and Stateless pods are rescheduled to an available spare node, if
available. Pods controlled by a stateful set are not rescheduled to other nodes during TMM. Daemon sets remain in Running
status on the node under TMM. Finally, the ObjectScale Software Bundle stateful CMO pods are rescheduled to an available
spare node, by deleting the PVC and pod. Once the TMM taint is removed from the node, the pending stateful pods previously
running on the node under TMM return to Running status on the original node.
In PMM, the ObjectScale Operator deletes all the pods and PVCs on the node.

Placing multiple nodes into maintenance mode

ObjectScale supports placing multiple nodes into maintenance mode simultaneously, depending on the size and the state of
the object store and ObjectScale. An object store configured with the recommended number of SS replicas can place up to
two nodes into maintenance mode. When a third node attempts to enter maintenance mode, it is either rejected from entering
maintenance mode or placed into a queue until another node exits maintenance mode. See alert messages within ObjectScale
Portal for details on status of nodes in or rejected from maintenance mode.
NOTE: Nodes cannot be placed into maintenance mode if any non-maintenance mode service procedures are pending or in
progress.
ObjectScale recommends that:
● Object stores with a 12+4 EC scheme should have at least 10 SS replicas

Temporary maintenance mode

Temporary maintenance mode (TMM) is a service procedure used to place a node into maintenance mode for maintenance
activities at the node level, such as software upgrades. While the node is in TMM, all user data remains accessible (read and
write).
During TMM, all object stores move to maintenance phase until the node is taken out of TMM. Once taken out of TMM, the
object store status returns to Available.

NOTE: If the node TMM for more than one hour, and it is running an SS replica, recovery begins for the data on that node.

For ObjectScale instances deployed on a Red Hat OpenShift cluster, manually place a node into TMM by placing a taint on the
node.
For the ObjectScale Software Bundle Cluster, you must enter TMM by making the node unschedulable and then manually
cordon the node within the CMO Platform. This step is not automatic and initiated by the cluster administrator similar to original
step used to enter TMM.
When TMM occurs, ObjectScale Operator reacts to the node taint and undertakes the following actions.
● Once placed into TMM, all stateless pods (ReplicaSet pods) are relocated to other available nodes in the cluster
automatically.
● Pods controlled by a DaemonSet may continue to run on the node while in TMM. This includes some CSI Bare-Metal,
Kubernetes, and platform-related pods continue running on the node under TMM.

Maintain ObjectScale 283

● ObjectScale and object store pods that have a persistent volume claim (PVC) which is bound to a persistent volume (PV) on
the node in TMM remain in the Pending state until the node is taken out of TMM. Pods controlled by a stateful set are not
rescheduled to other nodes.
If entering TMM is rejected or failed, manually return the node to an Available state.
● For ObjectScale instances deployed on a Red Hat OpenShift cluster, you must manually exit the TMM by making the node
schedulable. This step is not automatic and controlled by the OpenShift administrator similar to original step used to enter
TMM.
● For the ObjectScale Software Bundle, you manually uncordon the node using the CMO Platform API and then exit TMM by
making the node schedulable. This step is not automatic and controlled by the cluster administrator similar to original step
used to enter TMM.

Permanent maintenance mode

Permanent maintenance mode (PMM) is a service procedure that is used to place a node into maintenance mode permanently
for node removal. Permanent removal should only occur while the cluster is healthy. When you place a taint on the node to go
into PMM, then the ObjectScale Operator handles the taint and moves resources to an available node.

Maintenance activities for ObjectScale Software Bundle

For ObjectScale Software Bundle, placing nodes into maintenance modes and performing node maintenance activities, such as
adding or removing a node from the cluster, requires additional CMO Platform API calls be made after entering the node into an
ObjectScale maintenance mode.
Using the ObjectScale maintenance modes and/or the CMO Platform APIs allows you to perform these node activities on the
ObjectScale software bundle clusters.
● Place a node in an ObjectScale software bundle cluster into temporary maintenance mode
● Add a new node to the ObjectScale software bundle cluster
● Remove a node from the ObjectScale software bundle cluster
● Prepare a node for node hardware or software maintenance
● Replace a node within the ObjectScale software bundle cluster

Maintenance mode in ObjectScale on OpenShift

When performing certain maintenance tasks on ObjectScale in a OpenShift environment, follow these procedures:
● Place a node into temporary maintenance mode (ObjectScale on OpenShift)
● Remove a node from temporary maintenance mode (ObjectScale on OpenShift)
● Place a healthy node into permanent maintenance mode (ObjectScale on OpenShift)
● Place a failed node into permanent maintenance mode (ObjectScale on OpenShift)

Place a node into temporary maintenance mode (ObjectScale on OpenShift)

Use this manual process to place a healthy node into temporary maintenance mode (TMM). Use this process for ObjectScale
instances on a Red Hat OpenShift cluster.

Steps
1. Apply a taint to the node to be placed into temporary maintenance mode:

kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule

2. Verify that the PHASE of the cluster now displays Maintenance.

kubectl get ecs-cluster -n <NAMESPACE>

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API

ecs-cluster Maintenance 22/23 10.236.228.53:443 10.236.228.52:4443

The ObjectScale Portal shows the object store status as Maintenance.

284 Maintain ObjectScale

3. Once the taint has been applied to a node, the ObjectScale Operator creates a TMM service procedure. Retrieve the list of
service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure name:

kubectl get serviceprocedures -n <NAMESPACE>

NAME AGE
recovery-ecs-cluster-bk-bookie-0-18369f2d 32m
recovery-ecs-cluster-bk-bookie-1-426c0578 26h
recovery-ecs-cluster-influxdb-0-abee6329 34m
recovery-ecs-cluster-influxdb-2-1d470d3e 33m
recovery-ecs-cluster-zookeeper-4-c01cff83 33m
recovery-objectscale-manager-influxdb-0-42e4e0a9 46m
tmm-a4a9b606-4126-4914-b18c-27337e841f63 15m

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

NOTE: Do not delete the service procedure while it is running.

4. Monitor the status of the service procedure with the following command:

while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure transitions through various phases as it progresses. The Reason value for the TMM service procedure
should progress from NotStarted, In Progress, PostCheck, and finally to Success. TMM should enter Waiting
until the taint is removed, at which point it goes to Success.

5. Verify that only pods controlled by a DaemonSet remain running.

You may see other non-ObjectScale pods running on the node, such as metallb, OpenShift pods.

kubectl get pods --all-namespaces -o custom-

columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].k
ind,Started:status.startTime | grep <NODE_NAME>

baremetal-csi-node-56jb2 worker2.ocp4.cmo.com DaemonSet

2021-02-19T16:11:24Z
speaker-sg4ws worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:36:00Z
tuned-pwphr worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
dns-default-prbvg worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
node-ca-tpgwk worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
machine-config-daemon-nkdbx worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
node-exporter-wnltt worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
multus-m7pgs worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
network-metrics-daemon-xp68j worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
ovs-gm962 worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z
sdn-fcrkh worker2.ocp4.cmo.com DaemonSet
2021-02-11T14:13:05Z

6. Also, any pods previously running on the TMM node that belong to a StatefulSet enter the Pending state. These are pods
that have a persistent volume claim (PVC) that is bound to a persistent volume (PV) on the node in TMM.

Maintain ObjectScale 285

 There may also be stateless pods (that is ReplicaSet) in the Pending state if the pods cannot be relocated due to pod
anti-affinity rules. This is expected behavior.

kubectl get pods --all-namespaces=true -o custom-

columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].k
ind,Status:status.phase,Started:status.startTime | grep Pending

decks-support-store-0 <none> StatefulSet Pending <none>

objectscale-iam-atlas-1 <none> StatefulSet Pending <none>
objs-mgr-rsyslog-4 <none> StatefulSet Pending <none>

7. Verify the ObjectScale Portal UI shows that the node has entered TMM by reviewing the Monitoring > Alerts tab.

Results
The node is now in TMM.
NOTE: Ensure that you check the Alerts and Logs corresponding to ObjectScale Operator.

Remove a node from temporary maintenance mode (ObjectScale on OpenShift)

About this task

Use this process for ObjectScale instances on a Red Hat OpenShift cluster.

Steps
1. Remove the taint from the node in temporary maintenance mode.

kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule-

2. Verify that all nodes in the cluster are ready.

kubectl get nodes

NAME STATUS ROLES AGE VERSION

master0.ocp4.cmo.com Ready master,worker 8d v1.19.0+3b01205
master1.ocp4.cmo.com Ready master,worker 8d v1.19.0+3b01205
master2.ocp4.cmo.com Ready master,worker 8d v1.19.0+3b01205
worker0.ocp4.cmo.com Ready worker 8d v1.19.0+3b01205
worker1.ocp4.cmo.com Ready worker 8d v1.19.0+3b01205
worker2.ocp4.cmo.com Ready worker 8d v1.19.0+3b01205
worker3.ocp4.cmo.com Ready worker 8d v1.19.0+3b01205

3. Verify that the object store Phase returns to Available.

kubectl get ecs-cluster

4. Verify that the ObjectScale Portal UI shows that the node has returned from TMM by reviewing the Monitoring > Alerts
tab.

Results
All pods that were in the pending state are now running on the node as before.

286 Maintain ObjectScale

Place a healthy node into permanent maintenance mode (ObjectScale on OpenShift)
Use the manual process to place a healthy node into permanent maintenance mode (PMM). Use this process for ObjectScale
instances on a Red Hat OpenShift cluster.

About this task

Place the healthy node to be removed into permanent maintenance mode:

Steps
1. Apply a taint to the node to be removed:

kubectl taint node <NODE_NAME> node.dell.com/drain=drain:NoSchedule

Wait for the PMM service procedure to complete before moving on to the next steps.
2. Collect the UUID for the node to be removed from the cluster:

kubectl get csibmnodes

3. Remove the node from the OpenShift cluster.

kubectl delete node <NODE_NAME>

NOTE: Once you delete the node, it is no longer listed in the kubectl get nodes output.

For healthy node removal initiated by tainting the node, removal of CSI resources happens automatically.
This process includes removal of the following CSI resources, which completes the node removal process:
● The Bare-Metal node
● Available capacity
● Bare-Metal drive CRs

4. Optional: Monitor the automatic recovery of non-SS pods:

In order to ensure data protection, certain non-SS pods, such as the bookie, influxdb, and zookeeper pods, require recovery
after they are relocated. ObjectScale Operator initiates recovery for these pods automatically once the pods are removed
from the PMM node and started on another available node in the cluster.

kubectl get serviceprocedures -A -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message

Place a failed node into permanent maintenance mode (ObjectScale on OpenShift)

Use kubectl to manually place a failed node (that is powered off, dead, or otherwise inaccessible) into permanent maintenance
mode. Use this process for ObjectScale instances on a Red Hat OpenShift cluster.

About this task

NOTE: The removal process for a failed node is largely manual, and does not involve the ObjectScale Operator in the
same way that standard PMM does. However, during the node removal process, the ObjectScale Operator starts recovery
procedures for certain non-SS stateful pods, such as bookie, influxdb, and zookeeper. Recovery of the non-SS stateful pods
occurs automatically, and does not affect the procedure workflow.
Place the failed node to be removed in to permanent maintenance mode:

Maintain ObjectScale 287

Steps
1. Mark the failed node as unschedulable so that it is no longer available to run pods.

kubectl cordon <NODE_NAME>

2. Delete the pods from the node.

kubectl drain <NODE_NAME> --force --delete-local-data --ignore-daemonsets

3. Collect the UUID for the node to be removed from the cluster:

kubectl get csibmnodes

4. Remove the node from the OpenShift cluster.

kubectl delete node <NODE_NAME>

NOTE: Once you delete the node, it is no longer listed in the kubectl get nodes output.

5. Manually delete the PVCs bound to the failed node:

a. Get the names of the PVCs:

kubectl get pvc

b. Get the details for each of the PVCs:

kubectl describe <PVC_NAME>

c. Get the node for each of the PVCs:

for i in `kubectl get pvc --no-headers -o jsonpath="{.items[*].metadata.name}"`; do

echo "=== $i"; kubectl get pvc $i -o json | grep selected-node | grep -v "{}"; done

user1@hw-and-os-96:~> kubectl describe pvc data-0-ecs-cluster-ss-0

Name: data-0-ecs-cluster-ss-0
Namespace: default
StorageClass: csi-baremetal-sc-hdd
Status: Bound
Volume: pvc-8f28acda-f7e5-4efc-9b4b-d8e50e21e72f
Labels: app=ecs-cluster-ss
app.kubernetes.io/component=ss
app.kubernetes.io/name=ecs-cluster
app.kubernetes.io/namespace=default
component=ss
objectscale.dellemc.com/logging-inject=true
objectscale.dellemc.com/logging-release-name=ecs-cluster
operator=objectscale-operator
release=ecs-cluster
Annotations: pv.attach.kubernetes.io/ignore-if-inaccessible: yes
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi-baremetal
volume.kubernetes.io/selected-node: worker3.ocp4.cmo.com
volumehealth.storage.kubernates.io/health: accessible
volumerelease.csi-baremetal/support: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 11176Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: ecs-cluster-ss-0
Events: <none>

288 Maintain ObjectScale

 d. Once all PVC names for the node have been gathered, delete the PVCs:

kubectl delete pvc <PVC_NAME_1> <PVC_NAME_2> <PVC_NAME_N>

6. Patch and delete the volumes of the failed node:

a. Patch the volumes:

kubectl get volume | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl patch
volume --type merge -p '{"metadata":{"finalizers":null}}'

b. Remove the volumes:

kubectl get volume | grep <NODE_UUID>| awk '{print $1}' | xargs kubectl delete
volume

7. Patch and delete the LVGs of the failed node:

a. Patch the LVGs:

kubectl get lvg | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl patch lvg
--type merge -p '{"metadata":{"finalizers":null}}'

b. Remove the LVGs:

kubectl get lvg | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl delete lvg

8. Clean up all the CSI resources for the failed node:

a. Patch the LVGs:

kubectl get csibmnode | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl patch
csibmnode --type merge -p '{"metadata":{"finalizers":null}}'

b. Remove the LVGs:

kubectl get csibmnode | grep <NODE_UUID>| awk '{print $1}' | xargs kubectl delete
csibmnode

c. Delete the drive CRs:

kubectl get drive | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl delete drive

d. Delete the available capacity:

kubectl get ac | grep <NODE_UUID> | awk '{print $1}' | xargs kubectl delete ac

9. Remove the pending pods for all namespaces that are associated with ObjectScale and object stores:
a. Identify the pods to be deleted:

kubectl get pods | grep Pending

b. Delete each pod returned, that is associated with the removed node:

kubectl delete pods <PODS>

10. Finally, verify that all the resources have been successfully removed:
a. Check for Bare-Metal nodes:

kubectl get csibmnode | grep <NODE_UUID>

Maintain ObjectScale 289

 b. Check for available capacity:

kubectl get ac | grep <NODE_UUID>

c. Check for drive CRs:

kubectl get drive | grep <NODE_UUID>

11. Optional: Monitor the automatic recovery of non-SS pods:

In order to ensure data protection, certain non-SS pods, such as the bookie, influxdb, and zookeeper pods, require recovery
after they are relocated. ObjectScale Operator initiates recovery for these pods automatically once the pods are removed
from the PMM node and started on another available node in the cluster.

kubectl get serviceprocedures -A -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message

Temporary maintenance mode for ObjectScale Software Bundle

When performing certain maintenance tasks on the ObjectScale Software Bundle, follow these procedures to move the node
into and out of temporary maintenance mode:
● Place a node into temporary maintenance mode (ObjectScale Software Bundle)
● Remove a node from temporary maintenance mode (ObjectScale Software Bundle)

Place a node into temporary maintenance mode (ObjectScale Software Bundle)

Use this manual process to place a healthy node into temporary maintenance mode (TMM) within the ObjectScale Software
Bundle.

Steps
1. Apply a taint to the node to be placed into ObjectScale temporary maintenance mode:

kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule

2. Verify that the PHASE of the cluster now displays Maintenance.

kubectl -n <OBJECTSCALE_NAMESPACE> get ecs-cluster

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API

ecs-cluster Maintenance 22/23 10.236.228.53:443 10.236.228.52:4443

The ObjectScale Portal UI shows the object store status as Maintenance.

3. Once the taint has been applied to a node, the ObjectScale Operator creates the ObjectScale TMM service procedure.
Retrieve the list of service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure
name:

kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

NOTE: Do not delete the service procedure while it is running.

290 Maintain ObjectScale

4. Monitor the status of the service procedure with the following command:

while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure transitions through various phases as it progresses. The Reason value for the TMM service
procedure should progress from NotStarted, In Progress, PostCheck, Waiting, and finally to Success. A reason
of Success or Waiting indicates that the service procedure has completed without error, and the node is now in TMM.

5. Next, place the node into maintenance mode within the CMO Platform within the ObjectScale Software Bundle.

kubectl cordon <NODE_NAME>

6. Safely evict all your pods from the node:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

For example:

# kubectl drain hostname6 --ignore-daemonsets --delete-emptydir-data --force

WAINING: ignoring DaemonSet-managed Pods: cmo/metallb-speaker-ggrkq, cmo/whereabout-
whereabouts-58dss, calico-system/calico-node-8rmcp, default/csi-baremetal-node-9wg5w,
kube-system/rke2-ingress-nginx-controller-zvbnw, kube-system/rke2-multus-ds-5bv2x
evicting pod cmo/decks-support-store-0
pod/decks-support-store-0 evicted
node/hostname6 drained

7. Verify the status of the drained node:

kubectl get node <NODE_NAME>

For example:

# kubectl get node hostname6

NAME STATUS ROLES AGE VERSION
hostname14 Ready,SchedulingDisabled <none> 6d19h v1.24.7+rke2r1

8. Verify that all CMO component pods have been rescheduled to the other nodes.

kubectl get pod -n cmo | grep Pending

9. Verify the ObjectScale Portal UI shows that the node has entered TMM by reviewing the Monitoring > Issues tab.

Results
The node is now in TMM.
NOTE: Ensure that you check the Issues or Logs corresponding to ObjectScale Operator.

Remove a node from temporary maintenance mode (ObjectScale Software Bundle)

Use this manual process to return a healthy node from temporary maintenance mode (TMM) after completing the maintenance
activity within the ObjectScale Software Bundle.

About this task

Once you have finished the maintenance activity on the node in maintenance mode, return a node to normal status.

Maintain ObjectScale 291

Steps
1. Mark node as schedulable.

kubectl uncordon <NODE_NAME>

2. Remove the taint from the node in temporary maintenance mode.

kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule-

TMM service procedure should transition to Success.

3. Verify that there are no pending pods:

kubectl get pod -n cmo | grep Pending

4. Verify that all nodes in the cluster are ready.

kubectl get nodes

5. Verify that the object store Phase returns to Available.

kubectl get ecs-cluster

6. Verify that the ObjectScale Portal UI shows that the node has returned from TMM by reviewing the Monitoring > Events
tab.

Results
All pods that were in the Pending state are now running on the node as before.

Maintenance mode in ObjectScale on Appliance

When performing certain maintenance tasks on ObjectScale Appliance, follow these procedures:

Place a node into Maintenance Mode (ObjectScale Appliance)

Use this process to place a node into Maintenance Mode.

Steps
1. From the ObjectScale Portal user interface, click Nodes.
The list of Nodes that the user is authorized to view is displayed.
2. Select the Node to be placed into Maintenance Mode by clicking the radio button on its left side.
3. From the Action dropdown, click Enable Maintenance Mode.
A dialogue box is displayed to confirm placing the node into maintenance mode.
4. Click the checkbox to acknowledge the risks and click Enable.
A notification is displayed indicating that maintenance mode operation has been initiated.

Results
The selected node is now in Maintenance Mode.

Exit a Node from Maintenance Mode (ObjectScale Appliance)

About this task

Use this process to exit a node from Maintenance Mode for ObjectScale Appliance.

292 Maintain ObjectScale

Steps
1. From the ObjectScale Portal user interface, click Nodes.
The list of Nodes that the user is authorized to view is displayed.
2. Select a Node that is already in Maintenance Mode by clicking the radio button on its left side.
3. From the Action drop-down, click Disable Maintenance Mode.
A dialogue box is displayed to confirm the exit of the selected node from maintenance mode.
4. Click the checkbox to acknowledge the risks and click Disable.
A notification is displayed indicating that exit maintenance mode operation has been initiated.

Results
The node has exited Maintenance Mode.

Service procedures for cluster disks

Disk replacement service procedure

ObjectScale generates messages in ObjectScale Portal when failed persistent volumes (PVs) are found. Once ObjectScale finds
failed PVs, the disk replacement service procedure automatically begins to recover data on the failed drive. You can monitor the
status of the recovery and confirm a successful recovery using the Events within the ObjectScale Portal user interface.
There are two types of disk replacement service procedures ObjectScale uses when necessary. One method is used when the
disk contains a StorageServer (SS) or NVMeEngine pod, and the other is used when the disk contains only a non-SS pod.
When ObjectScale uses the disk replacement service procedure for SS pods, the status of the object store changes. This
affects the availability of maintenance mode service procedures until the disk replacement service procedure completes. When
ObjectScale uses the disk replacement service procedure for non-SS pods, ObjectScale creates a DR SP. This does not affect
your ability to place nodes into a maintenance mode.
Shown below is the high-level overview of the disk replacement service procedure.
● When a disk's health becomes FAILED or SUSPECT ObjectScale Portal displays an issue in the Alerts section with the
details of the degraded disk status.
● During the disk replacement service procedure for SS pods, the object store status changes from Available to
ReplacingPV. Otherwise, the object store status is unchanged.
● Then the data recovery portion of the service procedure is initiated automatically, if applicable.
○ If there is available capacity (free disks) in the node, a volume is re-created once the recovery is completed using a new
available disk in the system. The old disk is now waiting to be replaced (pod and object store are in a good state, and no
longer impacted).
○ If there is no available capacity on the node (no free disks), the system waits for the user to perform the necessary
actions required before initiating the volume recreation.
● The alert is updated with a message that the Disk is ready for replacement
● Once you receive this message, you can replace the hard drive in the node.
● After you replace the disk and ObjectScale completes the disk replacement SP, the alert message is updated to Disk has
been successfully replaced.

Automatic disk replacement service procedure

About this task

An automatic disk replacement procedure is implemented within ObjectScale and automatically handles disk failures. This
procedure details the process to locate failed Persistent Volumes and then successfully replace the PV.

Steps
1. From the ObjectScale Portal user interface, click Administration > ObjectScale.
The list of Object Stores in the selected namespace that the user is authorized to view is displayed.
2. Select the appropriate namespace from the namespace drop-down on the upper right the ObjectScale Portal user interface.
3. Optional: Locate the object store containing the failed disk on the object stores details page.

Maintain ObjectScale 293

 When a PV fails, the state and health of the object store(s) containing that PV will go into ReplacingPV and the disk
replacement service procedure begins.
4. Monitor the status of the process at Monitoring > Logs tab to view the system generated events for the disk replacement
process.
● If the system contains an available spare drive, the service procedure will progress to completion. Once the process has
completed, the object store(s) will return to the Started State and Available Health.
● If there are no available spare drives or otherwise insufficient capacity, the disk replacement service procedure generates
a warning events for Not enough capacity as it attempts to recreate the PVCs on the failed PV. Complete step 5 to
complete the replacement if this occurs.
5. After getting an event that Reason: DriveReadyForRemoval and have a new disk available, initiate the physical
replacement in OpenShift by placing a replacement=ready annotation on the failed/suspect disk.
a. Confirm the disk is in Released status.

kubectl get drives | grep <DRIVE_SERIAL_NUMBER>

b. Place the replacement=ready annotation on the failed/suspect disk.

annotate drives.csi-baremetal.dell.com <DRIVE_RESOURCE_ID> replacement=ready

c. Confirm that the disk is now in Removed status.

kubectl get drives | grep <DRIVE_SERIAL_NUMBER>

d. Confirm that the ISSUE has been updated with Reason: DriveReadyForPhysicalRemoval.
CAUTION: Do not physical replace the disk until the above WARNING event is displayed under the
respective ISSUE.

The disk LED is blinking. If you are unable to identify the disk to replace, you will need to determine another way to identify
the disk manually or visually, by using additional information located in the associated ISSUE events.
6. Remove and replace the failed drive with the new, clean drive. Afterwards, the ISSUE in ObjectScale Portal UI will be
auto-cleared by being set to Normal severity. Once the event Reason: "DriveSuccessfullyRemoved" occurs and
you have inserted a new drive into the node, the disk replacement service procedure has completed successfully and no
further actions is required.

Proactive Disk Removal Service Procedure (for Appliance and Software Bundle)

About this task

You can proactively trigger a disk removal procedure within ObjectScale for Appliance and Software Bundle deployment. This
helps in removing a healthy disk that is suspected to be failed.

Steps
1. From the ObjectScale Portal user interface, click Disks.
The list of Disks that the user is authorized to view is displayed.
2. Select the disk to be removed, and click Remove.
A dialogue box is displayed to acknowledge the risks and confirm disk removal.
3. Click Remove disk.
The health of the disk is changed to "BAD".

admin@lehi-dirt:~> kubectl get drive 69d56273-b15b-49d5-bd65-b75b525b3425 -o yaml

apiVersion: csi-baremetal.dell.com/v1
kind: Drive
metadata:
annotations:
health: bad
creationTimestamp: "2023-07-24T13:14:34Z"
generation: 10
labels:
app: csi-baremetal
app.kubernetes.io/name: csi-baremetal
name: 69d56273-b15b-49d5-bd65-b75b525b3425

294 Maintain ObjectScale

 resourceVersion: "775292"
uid: 35edbfa5-9e69-4a21-8504-97d22d88e485
spec:
Firmware: 1.1.1
Health: BAD
IsClean: true
NodeId: 8a4f12a5-e4b4-406f-b272-f920f1e826e4
......
Status: ONLINE
Type: NVME
UUID: 69d56273-b15b-49d5-bd65-b75b525b3425
Usage: IN_USE

4. The cluster operation CR gets created.

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
TriggerRemoval 69d56273-b15b-49d5-bd65-b75b525b3425 19s

5. The object store status moves to "ReplacingPV".

NOTE: This change is applicable only for SS or NVMeEngine pods. For non-SS pods, the object store status does not
change.

admin@lehi-dirt:~> kubectl get ecs -A

NAMESPACE NAME PHASE READY COMPONENTS S3 ENDPOINT
MGMT API
objectscale dirt-objectstore ReplacingPV 21/21
10.249.248.25:443 10.249.248.27:4443

6. The recovery of the disk starts.

admin@lehi-dirt:~> kubectl get serviceprocedure -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale recovery-dirt-objectstore-nvmeengine-1-541fc74e Recovery
Recovering objectscale/dirt-objectstore-nvmeengine-1 17m

7. Once the recovery is completed, the status changes to "ReadytoEject".

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
ReadyToEject 69d56273-b15b-49d5-bd65-b75b525b3425 128m

8. When the status turns "ReadytoEject", click Eject to blink the disk on the rack.
A dialogue box is displayed to confirm the eject disk procedure.
9. Click Eject disk.
The disk is ready to be physically removed.
10. Remove the disk physically.
You can physically locate the correct disk in two ways:
a. By checking the CSI-01 alert; see the Monitoring Events, Audits, and Alerts section for more details.
b. By using the following commands:
First define the drive name driveName=<drive name> , and then get the node and slot information using the below
commands:

#Node
nodeUUID=$(kubectl get drive ${driveName} -o yaml | grep NodeId | awk -F ': '

Maintain ObjectScale 295

 '{print$2}')
kubectl get csibmnode | grep $nodeUUID | awk '{print $3}'

#Slot
kubectl get drive ${driveName} -o yaml | grep Slot | awk -F ': ' '{print$2}'

Usage of the disk is changed to "Removed", and the procedure is complete.

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
Success 69d56273-b15b-49d5-bd65-b75b525b3425 138m

admin@lehi-dirt:~> kubectl get drive 69d56273-b15b-49d5-bd65-b75b525b3425 -o yaml

apiVersion: csi-baremetal.dell.com/v1
kind: Drive
metadata:
annotations:
health: bad
removal: ready
creationTimestamp: "2023-07-24T13:14:34Z"
generation: 10
labels:
.....
name: 69d56273-b15b-49d5-bd65-b75b525b3425
spec:
Firmware: 1.1.1
Health: BAD
IsClean: true
Status: ONLINE
Type: NVME
UUID: 69d56273-b15b-49d5-bd65-b75b525b3425
Usage: REMOVED
VID: "0x1179"

Reactive Disk Removal Service Procedure (for Appliance and Software Bundle)

About this task

The reactive disk removal service procedure is triggered automatically within ObjectScale when a disk fails.

Steps
1. CSI marks the status of failed disks from IN_USE to RELEASING.
2. The cluster operation CR gets created.

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
TriggerRemoval 69d56273-b15b-49d5-bd65-b75b525b3425 19s

3. The object store status moves to "ReplacingPV" .

admin@lehi-dirt:~> kubectl get ecs -A

NAMESPACE NAME PHASE READY COMPONENTS S3 ENDPOINT
MGMT API
objectscale dirt-objectstore ReplacingPV 21/21
10.249.248.25:443 10.249.248.27:4443

NOTE: This change is applicable only for SS pods. For non-SS pods, the object store status does not change.

296 Maintain ObjectScale

4. The recovery of the disk starts.

admin@lehi-dirt:~> kubectl get serviceprocedure -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale recovery-dirt-objectstore-nvmeengine-1-541fc74e Recovery
Recovering objectscale/dirt-objectstore-nvmeengine-1 17m

5. Once the recovery is completed, the status changes to "ReadytoEject".

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
ReadyToEject 69d56273-b15b-49d5-bd65-b75b525b3425 128m

6. When the status turns "ReadytoEject", clickEject to blink the disk on the rack.
A dialogue box is displayed to confirm the eject disk procedure.
7. Click Eject disk.
The disk is ready to be physically removed.
8. Remove the disk physically.
You can physically locate the correct disk in two ways:
a. By checking the CSI-01 alert; see the Monitoring Events, Audits, and Alerts section for more details.
b. By using the following commands:
First define the drive name driveName=<drive name> , and then get the node and slot information using the below
commands:

#Node
nodeUUID=$(kubectl get drive ${driveName} -o yaml | grep NodeId | awk -F ': '
'{print$2}')
kubectl get csibmnode | grep $nodeUUID | awk '{print $3}'

#Slot
kubectl get drive ${driveName} -o yaml | grep Slot | awk -F ': ' '{print$2}'

Usage of the disk is changed to "Removed", and the procedure is complete.

admin@lehi-dirt:~> kubectl get co -A

NAMESPACE NAME
TYPE STATUS OBJECT NAME
AGE
objectscale diskremoval-cfb0478d-cacb-4c3c-b612-659701949847 DiskRemoval
Success 69d56273-b15b-49d5-bd65-b75b525b3425 138m

admin@lehi-dirt:~> kubectl get drive 69d56273-b15b-49d5-bd65-b75b525b3425 -o yaml

apiVersion: csi-baremetal.dell.com/v1
kind: Drive
metadata:
annotations:
health: bad
removal: ready
creationTimestamp: "2023-07-24T13:14:34Z"
generation: 10
labels:
.....
name: 69d56273-b15b-49d5-bd65-b75b525b3425
spec:
Firmware: 1.1.1
Health: BAD
IsClean: true
Status: ONLINE
Type: NVME
UUID: 69d56273-b15b-49d5-bd65-b75b525b3425

Maintain ObjectScale 297

 Usage: REMOVED
VID: "0x1179"

Service procedures for cluster nodes

OpenShift

Perform a node replacement service procedure (ObjectScale on OpenShift)

To manually replace a node, complete these steps. Use this process for ObjectScale instances on a Red Hat OpenShift cluster.

Prerequisites
● Ensure that the replacement node has the same name and IP address as the node being replaced.
● If the replacement process takes longer than 1 hour (which is likely), recovery begins to run for the data on the replaced
node. However, it should stop once the node is replaced and operational.

Steps
1. Prepare the node for removal:
● If the node is healthy, follow Place a node into temporary maintenance mode (ObjectScale on OpenShift) to prepare the
node to be replaced by placing the node into TMM.
● If the node is in a failed
state, follow https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.6/
html/replacing_nodes/openshift_container_storage_deployed_using_local_storage_devices#replacing-a-failed-node-on-
bare-metal-user-provisioned-infrastructure_rhocs
a. From the service node, run the following command to mark the node as unscheduable:

kubectl cordon <NODE_NAME>

b. Remove the pods in Terminating state:

kubectl get pods -A -o wide | grep-i <NODE_NAME> | awk '{if ($4 ==

"Terminating") system ("kubectl -n " $1 " delete pods " $2 " --grace-period=0
" " --force ")}'

c. Drain the node by evacuating the pods from the node:

NOTE: To list the evacuated objects without performing the operation, include the --dry-run=client
parameter with the command below.

kubectl drain <NODE_NAME> --force --delete-local-data --ignore-daemonsets

2. Remove the node from the cluster:

kubectl delete node <NODE_NAME>

3. Physically remove and replace the failed node hardware. As you do so, ensure that:
● You move all the drives from the failed node into the new compute node. Then install and join it back to the OpenShift
cluster, following the steps outlined in the OpenShift documentation.
● The new node satisfies the requirements that are listed in the "Deployment pre-requisites for ObjectScale on OpenShift"
of the Dell ObjectScale Application Installation Guide for Red Hat OpenShift.
All the PVC bindings remain, all the stateful pods start on the new node.
4. Ensure that the new node has been added to the cluster, and all nodes are ready. For example:

kubectl get nodes

NAME STATUS ROLES AGE VERSION

master0.ocp4.cmo.com Ready master 15d v1.19.0+e49167a
master1.ocp4.cmo.com Ready master 15d v1.19.0+e49167a
master2.ocp4.cmo.com Ready master 15d v1.19.0+e49167a

298 Maintain ObjectScale

 worker0.ocp4.cmo.com Ready worker 15d v1.19.0+e49167a
worker1.ocp4.cmo.com Ready worker 46m v1.19.0+e49167a
worker2.ocp4.cmo.com Ready worker 15d v1.19.0+e49167a

5. Verify that CSI recognizes the node and it appears in the Bare-Metal node list. For example:

kubectl get csibmnodes

NAME UUID ADDRESSES

csibmnode-4f19a3e9-9c9b-40a8-... 4f19a3e9-9c9b-40a8-...
{"Hostname":"master0.ocp4.cmo.com","InternalIP":"10.236.224.60"}
csibmnode-a0dba2b4-5eab-4c34-... a0dba2b4-5eab-4c34-...
{"Hostname":"worker0.ocp4.cmo.com","InternalIP":"10.236.224.66"}
csibmnode-bb7dcedc-139b-4d8f-... bb7dcedc-139b-4d8f-...
{"Hostname":"master2.ocp4.cmo.com","InternalIP":"10.236.224.64"}
csibmnode-bdb9f0b8-f52d-4aaf-... bdb9f0b8-f52d-4aaf-...
{"Hostname":"worker1.ocp4.cmo.com","InternalIP":"10.236.224.68"}
csibmnode-de3eebf0-dfcd-41e9-... de3eebf0-dfcd-41e9-...
{"Hostname":"worker3.ocp4.cmo.com","InternalIP":"10.236.224.72"}
csibmnode-e820eea6-3145-4fb8-... e820eea6-3145-4fb8-...
{"Hostname":"master1.ocp4.cmo.com","InternalIP":"10.236.224.62"}
csibmnode-f21e396b-2d91-43d5-... f21e396b-2d91-43d5-...
{"Hostname":"worker2.ocp4.cmo.com","InternalIP":"10.236.224.70"}

6. Verify that the cluster is available. For example:

kubectl get ecs

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API

ecs-cluster Available 23/23 10.236.228.53:443 10.236.228.52:4443

7. Verify other features and components, including:

● S3 I/O.
● ObjectScale Portal.
● Kubectl command output.
● All pods, including any previously in the pending state, are now running.
● Pod restarts have not occurred or increased.

Software Bundle

Add a node (ObjectScale Software Bundle)

Follow this Node Addition service procedure to add a node to the ObjectScale Software Bundle cluster.

Prerequisites
Ensure that the new node is:
● Installed the same operating system version and networking
● Configured in a consistent manner as the other nodes within the ObjectScale Software Bundle

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)

Maintain ObjectScale 299

 export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-
urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. On the node, create the scaleup.json file with the necessary details for the node.
NOTE: When a node is added to a cluster, a situation may occur whereby the /etc/hosts file for the added node is
not updated correctly, which causes issues when the cluster is upgraded. To avoid failures during the upgrade process,
perform the following steps after adding a node:
a. Retrieve the helmrepo service IP address.

kubectl -n cmo get svc helmrepo

For example:

kubectl -n cmo get svc helmrepo

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helmrepo ClusterIP 172.43.174.187 <none> 30036/TCP 12d

b. Add an entry for the service to the /etc/hosts file of the added node. For example:

<CLUSTER_IP> helmrepo

For example:

172.43.174.187 helmrepo

Place this JSON payload in the node where we are going to perform the scale up of the node.

{
"credentials": [{
"name": "<HOSTNAME>",
"type": "password",
"password": "<PASSWORD>"
}],
"hosts": [{
"hostname": "<NODE_HOSTNAME>",
"managementhost": "<HOST_IP>",
"kuberneteshost": "<HOST_IP>",
"hostCredentials": "<HOST_CREDS>",
"topology": {
"role": "controlplane" or "worker"
}
}]
}

For example:

{
"credentials": [{
"name": "mykey1",

300 Maintain ObjectScale

 "type": "password",
"password": "ChangeMe"
}],
"hosts": [{
"hostname": "hostname6",
"managementhost": "10.236.227.213",
"kuberneteshost": "10.236.227.213",
"hostCredentials": "mykey1",
"topology": {
"role": "controlplane"
}
}]
}

4. Call the CMO Platform Manager API to initiate the scaling-up operation.
Run this command from the directory where the scaleup.json file exists.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://10.43.78.77:7070/v3/clusters/
nodes -v -k | json_pp
......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "286bdb32-ff07-4e46-947e-e4c9e9b98338",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/286bdb32-ff07-4e46-947e-e4c9e9b98338",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "scale"
}

5. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is 286bdb32-ff07-4e46-947e-e4c9e9b98338.
6. After performing the scale up API, check the status of the operation:
NOTE: The CMO Platform Manager TOKEN may expire, and need to be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

7. Confirm that the new node appears in the node list.

kubectl get node

Maintain ObjectScale 301

Results
You have successfully added the node to the ObjectScale Software Bundle. It can now be added to an existing object store
using the ObjectScale horizontal expansion service procedure.

Remove a node (ObjectScale Software Bundle)

Follow this Node Removal service procedure to remove a node from the ObjectScale Software Bundle cluster.

Prerequisites
Ensure that the ObjectScale Software Bundle is equipped with a spare node with enough drives for the pods. This spare node
receives the contents of the node that is placed into permanent maintenance mode.

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)
export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-
urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Apply a taint to the node to be removed:

kubectl taint node <NODE_NAME> node.dell.com/drain=drain:NoSchedule

4. View the status of the PMM service procedure.

Once the taint has been applied to a node, the ObjectScale Operator creates a PMM service procedure. Retrieve the list of
service procedures and locate the PMM service procedure with pmm- prefixed to the service procedure name.

kubectl get serviceprocedures

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

NOTE: Do not delete the service procedure while it is running.

302 Maintain ObjectScale

5. Monitor the status of the service procedure with the following command:

while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure transitions through various phases as it progresses. The Reason value for the PMM service procedure
should progress from NotStarted, In Progress, PostCheck, and finally to Success. A reason of Success indicates
that the service procedure has completed without error, and the node is now in PMM.

6. Once the PMM service procedure is successful, place the node into maintenance mode within the CMO Platform within the
ObjectScale Software Bundle.

kubectl cordon <NODE_NAME>

7. Safely evict all your pods from the node:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

8. Verify the status of the drained node:

kubectl get node <NODE_NAME>

NAME STATUS ROLES AGE VERSION

hostname6 Ready,SchedulingDisabled <none> 6d19h v1.24.7+rke2r1

9. Verify that all CMO component pods, except the DaemonSet-managed Pods, have been rescheduled to the other nodes.

kubectl get pod -n cmo -o wide | grep <NODE_NAME>

10. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle.
Place this JSON payload in the node where you are going to perform the scale down of the node.

{
"hosts": [{
"hostname": "<NODE_HOSTNAME>"
}],
"remove_os_packages": "true"
}

NOTE: If the remove_os_packages parameter is set to true, the OS packages are removed from the node. This
precludes the user from adding the node back to the cluster without reinstalling those OS packages.
For example:

{
"worker": [{
"hostname": "hostname6",
}],
"remove_os_packages": "true"
}

11. Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even
though the scale down happens successfully.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

Maintain ObjectScale 303

 For example:

......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "remove-node"
}

12. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
13. After performing the scale down API, check the status of the operation through the API below:
NOTE: The CMO Platform Manager TOKEN may expire, and be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

NOTE: In certain situations, the status may show as Failed when the failure node was removed successfully. Check
the node status.

14. Confirm that the node has been removed from the node list.

kubectl get node

Prepare a failed node for node hardware or software maintenance (ObjectScale

Software Bundle)
Follow this Node Reparation procedure for a failed node.

About this task

Stateful data on the node to be repaired are not deleted, and nor are stateful ObjectScale pods rescheduled to other nodes in
the cluster.

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

304 Maintain ObjectScale

 a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)
export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-
urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Safely evict all your pods from the node:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

For example:

# kubectl drain hostname6 --ignore-daemonsets --delete-emptydir-data --force

WAINING: ignoring DaemonSet-managed Pods: cmo/metallb-speaker-ggrkq, cmo/whereabout-
whereabouts-58dss, calico-system/calico-node-8rmcp, default/csi-baremetal-node-9wg5w,
kube-system/rke2-ingress-nginx-controller-zvbnw, kube-system/rke2-multus-ds-5bv2x
evicting pod cmo/decks-support-store-0
pod/decks-support-store-0 evicted
node/hostname6 drained

4. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle.
Place this JSON payload in one of the controlplane nodes where you will to perform the scale down of the node.

{
"hosts": [{
"hostname": "<NODE_HOSTNAME>"
}],
"remove_os_packages": "false"
}

5. Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even
though the scale down happens successfully.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {

Maintain ObjectScale 305

 "hosts" : {}
},
"id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "remove-node"
}

6. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
7. After performing the scale down API, check the status of the operation through the API below:
NOTE: The CMO Platform Manager TOKEN may expire, and be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

NOTE: In certain situations, the status may show as Failed when the failure node was removed successfully. Check
the node status.

8. Confirm that the node has been removed from the node list.

kubectl get node

9. Verify that the statefulset pods have move to Pending state after node removal:

kubectl get pods -o wide | grep -v Running

10. Fix the node while it is offline, and then go to the next step.
11. On the node, create the scaleup.json file with the necessary details for the node.
NOTE: When a node is added to a cluster, a situation may occur whereby the /etc/hosts file for the added node is
not updated correctly, which causes issues when the cluster is upgraded. To avoid failures during the upgrade process,
perform the following steps after adding a node:
a. Retrieve the helmrepo service IP address.

kubectl -n cmo get svc helmrepo

For example:

kubectl -n cmo get svc helmrepo

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helmrepo ClusterIP 172.43.174.187 <none> 30036/TCP 12d

b. Add an entry for the service to the /etc/hosts file of the added node. For example:

<CLUSTER_IP> helmrepo

306 Maintain ObjectScale

 For example:

172.43.174.187 helmrepo

Place this JSON payload in the node where we are going to perform the scale up of the node.

{
"credentials": [{
"name": "<HOSTNAME>",
"type": "password",
"password": "<PASSWORD>"
}],
"hosts": [{
"hostname": "<NODE_HOSTNAME>",
"managementhost": "<HOST_IP>",
"kuberneteshost": "<HOST_IP>",
"hostCredentials": "<HOST_CREDS>",
"topology": {
"role": "controlplane" or "worker"
}
}]
}

For example:

{
"credentials": [{
"name": "mykey1",
"type": "password",
"password": "ChangeMe"
}],
"hosts": [{
"hostname": "hostname6",
"managementhost": "10.236.227.213",
"kuberneteshost": "10.236.227.213",
"hostCredentials": "mykey1",
"topology": {
"role": "controlplane"
}
}]
}

12. Call the CMO Platform Manager API to initiate the scaling-up operation.
Run this command from the directory where the scaleup.json file exists.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://10.43.78.77:7070/v3/clusters/
nodes -v -k | json_pp
......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "286bdb32-ff07-4e46-947e-e4c9e9b98338",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/286bdb32-ff07-4e46-947e-e4c9e9b98338",
"rel" : "self"
},
"logs" : "",
"state" : "created",

Maintain ObjectScale 307

 "updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "scale"
}

13. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is 286bdb32-ff07-4e46-947e-e4c9e9b98338.
14. After performing the scale up API, check the status of the operation:
NOTE: The CMO Platform Manager TOKEN may expire, and need to be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

15. Confirm that the new node appears in the node list.

kubectl get node

16. Verify that pods can be rescheduled to this node:

kubectl get pod -A -o wide | grep <NODE_NAME>

Prepare a healthy node for node hardware or software maintenance (ObjectScale

Software Bundle)
Follow this Node Reparation procedure to repair a healthy node to fix a system disk, issues with node hardware or software, or
upgrade the node Operating System.

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)
export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-

308 Maintain ObjectScale

 urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Apply a taint to the node to be placed into ObjectScale temporary maintenance mode:

kubectl taint node <NODE_NAME> node.dell.com/drain=planned-downtime:NoSchedule

4. Verify that the PHASE of the cluster now displays Maintenance.

kubectl -n <OBJECTSCALE_NAMESPACE> get ecs-cluster

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API

ecs-cluster Maintenance 22/23 10.236.228.53:443 10.236.228.52:4443

The ObjectScale Portal UI shows the object store status as Maintenance.

5. Once the taint has been applied to a node, the ObjectScale Operator creates the ObjectScale TMM service procedure.
Retrieve the list of service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure
name:

kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

NOTE: Do not delete the service procedure while it is running.

6. Monitor the status of the service procedure with the following command:

while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure transitions through various phases as it progresses. The Reason value for the TMM service
procedure should progress from NotStarted, In Progress, PostCheck, Waiting, and finally to Success. A reason
of Success or Waiting indicates that the service procedure has completed without error, and the node is now in TMM.

7. Next, place the node into maintenance mode within the CMO Platform within the ObjectScale Software Bundle.

kubectl cordon <NODE_NAME>

8. Safely evict all your pods from the node:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

For example:

# kubectl drain hostname6 --ignore-daemonsets --delete-emptydir-data --force

WAINING: ignoring DaemonSet-managed Pods: cmo/metallb-speaker-ggrkq, cmo/whereabout-
whereabouts-58dss, calico-system/calico-node-8rmcp, default/csi-baremetal-node-9wg5w,
kube-system/rke2-ingress-nginx-controller-zvbnw, kube-system/rke2-multus-ds-5bv2x
evicting pod cmo/decks-support-store-0
pod/decks-support-store-0 evicted
node/hostname6 drained

Maintain ObjectScale 309

9. Verify the status of the drained node:

kubectl get node <NODE_NAME>

For example:

# kubectl get node hostname6

NAME STATUS ROLES AGE VERSION
hostname14 Ready,SchedulingDisabled <none> 6d19h v1.24.7+rke2r1

10. Verify that all CMO component pods have been rescheduled to the other nodes.

kubectl get pod -n cmo | grep Pending

11. Verify the ObjectScale Portal UI shows that the node has entered TMM by reviewing the Monitoring > Issues tab.
12. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle
cluster.
Place this JSON payload in a controlplane node where you are going to perform the scale down of the node.

{
"hosts": [{
"hostname": "<NODE_HOSTNAME>"
}],
"remove_os_packages": "false"
}

13. Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even
though the scale down happens successfully.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "remove-node"
}

14. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
15. After performing the scale down API, check the status of the operation through the API below:
NOTE: The CMO Platform Manager TOKEN may expire, and be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --

310 Maintain ObjectScale

 data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

NOTE: In certain situations, the status may show as Failed when the failure node was removed successfully. Check
the node status.

16. Confirm that the node has been removed from the node list.

kubectl get node

17. Perform any necessary maintenance on the node.

18. On the node, create the scaleup.json file with the necessary details for the node.
NOTE: When a node is added to a cluster, a situation may occur whereby the /etc/hosts file for the added node is
not updated correctly, which causes issues when the cluster is upgraded. To avoid failures during the upgrade process,
perform the following steps after adding a node:
a. Retrieve the helmrepo service IP address.

kubectl -n cmo get svc helmrepo

For example:

kubectl -n cmo get svc helmrepo

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helmrepo ClusterIP 172.43.174.187 <none> 30036/TCP 12d

b. Add an entry for the service to the /etc/hosts file of the added node. For example:

<CLUSTER_IP> helmrepo

For example:

172.43.174.187 helmrepo

Place this JSON payload in the node where we are going to perform the scale up of the node.

{
"credentials": [{
"name": "<HOSTNAME>",
"type": "password",
"password": "<PASSWORD>"
}],
"hosts": [{
"hostname": "<NODE_HOSTNAME>",
"managementhost": "<HOST_IP>",
"kuberneteshost": "<HOST_IP>",
"hostCredentials": "<HOST_CREDS>",
"topology": {
"role": "controlplane" or "worker"
}
}]
}

For example:

{
"credentials": [{
"name": "mykey1",
"type": "password",

Maintain ObjectScale 311

 "password": "ChangeMe"
}],
"hosts": [{
"hostname": "hostname6",
"managementhost": "10.236.227.213",
"kuberneteshost": "10.236.227.213",
"hostCredentials": "mykey1",
"topology": {
"role": "controlplane"
}
}]
}

19. Call the CMO Platform Manager API to initiate the scaling-up operation.
Run this command from the directory where the scaleup.json file exists.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://10.43.78.77:7070/v3/clusters/
nodes -v -k | json_pp
......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "286bdb32-ff07-4e46-947e-e4c9e9b98338",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/286bdb32-ff07-4e46-947e-e4c9e9b98338",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "scale"
}

20. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is 286bdb32-ff07-4e46-947e-e4c9e9b98338.
21. After performing the scale up API, check the status of the operation:
NOTE: The CMO Platform Manager TOKEN may expire, and need to be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

22. Confirm that the new node appears in the node list.

kubectl get node

312 Maintain ObjectScale

Replace a healthy node within the ObjectScale Software Bundle
Use this Node Replacement service procedure to replace a healthy node which has not failed, with issues including but not
restricted to the system disk failure or OS issues.

Prerequisites
Ensure that the new node has the same OS version and networking configuration as the other nodes within the ObjectScale
Software Bundle.

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)
export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-
urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Scale up a node (hostname6) with reference (hostref) to the old node.

Place this JSON payload in the node where you are going to perform the scale up of the node.

{
"credentials": [{
"name": "<HOSTNAME>",
"type": "password",
"password": "<PASSWORD>"
}],
"hosts": [{
"hostname": "<NODE_HOSTNAME>",
"managementhost": "<HOST_IP>",
"kuberneteshost": "<HOST_IP>",
"hostCredentials": "<HOST_CREDS>",
"topology": {
"hostref": "<OLD_NODE_NAME>" // This is only for replacing old node, the
resource from the old node will be preferentially schedule to the new node
}
}]
}

Maintain ObjectScale 313

 For example:

{
"credentials": [{
"name": "mykey1",
"type": "password",
"password": "ChangeMe"
}],
"hosts": [{
"hostname": "lehi-enterprise",
"managementhost": "10.236.227.213",
"kuberneteshost": "10.236.227.213",
"hostCredentials": "mykey1",
"hostref": "hostname4"
}
}]
}

4. Call the CMO Platform Manager API to initiate the scaling-up operation.
Run this command from the directory where the scaleup.json file exists.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://10.43.78.77:7070/v3/clusters/
nodes -v -k | json_pp
......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "286bdb32-ff07-4e46-947e-e4c9e9b98338",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/286bdb32-ff07-4e46-947e-e4c9e9b98338",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "scale"
}

5. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is 286bdb32-ff07-4e46-947e-e4c9e9b98338.
6. After performing the scale up API, check the status of the operation:
NOTE: The CMO Platform Manager TOKEN may expire, and need to be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

314 Maintain ObjectScale

7. Confirm that the new node appears in the node list.

kubectl get node

8. Confirm that daemonset pods are scheduled on the new node.

kubectl get pod -A -o wide | grep <NODE_NAME>

9. Apply a taint to the node to be removed:

kubectl taint node <NODE_NAME> node.dell.com/drain=drain:NoSchedule

10. View the status of the PMM service procedure.

Once the taint has been applied to a node, the ObjectScale Operator creates a PMM service procedure. Retrieve the list of
service procedures and locate the PMM service procedure with pmm- prefixed to the service procedure name.

kubectl get serviceprocedures

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n <OBJECTSCALE_NAMESPACE> describe serviceprocedures <SP_NAME>

NOTE: Do not delete the service procedure while it is running.

11. Monitor the status of the service procedure with the following command:

while true; do kubectl -n <OBJECTSCALE_NAMESPACE> get serviceprocedures -o custom-

columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage
dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure transitions through various phases as it progresses. The Reason value for the PMM service procedure
should progress from NotStarted, In Progress, PostCheck, and finally to Success. A reason of Success indicates
that the service procedure has completed without error, and the node is now in PMM.

12. Once the PMM service procedure is successful, place the node into maintenance mode within the CMO Platform within the
ObjectScale Software Bundle.

kubectl cordon <NODE_NAME>

13. Safely evict all your pods from the node:

kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force

14. Verify the status of the drained node:

kubectl get node <NODE_NAME>

NAME STATUS ROLES AGE VERSION

hostname6 Ready,SchedulingDisabled <none> 6d19h v1.24.7+rke2r1

15. Verify that all CMO component pods, except the DaemonSet-managed Pods, have been rescheduled to the other nodes.

kubectl get pod -n cmo -o wide | grep <NODE_NAME>

16. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle.
Place this JSON payload in the node where you are going to perform the scale down of the node.

{
"hosts": [{
"hostname": "<NODE_HOSTNAME>"
}],

Maintain ObjectScale 315

 "remove_os_packages": "true"
}

NOTE: If the remove_os_packages parameter is set to true, the OS packages are removed from the node. This
precludes the user from adding the node back to the cluster without reinstalling those OS packages.
For example:

{
"worker": [{
"hostname": "hostname6",
}],
"remove_os_packages": "true"
}

17. Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even
though the scale down happens successfully.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "remove-node"
}

18. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
19. After performing the scale down API, check the status of the operation through the API below:
NOTE: The CMO Platform Manager TOKEN may expire, and be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

NOTE: In certain situations, the status may show as Failed when the failure node was removed successfully. Check
the node status.

316 Maintain ObjectScale

20. Confirm that the node has been removed from the node list.

kubectl get node

21. Verify that the pods from the removed node have been rescheduled to the new node.

kubectl get pod -n cmo -o wide | grep <NEW_NODE_NAME>

Replace a failed node within the ObjectScale Software Bundle

Use this Node Replacement service procedure to replace a failed node.

Prerequisites
Ensure that the new node has the same operating system version and networking configuration as the other nodes within the
ObjectScale Software Bundle. Ensure that the system has an extra FTT quota, that is, if the system is FTT=1, ensure that there
are no extra nods down. If the system is FTT=2, the other down node size is <=1. Ensure that there are no other ongoing service
procedures or recoveries.
NOTE: If this FTT requirement is not met, do not proceed with these steps; call Dell Support.

About this task

When a node goes to failure, all pods on that node turn to terminating state. Stateless pods would be rescheduled to another
available node after five minutes, and stateful pods would keep terminating.

Steps
1. The ObjectScale Software Bundle CMO Platform Manager APIs require a keycloak token to authenticate the requests for
cluster management tasks.
The ObjectScale Software Bundle contains a CMO Platform Manager running on Kubernetes within the cluster that is used
to request cluster management tasks, like service procedures.

a. Collect the keycloak account information from the secret:

export KEYCLOAK_USER=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json | jq

-r '.data["keycloak-username"]' | base64 --decode)
export KEYCLOAK_PASSWORD=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json
| jq -r '.data["keycloak-password"]' | base64 --decode)
export KEYCLOAK_REALM=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-realm"]' | base64 --decode)
export KEYCLOAK_CLIENT=$(kubectl get secret keycloak-pm-auth-info -n cmo -o json |
jq -r '.data["keycloak-client"]' | base64 --decode)
export KEYCLOAK_CLIENT_SECRET=$(kubectl get secret keycloak-pm-auth-info -n cmo -o
json | jq -r '.data["keycloak-credentials-secret"]' | base64 --decode)

b. Set an environment variable for the access token:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --data-
urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-urlencode
password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

2. Collect the IP address of the CMO Platform Manager.

kubectl get services -n cmo platform-manager -o jsonpath='{.spec.clusterIP}'

3. Create the scaledown.json with the details of the node that you are removing from the ObjectScale Software Bundle.

Maintain ObjectScale 317

 Place this JSON payload in the node where you are going to perform the scale down of the node.

{
"hosts": [{
"hostname": "<NODE_HOSTNAME>"
}],
"remove_os_packages": "true"
}

NOTE: If the remove_os_packages parameter is set to true, the OS packages are removed from the node. This
precludes the user from adding the node back to the cluster without reinstalling those OS packages.
For example:

{
"worker": [{
"hostname": "hostname6",
}],
"remove_os_packages": "true"
}

4. Scale down the node using the CMO Platform Manager scale down API.
NOTE: If the node is unreachable (the logs read "Unreachable=1"), a scale down operation would report failure, even
though the scale down happens successfully.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request DELETE --data @scaledown.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {
"hosts" : {}
},
"id" : "ac2324c5-0112-45f3-83e9-4f018d24ca57",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/ac2324c5-0112-45f3-83e9-4f018d24ca57",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "remove-node"
}

5. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is ac2324c5-0112-45f3-83e9-4f018d24ca57.
6. After performing the scale down API, check the status of the operation through the API below:
NOTE: The CMO Platform Manager TOKEN may expire, and be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

318 Maintain ObjectScale

 NOTE: In certain situations, the status may show as Failed when the failure node was removed successfully. Check
the node status.

7. Confirm that the node has been removed from the node list.

kubectl get node

8. Scale up a node with reference to the old, removed node.

Place this JSON payload in the node where you are going to perform the scale up of the node.

{
"credentials": [{
"name": "<HOSTNAME>",
"type": "password",
"password": "<PASSWORD>"
}],
"hosts": [{
"hostname": "<NODE_HOSTNAME>",
"managementhost": "<HOST_IP>",
"kuberneteshost": "<HOST_IP>",
"hostCredentials": "<HOST_CREDS>",
"topology": {
"role": "<REMOVED_NODE_ROLE>"
}
}]
}

For example:

{
"credentials": [{
"name": "mykey1",
"type": "password",
"password": "ChangeMe"}],
"hosts": [{
"hostname": "hostname6",
"managementhost": "10.236.227.214",
"kuberneteshost": "10.236.227.214",
"hostCredentials": "mykey1",
"topology": {
"role": "worker"}
}]
}
}

9. Confirm that daemonset pods are scheduled on the new node.

kubectl get pod -A -o wide | grep <NODE_NAME>

10. Call the CMO Platform Manager API to initiate the scaling-up operation.
Run this command from the directory where the scaleup.json file exists.

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://<CMO_PLATFORM_MANAGER_IP>/v3/
clusters/nodes -v -k | json_pp

For example:

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request POST --data @scaleup.json https://10.43.78.77:7070/v3/clusters/
nodes -v -k | json_pp
......
{
"created_at" : "2023-04-15T11:35:35Z",
"completed_tasks" : 0,
"total_tasks" : 273,
"recap" : {

Maintain ObjectScale 319

 "hosts" : {}
},
"id" : "286bdb32-ff07-4e46-947e-e4c9e9b98338",
"link" : {
"href" : "https://0.0.0.0:8080/v1/status/286bdb32-ff07-4e46-947e-e4c9e9b98338",
"rel" : "self"
},
"logs" : "",
"state" : "created",
"updated_at" : "2023-04-15T11:35:36Z",
"playbook_id" : "scale"
}

11. Collect the "id" value from the returned output. You will use this value in the next step.
For previous example, the "id" value is 286bdb32-ff07-4e46-947e-e4c9e9b98338.
12. After performing the scale up API, check the status of the operation:
NOTE: The CMO Platform Manager TOKEN may expire, and need to be refreshed by running:

export TOKEN=$(curl -L -X POST https://keycloak-http.atlantic/auth/realms/

$KEYCLOAK_REALM/protocol/openid-connect/token -H 'Content-Type: application/x-www-
form-urlencoded' --data-urlencode client_id=$KEYCLOAK_CLIENT --data-urlencode
'grant_type=password' --data-urlencode client_secret=$KEYCLOAK_CLIENT_SECRET --
data-urlencode 'scope=openid' --data-urlencode username=$KEYCLOAK_USER --data-
urlencode password=$KEYCLOAK_PASSWORD | jq -r '.access_token')

curl --header "Content-Type: application/json" --header "Authorization: Bearer

$TOKEN" --request GET https://<CMO_PLATFORM_MANAGER_IP>/v1/status/<ID> -k | jq

When the operation is finished, the operation "state" is marked as "complete".

13. Confirm that the new node appears in the node list.

kubectl get node

NOTE: Although the status of this operation may appear as failed, but the failure node could be removed successfully.
Check the node status.
14. Delete the PVC, volumes, and LVGs of stateful pods on the removed node.
Retrieve all PVCs bound to the node to be removed.
NOTE: The node name is listed as part of the volume.kubernetes.io/selected-node annotation in the
describe output of each PVC.

The PVC names and the described details are obtained with the following commands.
a. Get PVC names:

kubectl get pvc

b. Get the details for each listed PVC:

kubectl describe pvc <PVC_NAME>

c. Get the node for each listed PVC:

PVCs are namespace-scoped resources. Repeat this step for all namespaces used by ObjectScale.

for i in `kubectl get pvc --no-headers -o jsonpath="{.items[*].metadata.name}"`; do

echo "=== $i"; kubectl get pvc $i -o json | grep selected-node | grep -v "{}"; done

320 Maintain ObjectScale

 d. Patch and remove volumes:

kubectl get volume | grep <NODE_ID> | awk '{print $1}' | xargs kubectl patch volume
--type merge -p '{"metadata":{"finalizers":null}}'

kubectl get volume | grep <NODE_ID> | awk '{print $1}' | xargs kubectl delete volume

e. Patch and remove volumes:

kubectl get lvg | grep <NODE_ID> | awk '{print $1}' | xargs kubectl patch lvg --
type merge -p '{"metadata":{"finalizers":null}}'

kubectl get lvg | grep <NODE_ID> | awk '{print $1}' | xargs kubectl delete lvg

15. Clean up CSI resources.

More cleanup steps are required for failed node removal, and may otherwise be required if the PMM procedure fails.
a. Delete CSI Bare-Metal Node:

kubectl get csibmnode | grep <NODE_ID> | awk '{print $1}' | xargs kubectl delete
csibmnode

b. Delete Drive CRs:

kubectl get drive | grep <NODE_ID> | awk '{print $1}' | xargs kubectl delete drive

c. Delete Available Capacity:

kubectl get ac | grep <NODE_ID> | awk '{print $1}' | xargs kubectl delete ac

16. Verify Resource Removal.

a. Check for CSI Bare-Metal Node:

kubectl get csibmnodes | grep <NODE_ID>

b. Check for available capacity:

kubectl get ac | grep <NODE_ID>

c. Check for drive CRs:

kubectl get drive | grep <NODE_ID>

17. Delete pending stateful pods.

kubectl get pods -o wide -A | grep Pending

NOTE: After the removal of a failed node, there may be some pods left in the Pending state. These are likely
StatefulSet pods that were previously running on the removed node. This includes SS, influxdb, bookie, and atlas pods.
Once deleted, they, along with their associated volumes, are re-created on another available node.

Maintain ObjectScale 321

Appliance

Add a node (ObjectScale Appliance)

This section describes the Node Addition service procedure to add a node to the ObjectScale Appliance cluster.

Prerequisites
Ensure that the new node is:
● Installed with the same operating system version and networking.
NOTE: If there is a mismatch in the version of the operating system, contact Dell Support for guidance on reimaging the
operating system to the correct version.
● Configured in a consistent manner as the other nodes within the ObjectScale Appliance.
If you are adding the deployment node to the cluster, stop the containers in the deployment node with cd /var/atlantic/
files and sudo sh stopcontainers.sh commands before adding it.

Steps
1. From the ObjectScale Portal user interface, click Nodes.
The list of Nodes that the user is authorized to view is displayed.
2. Click Add.
The Add Node dialogue box is displayed.
3. In the General section, complete the required fields, and click Next.

Option Description
Node Name Enter a name for the new node.
NOTE:
● Node name can include ASCII(7) letters from "a to z", numbers from "0-9", and hyphen.
● Node name cannot start or end with a hyphen.

Host IP Enter the reachable Host IP.

Role Choose a role for the new node from Worker or Control Plane options.
4. In the Labels section, optionally add up to five labels by completing the required fields, and click Next.

Option Description
Name Enter a name for the label. Label should follow the Kubernetes label syntax. See "Labels and Selectors" section in
Kubernetes documentation for details.
Value Enter a value for the label.

● Click Add Label to add another label.

● Click Delete to delete a label.
5. In the Self Encrypting Drive section, add the required configuration details, and click Next.

Option Description
KMIP Details Enter Username, Password, and FQDN. Provide the username and password for an iDRAC user that is
used to create and access keys from the external Key Management Server (KMS). The FQDN is the fully
qualified domain name of the KMS server.
KMS Server Enter KMS Admin Username and KMS Admin Password. Provide credentials for an admin on the
Details external KMS. The admin must have User admin and CA cert admin capabilities to create a user and sign
certificates.
6. In the Review section, confirm the configuration summary, and click Save.
● Click Edit to make changes.
The node addition progress can be monitored from the Nodes section in the ObjectScale portal user interface.

322 Maintain ObjectScale

Results
The node is successfully added to the ObjectScale Appliance.

Next steps
If iDRAC IP is not already configured for the newly added node, it must be configured manually using server patch APIs. See
Updating iDRAC IPs Using Server Patch API for ObjectScale Appliance for steps.

Node Replacement on ObjectScale Appliance

This section describes the service procedure to replace a node on ObjectScale Appliance.

About this task

Steps
1. Add a new node, if there are no spare nodes in the cluster.
See the "Add a Node on ObjectScale Appliance" service procedure above for steps.
2. Remove the node to be replaced.
a. From the ObjectScale Portal user interface, click Nodes.
The list of nodes you are authorized to view is displayed.
b. Select the node to replace and click Remove.
A dialogue box is displayed to acknowledge the risks and confirm node removal.
c. Select Remove Data Disk and I understand the risk of removing the node check boxes, and click
Remove Node.
3. Replace a failed node.
See the "Replace a failed node within the ObjectScale Software Bundle" service procedure for steps.

Node Repair on ObjectScale Appliance

This section describes the node repair procedure on ObjectScale Appliance.

Prerequisites

Steps
1. Remove the node to be repaired.
a. From the ObjectScale Portal user interface, click Nodes.
The list of nodes you are authorized to view is displayed.
b. Select the node to repair and click Remove.
A dialogue box is displayed to acknowledge the risks and confirm node removal.
c. Select the I understand the risk of removing the node check box, and click Remove Node.
NOTE: In this case, do not check the Remove Data Disk checkbox.

2. Repair the failed node.

3. Add the repaired node back.
If the node addition procedure is stuck at the DiscoverInventory stage, check node discovery status using kubectl
get server -A. If the status is Failed, perform the following steps:
a. Reset iDRAC.
b. Reset hosa and ism service on the node.

sudo systemctl status hosa.service dcismeng.service

sudo systemctl restart hosa.service dcismeng.service

c. Delete the node addition CR that is stuck at DiscoverInventory phase.

Maintain ObjectScale 323

 d. Edit the CPC CR, remove the entry of the node in the Spec.Nodelist.
e. Retry the node addition procedure through API or the ObjectScale Portal user interface.

Failed Node Repair on ObjectScale Appliance

This section describes the failed node repair procedure on ObjectScale Appliance.

Steps
1. Remove the node to be repaired.
a. From the ObjectScale Portal user interface, click Nodes.
The list of nodes you are authorized to view is displayed.
b. Select the node to repair and click Remove.
A dialogue box is displayed to acknowledge the risks and confirm node removal.
c. Select the I understand the risk of removing the node check box, and click Remove Node.
NOTE: In this case, do not check the Remove Data Disk checkbox.

2. Confirm that the node has been removed using kubectl get node and kubectl get co -n <namespace> |
grep NodeRemoval | grep <nodename>.
If the node is removed from the nodeList and the node removal CO is in Failed status, then follow steps 3 to 8.
3. Reimage the operating system.
4. Check the HostCfgTemplate using kubectl get hct -A.
5. Edit the HostCfgTemplate to remove the given node in serverList using kubectl edit hct -n <namespace>.
6. Monitor the HostCfgProfile for the given node by using kubectl get hcp -A.
● If the HostCfgProfile for the given node is automatically deleted, go to the next step.
● If the HostCfgProfile for the given node has a failed status such as Spec-Reset-Failed, forcibly remove the
HostCfgProfile and related resources.
7. Patch CPC to remove the node entry from Spec.nodeList.
The node entry is cleaned up from CPC.Status's nodeList
8. Add the repaired node back.
After the node is repaired and added back to the cluster, sometimes the CSI pod may not start up before other stateful
pods, and then all the volumes will be in Failed status. In such a scenario, perform the below clean up steps:
Ensure that the system has an extra FTT quota, that is if the system is FTT=1, ensure that there are no extra nods down.
If the system is FTT=2, the other down node size is <=1. Ensure that there are no other ongoing service procedures or
recoveries.
NOTE: If this FTT requirement is not met, do not proceed with these steps.

a. Get the FAILED volume information.

Note down the results, especially NAME, STORAGE_CLASS, LOCATION, and related POD NAME.

kubectl get volume -A | grep FAILED

// get the failed volume information, note down the POD NAME which is under
spec.Owners
kubectl get volume <volume name> -n <objectscale-ns> -o yaml

b. Delete all the FAILED volumes.

kubectl delete volume $(kubectl get volume -n <objectscale-ns> | grep FAILED | awk
'{print $1}') -n <objectscale-ns>

c. For volume storage class, if it is a hard drive, NVMe, or SSD, delete the pvc and pod directly.

// get pvc information, note down the NAME

kubectl get pvc -n <objectscale-ns> | grep <volume name>

// remove the finalizers of the pvc

kubectl patch pvc <pvc name> -n <objectscale-ns> -p '{"metadata":{"finalizers":
[]}}' --type=merge

324 Maintain ObjectScale

 // delete the pvc
kubectl delete pvc <pvc name> -n <objectscale-ns>

// delete the pod

kubectl delete pod <pod name> -n <objectscale-ns>

d. For LVG storage classes, first delete lvg, ac, and drive; and then delete pvc and pod.

// get LVG/AC/DRIVE CR information, note down the LVG name, LVG locations, AC name
and drive name
// lvg volumes locations are lvg names
// the location in lvg locations is the drive name related
kubectl get lvg | grep <volume location>
kubectl get ac | grep <lvg name>
kubectl get drive | grep <lvg location>

// remove the finalizers of the lvg and delete the lvg

kubectl patch lvg <lvg name> -p '{"metadata":{"finalizers":[]}}' --type=merge
kubectl delete lvg <lvg name>

// delete the ac
kubectl delete ac <ac name>

// delete the drive

kubectl delete drive <drive name>

// get pvc information, note down the NAME

kubectl get pvc -n <objectscale-ns> | grep <volume name>

// remove the finalizers of the pvc

kubectl patch pvc <pvc name> -n <objectscale-ns> -p '{"metadata":{"finalizers":
[]}}' --type=merge

// delete the pvc

kubectl delete pvc <pvc name> -n <objectscale-ns>

// delete the pod

kubectl delete pod <pod name> -n <objectscale-ns>

If the node addition procedure is stuck at the DiscoverInventory stage, check node discovery status using kubectl
get server -A. If the status is Failed, perform the following steps:
a. Reset iDRAC.
b. Reset hosa and ism service on the node.

sudo systemctl status hosa.service dcismeng.service

sudo systemctl restart hosa.service dcismeng.service

c. Delete the node addition CR that is stuck at DiscoverInventory phase.

d. Edit the CPC CR, remove the entry of the node in the Spec.Nodelist.
e. Retry the node addition procedure through API or the ObjectScale Portal user interface.

Next steps
● After the cleanup steps, wait for all pods to restart.
● Do not trigger other service procedures immediately after the cleanup as the data needs time to get balanced.
.

Maintain ObjectScale 325

Troubleshooting Service Procedures

View service procedure status

You can view the status of the service procedure with:

kubectl get serviceprocedures <NAME> -n <NAMESPACE>

Permanent Maintenance Mode service procedure status

● When the PMM service procedure is in progress:
○ Phase is changed to "Maintenance"
○ status.conditions.maintenance.type = "in progress"OR
○ status.conditions.maintenance.type = "completed" (All migrations are completed but taints still on a node)
● Once the PMM service procedure is completed:
○ Phase is changed to "Available"
○ status.conditions.maintenance.type = "success"
● If the PMM service procedure failed:
○ Alert is sent
○ status.conditions.maintenance.type = "failed"

Temporary Maintenance Mode service procedure status

● When the TMM service procedure is in progress:
○ Phase is changed to "Maintenance"
○ status.conditions.maintenance.type = "in progress"
● Once the TMM service procedure is completed:
○ "Waiting" phase occurs when the operator has completed the TMM procedure, but before the taint has been removed
from the node.
○ Then, Phase is changed to "Available"
○ status.conditions.maintenance.type = "success"
● If the TMM service procedure failed:
○ Alert is sent
○ status.conditions.maintenance.type = "failed"

Disk Replacement service procedure status

● When the replacement is in progress:
○ Phase is changed to "ReplacingPV"
○ status.conditions.replacement.type = "in progress"OR
○ status.conditions.replacement.type = "pvc deleted"(PVCs were deleted and we need to delete ss one more time)
● Once the replacement is completed:
○ Phase is changed to "Available"
○ status.conditions.replacement.type = "success"
● If the replacement failed:
○ Alert is sent
○ status.conditions.replacement.type = "failed"

Vertical expansion service procedure status

● When the vertical expansion service procedure is in progress:
○ Phase is changed to "Expansion"
○ status.conditions.expansion.type = "in progress"
● Once the vertical expansion service procedure is completed:

326 Maintain ObjectScale

 ○ Phase is changed to "Available"
○ status.conditions.expansion.type = "success"
● If the vertical expansion service procedure failed:
○ Alert is sent
○ status.conditions.expansion.type = "failed"

Horizontal expansion service procedure status

● When the horizontal expansion service procedure is in progress:
○ Phase is changed to "Expansion"
○ status.conditions.expansion.type = "in progress", status.conditions.updatingTopology.type = "in progress"
○ When new pod is Running, status.conditions.expansion.type = "succeed" and phase is changed to "UpdatingTopology"
● Once the horizontal expansion service procedure is completed:
○ Phase is changed to "Available"
○ status.conditions.expansion.type = "success"
○ status.conditions.updatingTopology.type = "success"
● If the horizontal expansion service procedure failed:
○ Alert is sent
○ status.conditions.expansion.type = "failed"
or
status.conditions.updatingTopology.type = "failed"

Service Procedure states

Retrieve details on the service procedures using:

kubectl describe serviceprocedures <SERVICE_PROCEDURE_NAME> -n <NAMESPACE>

apiVersion: ecs.dellemc.com/v1beta3
kind: ServiceProcedure
metadata:
label:
app: ecs-release-name # in case for a specific service procedure
spec:
type: Enum(PermanentMaintenanceMode) # A type of service procedure
diskInfo:
name: name of replacing disk # Contains K8s PVC Name
uuid: UUID of the replacing disk (if applicable) # On Openshift it's resolving from
the Volume CRD
nodeInfo:
name: name of replacing node # Contains K8s Node Name
uuid: UUID of the tainted node # Contains K8s Node UID
status:
reason: Enum(In Progress, Success, Failed, Recovering, Rollback, Abort) # current
actual state
message: <short message what is going on. errors for example> # message what is going
on for rightnow.

The Service Procedure custom resource (CR) can have the following states in the status.reason field:
1. Created - New SP CR recently created when the service procedure event was detected. It should have filled spec.type and
spec.diskInfo or spec.nodeInfo fields.
2. NotStarted - A state of the SP with passed pre-checks. Ready for further processing.
3. Recovering - A state applicable only for components where recovery scripts are available. SP CR is in Recovering state
after Created and before In Progress.
4. In Progress - A state of the processing SP CR. In general, this occurs after the Created state.
5. PostCheck - A state of the SP after main processing. The SP operator runs post checks until SP's post-check fails (if one
of handling pods in the Failed state) or succeed (is all handling pods in a Running'state).
6. Failed - Terminated state of the SP CR in case of any failure during SP processing or failed post-check.
7. Rejected - Terminated state of the SP CR if one of pre-checks failed and further processing is not allowed.

Maintain ObjectScale 327

8. Success - Terminated state of the succeed SP CR.

Updating iDRAC IPs Using Server Patch API for ObjectScale

Appliance
The following sections describe how to update iDRAC IP using Server Patch API after the deployment of ObjectScale Appliance.
These steps are required when:
● There is a change in iDRAC connections.
● You need to modify the iDRAC IP.
● A new node is added to the cluster post deployment (either the service node used for deployment or any other fresh node).
NOTE: After deployment, iDRAC should not be updated via external tools. All updates to follow below steps.

Set Static IP using API for iDRAC

This topic describes how to update static IP using API for iDRAC after the deployment of ObjectScale Appliance.

Steps
1. Set external static IP when iDRAC is connected to an external management network and set internal static IP when iDRAC is
connected to ObjectScale backend network.
● To set external static IP when iDRAC is connected to an external management network, use the following
command to patch:kubectl -n gc patch server <node_name> --type merge --patch-file
static_ip_patch.yaml. Below is a sample payload for setting externally accessible static iDRAC IP:

provo-enterprise:~/payload # cat static_ip_patch.yaml

spec:
bmc:
- nicSettings:
- ipV4Add:
- address: "10.236.71.25"
gateway: "10.236.71.1"
subnetMask: "255.255.255.0"
DNSFromDHCP: "Disabled"
DHCPEnable: "Disabled"
reInitialize: true
provo-enterprise:~/payload # kubectl -n gc patch server murray-enterprise --type
merge --patch-file static_ip_patch.yaml
server.mw.dell.com/murray-enterprise patched
provo-enterprise:~/payload #

● To set internal static IP when iDRAC is connected to ObjectScale backend network, use the following
command to patch:kubectl -n gc patch server <node_name> --type merge --patch-file
static_internal_ip_patch.yaml
To determine the internal iDRAC IP to be set, you can check the private IP of the node using ip addr command and
derive the iDRAC IP using the private IP as follows.
NOTE: The Private interface IP is always 169.254.<rack_id>.<100+nodeid> and it gets automatically
assigned.
If the IP assigned to the private interface is 169.254.3.108, then the iDRAC IP would be 169.254.3.158 (hint: the
last octet of iDRAC IP is 50+last octet of the private IP).
You can also determine the internal iDRAC IP to be set by executing the below script idrac.sh in /var/atlantic/
files of each node:

admin@provo-cactusjack:~> cd /var/atlantic/files/ admin@provo-cactusjack:/var/

atlantic/files> sudo sh idrac.sh
169.254.6.151/17
admin@provo-cactusjack:/var/atlantic/files>

Gateway to be provided in the payload is always 169.254.0.1and the subnet mask to be provided is always
255.255.128.0.

328 Maintain ObjectScale

 Below is a sample payload for setting internal iDRAC IP:

provo-enterprise:~/payload # cat static_internal_ip_patch.yaml

spec:
bmc:
- nicSettings:
- ipV4Add:
- address: "169.254.2.152"
gateway: "169.254.0.1"
subnetMask: "255.255.128.0"
DNSFromDHCP: "Disabled"
DHCPEnable: "Disabled"
reInitialize: true
provo-enterprise:~/payload # kubectl -n gc patch server murray-enterprise --type
merge --patch-file static_internal_ip_patch.yaml
server.mw.dell.com/murray-enterprise patched
provo-enterprise:~/payload #

2. Wait till the server status is Ready.

provo-enterprise:~/payload # kubectl get servers -A

NAMESPACE NAME POWER STATUS PROVISIONING STATUS COMMAND
EXECUTING BMCENDPOINT SUBSCRIPTION_RECONCILE_STATUS
gc layton-enterprise On Ready
NotRun
gc lehi-enterprise On Ready
NotRun
gc logan-enterprise On Ready
NotRun
gc murray-enterprise On Busy
CollectingInventory NotRun
gc ogden-enterprise On Ready
NotRun
gc orem-enterprise On Ready
NotRun
gc sandy-enterprise On Ready
NotRun
provo-enterprise:~/payload #
provo-enterprise:~/payload # kubectl get servers -A
NAMESPACE NAME POWER STATUS PROVISIONING STATUS COMMAND
EXECUTING BMCENDPOINT SUBSCRIPTION_RECONCILE_STATUS
gc layton-enterprise On Ready
NotRun
gc lehi-enterprise On Ready
NotRun
gc logan-enterprise On Ready
NotRun
gc murray-enterprise On Ready
NotRun
gc ogden-enterprise On Ready
NotRun
gc orem-enterprise On Ready
NotRun
gc sandy-enterprise On Ready
NotRun

3. Do an inventory refresh using the following command:

NOTE: This is a mandatory step after setting any static IP for iDRAC.

kubectl patch server orem-enterprise --type='json' -p='[{"op": "add", "path": "/spec/

reInitialize", "value":true}]' -n gc

4. Wait till the server status is Ready.

provo-enterprise:~/payload # kubectl get servers -A

NAMESPACE NAME POWER STATUS PROVISIONING STATUS COMMAND
EXECUTING BMCENDPOINT SUBSCRIPTION_RECONCILE_STATUS
gc layton-enterprise On Ready

Maintain ObjectScale 329

 NotRun
gc lehi-enterprise On Ready
NotRun
gc logan-enterprise On Ready
NotRun
gc murray-enterprise On Busy
CollectingInventory NotRun
gc ogden-enterprise On Ready
NotRun
gc orem-enterprise On Ready
NotRun
gc sandy-enterprise On Ready
NotRun
provo-enterprise:~/payload #
provo-enterprise:~/payload # kubectl get servers -A
NAMESPACE NAME POWER STATUS PROVISIONING STATUS COMMAND
EXECUTING BMCENDPOINT SUBSCRIPTION_RECONCILE_STATUS
gc layton-enterprise On Ready
NotRun
gc lehi-enterprise On Ready
NotRun
gc logan-enterprise On Ready
NotRun
gc murray-enterprise On Ready
NotRun
gc ogden-enterprise On Ready
NotRun
gc orem-enterprise On Ready
NotRun
gc sandy-enterprise On Ready
NotRun

5. Verify that IP is set via ipmitool.

For this, SSH to the node whose iDRAC IP needs to be verified, and execute the following command:

admin@murray-enterprise:~> sudo ipmitool lan print

Set in Progress : Set Complete
Auth Type Support : MD5
Auth Type Enable : Callback : MD5
: User : MD5
: Operator : MD5
: Admin : MD5
: OEM :
IP Address Source : Static Address
IP Address : 10.236.71.25
Subnet Mask : 255.255.255.0
MAC Address : b0:4f:13:b0:2c:d2
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 10.236.71.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : Xaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
admin@murray-enterprise:~>

330 Maintain ObjectScale

Enabling DHCP IP through API for iDRAC for ObjectScale Appliance
This topic describes how to enable DHCP IP when iDRAC is connected to an external management network.

Steps
1. Enable DHCP IP using API for iDRAC.
Command to patch: kubectl -n gc patch server <node_name> --type merge --patch-file
dhcp_patch.yaml.

provo-enterprise:~/payload # cat dhcp_patch.yaml

spec:
bmc:
- nicSettings:
- ipV4Add:
DNSFromDHCP: "Enabled"
DHCPEnable: "Enabled"
reInitialize: true
provo-enterprise:~/payload # kubectl -n gc patch server ogden-enterprise --type merge
--patch-file dhcp_patch.yaml
server.mw.dell.com/ogden-enterprise patched
provo-enterprise:~/payload #

2. Wait till the server status is Ready.

provo-enterprise:~/payload # kubectl get servers -A

NAMESPACE NAME POWER STATUS PROVISIONING STATUS COMMAND
EXECUTING BMCENDPOINT SUBSCRIPTION_RECONCILE_STATUS
gc layton-enterprise On Ready
NotRun
gc lehi-enterprise On Ready
NotRun
gc logan-enterprise On Ready
NotRun
gc murray-enterprise On Ready
NotRun
gc ogden-enterprise On Ready
NotRun
gc orem-enterprise On Ready
NotRun
gc sandy-enterprise On Ready

3. Verify that IP is set via ipmitool.

For this, SSH to the node whose iDRAC IP needs to be verified, and execute the following command:

admin@ogden-enterprise:~> sudo ipmitool lan print

Set in Progress : Set Complete
Auth Type Support : MD5
Auth Type Enable : Callback : MD5
: User : MD5
: Operator : MD5
: Admin : MD5
: OEM :
IP Address Source : DHCP Address
IP Address : 10.236.71.21
Subnet Mask : 255.255.255.0
MAC Address : b0:4f:13:b0:2c:b4
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 10.236.71.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : Xaaaaaaaaaaaaaa

Maintain ObjectScale 331

 : X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
admin@ogden-enterprise:~>

Next steps
NOTE: After setting the static/DHCP IP, values may not show up correctly in the STATUS fields of the server CR
immediately. server-refresh-inventory job runs every one hour and refreshes the status with correct details.

Shutting Down and Restarting ObjectScale

The below steps should be performed before shutting down and restarting the ObjectScale cluster.

Steps
1. Set the default namespace and alias.

alias k="kubectl -n $(kubectl get pods -A | grep -m 1 -E 'platform|pgo|helmrepo' |

cut -d' ' -f1)"
kubectl config set-context default --namespace=$(kubectl get pods -A | grep -m 1 -E
'platform|pgo|helmrepo|docker' | cut -d' ' -f1)

2. Check to ensure that the pgo controller pod and PostgreSQL database pods are up and running.

echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-

headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance"
--no-headers -o name) | xargs kubectl get -o wide

Sample output:

NAME READY STATUS RESTARTS AGE IP

NODE NOMINATED NODE READINESS GATES
pgo-6657877fdb-78hk6 1/1 Running 0 2d15h 192.168.0.192
master2.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-75d9-0 4/4 Running 0 2d15h 192.168.4.93
worker1.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-grc4-0 4/4 Running 0 2d15h 192.168.4.95
worker1.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-wm2p-0 4/4 Running 0 2d15h 192.168.4.94
worker1.ocp4.my-cluster.com <none> <none>

3. Gracefully shut down the PostgreSQL database HA cluster.

k patch $(k get postgrescluster -o name) --type merge --patch '{"spec":{"shutdown":

true}}'

4. Verify that the PostgreSQL is shut down.

echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-

headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance"
--no-headers -o name) | xargs kubectl get -o wide

Only pgo pods should be running, sample out below:

NAME READY STATUS RESTARTS AGE IP

NODE NOMINATED NODE READINESS GATES
pgo-6657877fdb-78hk6 1/1 Running 0 2d15h 192.168.0.192
master2.ocp4.cluster.com <none> <none>

5. After ensuring that the PostgreSQL pods are not running, shut down or restart the ObjectScale cluster nodes.

332 Maintain ObjectScale

6. After all ObjectScale cluster nodes are up and running, run the following command to start the PostgreSQL cluster.

k patch $(k get postgrescluster -o name) --type merge --patch '{"spec":{"shutdown":

false}}'

7. Verify that the PostgreSQL pods are up and running.

echo $(kubectl get pods -l="postgres-operator.crunchydata.com/control-plane=pgo" --no-

headers -o name && kubectl get pods -l="postgres-operator.crunchydata.com/instance"
--no-headers -o name) | xargs kubectl get -o wide

Sample output:

NAME READY STATUS RESTARTS AGE IP

NODE NOMINATED NODE READINESS GATES
pgo-6657877fdb-78hk6 1/1 Running 0 2d18h 192.168.0.192
master2.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-75d9-0 4/4 Running 0 2d18h 192.168.4.93
worker1.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-grc4-0 4/4 Running 0 2d18h 192.168.4.95
worker1.ocp4.my-cluster.com <none> <none>
postgres-ha-postgres-wm2p-0 4/4 Running 0 2d18h 192.168.4.94
worker1.ocp4.my-cluster.com <none> <none

Maintain ObjectScale 333