Rest API Developers Guide
Rest API Developers Guide
Developer’s Guide
Release 21.2
Contents
1.0 INTRODUCTION........................................................................................ 8
1.1 Prerequisites ..................................................................................................................... 8
12.0 TENANT................................................................................................... 41
12.1 Retrieve Tenant Resources ........................................................................................... 42
16.3 Determining the List of all System Audit-Event-Types grouped by type property 170
16.4 Determining What Types are Available at the System Level ................................... 172
16.5 Determining What Categorized Types are Available at the System Level ............. 174
16.7 Getting the Audits that Have Not Been Archived Yet ............................................... 180
16.8 Viewing the Audits that Have Been Archived ............................................................ 180
17.6 Creating Purge Lists Using the Generate Purge List Job ........................................ 218
18.5 Removing One or More Items from a Hold Set .......................................................... 260
23.5 Viewing job instance information for jobs that use batch ....................................... 307
Note about running the examples:You may copy/paste all the provided examples, but the JSON
Web Token cannot be copied/pasted for two reasons:
1. JSON Web Tokens do expire. JSON Web token are explained later in this guide. So you
will need to get a fresh token if you intend to run provided examples in your own
environment.
2. JSON Web Tokens included in most examples have been shortened from their original
length to keep examples shorter and clearer.
Rest-api:
GET Error! Hyperlink reference not valid.
headers:
Authorization: <xxxxxxx>
Accept:application/hal+json
A new REST API was added to the AIP collection resource to apply-retention.
This API is described in 17.4.2.
The rest API for managed-items (from the application link has been changed its
behavior)
• If no parameters are passed, the list only returns a managed item (if it
exists) for the application. Previously, it would return all the managed
items in the application
• If the type parameters are passed, the external Id must also be passed,
otherwise a Bad Request will be returned. Previously, it would return all
the managed items in the application (which was incorrect).
As shown on the diagram above, clients only interact directly with the REST
(API) Layer. Since the majority of the resources in REST API require
authorization, the REST Client first needs to obtain an
{
"name": "InfoArchive Home Resource",
"_links": {
"self": {
"href": "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info": {
"href": "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants": {
"href": "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant": {
"href": "http://localhost:8765/systemdata/tenants/41ceb11f-2943-4a4d-9fb4-
bb198509942e"
}
….. (some links removed for clarity of this example)
}
}
As displayed above, the response consists of two parts: _links node (containing
an array of link elements) and the main part of the resource. In this example, the
main part is very simple and contains a single attribute name but, in other cases,
resources will have a more extensive set of metadata/attributes.
Each link (element contained in the _links node) element consists of the
following structure:
• Link relation name (for example, self, http://identifiers.emc.com/tenants,
etc.)
• href key and its value
When traversing the InfoArchive REST API, clients typically interacting with a
given resource will be able to navigate the system further by selecting one of the
link’s elements from an available array of links and then using the value of that
link relation contained in the href key to make another call.
In some cases, a client may choose to append the URI query parameters to the
value of the href key but should never attempt to modify the path element of the
URI.
It is also worth noting that actual links that are available (and returned to the
client) are based on the client’s role, which translates directly into authorization.
To simplify client’s interaction and to remove potential ambiguity, only links that
the client is authorized to interact with, are returned. If the client is not authorized
for a given resource, it will not see its link relation. Each resource is also
checking authorization upon execution so, even if the client was able to “guess”
or “figure out” a URI to a resource, it will still not be able to interact with it if it
does not have the proper authorization level. See section 6.0 for more details on
roles and permissions.
4.1 REST and CRUD Operations
REST interactions happen over HTTP and REST leverages that protocol to
create a uniform API. The interactions with resources use more than just a
simple GET HTTP verb. In fact, the IA REST API utilizes GET, PUT, POST and
DELETE verbs.
What is safe and what is idempotent?
In REST, some methods are safe and some are idempotent. A method that is safe does not modify
the resource. This is a property that is very important for caching, for example.
If a method is idempotent, it means that the outcome on the server side will be exactly the same,
regardless of whether the method was called once or many times. This property is very important
to understand, as it will help you in situations where the connection is interrupted, for example.
Knowing that the method is idempotent will allow you to resend the request again without fear of
additional consequences.
The interaction with IA REST API over HTTP is as follows:
NOTE: The Gateway (server side part of InfoArchive Web Application aka IAWA)
component can also dispatch REST API calls to InfoArchive Server. It is
encouraged to route the REST API calls through the Gateway. To do this all the
REST API calls need to use the URL of Gateway (or a load balancer in-front of
Gateways) and prefix the REST API paths with /restapi/. For example:
http(s)://gateway-host:gateway-port/[context-path/]/restapi/services
• POST represents all other actions. This method is the most versatile. It is
also neither safe nor idempotent. Depending on the action, it may require
a payload. The Response will depend on the type of action. There is a
paradigm of creating new objects by sending a POST request to the
collection that the resource will be a member of. In such case, the payload
would be a new resource representation and the response would be 201
Created. For example, to create a new search resource, we would send a
POST request to the searches collection, passing new search
representation in the payload to the POST.
http(s)://gateway-host:gateway-port/[context-path/]/restapi/services
The Gateway uses Zuul proxy to dispatch/route the REST API calls to
InfoArchive Server. There is a 60 second timeout on the REST API calls routed
through Gateway. The timeout is configurable via the Gateway’s application.yml
file:
infoarchive/config/iawebapp/application.yml
X-SOCKET-TIMEOUT-MILLIS: 120000
There is a limitation while uploading/downloading large files going via the Gateway.
To workaround this limitation, prefix the REST API call with /zuul/ like so:
http(s)://gateway-host:gateway-port/[context-
path]/zuul/restapi/services
For an overview of the negotiation process to obtain JWT, refer to section 3.1.
Each REST call needs to be authenticated. The IA Server supports different
authentication profiles, but this guide will focus on the most popular and highly
recommended JSON Web Token (JWT). If the IA Server is configured with a
JWT profile, which is specified in server’s application.yml file, each REST
request needs to have a JWT token provided in the Authorization request
header. The format is as follows:
Request Header: Authorization
Header value: Bearer [token here]
For example, using your favorite REST client, (you should be able to use any
REST client), set the Authorization request header so that value of the
Authentication header looks like the one below:
If the Authorization header is not set, or the token is not valid, you will receive a
401-error code response from the server instead of a successful code.
Note: If you are using OTDS SSO, in this mode OTDS is acting as a OAuth2
server. The OTDS is responsible for OAuth2 clients. To define new OTDS
Clients, you have to use OTDS Admin Web Application.
Compared to obtaining a JWT for the first time, in this instance, the payload is
longer since it includes the refresh_token and the grant_type value
(refresh_token). However, the username, password client_id or scope are
not required.
The refresh_token must be properly protected and stored securely.
5.3 Putting it all Together
Now that we have basics of REST and authentication behind us, this section
documents a simple interaction example, from the client’s perspective, on how to
get to Product-Info resource. Product-info resource is an informatory resource
about the IA Server, and it is accessible directly from the Home Resource.
Once a JSON Web Token (JWT) has been obtained, the client can make a GET
request to the Home Resource. The IA Home Resource supports HAL Media
Type so we can do the following:
• Prepare a GET request to http://localhost:8765/services, setting the
Accept header to application/hal+json and Authorization header set to
JWT we just obtained.
The following is an example using curl:
NOTE: In the example above, the value of the Authorization header was shortened. Most of the
examples throughout this guide will have similar, shorter versions of the Authorization header.
• The server then returns a 200 OK Response with HAL body. The following
is an example of the response:
{
"name": "InfoArchive Home Resource",
"_links": {
"self": {
"href": "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info": {
"href": "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants": {
"href": "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant": {
"href": "http://localhost:8765/systemdata/tenants/41ceb11f-2943-4a4d-9fb4-
bb198509942e"
},
"http://identifiers.emc.com/federations": {
"href": "http://localhost:8765/systemdata/federations"
},
"http://identifiers.emc.com/xdb-databases": {
"href": "http://localhost:8765/systemdata/xdb-databases"
},
"http://identifiers.emc.com/file-system-roots": {
"href": "http://localhost:8765/systemdata/file-system-roots"
},
"http://identifiers.emc.com/crypto-objects": {
"href": "http://localhost:8765/systemdata/crypto-objects"
},
"http://identifiers.emc.com/groups": {
"href": "http://localhost:8765/systemdata/groups"
}
}
}
The next step is to parse the returned JSON data looking for a specific link
relation, such as product-info (http://identifiers.emc.com/product-info).
Then refer to the value of the href attribute of that element (for example,
http://localhost:8765/product-info).
• The following step is to issue a GET request to that URI, setting the
Accept header to application/hal+json and setting Authorization header
to the latest JWT value. The server should return a 200 OK Response with
the body describing vital server resources, including buildProperties,
systemProperties and runtimeProperties.
The following is an example of the request using curl:
• Finally, examine the _links node for potential paths that we can be
pursuing from this point on. In this example, we reached the leaf so the
only path available to us is to turn back to Home Resource.
The permissions model is based on the concept of roles and actions. There is
predefined number of roles in the system, which can be found in the server’s
application.yml file, and are displayed in the illustration above as a subset of
these. Each role is assigned several actions it supports. Again, both roles and
actions are defined in the aforementioned application.yml file.
Each individual user should be assigned a role (one or more) and these roles will
be encoded in the JSON Web Token, which happens during
Authentication/Authorization request. The system will allow/disallow the user’s
request based on her or his role membership(s). Specifically, for each request,
the server will check whether any of the roles assigned to the current user allows
for a given action. If none of the roles support the given action, the request is
denied. In most cases, the server will indicate this by responding to the request
with a HTTP error 403 – forbidden (access).
For example, only users with Developer’s role permissions can create search
resources, or only users in either the Administration or Developer roles can
create application resources.
Since most of the IA REST APIs is controlled by link relations, in some cases,
applicable link relations are only served if users own the specific role. However,
even if the link relation is present, that does not always guarantee that the given
user will be permitted to invoke certain functionality, as there are additional
There are two main types of resources: single resources and collections. What
we have seen so far (Home Resource and Product Info) are examples of single
resources. They are characterized by a simple, two-tone structure: resource itself
and _links node.
While Home and Product-Info are not part of any specific collection, the majority of Resources
in InfoArchive are usually part of some type (usually homogenous) collection.
REST has a paradigm in which we create a new item in the collection by issuing
a POST to the collection itself, where the payload contains the item to be
created. The payload should contain all of the necessary attributes for the item to
be created. Any item omitted from the payload will get its default value. If the
For resources that could have large numbers, spEL is not supported. This
includes:
• AIPs
• Managed Items
• Retained Sets
• Hold Sets
7.2.1 Common SpEL Expressions
When you update a resource, you must first GET the resource. When you get
the resource, there will be an ETAG in the header that is used to prevent
concurrent updates overriding changes.
First, we get the resource, in this case we will get the Audit Application
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9…' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207/applications?spel=%3F%5Bname%3D%3D%27Audit%27%5D%0D%0A
From the response you will need the self link, and do a GET. In the response
header, you will get an ETAG value.
etag:"564e1971233e098c26d412f2d4e652742355e616fed8ba88fc9750f869aac1c29cb944175c3
74a7b6769989aa7a4216198ee12f53bf7827850dfe28540587a97"
This etag must be included in the header when doing the PUT command.
8.0 Responses
All server responses consist of a response status code (standard HTTP Status
Codes), response headers and sometimes response bodies, if applicable.
Response code is a first indicator with respect to what state we are currently
dealing with – was our request processed OK or did it fail. Typically, any
response that was processed correctly will contain Status Code from the 2xx
family (for example, 200 OK, 201 Created, 202 Accepted or 204 No Content).
In addition to Status Codes from the 2xx family, we should be generally ready to
{
"id": "feb361ce-6d81-4d29-967f-9b5e496e1559",
"createdBy": "[email protected]",
"createdDate": "2018-04-19T10:42:49.197-04:00",
"lastModifiedBy": "[email protected]",
"lastModifiedDate": "2018-04-19T10:42:49.197-04:00",
"version": 1,
"name": "PhoneCalls-aic",
"_links": {
"self": {
"href": "http://127.0.0.1:8765/systemdata/aics/feb361ce-6d81-4d29-967f-9b5e496e1559"
},
"http://identifiers.emc.com/update": {
"href": "http://127.0.0.1:8765/systemdata/aics/feb361ce-6d81-4d29-967f-9b5e496e1559"
},
"http://identifiers.emc.com/dip": {
"href": "http://127.0.0.1:8765/systemdata/aics/feb361ce-6d81-4d29-967f-
9b5e496e1559/dip",
"deprecation": "http://127.0.0.1:8765/deprecated"
}
}
}
Some REST calls may result in errors. If so, the error response code will shine a
light on the type of problem. Since all response codes are HTTP Status Codes,
when it comes to errors, we typically divide them into two sections:
• 4xx family – client-side errors: If we receive a 4xx type error, this is, in
most cases, the client’s fault and client should figure out what went wrong
and attempt to resolve the problem. Examples here would be non-JSON
compliant payload for POST, or missing a mandatory JSON attribute (both
would be 400), or PUT where version attribute is not the latest, would fail
with a 409 error. It is the client’s responsibility to figure out how to fix this
type of problem. There are usually helpful hints in the error response body
(seen next section).
• 5xx family – server-side errors. This indicates that the request was fine,
but something went wrong on the server and the request failed.
Depending on the failure, the request may or may not need to be
repeated. For example, if the status code is 500, the client typically does
not need to repeat the request, as the request is likely to fail again.
However, if the status code was 503 – the failure could have been due to
the resource contention issue (for example, lock) and repeating the
request may be successful.
{
"_errors": [{
"error": "name",
"errorCodes": [],
"message": "may not be empty",
"localizedMessage": "may not be empty"
}],
"_links": {}
}
As of this writing, there are two attributes in the JSON error response that will
help determine what went wrong: error and message. The array errorCodes is
currently reserved for future use and, for most errors, will not provide a specific
error code yet.
12.0 Tenant
Tenant is a top-level resource and a collection that can be found right off the top
of the tree via the Home Resource. Executing a GET action on a Home
Resource will return a payload that will include the tenant’s resources
As shown above, only users in the Administrator’s role can create tenant
resources.
Note: If leveraging the exercise example in this guide, ensure that the correct Scope was used in
order to obtain a JWT with Administration permission with the appropriate rights, as described
in Section 4.1 of this guide. The Authorization Method example used in this guide originally
requested a “Search” Scope level access when, in this case, an Administration permission is
required.
After a successful call, the IA server will return a 201 Created response that
includes the newly created Application Resource, similar to the following
example:
Note that this REST only creates the application, to effectively use it requires
creating associated spaces and stores. Look at the sample applications to get a
better understanding of all the resources required to build an application.
Note: All of the following objects should be preconfigured and properly set up in the
system:
- xDB Federation (s)
- Database (s)
- Storage (s)
- Store (s)
- Space (s)
- File System Roots
Moreover, it is required to have the tenant and application properly configured.
When describing ingestion. we only focus on the way to retrieve the references and URI
to the configuration objects, if required.
The following scheme illustrates the REST structure to the endpoints, where the
data preservation configuration objects can be found and created:
If using the AGGREGATE mode, preserve the link to “self” relation to use it in the
holding configuration object.
13.3.1.3 PDI-Schema
Another object that has to be configured in a SIP application is the PDI-schema.
This object describes the xsd schema of PDI files with structured content.
Obtain the link to the collection of object with objects by issuing a GET on the
href’s value for the link relation http://identifiers.emc.com/pdi-schemas from the
application root.
You may temporarily preserve the URI to “self” relation to use in the holding
configuration.
It is also required that you pass a file that contains the xsd schema itself. Find the
href’s value for the http://identifiers.emc.com/contents link relation, as marked by
yellow in the above response.
And now it is required to issue a POST on the found URI with the multipart form-
data type. For example:
13.3.1.4 Ingest
Obtain the link to the collection of object with objects by issuing a GET on the
href’s value for the link relation http://identifiers.emc.com/ingests from the
application root.
In case of a successful server response, you will get a payload similar to the
following:
{
"id" : "5545fb55-50dc-430d-92eb-a1eb46400f60",
"createdBy" : "[email protected]",
"createdDate" : "2017-03-23T17:30:50.805+03:00",
"lastModifiedBy" : "[email protected]",
"lastModifiedDate" : "2017-03-23T17:30:50.805+03:00",
"version" : 1,
"name" : "ingest_node_01",
"application" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424",
"logLevel" : "INFO",
"enumerationCutoffDays" : 30,
"enumerationMaxResultCount" : 10,
"enumerationMinusRunning" : true,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/ingest-nodes/5545fb55-50dc-430d-92eb-a1eb46400f60"
},
"http://identifiers.emc.com/update" : {
"href" : "http://localhost:8765/systemdata/ingest-nodes/5545fb55-50dc-430d-92eb-a1eb46400f60"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/ingest-nodes/5545fb55-50dc-430d-92eb-a1eb46400f60"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424"
},
"http://identifiers.emc.com/enumeration" : {
"href" : "http://localhost:8765/systemdata/ingest-nodes/5545fb55-50dc-430d-92eb-a1eb46400f60/aips"
}
}
}
The ingest node URI can be found as the value of the node: “_links”->”self”-
>”href” is used in Holding object (highlighted).
13.3.1.7 Holding
To create a holding configuration object, it is required to find a link to where to
issue a POST request.
Let’s start from application root and find http://identifiers.emc.com/holdings link
relation.
13.3.1.9 Confirmation
There is a link relation http://identifiers.emc.com/confirmations rooted from the
application. We need to find it and extract an href’s value for the URI, as it refers
to a collection on confirmations.
Let’s do a GET method on the found URI.
Alternatively, you can import a ZIP file that contains the configuration YAML file
plus any external files that are referenced by the YAML:
In order to delete the table, for the application, need to get the table link by doing
GETs on the databases, and then in schema, and then in the tables link.
Here is an example of the REST request to list the tables for a schema
This is response
…,
"pathValueIndexList" : [ ],
"schema" : "http://localhost:8765/systemdata/schemas/ed92fd35-7001-4042-a8fb-529a0f2306f5",
"lastIngestionDate" : "2019-08-27T17:46:07.5536844-04:00",
"projectedDispositionDate" : null,
"underHold" : false,
"underRetention" : false,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/tables/AAAADA"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/tables/AAAADA"
},
"http://identifiers.emc.com/delete-content" : {
"href" : "http://localhost:8765/systemdata/tables/AAAADA/content"
},
"http://identifiers.emc.com/schema" : {
"href" : "http://localhost:8765/systemdata/schemas/ed92fd35-7001-4042-a8fb-
529a0f2306f5"
},
"http://identifiers.emc.com/chain-of-custody" : {
"href" : "http://localhost:8765/systemdata/tables/AAAADA/chain-of-custody"
},
"http://identifiers.emc.com/ingest" : {
"href" : "http://localhost:8765/systemdata/tables/AAAADA/content"
}
}
} ]
13.4.1.1 Load balance the xDB libraries after xDB library policy update
The command launches the load balancing as a background process after the
xDB library policy configuration has been updated.
The use case for this process is to allow the data nodes to be load balanced
using a command, after the configuration has been updated.
So typical example is the data node is full, then the administrator decides to add
Rest API:
POST "http://localhost:8765/systemdata/applications/<applicaition-id>/cache-out-
application"
The Post body is optional and the below default values will be used if none
specified
"name:: <application name>_CacheOutData_timestamp
"createBackups" : true which means backup before caching out
Note:
It is recommended to always let backup creation to happen, to have up to content
available as backup (The backed-up content is used for cache-in operation
subsequently).
User needs to ensure that the existing backup is up to date if opting to skip the
backup, If not, it is recommended to always request a backup.
Example:
curl -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6…-H 'Accept:
application/hal+json' localhost:8765/systemdata/applications/ac66619f-2c63-40e6-86fd-
ffeab754126f/cache-out-application
Rest API:
POST "http://localhost:8765/systemdata/applications/<applicaiton-id>/cache-in-
application"
{ "name" : "mycacheInorderitem" }
Note: The post body is optional and default name "<application
name>_CacheInData_timestamp " will be used if none specified
Example:
curl -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6…-H 'Accept:
application/hal+json' localhost:8765/systemdata/applications/ac66619f-2c63-40e6-86fd-
ffeab754126f/cache-in-application
13.5.3 Errors
As this feature is specific for table-based application, trying to cache-out SIP
based application will result in error:
{
"_errors" : [ {
"error" : "A conflict occurred.",
"errorCodes" : [ ],
"message" : "Cannot cache-out data from application when indexing job is underway",
"localizedMessage" : "Cannot cache-out data from application when indexing job is
underway"
} ],
"_links" : { }
}
14.0 Ingestion
As far as InfoArchive supports two kinds of application for archiving: Table based
and SIP based, then there are two different mechanism of data ingestion to each
type of applications. Both ingestion ways are covered in the chapter.
When planning about data ingestion from the REST perspective, it is
recommended to start working at the application level via “Home Resource” ->
“Tenant” -> “Application”.
14.1 SIP Ingestion
Using the SIP based demo application named PhoneCalls as an example, the
following REST Links are available.
From that application root, we have following REST links available.
It contains only the page information and the number of relation links to perform
different actions: “receive” and “ingest-direct”.
"aggregate" : false,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424"
},
"http://identifiers.emc.com/contents" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/contents"
},
"http://identifiers.emc.com/ris" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/ris"
},
"http://identifiers.emc.com/invalid" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/invalid"
},
"http://identifiers.emc.com/transform" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/transform"
},
"http://identifiers.emc.com/xdb-library" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/xdb-libraries/8890e309-1168-4608-a551-
a0b230bf2406"
},
"http://identifiers.emc.com/backup" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/backup"
},
"http://identifiers.emc.com/recovery" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/recovery"
},
"http://identifiers.emc.com/restore" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/restore"
},
"http://identifiers.emc.com/detach" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/detach"
},
"http://identifiers.emc.com/ci-crypto-object" : {
"href" : "http://localhost:8765/systemdata/crypto-objects/32d47a3f-0609-430b-bf69-6a7ebcf5746a"
},
"http://identifiers.emc.com/pdi-crypto-object" : {
"href" : "http://localhost:8765/systemdata/crypto-objects/32d47a3f-0609-430b-bf69-6a7ebcf5746a"
},
"http://identifiers.emc.com/sip-crypto-object" : {
"href" : "http://localhost:8765/systemdata/crypto-objects/32d47a3f-0609-430b-bf69-6a7ebcf5746a"
},
"http://identifiers.emc.com/groups" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/6728b300-0383-497b-9faa-
e205b8433aff/groups"
}
}
}
The number of links (and the link relations) of the ingested AIP object have been
changed in comparison to the reception state.
However, real life requirements dictate optimizing the process to increase batch
ingestion performance. That translates into first receiving several SIP files
following the instructions shown in Section 12.2.2.1 (Reception Step).
Once the reception step is completed for several SIPs, the Enumeration
Resource is leveraged to create a collection of SIPs that status is “Waiting
Ingestion”.
The enumeration resource eliminates the need for leveraging the
“http://identifiers.emc.com/aips” URI previously covered in section 12.1.2, which
would have resulted in getting all of the application’s AIPs, regardless of status,
and then the need to examine each individual AIP.
Instead, the Enumeration response includes an http://identifiers.emc.com/ingest
relation for each in individual AIPs in the collection greatly simplifying
development and increasing performance.
See the example enumeration response below (abbreviated):
If successful, the response is the same AIP, but with updated attributes.
"http://identifiers.emc.com/ingest-direct": {
"href": "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-
8bc4-d74de313a424/aips?ingestDirect=true"
}
The direct ingest is performed by POST request to the above link, but it is
required to set up several important parameters to set up form-data:
- -F “format=sip_zip” – it is a mandatory parameter that sets up the SIP
format to be received. In the below example, it is “sip_zip”. Receiver
node should support provided format.
- -F “[email protected]” - a parameter that contains a
path to the received SIP package. NOTE, that in the below example,
the SIP package is located in the same folder as curl executed.
{
"id" : "88556ddc-3a1a-4bb6-9400-fb9bb2240af3",
"aipId" : "88556ddc-3a1a-4bb6-9400-fb9bb2240af3",
"sipProductionDate" : "2001-12-01T00:00:00+01:00",
"sipSeqno" : 1,
"sipIsLast" : true,
"sipAiuCount" : 3,
"sipPageCount" : 0,
"sipPdiHashAlgorithm" : null,
"sipPdiHashEncoding" : null,
"sipPdiHash" : null,
"dirty" : false,
"ingestDeadlineDate" : "2017-03-23T20:21:06.658+03:00",
………….
"returnCode" : "OK",
"returnMsg" : "",
"projectedDispositionDate" : null,
"underHold" : false,
"underRetention" : false,
"permission" : {
"groups" : [ ]
},
"state" : "Completed",
"xdbLibraryDetached" : false,
"openAggregate" : false,
"phase" : "Completed",
"pdiCryptoObjectName" : "PhoneCalls-crypto-object",
"sipCryptoObjectName" : "PhoneCalls-crypto-object",
"xdbLibrarySize" : 249856,
"validAggregate" : false,
"xdbLibraryIndexSize" : 53248,
"ciCryptoObjectName" : "PhoneCalls-crypto-object",
"xdbLibraryName" : "b4419f0e-9067-42b1-8ad0-c62cfe76d34e",
"aggregate" : false,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/88556ddc-3a1a-4bb6-9400-
fb9bb2240af3"
},
"http://identifiers.emc.com/delete" : {
"href" : http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/88556ddc-3a1a-4bb6-9400-
fb9bb2240af3
…..
"href" : "http://localhost:8765/systemdata/crypto-objects/32d47a3f-0609-430b-bf69-6a7ebcf5746a"
},
"http://identifiers.emc.com/groups" : {
"href" : "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-8bc4-d74de313a424/aips/88556ddc-3a1a-4bb6-9400-
fb9bb2240af3/groups"
}
}
}
"http://identifiers.emc.com/ingest-async ": {
"href": "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-
8bc4-d74de313a424/aips/ingest-async?sip&format&name&eligibilityDate }
Parameters sip and format are mandatory, and their usage is exactly the same
than for Reception.
Parameters name and eligibilityDate are optional:
• name : name of the Ingestion Order Item that will be returned. If no name
is specified, the system name the Order Item with the name of the AIP.
• eligibilityDate: minimum date on which the Ingestion Order Item created
should be taken into account by the IA Server.
Result of a successful execution of the Rest Call is
• HTTP Code ACCEPTED (202)
• A JSON representation of the created Ingestion Order Item.
"http://identifiers.emc.com/ingest-async ": {
"href": "http://localhost:8765/systemdata/applications/7a7b9d20-e31f-4d5c-
8bc4-d74de313a424/aips/41a054af-0e27-4a6c-9cee-80c223aa3203/ingest-
async?sip&format&name&eligibilityDate }
At that point, we have covered the data ingestion for both the SIP application
type and table application types. And we are ready to perform the search on the
data. So, we are moving to the next chapter - Searches.
14.3.2 Attachment ingest
Here is an example of an XML file with references to attachments:
As you can see we use “form” option of data upload with curl – and provide all
three references. All these files are attached to the POST request and are
uploaded to the server. If everything goes well we’re getting 201 Created
response (note – you will need to add “-i” option to curl to actually see the
response headers, which are normally suppressed):
HTTP/1.1 201
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Expires: 0
Pragma: no-cache
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
RequestID: e3b34451-1372-4856-94bf-afdf07118d1b
Date: Fri, 14 Jun 2019 14:46:29 GMT
X-Content-Type-Options: nosniff
Content-Length: 0
Effectively, we need to do a GET on the search resources that are available from
an application.
To help facilitate this procedure, install the sample Audit application, which has
some default searches available.
From the applications link, find the searches link
The following (optional) filters can be used to further refine the collection of
searches:
Even though a search is returned, there may be no search sets available to run.
If the search is not published, only developers can run the search. It is also
possible that the permission set during search composition has limited the
access of the search.
TIP: in the current release of InfoArchive, we typically deal with just a single tenant.
There is a link relation http://identifiers.emc.com/tenant to point to a tenant resource
instance directly from the Home Resource. So, alternatively, we could look for that link
relation instead of looking for collection of tenants. This would be a shortcut of sort.
TIP – it may be handy to keep reference to our instance of application we just found - even if we
choose not to use it directly right now. It’s probably worthwhile to keep it around as we may
need to use it later. In general, due to the discovery process involved in finding resources, it is a
good idea to keep some key resource references so we won’t have to look for them again in near
future.
For defining the search, we need to retrieve and only select the Queries that are
bounded with a certain AIC. Otherwise, you may select an inappropriate Query
object for that given AIC, which will lead to from search configuration.
TIP – since the curl tool is a command line interface, we have to make sure that it is
parsed correctly. For example, parameters can’t contain new line characters because it
will prevent curl from working as expected. You’ll have to remove new lines and any
other special formatting characters that my not work in the command line tool. Typically,
this boils down to removing line breaks.
{
"_errors": [{
"error": "XQuery",
"errorCodes": [],
"message": "Search failed create XQuery, please check XQuery for possible
issues: query:1:6:XQUERY_PARSE_ERROR: no viable alternative at input 'blah' (NCName)
(XPST0003)",
"localizedMessage": "Search failed create XQuery, please check XQuery for
possible issues: query:1:6:XQUERY_PARSE_ERROR: no viable alternative at input 'blah'
(NCName) (XPST0003)"
}],
"_links": {}
}
Luckily for us, the error is fairly descriptive and points to the issues with parsing
our dummy xQuery. Let’s fix it now.
NOTE – when collecting search results from an asynchronous search, the results are already
waiting to be retrieved so, technically, the payload (search criteria) to the above call is optional.
However, as a matter of good style, it is a good idea to pass them. As the search results sit in a
temporary cache, if the client waits long enough to retrieve the results, it is possible for the
cache to expire and the results will vanish. In that case, if we do pass search criteria, the system
will re-execute our search and be able to deliver proper results to us, even if they expired from
the cache.
In response to our call, we will get back results similar to the ones below:
}]
},
"_links": {
"self": {
"href": "http://localhost:8765/systemdata/search-compositions/da1cf6cd-bab3-49f3-b870-
25c555336cc8/search-results/c73c9e7e-4a90-409a-a3b1-ee433c1357f2"
},
"http://identifiers.emc.com/export": {
"href": "http://localhost:8765/systemdata/search-compositions/da1cf6cd-bab3-49f3-b870-
25c555336cc8/search-results/c73c9e7e-4a90-409a-a3b1-ee433c1357f2/export"
}
},
"page": {
"size": 10,
"totalElements": 1,
"totalPages": 1,
"number": 0
}
}
As you can see in the last response example, when we look at search-result
resource (regardless of how search was executed – in synchronous or
asynchronous mode) there is a link relation export.
The export of search results is somewhat similar to executing search in an
asynchronous way – in a sense that it is also a background process. Because it
is a background process, we’ll be interacting with the order-item resource again.
We’ll have to create an order-item resource that will be responsible for gathering
search results and converting them to some external format so that they can be
exported out of the InfoArchive system.
If you look at the order-item resource, you will recall that one of the attributes
there is type. You can see in our examples when we create background search,
its value is SEARCH. Similarly, when we create an order-item for export, its value
should be EXPORT. Let’s see do that.
To create export order-item, we need to do a POST to link relation export.
NOTE
-o mydownload.gzip – this is a curl specific way of downloading binary data to the local folder.
File upload and download is a bit more complicated type of interaction with any server so the
actual negotiation between client and server may be little bit different, depending on the client.
For example, when using some other client you may need to specifically set Accept header:
Accept: application/octet-stream
Furthermore, you can see we also used URI query parameter filename. This parameter is used
by IA REST API to set Content-Disposition header like this:
Content-Disposition: attachment; filename="someName.csv.gz”
This can be leveraged, for example, by the Javascript in your browser to automatically download
content to user’s file system.
In addition, the following headers are set and might be useful:
Accept-Ranges: bytes
Content-Range: bytes 0-863920/863921
Content-Length: 863921
They could be leveraged, for example, to validate if entire content has been collected.
From the response, find the link relation for the audit event types (removed one
of the two values from the output for brevity/clarity).
{
"_embedded" : {
"valueLists" : [ {
"id" : "73881e50-e688-429f-b1e9-01194c2e3cbf",
"createdBy" : "[email protected]",
"createdDate" : "2017-10-12T14:23:22.959-04:00",
"lastModifiedBy" : "[email protected]",
"lastModifiedDate" : "2017-10-12T14:23:23.021-04:00",
"version" : 2,
"name" : "event-types",
"application" : "http://localhost:8765/systemdata/applications/4173bd70-fd98-4534-b111-
6646a554dea4",
"description" : "Event Types",
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/value-lists/73881e50-e688-429f-b1e9-
01194c2e3cbf"
},
"http://identifiers.emc.com/value-list-content" : {
"href" : "http://localhost:8765/systemdata/value-lists/73881e50-e688-429f-b1e9-
01194c2e3cbf/content"
},
"http://identifiers.emc.com/update" : {
"href" : "http://localhost:8765/systemdata/value-lists/73881e50-e688-429f-b1e9-
01194c2e3cbf"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/value-lists/73881e50-e688-429f-b1e9-
01194c2e3cbf"
},
"http://identifiers.emc.com/value-list-query" : {
"href" : "http://localhost:8765/systemdata/value-lists/73881e50-e688-429f-b1e9-
01194c2e3cbf"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8765/systemdata/applications/4173bd70-fd98-4534-b111-
6646a554dea4"
}
}
}, …
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/applications/4173bd70-fd98-4534-b111-
6646a554dea4/value-lists?page=0&size=10"
},
"http://identifiers.emc.com/add" : {
"href" : "http://localhost:8765/systemdata/applications/4173bd70-fd98-4534-b111-
6646a554dea4/value-lists"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8765/systemdata/applications/4173bd70-fd98-4534-b111-
6646a554dea4"
}
},
"page" : {
"size" : 10,
"totalElements" : 2,
"totalPages" : 1,
"number" : 0
}
}
IF a get is done on the content and the content type is set to xml, the content can
be viewed for the value list.
The next chapters will explain how to execute a cross application search and
how to get the results.
15.11.1 Executing a synchronous
cross-application search
The synchronous execution is not the preferred way to execute a cross
application search. This kind of search is (most of the time) long; so, the
synchronous execution will not be terminated before the maximum execution
{
"crossApplicationSearch" : true,
"totalElements" : 6,
"executionTime" : 933,
"results" : [ {
"label" : "TextFieldFirstName (Set 1)",
"totalElements" : 0,
"executionTime" : 394,
"hideColumns" : null,
"debug" : false,
"partial" : false,
"_links" : {
"http://identifiers.emc.com/search-result" : {
"href" : "http://localhost:8080/restapi/systemdata/search-results/687fc710-8790-4988-
aabf-f0d374653439"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8080/restapi/systemdata/applications/b83f252e-7684-4505-9100-
f578ee8bd804"
},
"http://identifiers.emc.com/search" : {
"href" : "http://localhost:8080/restapi/systemdata/searches/9be9ebd2-b0ad-4c43-8fb6-
457fb2bae3be"
},
"http://identifiers.emc.com/search-composition" : {
"href" : "http://localhost:8080/restapi/systemdata/search-compositions/e091060d-6264-
4edf-8482-4add744e404f"
}
},
"applicationName" : "PhoneCalls",
"applicationCacheState" : "CACHED_IN",
"applicationArchiveType" : "SIP",
"searchName" : "TextFieldFirstName",
"searchCompositionName" : "Set 1"
}, (…)
From the root level, it is possible to get the system level audits by posting to the
audits link rel. All link rel specific to EMC will have this prefix:
http://identifiers.emc.com/
Similarly, to the tenant audit and event types are anchored from the tenant.
Finally, each application can have its own audits. In order to disable an
application audit, the audit must be disabled at both the tenant and the
application level.
Note that links can always change, and the appropriate URLs can be discovered
from the /services endpoint.
From the response, find the link relation for the audit event types:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
"http://identifiers.emc.com/audit-event-types" : {
"href" : "http://localhost:8765/systemdata/audit-event-types"
},
"http://identifiers.emc.com/audit-extended-event-types": {
"href": "http://localhost:8765/systemdata/audit-extended-event-types"
},
"http://identifiers.emc.com/audit-constants" : {
"href" : "http://localhost:8765/systemdata/audit-constants"
},
"http://identifiers.emc.com/audit-constants-names": {
"href" : "http://localhost:8765/systemdata/audit-constants-names"
},
"http://identifiers.emc.com/audits" : {
"href" : "http://localhost:8765/systemdata/audits"
},
…
}
}
To enable or disable the particular audit, post to the either the disable or enable
link. That same procedure can be used to set the tenant or application audits by
following the linkrels from the tenant or application.
Notice that the enabled in the response changed from false to true.
From the response, find the link relation for the audit event types:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
"http://identifiers.emc.com/audit-event-types" : {
"href" : "http://localhost:8765/systemdata/audit-event-types"
},
"http://identifiers.emc.com/audit-extended-event-types": {
"href": "http://localhost:8765/systemdata/audit-extended-event-types"
},
"http://identifiers.emc.com/audit-constants" : {
"href" : "http://localhost:8765/systemdata/audit-constants"
},
"http://identifiers.emc.com/audit-constants-names": {
"href" : "http://localhost:8765/systemdata/audit-constants-names"
},
"http://identifiers.emc.com/audits" : {
"href" : "http://localhost:8765/systemdata/audits"
},
…
}
}
From the response, find the link rel for the audit constants:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
"http://identifiers.emc.com/audit-event-types" : {
"href" : "http://localhost:8765/systemdata/audit-event-types"
},
"http://identifiers.emc.com/audit-extended-event-types": {
"href": "http://localhost:8765/systemdata/audit-extended-event-types"
},
"http://identifiers.emc.com/audit-constants" : {
"href" : "http://localhost:8765/systemdata/audit-constants"
},
"http://identifiers.emc.com/audit-constants-names": {
"href" : "http://localhost:8765/systemdata/audit-constants-names"
},
"http://identifiers.emc.com/audits" : {
"href" : "http://localhost:8765/systemdata/audits"
},
…
}
}
The category is used by the IAWA client to categorize the system audits.
From the response, find the link rel for the audit constants:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
"http://identifiers.emc.com/audit-event-types" : {
"href" : "http://localhost:8765/systemdata/audit-event-types"
},
"http://identifiers.emc.com/audit-extended-event-types": {
"href": "http://localhost:8765/systemdata/audit-extended-event-types"
},
"http://identifiers.emc.com/audit-constants" : {
"href" : "http://localhost:8765/systemdata/audit-constants"
},
"http://identifiers.emc.com/audit-constants-names": {
"href" : "http://localhost:8765/systemdata/audit-constants-names"
},
"http://identifiers.emc.com/audits" : {
"href" : "http://localhost:8765/systemdata/audits"
},
…
}
}
From the response, find the link relation for the tenant:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
}
Tip: we could have followed the tenants link relation and got the first tenant but
following this link relation saves a few REST calls
Now we will follow this link but add a spel expression to limit the applications, as
we want to focus on the Audit application (the spEL expression isn’t strictly
necessary if there is only one application, but ensures we get the one).
The syntax is a little tricky, but this is the spEL expression before encoding:
spel=?[name=='Audit']
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9…' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207/applications?spel=%3F%5Bname%3D%3D%27Audit%27%5D%0D%0A
The syntax is a little tricky, so this is the spEL expression before encoding:
spel=?[type=='hold' and name==’apply’].
Note that the audit-event-types should not be splitting across multiple lines
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9…' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/applications/8701a41a-e195-41f1-a330-
6c4894b30b13/audit-event-
types?spel=%3F%5Bname%3D%3D%27apply%27%20and%20type%3D%3D%27hold%27%5D
The response should look like this (only one audit returned):
{
"_embedded" : {
"auditEventTypeResults" : [ {
"id" : "ab09d3ff-0022-4a23-8507-d8ba9b5d7b33",
"type" : "hold",
"name" : "apply",
"applicationId" : "8701a41a-e195-41f1-a330-6c4894b30b13",
"tenantId" : "66139448-0814-4c73-9ffd-5810c64c1207",
"enabled" : true,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/ab09d3ff-0022-
4a23-8507-d8ba9b5d7b33"
},
"http://identifiers.emc.com/disable" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/ab09d3ff-0022-
4a23-8507-d8ba9b5d7b33/disable"
},
"http://identifiers.emc.com/enable" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/ab09d3ff-0022-
4a23-8507-d8ba9b5d7b33/enable"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/applications/8701a41a-e195-41f1-a330-
6c4894b30b13/audit-event-
types?spel=%3F%5Bname%3D%3D%27apply%27%20and%20type%3D%3D%27hold%27%5D"
}
},
"page" : {
"size" : 10,
"totalElements" : 1,
"totalPages" : 1,
"number" : 0
}
}
16.7 Getting the Audits that Have Not Been Archived Yet
It is possible via the rest to get the audits at either the system, tenant, or
application level by doing a GET on the audit’s link relationNote that each time
the Archive Audits job is run, this list is cleared out.
This REST API has been deprecated and it is recommended instead to archive
the audits and view the archived audits.
Note that it is important the on the xform additional criteria that distinguishes
between the System, Tenant, and Application Audits searches. If that additional
criteria is not passed, the search will return all three types of audits.
For example, the following criteria returns the Application audits only
<data><criterion><name>producer</name>
<operator>NOT_EQUAL</operator>
<value>System</value><value>INFOARCHIVE</value>
</criterion></data>
The following criteria returns the System audits only
<data><criterion><name>producer</name>
<operator>EQUAL</operator><value>System</value>
</criterion></data>
There are some caveats with custom audits that the GUI cannot be used to
enable/disable these audit event types.
16.9.1 Registering an Event Type
You need to decide if the event type is going to be at the system, tenant,
or application level. Depending on the level, there are three REST
endpoints for each.
For simplicity, this example will create the audit event type at the system
level.
curl -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9….' -H 'Accept: application/hal+json' -H
'Content-Type: application/hal+json' -d '{"enabled": true, "name" : "MyEventName", "type" :
"MyEventType" }' localhost:8765/systemdata/audit-event-types
Note that you do not need to pass the tenant or application id for creating an
application audit event, as this information is inferred from the path.
The response should look like the following (only one audit returned):
{
"id" : "af78176c-420f-4f88-b330-697ed01d1b4c",
"type" : "MyEventType",
"name" : "MyEventName",
"applicationId" : null,
"tenantId" : null,
"enabled" : true,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/af78176c-420f-4f88-
b330-697ed01d1b4c"
},
"http://identifiers.emc.com/disable" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/af78176c-420f-4f88-
b330-697ed01d1b4c/disable"
},
"http://identifiers.emc.com/enable" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/af78176c-420f-4f88-
b330-697ed01d1b4c/enable"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/audit-event-types/af78176c-420f-4f88-
b330-697ed01d1b4c"
}
}
}
Any event types can be deleted, so use caution with this API.
16.9.3 Creating an Audit
This is normally done automatically but, for a custom audit, a REST call is
required to do the audit. If the custom code was on the server, there is no need
to go through REST.
Again, there are three different endpoints, depending on if it is a system, tenant,
or application audit.
For simplicity, this example will create the audit at the system level.
curl -X POST -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9….' -H 'Accept: application/hal+json' -H
'Content-Type: application/hal+json' -d '{"eventName" : "MyEventName", "eventType" : "MyEventType",
"eventSource" : "SOURCE", "applicationName" : "MyApp", "auditedObjectId" : "74bd48f5-b891-44bd-
a2c3-9d7df6aaed5b", "supplementalData" : {"field1" : "value1", "field2" : "value2"} }' lo-
calhost:8765/systemdata/audits
The supplemental data is effectively a map of string, and the key cannot have a
space, but the value can.
The createdBy will be the user that is making the rest call. One possibility is to
specify in the supplementalData who actually performed the action, as creating
audits is restricted to the Administrator role.
Normally the applicationName would not be set for a system audit and usually
would not be passed in the payload.
Tenant resources:
• Retention policies can be used by multiple applications as well as holds.
• Retention policy categories are automatically created when creating a
retention policy
• Events are automatically created when either event or mixed retention
policies are created
Application resources:
• Retained sets associate managed items to retention policies. Retained
sets always have at least one item in the set.
• Holds sets store which items have a hold applied
• Purge-lists are created automatically and are used to track the disposition
process
From now on, this document assumes that you know how to find the tenant link
from the main services page.
To get the retention-policies link do a get on the tenant, use the following:
{
"id" : "66139448-0814-4c73-9ffd-5810c64c1207",
"createdBy" : "system",
"createdDate" : "2017-02-24T10:38:02.448-05:00",
"lastModifiedBy" : "system",
"lastModifiedDate" : "2017-02-24T10:38:02.448-05:00",
"version" : 1,
"name" : "INFOARCHIVE",
"permission" : {
"groups" : [ ]
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
"http://identifiers.emc.com/groups" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207/groups"
},
"http://identifiers.emc.com/applications" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207/applications"
},
…
"http://identifiers.emc.com/retention-policies" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207/retention-policies"
},…
}
Note that a retained-sets link relation provides an ability to view the retained sets
that are created when a retention policy is created.
{
"name": "Event",
"description": "",
"category": "",
"agingStrategy": {
"type": "EVENT",
"conditions": ["NoLongerNeeded"],
"agingPeriod": {
"units": "YEARS",
"value": 0
},
"cutoff": {
"type": "ANNUAL",
"monthDay": "--07-01"
}
},
"dispositionStrategy": {
"type": "DESTROY_ALL"
},
"approval": {
"approver": "",
"approvedDate": ""
},
"note": "",
"dispositionBlocked": false,
"customAttributes": {
"RECORD-CLASS-CODE": "ACC0001",
"RECORD-CLASS-NAME": "Accounts Payable",
"RECORD-RETENTION-MARKET": "CAD",
"RETENTION-EVENT-DESC": "The retention period begins on the last day of the fiscal
year in which the record is created."
}
}
If a retention policy can be modified, when doing a get on the retention policy, a
link relation for the update will be returned if the retention policy can be updated.
Retention polices that are in use have a limitation on what can be changed.
The following are rules governing retention policies in use:
• Cannot have the name changed
• Cannot changed the agingStrategy to a different type (for example,
change from DURATION to FIXED_DATE)
• Cannot change conditions for event or mixed retention policies
For simplicity, the databases, schemas, and tables are shown. You obviously
need to find the appropriate database from the collection and, similarly, for the
schemas and tables.
In the following procedure, we provide the step to get the href for an application
(in this case Audit). We will need this as the object that we want to apply
retention to.
The call to apply retention is done against the retention policy. We assume that
you have looked up the retention-policies link from the tenant.
abc
There are now two APIs, you can either call apply (synchronous) or apply-async.
retainedSetDescr N Description
iption
ageIndividually N If multiple
items are
added to
the set,
how do
they age
If items are to age individually (currently only for records), then the baseDate and
eventContext need to be set for each item, depending on the retention strategy.
Here is an example of the payload for a mixed retention policy where both items
are put in the same set, but age individually. Both records could share the same
context if the same event governs both.
It is not possible to put two applications into the same set as, by definition, they
will be in separate applications.
Some tips:
- If you get violation of uniqueness constraint, then the purge set name was
already in use.
- The response returns information about the set. If the set is not aging
individually, a qualification date may be calculated for the set.
- We can confirm that the correct retention policy was used.
The general steps are generic for any job, but these instructions focus on
applying retention to records (as the parameters required differ depending on the
job )
For the first step, a job definition needs to exist that indicates
- Which search to run (a search name and search set) which also indicates
which fields to be shown later when these records show up in purge lists
- Which application is the search defined in
- What search criteria to use
- Which retention policy to apply
- Which attribute to use as the base date (for duration-based retention
policies)
Parameter Description
searchName This is the name of the search to execute. The search must be
defined in the application and must be in the “ready” state.
<data><criterion><name>CustomerFirstName</name><operator>S
TARTS_WITH</operator><value>Morgan</value></criterion></da
ta>
This is the same criteria used by any search and you can use Chrome
F12 for a search via the GUI to see the format
retentionPolic All four types of retention policies are supported. Depending on the
yName type of policy, other properties are required to be set.
contextType Used for Event policies. There are two possible values
• Attribute: The job gets the context from an attribute in the data.
The attribute is set in the context field.
• Fixed: The job uses the value in this field as the context for the
events.
For Example, if you want to have all records for an employee
age for five years after she/he has left the company. You would
make the context the employee ID, since it will be the same for
all documents. InfoArchive groups all documents with the same
context together and you would trigger the event using this
context. All records associated with this context (i.e., employee #)
will be eligible for disposition in five years
context The value is dependent on the contextType. Either the attribute (its
value) to be used as the context or a value entered into this field will
be used as a context for all records that are returned by the search.
retentionDate This value is used for Duration and Duration portion of Mixed
Attribute retention policies. Duration policies that are applied to records need
to have a date to calculate the age of the record. The date will be
taken from the attribute specified in this field. If the attribute does
not have a date, the record will be skipped meaning a policy will not
be applied
trigger Event policies can be triggered using the records data. There are two
possible values for this property:
• True: The job will attempt to trigger the event policy that was
applied to the records.
• False:
Set the value to false if not using either event or mixed retention
policy
triggerCheck The job has the ability to use an attribute to determine if the trigger
Attribute should be performed. For example, if the data has a field that
triggerCheck This is value to check against for whether the event should be
Value triggered.
triggerDateAtt The job requires a date to trigger the event. This property is an
ribute attribute name where the job will get the trigger date. For example,
if the event is to keep all employee records five years after the
employee leaves the company. There would be an attribute called
’terminationDate’ that contains the date the employee left the
company. Putting terminationDate in the TriggerDateAttribute
property would instruct the job to fetch the trigger date from the
terminationDate attribute.
searchSet Set 1 Must be the search set name (by default, Set 1,
space is required)
applications Link for the application Link for the application where the records reside.
Using the following command to create the job definition (simplified version,
doesn’t allow manualRetry, removes values:
This is the full example providing all of the values for creating a job
curl -X POST -d'{"name":"My Apply Retention Policy to Records","handlerName":"Apply-
PolicyToRecordsJob","description":"Applies a specific retention policy to records
matching a search criteria ..","inactive":false,"readOnly":false,"cancelE-
nabled":false,"manualRetryEnabled":true,"repeatable":true,"applica-
tionScoped":true,"systemScoped":false,"tenantScoped":false,"interval":0,"max-
Attempts":0,"expirationInterval":600000,"rescheduleInterval":60000,"cronExpres-
sion":"","logLevel":"INFO","properties":{"searchName":"FirstName_Opera-
tor","searchSet":"Set 1","searchCriteriaFile":"<data> <criterion><name>CustomerFirst-
Name</name><operator>STARTS_WITH</operator><value>Morgan</value></crite-
rion></data>","retentionPolicyName":"Audit-policy","contextType":"attribute","con-
text":"","retentionDateAttribute":"CallEndDate","trigger":"false","triggerCheckAttrib-
ute":"","triggerCheckValue":"","triggerDateAttribute":""},"applications":["http://lo-
calhost:8080/restapi/systemdata/applications/5b0ce807-ce0c-4d2a-97b5-
3776e2d32fe6"],"internal":false} ' -H 'Authorization: Bearer eyJhbG… -H 'Accept: ap-
plication/hal+json' -H 'Content-Type: application/hal+json' http://localhost:8765/sys-
temdata/job-definitions
Key points:
The application link is the same as what was specified when creating the job
definition. The reason for including the application link on the run is that in the
job definition it is possible to scope the definition to multiple applications but in
this case running a job instance is for a particular application.
The section 23.5 give details for viewing details of a job instance, and following
other links to get more information. The GUI can also be used to view
information about your job. Polling can be done by doing a GET on the job
instance link.
Some other points of interest:
- If this was a one-shot operation to apply retention to a search, if you want
to clean everything up, the job instances need to be removed from the job
definition first and then the job definition can be deleted.
o Job instances can only be deleted after they have completed,
failed, or expired
- It is possible to modify the job using a PUT and change the criteria,
retention policy to do a different search. Each job instance tracks what
parameters were used for the run.
It is possible to use an event retention policy (specifying which field sets up the
context for the event. If the job was configured to trigger events, the “Process
Retention Events” job must be run.
A new link relation has been added to the AIP collection to allow retention to be
applied.
The parameters for this post are the same as for a GET on packages,
The expected body has three parameters
• retentionPolicyName
• retainedSetName
• retainedSetDescription
The first two parameters are mandatory and the retainedSetName must not
already exist.
So, when items are put under retention, managed items are created as well as
retained sets.
Type Fields
Tip: For records, ensure that the search is always published. Otherwise, only the
ID of the record is returned in the supplemental data.
The following is the request. We specifically specify the type and the externalId of
the resource we want to check if it is managed:
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9… -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/applications/8701a41a-e195-41f1-a330-
6c4894b30b13/managed-
items?type=application&externalId=http://localhost:8765/systemdata/applications/8701a41a-e195-41f1-
a330-6c4894b30b13
There are some key points about this API that need highlighting:
The external ID passed is the href for applications, tables, and AIPs
(packages).
The external ID passed for records is the row identifier returned from
searches.
Let’s run the job. We post to the job-instances link off the job-definition: Note
running a job can be found in 23.4.
Wait a few minutes and do a get on our job instance (note that job may be still be
r:
Interesting points:
There is a link to get at the logs (useful if something went wrong):
This job doesn’t log a lot, but other jobs could have more information. This is the
information is shown in the UI when clicking on the information for a job instance.
Now we look at our purge lists. This document will assume that you can find the
link for the PhoneCalls application (see example on how we found the Audit
application in the Audit section).
"id" : "fc6ff1f8-48bd-4de0-8b97-6b7c0399d24c",
"createdBy" : "system",
"createdDate" : "2017-03-01T16:30:30.053-05:00",
"lastModifiedBy" : "system",
"lastModifiedDate" : "2017-03-01T16:30:30.337-05:00",
"version" : 2,
"tenantId" : "66139448-0814-4c73-9ffd-5810c64c1207",
"name" : "Purgelist-1",
"filterName" : "c565f2d1-b32e-42ae-acbc-2476b70b1166",
"status" : "UNDER_REVIEW",
"count" : 10,
"retentionPolicy" : "http://localhost:8765/systemdata/retention-
policies/1e74f1d8-522a-497a-a13b-18bba5ac0a70",
"dispositionSummary" : null,
"dispositionDate" : null,
"approvedDate" : null,
"rejectedDate" : null,
"cancelledDate" : null,
"retentionPolicyName" : "PhoneCalls-policy",
"type" : "aip",
"externalReference" : null,
"customAttributes" : null,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/purge-candidate-lists/fc6ff1f8-
48bd-4de0-8b97-6b7c0399d24c"
},
"http://identifiers.emc.com/approve" : {
"href" : "http://localhost:8765/systemdata/purge-candidate-lists/fc6ff1f8-
48bd-4de0-8b97-6b7c0399d24c/approve"
},
"http://identifiers.emc.com/reject" : {
"href" : "http://localhost:8765/systemdata/purge-candidate-lists/fc6ff1f8-
48bd-4de0-8b97-6b7c0399d24c/reject"
},
"http://identifiers.emc.com/retention-policy" : {
"href" : "http://localhost:8765/systemdata/retention-policies/1e74f1d8-522a-
497a-a13b-18bba5ac0a70"
},
"http://identifiers.emc.com/managed-items" : {
"href" : "http://localhost:8765/systemdata/purge-candidate-lists/fc6ff1f8-
48bd-4de0-8b97-6b7c0399d24c/managed-items"
}
}
} ]
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/applications/c565f2d1-b32e-42ae-acbc-
2476b70b1166/purge-candidate-lists"
}
},
"page" : {
"size" : 10,
"totalElements" : 1,
"totalPages" : 1,
"number" : 0
}
}
If the GeneratePurgeCandidateList job was run again before the lists were
approved and the disposition job ran, then the old list is marked cancelled
and items that are eligible would be put into a new purge list.
Interesting points:
- If we made a mistake and approved the wrong purge list, the revoke link
will work as long as the DisposePurgeCandidateList job hasn’t already
started running.
The approve/reject/revoke links require the user to be a Retention Manager.
From the response, find the link relation for the job-definitions:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},…
"http://identifiers.emc.com/job-definitions" : {
"href" : "http://localhost:8765/systemdata/job-definitions"
},
…
}
Let’s now do a get on our Purge list and see the impact:
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9.. ' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/purge-candidate-lists/fc6ff1f8-48bd-4de0-
8b97-6b7c0399d24c
And here is the response:
We can see here (we only show one of the 10 items) that the status of the
AIP changed to purge (this is specific to AIP disposition).
From the response, find the link rel for the job-definitions:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},…
"http://identifiers.emc.com/job-definitions" : {
"href" : "http://localhost:8765/systemdata/job-definitions"
},
…
}
This job runs fairly quickly, and you can follow the self-link to see when it
finishes.
We now want to run the Disposition job again (we got this link earlier when we
first ran disposition):
Wait a few minutes, and let’s repeat the AIPs linkrel call:
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9….' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/applications/c565f2d1-b32e-42ae-acbc-
2476b70b1166/aips
In the section for creating retention policies, there is an example for creating an
event-based retention policy. If the PhoneCalls granular application is installed,
two addition policies are installed.
Note that this call no longer runs requalification for records associated with the
event. Instead, after all of the events have been fulfilled, run the “Process
Retention Events” job.
Holds are per tenant and there are two types of holds: permanent and legal.
Functionally, they are same in that the holds prevent the item from being either
deleted or disposed. The main difference is a legal hold is typically temporary
and the intent is that at some point the hold will be removed.
Of particular note is the apply link relation. This allows the hold to be applied to
an object. The operation is done to the hold rather than object being protected.
Let’s assume that you have at least one application (say Audit) and we will apply
the hold to the application.
We need to look up the href for the Audit application (this is included in other
examples):
http://localhost:8765/systemdata/applications/8701a41a-e195-41f1-a330-
6c4894b30b13
Because we only applied a hold to one object, the request should be done
quickly. We can see the status of the order item.
Here is the call to look at the status of the order item:
A very important point is that it is the responsibility of the caller to ensure that the
item to protect is a valid item. For applications, tables, and packages, this is the
href. There is no check that the item is still actually in the system.
For table rows and AIUs, the id is the row identifier returned in the search.
So, we can see the name of the hold set and how many items are in the set.
On this hold set, a DELETE on the delete link relation will remove the entire hold
set and, for the call, a reason can be given which will be part of the audit (if
enabled).
A DELETE call to the delete link relation will remove the item from the hold set. If
the hold set would no longer have any items, the hold set is automatically
cleaned up.
These are the resources for the metrics, and they are per tenant.
From the metrics link, there is a refresh linkrel that can be used to update the
metrics. This linkrel will now return a 410(GONE) response. Normally, the
customer would use the Refresh Metrics job to automatically refresh the metrics.
From the metrics link, there are two sub-resources for viewing either the
compliance-metrics or the storage metrics. This information is shown in the GUI
through the dashboards.
Note that as of 16EP5, additional metrics are available as part of the Reports
application. It is possible to use REST calls to run searches to get information
about the application.
From the response, find the link rel for the tenant:
{
"name" : "InfoArchive Home Resource",
"_links" : {
"self" : {
"href" : "http://localhost:8765/services"
},
"http://identifiers.emc.com/product-info" : {
"href" : "http://localhost:8765/product-info"
},
"http://identifiers.emc.com/tenants" : {
"href" : "http://localhost:8765/systemdata/tenants"
},
"http://identifiers.emc.com/tenant" : {
"href" : "http://localhost:8765/systemdata/tenants/66139448-0814-4c73-9ffd-
5810c64c1207"
},
…
}
Tip: We could have followed the tenant’s link and got the first tenant, but
following this linkrel saves a few REST calls:
curl --location --request GET 'http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-
b147c8dacf21' \
--header 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9…’
The response will look like this if the metrics were refreshed:
{
"createdBy" : "system",
"createdDate" : "2021-03-09T16:18:41.4244618-05:00",
"id" : "3b9a82e0-abbc-4ccc-abb9-bdd23f352cfb",
"tenant" : "http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-b147c8dacf21",
"applicationCount" : 4,
"retentionPolicyCount" : 2,
"holdCount" : 2,
"purgeCandidateListCount" : 1,
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-
b147c8dacf21/metrics"
},
"http://identifiers.emc.com/refresh" : {
"href" : "http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-
b147c8dacf21/metrics"
},
"http://identifiers.emc.com/compliance-metrics" : {
"href" : "http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-
b147c8dacf21/compliance-metrics"
},
"http://identifiers.emc.com/storage-metrics" : {
"href" : "http://localhost:8765/systemdata/tenants/205ae826-a096-4684-b431-
b147c8dacf21/storage-metrics"
}
}
}
Tip: If the createdBy is set to null, it means that the refresh was never done for
the metrics.
Do not post to refresh link relation, as it will always return Gone (410) response.
This section of the response shows the detail storage break down per
application:
Both compliance metrics and storage metrics have an export link relation which
allows us to export the metrics data into zip file which contains some csv file
inside.
It is possible to issue a GET request to the URI (marked by yellow on the above
listing) related to the crypto object. The example of the request is below:
Please note “inUse” system attribute, in our case with “false” value. CryptoObject
that are not (yet) used can be updated/deleted/etc. – since they haven’t been
used by any process. Any CryptoObject that has been used for any type of
encryption is protected and cannot be updated or deleted.
Note – as expected, our “keySize” attribute has been updated to requested value
of 128 instead of 256.
Application
(SIP)
Holding-
Pdi-cryptos
cryptos
Application
(Table)
Database-
cryptos
structured- unstructured-
crypto-object crypto-object
21.1 Overview
Users can do operations that may take a long time to execute. For that purpose,
there is an API to get information about order items that the user initiated.
Here is sample of a response, assuming the user has done a background search
against the Audits:
…
}
}]
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/tenants/f94634e5-5012-4fac-bf01-8471b9a94a39/order-items?page=0&size=10"
}
},
"page" : {
"size" : 10,
"totalElements" : 1,
"totalPages" : 1,
"number" : 0
}
}
The example has removed some of the information in the return for more clarity.
In this case, because the request was done so quickly, the request was
complete.
In this case, there were no log messages and, again, this can be influenced by
the server’s logging level.
As it can be seen above, from browser’s perspective, all requests are always
going to the same host (of origin) – proxy. It is proxy’s job to dispatch them from
there, but since browser is not aware of the dispatching, no security issues will
arise.
There are many different types of proxies that can be utilized for this, ranging
from very simple ones that only do request forwarding, to much more
sophisticated that are also capable of updating response payloads for the
purpose of patching links.
I’ll describe in more detail example of a setup with a very simple, request
forwarding proxy.
22.1.2 Our setup
We’ll setup a very simple scenario:
• Tomcat running on port 7070 hosting our proxy. This instance can also be
used to host our client application
• IA Server running on port 8765
Our middle man will be Apache Tomcat (for purpose of cross verifying
configurations, we’re using Tomcat version 7.0.59.
Our Tomcat is hosting both: our sample application as well as proxy. We’ve
configured our Tomcat to run on port 7070. Our InfoArchive Server runs
separately, in a different container, on port 8765.
As you can see, even though our request went through the proxy/middle man
(port 7070), the response links are still pointing back directly to IA Server
(running on port 8765). That is pretty much expected as we did not do anything
to patch the links from response’s payload but if client acts directly on any of
these links, it will run immediately into the CORS issue. This can be remedied in
two ways:
1. We use more sophisticated proxy that will patch those links automatically
function replaceBaseUrlForProxy(url) {
var parser = document.createElement('a');
parser.href = url;
return ‘http://localhost:7070/iarest’ + parser.pathname + parser.search;
}
Calling replaceBaseUrlForProxy() for each of the unpatched links will point them
to the proxy avoiding CORS issues.
22.1.3 Testing it out
First, let’s test IA Server to ensure it is running. We can issue a GET request to
http://localhost:8765/services and we should get back OK 200 response with
JSON payload of Home resource
Next we can test our proxy. We can issue a GET request to
http://localhost:7070/iarest/services and we should get back OK 200 response
with JSON payload of Home resource.
Yes, both of these URIs, from client’s perspective, should return exactly same
thing, since the 2nd request hits our proxy and gets forwarded to the URI from 1st
test.
If the request to IA Server is not working, obviously we have an issue running IA
Server so this should be sorted out first, until we get successful response.
Only once IA Server is working fine, we can consider testing our proxy. If the
proxy doesn’t respond as expected, we’ll have to debug it. This is done typically
following these generic points:
• Ensure Tomcat instance on port 7070 is running.
• Once we know Tomcat is running, check the status of the proxy. You’ll
need to confirm that your proxy has been setup correctly and is actually
running. Depending on your proxy you can probably check Tomcat’s logs
and perhaps log of the proxy itself to determine the issue.
Job Job
Definitions Instances
Job Job
Definition Instance
Job Order
Instances Items
Job definitions indicate which jobs can be done. Job Instances represent a run
of the job. Job instances can be scoped to the system, tenant, or application.
When accessing via rest, the resources are accessed directly of the home
resource.
23.1 Role Permission
Job definitions can be viewed from following the job-definitions link relation off
the home resource:
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9….' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/job-definitions
So, for this example, we see the Commit job. There is a link relation to show the
job instances associated with this job.
Activating or inactivating stops the schedule for jobs running on a schedule. If the
Because this job is expected to be run manually, the interval is 0 and the cron
expression is null.
We can see when it was scheduled (this job was run ad hoc) and its status.
For some jobs, there can be a restart linkrel if the job supports retry. To do a
restart, POST with no payload.
One point that might be surprising is that job instance didn’t define any
properties. Often the properties are specified on the job definition.
23.4 Running or scheduling a job
To run or schedule a job, a post is done to the job instances on the job definition
you are running. The expectation is that the job definition is not suspended.
Note that it possible to only delete the job instance if hasn’t started or has
finished running. Note that job-children will only return items if the job spawned
other jobs.
If you wanted to instead schedule the job, change the “now” parameter to false.
23.5 Viewing job instance information for jobs that use batch
This is very similar to viewing information about batch items for order items.
To speed this up, we will use the GUI to run the apply hold via the rule job. This
screenshot shows how to configure the job:
This command gets all job instances, we could have followed the link relation on
the job definition.
Here is a sample output (assuming the job had finished with success)
If that job instance had failed, -the restart link relation would be available.
Here is a sample output (assuming the job had finished with success)
So, this will look similar to later on when we show an order item for a background
task.
Logs are available for both the order item and the batches.
Finally, for clarity, let’s follow that context link relation
curl -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiJ9….' -H 'Accept: application/hal+json' -H 'Content-
Type: application/hal+json' http://localhost:8765/systemdata/order-items/dfd6b6ca-836a-4961-9a8d-
c123b2265e00/context
Here is a sample output
{
"id" : "66253064-5be1-46a2-a9c5-3a6c9dd921b5",
"createdBy" : "system",
"createdDate" : "2017-10-12T16:33:54.14-04:00",
"lastModifiedBy" : "system",
"lastModifiedDate" : "2017-10-12T16:34:16.1-04:00",
"version" : 4,
"contextId" : "973ef5b8-9164-4ae5-90b7-721790a86831",
"totalSize" : 38,
"batchSize" : 1000,
"timeOut" : 0,
"done" : true,
"retriedAfterLockNotGranted" : false,
"retryable" : true,
"batchCount" : 1,
"totalExecutionTime" : 4166,
"completedCount" : 1,
"failedCount" : 0,
"inProgressCount" : 0,
"inQueueCount" : 0,
"waitingCount" : 0,
"orderItem" : "http://localhost:8765/systemdata/order-items/dfd6b6ca-836a-4961-9a8d-
c123b2265e00",
"searchResult" : "http://localhost:8765/systemdata/search-results/4d0f3841-f967-
41cf-a269-343462d28405",
"searchComposition" : null,
"searchCriteria" : null,
"spelExpression" : null,
"items" : [ ],
"parameters" : {
"developerMode" : "false",
"tenantId" : "83d23361-99a9-44c4-bb31-6857a5f02cf5",
"ruleName" : "",
"applicationId" : "14e259ef-df86-41d3-a723-964f5aba5b61"
},
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/order-items/66253064-5be1-46a2-a9c5-
3a6c9dd921b5/context"
},
"http://identifiers.emc.com/order-item" : {
"href" : "http://localhost:8765/systemdata/order-items/dfd6b6ca-836a-4961-9a8d-
c123b2265e00"
}
}
}
So, the context for the order is interesting. The totalSize and batchSize should be
clear on how many items needed to be processed.
The retriedAfterLockNotGranted normally isn’t set for an order item but could be
set for the context for a batch and basically is meant to avoid contention when
multiple batches are running concurrently.
Tenants
Applications
Rules
Rules can be used to apply retention, apply holds, and trigger events.
Rules are usually acted on by running the associated job.
24.1 Role Permission
Find the link relation for rules for that application and do a GET.
The example has omitted one of the 3 rules that would have been returned.
Supported parameters:
The body of the request on the post includes the name and type (both are
mandatory). Type is one of APPLY_RETENTION, APPLY_HOLD,
TRIGGER_EVENT.So how to do this is practice, first create the rule:
curl -X POST -H 'Content-Type: application/hal+json' -H 'Authorization: Bearer …' -H
'Accept: application/hal+json' -d '{"name":"test", "type" : "APPLY_RETENTION"}'
http://localhost:8765/systemdata/applications/07a0cf07-5dca-4d5c-831d-
b7110a8303d4/rules
You will be posting to the link from the contents link where you pass the file store
and file information.
So, the size is how many characters (this value is in bytes) that were in the file
that was uploaded, and you can follow the content-download linkrel to ensure
that the rule was uploaded correctly.
In InfoArchive, tasks can potentially take a large amount of time. An order item
tracks a particular operation and can indicate the status (completed, in progress,
queued, …)
Some tasks may act on a large collection of resources, and batching can be used
for scalability
Here is an architecture diagram for order items
System Order
Tenants
Items
Tenant
Order Items
Batch items
Jobs also may cause order items to be created, and these order items can be
viewed either from the system order items or from the job instance that created
the order item.
Order items can be seen by any user that can do an operation that uses
background processing.
For our examples, we can use the PhoneCallsGranular application. For brevity,
we will assume that the background search was already done from the IAWA
GUI.
The search result indicates where the search results can be found. Note that the
search result will not remain forever and when the background task is cleaned
up, the search result would also be cleaned.
It should be noted that depending on the order item, a link relation may be
available to restart the order item. Currently only a few operations support this
(such as applying hold), and this operation is only available if the operation failed
(EXCEPTION state).
25.3 Viewing batches associated with order items
Certain operations such applying holds use the batch framework and in this
section we will explore actually getting more information.
For this example, we will assume that someone used the GUI to apply a hold to
the search results in our previous background query.
Do a get on the link relation my-order-items from the tenant. This is the same
call as before.
{
"_embedded" : {
"orderItems" : [ {
"id" : "aacc76a7-c00b-4817-bce4-0e5e7fd34cff",
"createdBy" : "[email protected]",
"createdDate" : "2017-10-12T15:56:44.75-04:00",
"lastModifiedBy" : "system",
"lastModifiedDate" : "2017-10-12T15:57:13.044-04:00",
"version" : 11,
"name" : "Test_2017-10-12T15:56:44.745-04:00",
"type" : "APPLY_HOLD",
"application" : "http://localhost:8765/systemdata/applications/14e259ef-df86-41d3-a723-
964f5aba5b61",
"search" : null,
"searchComposition" : null,
"searchResult" : null,
"collection" : null,
"matter" : null,
"aip" : null,
"table" : null,
"holding" : null,
"xdbLibrary" : null,
"store" : null,
"purgeCandidateList" : null,
"log" : "http://localhost:8765/systemdata/applications/14e259ef-df86-41d3-a723-
964f5aba5b61/contents/a45af160-1f41-4c8c-89ac-adf718919c04",
"holdOperation" : null,
"items" : null,
"payload" : null,
"priority" : 0,
"eligibilityDate" : "2017-10-12T15:57:02.77-04:00",
"state" : "COMPLETE",
"userName" : "[email protected]",
"consumerContext" : null,
"dipCount" : 0,
"dipCrypto" : false,
"startDate" : "2017-10-12T15:56:52.077-04:00",
"endDate" : "2017-10-12T15:57:12.451-04:00",
"duration" : 20374,
"retentionDate" : "2017-10-13T15:56:44.745-04:00",
…,
"stateDescription" : "1 batch has completed",
…
"applicationName" : "PhoneCallsGranular",
"_links" : {
"self" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff"
},
"http://identifiers.emc.com/application" : {
"href" : "http://localhost:8765/systemdata/applications/14e259ef-df86-41d3-a723-
964f5aba5b61"
},
"http://identifiers.emc.com/log" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff/log"
},
"http://identifiers.emc.com/download-execution-log" : {
"href" : "http://localhost:8765/systemdata/applications/14e259ef-df86-41d3-a723-
964f5aba5b61/contents/a45af160-1f41-4c8c-89ac-adf718919c04/download?filename=Test_2017-10-
12T15:56:44.745-04:00.log.gz"
},
"http://identifiers.emc.com/delete" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff"
},
"http://identifiers.emc.com/batches" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff/batches"
},
"http://identifiers.emc.com/context" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff/context"
},
"http://identifiers.emc.com/groups" : {
"href" : "http://localhost:8765/systemdata/order-items/aacc76a7-c00b-4817-bce4-
0e5e7fd34cff/groups"
}
}
},
One thing that is worth mentioning is that the holdOperation field is null. In
previous versions, this would have contained information about the operation, if
the operation was batched, that information can now be found by following the
context link relation.
The state description indicates the overall status for the operation, in this case
only one batch was created (the batch size was 1000 and the example only
applied to 36 records).
So, if we wanted to view the logs for the order item, we could follow the link
relation for logs (or if we wanted to view the diagnostic logs, we could follow the
download-execution-logs link rel, doing a get).
We will now view the batches by following the link rel for batches:
Looking at the output, we can see that this was the first batch and see the size of
items (by default the batches have a maximum size of 1000)
let $query-str := concat("for $elem in /BASEBALL/MASTER/ROW ", $whereClause, " return $elem")
let $rows :=
for $elem in $main-query
let $fname := $elem/NAMEFIRST/text()
let $lname := $elem/NAMELAST/text()
let $bmonth := $elem/BIRTHMONTH/text()
let $bday := $elem/BIRTHDAY/text()
let $height := $elem/HEIGHT/text()
let $weight := $elem/WEIGHT/text()
let $debut := $elem/DEBUT/text()
let $finalGame := $elem/FINALGAME/text()
return
<row id="{string($elem/@table:id)}">
<column name="lastName">{ $lname }</column>
<column name="firstName">{ $fname }</column>
<column name="birthYear">{ $elem/BIRTHYEAR/text() }</column>
<column name="birthMonth">{ $bmonth }</column>
<column name="birthDay">{ $bday }</column>
<column name="height">{ $height }</column>
<column name="weight">{ $weight }</column>
<column name="debut">{ $debut }</column>
<column name="finalGame">{ $finalGame }</column>
</row>