integration-docs
Loading

Microsoft Office 365 Metrics Integration

Version 1.0.0 (View all)
Compatible Kibana version(s) 8.16.0 or higher
9.0.0 or higher
Supported Serverless project types
What's this?
Security
Observability
Subscription level
What's this?
Basic
Level of support
What's this?
Elastic

This integration uses the Microsoft Graph API and Microsoft Management API to collect essential metrics from Microsoft Office 365, offering detailed insights into user activity, application usage, and service health.

The following data can be collected with the Microsoft Office 365 Metrics integration:

Report API Data-stream Name Aggregation Level Required permissions
Microsoft 365 Active Users Service User Counts reportRoot: getOffice365ServicesUserCounts Microsoft 365 Active Users metrics Period-based Reports.Read.All
Microsoft 365 Groups Activity Group Detail reportRoot: getOffice365GroupsActivityDetail Microsoft 365 Groups Activity Group Detail Day-based Reports.Read.All
OneDrive Usage Account Detail reportRoot: getOneDriveUsageAccountDetail Microsoft 365 OneDrive Usage Account Detail Day-based Reports.Read.All
OneDrive Usage Account Counts reportRoot: getOneDriveUsageAccountCounts Microsoft 365 One Drive Usage metrics Period-based Reports.Read.All
OneDrive Usage File Counts reportRoot: getOneDriveUsageFileCounts Microsoft 365 One Drive Usage metrics Period-based Reports.Read.All
OneDrive Usage Storage reportRoot: getOneDriveUsageStorage Microsoft 365 One Drive Usage metrics Period-based Reports.Read.All
Outlook Activity Counts reportRoot: getEmailActivityCounts Microsoft 365 Outlook Activity metrics Period-based Reports.Read.All
Outlook App Usage Version Counts reportRoot: getEmailAppUsageVersionsUserCounts Microsoft 365 Outlook App Usage Version Counts metrics Period-based Reports.Read.All
Outlook Mailbox Usage Quota Status Mailbox Counts reportRoot: getMailboxUsageQuotaStatusMailboxCounts Microsoft 365 mailbox usage quota status metrics Period-based Reports.Read.All
Outlook Mailbox Usage Detail reportRoot: getMailboxUsageDetail Microsoft 365 mailbox usage detail metrics Period-based Reports.Read.All
SharePoint Site Usage Storage reportRoot: getSharePointSiteUsageStorage Microsoft 365 Sharepoint Site Usage metrics Period-based Reports.Read.All
SharePoint Site Usage Detail reportRoot: getSharePointSiteUsageDetail Microsoft 365 Sharepoint Site Usage metrics Period-based Reports.Read.All
Teams Device Usage User Counts reportRoot: getTeamsDeviceUsageUserCounts Microsoft 365 Teams Device Usage User Counts metrics Period-based Reports.Read.All
Teams User Activity User Counts reportRoot: getTeamsUserActivityUserCounts Microsoft 365 Teams User Activity User Counts metrics Period-based Reports.Read.All
Teams User Activity User Detail reportRoot: getTeamsUserActivityUserDetail Microsoft 365 Teams User Activity User Detail Day-based Reports.Read.All
Viva Engage Groups Activity Group Detail reportRoot: getYammerGroupsActivityDetail Microsoft 365 Viva Engage Groups Activity Day-based Reports.Read.All
Viva Engage Device Usage User Counts reportRoot: getYammerDeviceUsageUserCounts Microsoft 365 Viva Engage Device Usage User Counts metrics Period-based Reports.Read.All
Service Health reportRoot: getServiceHealth Microsoft 365 Service Health metrics No aggregation ServiceHealth.Read.All
Subscriptions subscribedSkus, subscriptions Microsoft 365 Subscriptions metrics No aggregation LicenseAssignment.Read.All
Teams Call Quality reportRoot: callRecords Microsoft 365 Teams Call Quality metrics No aggregation CallRecords.Read.All
Tenant Settings organization, adminReportSettings Microsoft 365 Tenant Settings No aggregation Organization.Read.All, ReportSettings.Read.All, Directory.Read.All
App Registrations List Applications Microsoft 365 App Registrations No aggregation Application.Read.All, User.Read(delegated)
Entra Features Organization, PremisesSync Microsoft 365 Entra Connect No aggregation Organization.Read.All, User.Read(delegated)
Entra ID users user, riskDetection Microsoft 365 Entra Connect User metrics No aggregation User.Read.All, IdentityRiskEvent.Read.All
Entra Agent agent Microsoft 365 Entra Agent metrics No aggregation RBAC role
Entra Alerts alerts Microsoft 365 Entra Alerts metrics No aggregation RBAC role

To use this package you need to enable datastreams you want to collect metrics for and register an application in Microsoft Entra ID (formerly known as Azure Active Directory).

Please make sure that the services for the enabled data streams are set up in Azure prior to using the integration.

Once the application is registered, configure or note the following to setup O365 metrics Elastic integration:

  1. Note Application (client) ID and the Directory (tenant) ID in the registered application's Overview page.
  2. Create a new secret to configure the authentication of your application.
    • Navigate to the Certificates & Secrets section.
    • Click New client secret and provide some description to create new secret.
    • Note the value which is required for the integration setup.
  3. Add permissions to your registered application.
    • Select and add the appropriate permissions from the available tiles.
    • For this package, we primarily use Graph APIs, so you can choose Microsoft Graph, which will display the Delegated and Application permission sections.
    • Refer to the Required Permissions column in the table under What data does this integration collect? section to identify the permissions required for each data stream and select accordingly. You can also refer to the Permissions section in the API documentation for each data stream to determine the necessary permissions.
    • Make sure Reports.Read.All from Microsoft Graph is added, as most APIs are report-based.
    • After the permissions are added, the admin will need to grant consent for a few permissions.

Once the secret is created and permissions are granted by admin, setup Elastic Agent's Microsoft O365 integration:

  • Click Add Microsoft Office 365.
  • Enable Collect Office 365 metrics via Graph API using CEL Input.
  • Add Directory (tenant) ID noted in Step 1 into Directory (tenant) ID parameter. This is required field.
  • Add Application (client) ID noted in Step 1 into Application (client) ID parameter. This is required field.
  • Add the secret Value noted in Step 2 into Client Secret parameter. This is required field.
  • Oauth2 Token URL can be added to generate the tokens during the oauth2 flow. If not provided, above Directory (tenant) ID will be used for oauth2 token generation.
  • Modify any other parameters as necessary.

Some data streams ingest data aggregated by a period, while other data streams ingest data aggregated by day, that is, daily.

For Period-based data streams, you have to configure the option Period during the setup. The supported values are: D7, D30, D90, and D180. As Day-based data streams ingest aggregated data per day, you have to configure the option Initial Interval to indicate how many days back you want to fetch the data. The supported values are from 1 to 28.

Microsoft 365 reports are typically available within 48 hours, but sometimes might take several days. As stated in the MS documentation, data quality is ensured by performing daily validation checks. During this process, you might notice differences in historical data in Microsoft 365 Reports in admin center.

To make sure there are no gaps and historical data is ingested into Elastic, the Microsoft Office 365 Metrics integration allows you to set the Sync Days in the past parameter for Day-based data streams. You can use this parameter to re-fetch the Microsoft 365 reports starting from N days in the past. By default, this paramater is 3. You can gradually increase this value if you see any discrepancies between Microsoft Reports and Elastic data (maximum value allowed is 28).

When data is re-fetched on same dates, the ingested data may be duplicated because Elastic data streams work in append-only design. For example, you may see duplicated documents in Elastic on the source data stream backed indices per resource (user/group/site) per report date. To maintain only the latest copy of the document, the Microsoft Office 365 Metrics integration installs Latest Transforms, one per report. These latest transform periodically pulls the data from the source data-stream-backed indices into a destination non-data-stream backed index. Therefore, the destination index only contains a single (latest) document per resource (user/group/site) per report date. Inside the reports dataset, you can distinguish between source and destination indices using the field labels.is_transform_source. This is set to true for source data-stream-backed indices and false for destination (latest) indices.

When searching for data, use a filter labels.is_transform_source: false to avoid getting duplicates. The Microsoft Office 365 Metrics integration dashboards has also this filter to show only the latest datapoints.

As the latest data is available in the destination indices, the source data-stream-backed indices are purged based on the ILM policy metrics-o365_metrics.<data_stream>-default_policy.

o365.metrics.report.name Source filter Source indices Destination filter Destination indices Destination alias
Microsoft 365 Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.groups_activity_group_detail_latest-* metrics-o365_metrics.groups_activity_group_detail_latest
OneDrive Usage Account Detail labels.is_transform_source: true metrics-o365_metrics.onedrive_usage_account_detail-* labels.is_transform_source: false metrics-o365_metrics.onedrive_usage_account_detail_latest-* metrics-o365_metrics.onedrive_usage_account_detail_latest
Teams User Activity User Detail labels.is_transform_source: true metrics-o365_metrics.teams_user_activity_user_detail-* labels.is_transform_source: false metrics-o365_metrics.teams_user_activity_user_detail_latest-* metrics-o365_metrics.teams_user_activity_user_detail_latest
Viva Engage Groups Activity Group Detail labels.is_transform_source: true metrics-o365_metrics.viva_engage_groups_activity_group_detail-* labels.is_transform_source: false metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest-* metrics-o365_metrics.viva_engage_groups_activity_group_detail_latest

Note: Sync Days in the past and Latest Transforms are only used in Day-based data streams, that is, for data streams aggregated per day.

By default, for all Microsoft 365 usage reports, the user names, emails, group, or site information are anonymized by Microsoft using MD5 hashes. You can revert this change for a tenant and show identifiable user, group, and site information if your organization allows it. To do this, follow these steps:

  1. Login to Microsoft 365 admin center
  2. Navigate to Settings -> Org Settings -> Services.
  3. Select Reports.
  4. Deselect the option Display concealed user, group, and site names in all reports, and save your changes.

Uses the Microsoft 365 Graph API to retrieve metrics from Microsoft 365.

Get details about Active Users Services User Count from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about users in Microsoft Entra ID.

Get details about Mailbox Usage Quota Status from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Mailbox Usage Detail from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Microsoft 365 groups activity by group from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by account from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by account counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by file counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about OneDrive usage by storage from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Outlook Activity from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Microsoft Outlook App Usage Version Counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about SharePoint Site Usage Detail from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about SharePoint Site Usage Storage from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Teams User Activity User Counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Teams User Activity User Detail from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Yammer Groups Activity Group Detail by group from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Yammer Device Usage User Counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Teams Device Usage User Counts from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Service Health from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Subscriptions from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Teams Call Quality from Microsoft Graph API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about tenant settings in Microsoft Entra ID.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about apps registered in Microsoft Entra ID. Microsoft API.

ECS Field Reference

Refer to the following document for detailed information on ECS fields.

Get details about Entra Features. Microsoft API.

Refer to the following document for detailed information on ECS fields.

Get details about Entra Agent. Microsoft Docs.

Refer to the following document for detailed information on ECS fields.

Get details about Entra Alerts. Microsoft Docs.

Refer to the following document for detailed information on ECS fields.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.