Log Insight Getting Started Guide
Log Insight Getting Started Guide
Log Insight Getting Started Guide
You can find the most up-to-date technical documentation on the VMware website at:
https://docs.vmware.com/
If you have comments about this documentation, submit your feedback to
[email protected]
VMware, Inc.
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com
Copyright © 2014–2018 VMware, Inc. All rights reserved. Copyright and trademark information.
VMware, Inc. 2
Contents
VMware, Inc. 3
Getting Started for vRealize Log Insight
Getting Started for vRealize Log Insight provides information about deploying and configuring
®
VMware vRealize™ Log Insight™, including how to size the vRealize Log Insight virtual appliance to
receive log messages.
Use this information when you want to plan or install your deployment. This information is written for
experienced Linux and Windows system administrators who are familiar with virtual machine technology
and data center operations.
VMware, Inc. 4
Before You Install
vRealize Log Insight 1
To start using vRealize Log Insight in your environment, you must deploy the vRealize Log Insight virtual
appliance and apply several basic configurations.
n Security Requirements
n Product Compatibility
n Minimum Requirements
n Sources that write log files and can run the vRealize Log Insight agent.
n Sources that can post log data with HTTP or HTTPS through the REST API. See
https://www.vmware.com/go/loginsight/api.
The vSphere log parser allows you to import vSphere log bundles in vRealize Log Insight.
Note Although vRealize Log Insight can handle historic data and real-time data simultaneously, you are
advised to deploy a separate instance of vRealize Log Insight to process imported log files.
See Import a Log Insight Archive into vRealize Log Insight in Administering vRealize Log Insight.
VMware, Inc. 5
Getting Started
Security Requirements
To ensure that your virtual environment is protected from external attacks, you must observe certain rules.
IT decision makers, architects, administrators, and others who must familiarize themselves with the
security components of vRealize Log Insight must read the security topics in Administering vRealize Log
Insight.
These topics provide concise references to the security features of vRealize Log Insight. Topics include
the product external interfaces, ports, authentication mechanisms, and options for configuration and
management of security features.
For information about securing your virtual environment, see the VMware vSphere Security Guide and the
Security Center on the VMware Web site.
Product Compatibility
vRealize Log Insight collects data over the syslog protocol and HTTP; can connect to vCenter Server to
collect events, tasks, and alarms data; and can integrate with vRealize Operations Manager to send
notification events and enable launch in context. Check the VMware vRealize Log Insight Release Notes
for latest updates on supported product versions.
Syslog Feeds
vRealize Log Insight collects and analyzes syslog data over the following ports and protocols:
n 514/UDP
n 514/TCP
n 1514/TCP (SSL)
You must configure environment components such as operating systems, applications, storage, firewalls,
and network devices to push their syslog feeds to vRealize Log Insight.
API Feeds
The vRealize Log Insight Ingestion API collects data over the following ports and protocols.
n 9000/TCP
VMware, Inc. 6
Getting Started
n 9543/TCP (SSL)
vSphere Integration
You can configure vRealize Log Insight to pull data for tasks, events, and alarms that occurred in one or
more vCenter Server instances. vRealize Log Insight uses the vSphere API to connect to vCenter Server
systems and collect data.
You can configure ESXi hosts to forward syslog data to vRealize Log Insight.
For compatibility information with specific versions of vCenter Server and ESXi, see the VMware Product
Interoperability Matrixes.
For information about connecting to a vSphere environment, see Connect vRealize Log Insight to a
vSphere Environment.
All supported versions of vCenter Operations Manager support notifications as well as Launch in Context.
n vRealize Log Insight can send notification events to vRealize Operations Manager.
See Configure vRealize Log Insight to Send Notification Events to vRealize Operations Manager.
n The launch in context menu of vRealize Operations Manager can display actions related to
vRealize Log Insight.
See Enable Launch in Context for vRealize Log Insight in vRealize Operations Manager.
Minimum Requirements
VMware distributes vRealize Log Insight as a virtual appliance in OVA file format. Various resources and
applications must be available for the virtual appliance to run successfully. For the most up-to-date
information about requirements, check the latest release notes.
Virtual Hardware
During deployment of the vRealize Log Insight virtual appliance, you can select from preset configuration
sizes according to the ingestion requirements for your environment. The presets are certified size
combinations of compute and disk resources, though you can add extra resources afterward. A small
configuration, described in the following table, consumes the fewest resources while remaining
supported. An extra-small configuration is also available, but it is suitable only for demos.
For complete resource requirements based on ingestion requirements, see Sizing the vRealize Log
Insight Virtual Appliance
VMware, Inc. 7
Getting Started
Memory 8 GB
vCPU 4
Supported Browsers
You can use one of the following browsers to connect to the vRealize Log Insight web user interface.
More recent browser versions also work with vRealize Log Insight, but have not been validated.
Note
n Internet Explorer Document mode must be set to Standards Mode. Other modes are not
supported.
n To use Internet Explorer with the vRealize Log Insight web client, Windows local storage integrity
level must be configured as low.
Account Passwords
Type Requirements
Root Unless you specify a root password or use guest customization during the deployment of the OVA, the
default credentials for the root user on the vRealize Log Insight virtual appliance are root/<blank>.
You are prompted to change the root account password when you first access the vRealize Log Insight
virtual appliance console.
User Account User accounts that you create in vRealize Log Insight 3.3 and later require a strong password. The
password must be at least 8 characters long and contain one uppercase character, one lowercase
character, one number, and one special character.
VMware, Inc. 8
Getting Started
Integration Requirements
Product Requirement
vCenter Server To pull events, tasks, and alarms data from a vCenter Server, you must provide a set of user
credentials for that vCenter Server. The minimum role required to register and unregister vRealize Log
Insight with a vCenter Server is Read-only. The role must be set at the vCenter Server level and
propagated to child objects. To configure ESXi hosts that a vCenter Server manages,
vRealize Log Insight requires additional privileges.
vSphere ESXi vSphere ESXi 6.0 update 1 or later is required to establish SSL connections to vRealize Log Insight.
vRealize Operations To enable notification events and the launch-in-context functionality in a vRealize Operations Manager
Manager instance, you must provide user credentials for that vRealize Operations Manager instance.
Port Protocol
22/TCP SSH
80/TCP HTTP
443/TCP HTTPS
Note External load balancers are not supported for use with vRealize Log Insight, including
vRealize Log Insight clusters.
Single Node
A basic vRealize Log Insight configuration includes a single node. The log sources are applications, OS
logs, virtual machine logs, hosts, the vCenter Server, virtual or physical switches and routers, storage
hardware, and so on. Log streams are transported to the vRealize Log Insight node using syslog (UDP,
TCP, TCP+SSL) or CFAPI (the vRealize Log Insight native ingestion protocol over HTTP or HTTPS),
either directly by an application, syslog concentrator, or the vRealize Log Insight agent installed on the
source.
VMware, Inc. 9
Getting Started
It is a best practice for single-node deployments to use the vRealize Log Insight integrated load balancer
(ILB) and to send queries and ingestion traffic to the ILB. This does not incur overhead and simplifies
configuration if you want to add nodes to create a cluster for your deployment in the future.
Clusters
The use of clusters is recommended for production environments. Two-node clusters are not supported.
Nodes in clusters should all be of the same size and in the same datacenter. The ILB used with clusters
requires that nodes be in the same L2 network.
A minimum of medium-sized nodes are recommended for production environments. If you anticipate
working with a high number of concurrent queries, including alerts, consider using large-sized nodes. For
information about sizing, see Sizing the vRealize Log Insight Virtual Appliance.
Single Cluster
A vRealize Log Insight single cluster configuration includes a minimum of three nodes to twelve nodes
and uses the ILB.
A single log message is present in only one location within the cluster. The cluster remains up and
available to ingest data and serve queries during the temporary unavailability of any single node results in
the cluster. Removal and reintroduction of a cluster node is not supported. A minimum of three healthy
nodes is required, or, four nodes to support a single-node failure scenario.
A vRealize Log Insight cluster's virtual IP uses a Linux Virtual Server in Direct Server Return Mode (LVS-
DR) for load balancing. Direct Server Return is more efficient than routing all response traffic through a
single cluster member. However, it also resembles spoofed traffic, which NSX Distributed Firewall blocks.
The vRealize Log Insight virtual machines must be excluded from VMware NSX Distributed Firewall
Protection.
The design is extended through the addition of multiple forwarder clusters at remote sites or clusters.
Each forwarder cluster is configured to forward all its log messages to the main cluster and users connect
to the main cluster, taking advantage of CFAPI for compression and resilience on the forwarding path.
Forwarder clusters configured as top-of-rack may be configured with a larger local retention.
VMware, Inc. 10
Getting Started
Standalone Deployment
You can change the appliance settings to meet the needs of the environment for which you intend to
collect logs during deployment.
vRealize Log Insight provides preset VM sizes that you can select from to meet the ingestion
requirements of your environment. These presets are certified size combinations of compute and disk
resources, though you can add extra resources afterward. A small configuration consumes the fewest
resources while remaining supported. An extra small configuration is suitable only for demos.
You can use a syslog aggregator to increase the number of syslog connections that send events to
vRealize Log Insight. However, the maximum number of events per second is fixed and does not depend
on the use of a syslog aggregator. A vRealize Log Insight instance cannot be used as a syslog
aggregator.
VMware, Inc. 11
Getting Started
n Each ESXi host sends up to 10 messages per second with an average message size of 170
bytes/message, which is roughly equivalent to 150 MB/day/host.
Note For large installations, you must upgrade the virtual hardware version of the vRealize Log Insight
virtual machine. vRealize Log Insight supports virtual hardware version 7 or later. Virtual hardware version
7 can support up to 8 virtual CPUs. Therefore, if you plan to provision 16 virtual CPUs, you must upgrade
to virtual hardware version 8 or later for ESXi 5.x. You use the vSphere Client to upgrade the virtual
hardware. If you want to upgrade virtual hardware to the latest version, read and understand the
information in the VMware knowledge base article Upgrading a virtual machine to the latest hardware
version (1010675) .
Cluster Deployment
Use a medium configuration, or larger, for the master and worker nodes in a vRealize Log Insight cluster.
The number of events per second increases linearly with the number of nodes. For example, in a cluster
of 3-12 nodes (two-node clusters are not supported), the net in a 12-node cluster is 180,000 events per
second (EPS), or 2.7 TB of events per day.
Procedure
1 Install the vRealize Log Insight Management Pack into vRealize Operations Manager.
The vRealize Log Insight Management Pack is required for the Launch in Context functionality
between the two products. The vRealize Log Insight Management Pack is available with the
vRealize Operations Manager download file or on the VMware Solution Exchange website.
VMware, Inc. 12
Getting Started
3 Configure vRealize Log Insight alerts to forward information to vRealize Operations Manager.
See Configure vRealize Log Insight to Send Notification Events to vRealize Operations Manager in
Administering vRealize Log Insight.
4 Enable vRealize Operations Launch In Context to query logs in vRealize Log Insight.
See Enable Launch in Context for vRealize Log Insight in vRealize Operations Manager in
Administering vRealize Log Insight.
VMware, Inc. 13
Life Cycle of an Event 2
Understanding how vRealize Log Insight works with messages and events is key to product usage.
The end-to-end life cycle of a log message or event includes multiple stages in vRealize Log Insight from
agent read, parse, ingestion, indexing (buckets), alerting, query, archive (bucket seal and ship), and
deletion.
2 It is picked up and sent to vRealize Log Insight (inside and/or outside vRealize Log Insight) in one of
the following ways:
n By a vRealize Log Insight agent using ingestion API or syslog
n If the event is declined, the client handles declines by means of UDP drops, TCP with protocol
settings, or CFAPI with a disk-backed queue.
4 The event is passed through the vRealize Log Insight ingestion pipeline, from which the following
steps occur:
n A keyword index is created or updated. The index is stored in proprietary format on local disk.
n The event is stored in compressed proprietary format on the local disk in a bucket.
VMware, Inc. 14
Getting Started
Event Storage
Each event is stored in a single on-disk bucket. When working with buckets, be aware of the following
behaviors and characteristics.
n Buckets can reach a maximum size of 1 GB. When a bucket reaches 1 GB, it is sealed and can no
longer be written to and is marked as to be archived. After a sealed bucket is archived, it is marked as
archived. This means an event may be retained locally and in the archives at the same time.
n Buckets are not replicated across vRealize Log Insight nodes. If you lose a node then you lose the
data on that node.
n vRealize Log Insight deletes old buckets when available space on the /storage/core partition is less
than 3%. Deletion is done using a FIFO model.
Note A near-full /storage/core partition is usual and expected. That partition should never reach
100% because vRealize Log Insight manages that partition. However, you should not attempt to store
data on that partition as it may interfere with the old bucket delete process.
Event Management
Be aware of the following characteristics and behaviors of vRealize Log Insight events and event
management.
n After an event is deleted locally it can no longer be queried unless it is imported from the archive
using the command-line interface.
n After all events for a machine learning cluster are deleted from vRealize Log Insight, the cluster is
removed.
VMware, Inc. 15
Getting Started
n vRealize Log Insight automatically rebalances all incoming events fairly across nodes in the cluster.
For example, even if a node is explicitly sent to an event, it may not be the node to ingest the event.
n Event metadata is stored in a proprietary format on a single vRealize Log Insight node and not in a
database.
VMware, Inc. 16
Installing vRealize Log Insight 3
vRealize Log Insight is delivered as a virtual appliance that you deploy in your vSphere environment.
After reviewing Sizing the vRealize Log Insight Virtual Appliance, proceed to Deploy the vRealize Log
Insight Virtual Appliance. Whether you have a single node deployment or a clustered deployment, follow
the standard OVF deployment procedure described in this section.
Prerequisites
n Verify that you have a copy of the vRealize Log Insight virtual appliance .ova file.
n Verify that you have permissions to deploy OVF templates to the inventory.
n Verify that your environment has enough resources to accommodate the minimum requirements of
the vRealize Log Insight virtual appliance. See Minimum Requirements.
n Verify that you have read and understand the virtual appliance sizing recommendations. See Sizing
the Log Insight Virtual Appliance.
Procedure
3 On the Select Configuration page, select the size of the vRealize Log Insight virtual appliance based
on the size of the environment for which you intend to collect logs.
VMware, Inc. 17
Getting Started
vRealize Log Insight provides preset VM sizes that you can select from to meet the ingestion
requirements of your environment. These presets are certified size combinations of compute and disk
resources, though you can add extra resources afterward. A small configuration consumes the fewest
resources while remaining supported. An extra small configuration is suitable only for demos.
You can use a syslog aggregator to increase the number of syslog connections that send events to
vRealize Log Insight. However, the maximum number of events per second is fixed and does not
depend on the use of a syslog aggregator. A vRealize Log Insight instance cannot be used as a
syslog aggregator.
Note If you select Large, you must upgrade the virtual hardware on the vRealize Log Insight virtual
machine after the deployment.
n Thick Provision Lazy Zeroed creates a virtual disk in a default thick format. Space required for
the virtual disk is allocated when the virtual disk is created. The data remaining on the physical
device is not erased during creation, but is zeroed out on demand later, on first write from the
virtual appliance.
n Thick Provision Eager Zeroed creates a type of thick virtual disk that supports clustering
features such as Fault Tolerance. Space required for the virtual disk is allocated at creation time.
In contrast to the flat format, the data remaining on the physical device is zeroed out when the
virtual disk is created. It might take much longer to create disks in this format than to create other
types of disks.
Important Deploy the vRealize Log Insight virtual appliance with thick provisioned eager zeroed
disks whenever possible for better performance and operation of the virtual appliance.
n Thin Provision creates a disk in thin format. The disk grows as the data saved on it grows. If
your storage device does not support thick provisioning disks or you want to conserve unused
disk space on the vRealize Log Insight virtual appliance, deploy the virtual appliance with thin
provisioned disks.
Note Shrinking disks on the vRealize Log Insight virtual appliance is not supported and might result
in data corruption or data loss.
VMware, Inc. 18
Getting Started
5 (Optional) On the Setup networks page, set the networking parameters for the vRealize Log Insight
virtual appliance.
If you do not provide network settings, such as an IP address, DNS servers, and gateway information,
vRealize Log Insight utilizes DHCP to set those settings.
Caution Do not specify more than two domain name servers. If you specify more than two domain
name servers, all configured domain name servers are ignored in the vRealize Log Insight virtual
appliance.
6 (Optional) On the Customize template page, set network properties if you are not using DHCP.
7 (Optional) On the Customize template page, select Other Properties and set the root password for
the vRealize Log Insight virtual appliance.
The root password is required for SSH. You can also set this password through the VMware Remote
Console.
For information on deploying virtual appliances, see the User's Guide to Deploying vApps and Virtual
Appliances.
After you power on the virtual appliance, an initialization process begins. The initialization process
takes several minutes to complete. At the end of the process, the virtual appliance restarts.
9 Navigate to the Console tab and check the IP address of the vRealize Log Insight virtual appliance.
What to do next
n If you want to configure a standalone vRealize Log Insight deployment, see Configure New Log
Insight Deployment.
The vRealize Log Insight Web interface is available at https://log-insight-host/ where log-insight-host
is the IP address or host name of the vRealize Log Insight virtual appliance.
All settings that you modify during the initial configuration are also available in the Administration web
user interface.
VMware, Inc. 19
Getting Started
For information about the trace data that vRealize Log Insight might collect and send to VMware when
you participate in the Customer Experience Improvement Program, see Chapter 4 The Customer
Experience Improvement Program.
Prerequisites
n In the vSphere Client, note the IP address of the vRealize Log Insight virtual appliance. For
information about locating the IP address, see Deploy the vRealize Log Insight Virtual Appliance.
n Verify that you are using a supported browser. See Minimum Requirements.
n Verify that you have a valid license key. You can request an evaluation or permanent license key
through your account on My VMware™ at https://my.vmware.com/.
n If you want to use local, vCenter Server, or Active Directory credentials to integrate
vRealize Log Insight with vRealize Operations Manager, verify that these users are imported in
vRealize Operations Manager Custom user interface. For instructions about configuring LDAP, see
the vRealize Operations Manager documentation.
Procedure
1 Use a supported browser to navigate to the web user interface of vRealize Log Insight.
The URL format is https://log_insight-host/, where log_insight-host is the IP address or host name of
the vRealize Log Insight virtual appliance.
3 Set the password for the Admin user and click Save and Continue.
Optionally, you can provide an email address for the admin user.
4 Enter the license key, click Add License Key, and click Save and Continue.
5 On the General Configuration page, type the email address to receive system notifications from
vRealize Log Insight.
6 If you are using webhooks to send notifications to vRealize Operations Manager or a third-party
application, enter a space-separated list of URLs in the Send HTTP Post System Notifications To
field.
7 (Optional) To leave the Customer Experience Improvement Program, deselect the Join the VMware
Customer Experience Program option. Click Save and Continue.
VMware, Inc. 20
Getting Started
8 On the Time Configuration page, set how time is synchronized on the vRealize Log Insight virtual
appliance and click Test.
Option Description
NTP server (recommended) By default, vRealize Log Insight is configured to synchronize time with public NTP
servers. If an external NTP server is not accessible due to firewall settings, you
can use the internal NTP server of your organization.
Use commas to separate multiple NTP servers.
ESX/ESXi host If no NTP servers are available, you can sync the time with the ESXi host where
you deployed the vRealize Log Insight virtual appliance.
10 To enable outgoing alert and system notification emails, specify the properties of an SMTP server.
To verify that the SMTP configuration is correct, type a valid email address and click
Test.vRealize Log Insight sends a test email to the address that you provided.
After the vRealize Log Insight process restarts, you are redirected to the Dashboards tab of
vRealize Log Insight.
What to do next
n
Go to the Administration page by selecting the drop-down menu icon in the navigation bar. From
the vSphere Integration page, configure vRealize Log Insight to pull tasks, events, and alerts from
vCenter Server instances, and to configure ESXi hosts to send syslog feeds to vRealize Log Insight.
n See the topic Assign a Permanent License to vRealize Log Insight in Administering
vRealize Log Insight.
n See the topic Install the vRealize Log Insight Adapter in vRealize Operations Manager Standalone in
Administering vRealize Log Insight.
n Install the vRealize Log Insight Windows Agent to collect events from Windows event channels,
Windows directories, and flat text log files. See the topic Installing the vRealize Log Insight Windows
Agent as a Windows Service in Administering vRealize Log Insight.
VMware, Inc. 21
Getting Started
vRealize Log Insight can scale out by using multiple virtual appliance instances in clusters. Clusters
enable linear scaling of ingestion throughput, increase query performance, and allow high-availability
ingestion. In cluster mode, vRealize Log Insight provides master and worker nodes. Both master and
worker nodes are responsible for a subset of data. Master nodes can query all subsets of data and
aggregate the results. You might require more nodes to support site needs. You can use up to twelve
nodes in a cluster.
Important A vRealize Log Insight cluster must have a minimum of three nodes.
Prerequisites
n In the vSphere Client, note the IP address of the worker vRealize Log Insight virtual appliance.
n Verify that you have the IP address or host name of the master vRealize Log Insight virtual appliance.
n Verify that you have an administrator account on the master vRealize Log Insight virtual appliance.
n Verify that the versions of the vRealize Log Insight master and worker nodes are in sync. Do not add
an older version vRealize Log Insight worker to a newer version vRealize Log Insight master node.
n You must synchronize the time on the vRealize Log Insight virtual appliance with an NTP server. See
Synchronize the Time on the Log Insight Virtual Appliance.
n For information on supported browser versions, see the vRealize Log Insight Release Notes.
Procedure
1 Use a supported browser to navigate to the Web user interface of the vRealize Log Insight worker.
The URL format is https://log_insight-host/, where log_insight-host is the IP address or host name of
the vRealize Log Insight worker virtual appliance.
3 Enter the IP address or host name of the vRealize Log Insight master and click Go.
The worker sends a request to the vRealize Log Insight master node to join the existing deployment.
5 Log in as an administrator.
6 Click Allow.
The worker node joins the existing deployment and vRealize Log Insight begins to operate in a
cluster.
What to do next
n Add more worker nodes as needed. The cluster must have a minimum of three nodes.
VMware, Inc. 22
The Customer Experience
Improvement Program 4
This product participates in VMware’s Customer Experience Improvement Program (“CEIP”).
Details regarding the data collected through CEIP and the purposes for which it is used by VMware are
set forth at the Trust & Assurance Center at http://www.vmware.com/trustvmware/ceip.html.
To join or leave the CEIP for this product, see "Join or Leave the VMware Customer Experience Program"
in Administering vRealize Log Insight.
VMware, Inc. 23