setting up the virtual machine

Install Virtual Box (or VMWare Workstation should work as well)

Go to File, Import Appliance, and point the wizard at the .ova file
(you can copy the file locally first if you want).

login with learnsoquery/learnsoquery, or

After install and start, you can access your virtual box locally via ssh

ssh -p 2222 [email protected]

Uptycs Intro to osquery

PART ONE
 Why osquery?
Open-source endpoint
Originally developed at Facebook
Ask Questions and Get Answers!
Turns system calls into virtual SQL tables
(Structured Query Language)

Universal Endpoint
Cross-platform
Cross-virtualization (level)
Cross-maturity
 Design Principles of osquery
Read Only
Only modifies files needed to run
Polite & respectful of privacy
Developer choice of tables/content
Non-intrusive
user mode, controls impact on machine

Universal language of SQL

Tries to be consistent across all OS
(when possible)
 Abstract the Operating Systems to SQL
ps -ef | grep apached
or
ps -C apached
or
ps ax | grep apache

But if you standardize on SQL

Select * from processes where name like ‘%apache%’

 Universal vs. OS Specific
Osquery was developed on *nix systems
Reverse problem of a lot of security software
Ported to Windows later on
Tries to be universal, but there are specific idiosyncrasies structures of
different OS that make this hard

macOS and Windows have the most OS specific tables

Windows is structured differently, period
macOS has a lot of customized features that are not standard *nix
More “custom” work has gone into tables for Windows & macOS
 Overview of Tables in osquery
Using SQL instead of System Utilities ==
Over 200 tables, some OS-specific

ALSO – "Special Tables"

osquery_ tables
How to capture events and use _event tables
Add-on Utilities
Augeas, Prometheus, Docker, Extensions and more!
acpi_tables curl etc_protocols launchd plist system_controls
ad_config curl_certificate etc_services launchd_overrides power_sensors system_info
alf device_file event_taps listening_ports preferences temperature_sensors
alf_exceptions device_firmware extended_attributes lldp_neighbors process_envs time
alf_explicit_auths device_hash fan_speed_sensors load_average process_events time_machine_backups
alf_services device_partitions file logged_in_users process_memory_map time_machine_destinations
app_schemes disk_encryption file_events magic process_open_files uptime
apps disk_events firefox_addons managed_policies process_open_sockets usb_devices
apt_sources dns_resolvers gatekeeper mounts processes user_events
arp_cache docker_container_labels gatekeeper_approved_apps nfs_shares prometheus_metrics user_groups
asl docker_container_mounts groups nvram python_packages user_interaction_events
augeas docker_container_networks hardware_events opera_extensions quicklook_cache user_ssh_keys
authorization_mechanisms docker_container_ports hash os_version routes users
authorizations docker_container_processes homebrew_packages osquery_events safari_extensions virtual_memory_info
authorized_keys docker_container_stats interface_addresses osquery_extensions sandboxes wifi_networks
block_devices docker_containers interface_details osquery_flags shared_folders wifi_status
browser_plugins docker_image_labels iokit_devicetree osquery_info sharing_preferences wifi_survey
carbon_black_info docker_images iokit_registry osquery_packs shell_history xprotect_entries
carves docker_info kernel_extensions osquery_registry signature xprotect_meta
certificates docker_network_labels kernel_info osquery_schedule sip_config xprotect_reports
chrome_extensions docker_networks kernel_panics package_bom smbios_tables yara
cpu_time docker_version keychain_acls package_install_history smc_keys yara_events
cpuid docker_volume_labels keychain_items package_receipts startup_items
crashes docker_volumes known_hosts pci_devices sudoers
crontab etc_hosts last platform_info suid_bin
 Osquery Files
Binaries: /usr/bin/
osqueryi
osqueryd
Config: /etc/osquery/
osquery.conf
osquery.flags
Database /var/osquery/
osquery.db
Logs /var/log/osquery
osquery.INFO
In depth config docs: https://osquery.readthedocs.io/en/stable/deployment/configuration/
 Osquery Flags
There are a LOT of flags – too many to cover here*

osquery> select count(*) from osquery_flags;

count(*) = 169

--verbose & --config-check

debugging

--config_path & --flagfile

configuration

--disable_events
event listeners
* https://github.com/facebook/osquery/blob/master/docs/wiki/installation/cli-flags.md
 Queries vs Query Packs
you'll hear references to both queries and query packs

A query is a set SQL request to get a specific set of data:

select * from processes

A query pack is a group of queries designed to be

distributed together.
 Query Packs
JSON with queries and configuration in them
Intervals to run the queries
What types of actions or modes to use
What platform or version to run on
Other configuration options
* https://osquery.readthedocs.io/en/stable/deployment/configuration/#query-packs
 Summary
Why osquery?
Design principles
Moving from diverse output to SQL
Tables
Configuration Flags
Query Packs
Uptycs Intro to osquery

PART ONE – THE END

Uptycs Intro to osquery

PART TWO
 setting up the virtual machine
Install Virtual Box (or VMWare Workstation should work as well)

Go to File, Import Appliance, and point the wizard at the .ova file
(you can copy the file locally first if you want).

login with learnsoquery/learnsoquery, or

After install and start, you can access your virtual box locally via ssh

ssh -p 2222 [email protected]

 Install & Verify
Osquery is already on your VM at /var/tmp/osquery*

To install
sudo dpkg -i /var/tmp/osquery.deb

To verify
$ osqueryi <enter>
osquery> 
osquery> select * from uptime; <enter>
*Otherwise, the best place to start is at https://osquery.io/downloads/
 Shockingly Easy (hopefully)
So, that took what, 2 minutes?

Very easy to get osquery deployed on a host

Works with almost all Package Managers

Used to be a bit trickier on Windows

Likely need to do code signing if you “roll your own.”

 osquery Shell Commands
If you’re not still in osquery, go ahead and type
osqueryi again.

Type .help at the osquery> prompt to see

some of the osquery shell commands.
 Some osquery Shell Commands
.exit / .quit - gets you out of the shell
.show & .features – show you some of osquery’s settings in one
place
.mode pretty – “default” view that shows SQL “tables”
.mode line – different view that shows one result per line
.tables $search – lists (some) tables currently available in
osquery
.schema $table -- shows you * trythe buildtable_info('users');
also: PRAGMA statement for a table*
 SQLite Syntax
osquery uses SQLite* as its SQL interpreter

only SELECT statements

Read Only (in osquery core, at least)

a few functions don’t work, most do * https://www.sqlite.org/lang.html

 Simple Queries
select * from <table_name>;

osquery> select * from uptime;  protip: the semicolon

osquery> select
    ...> *
    ...> from
    ...> uptime
    ...> ;
 Why Run osquery with sudo?
Try this:

osqueryi> select * from shadow;

What results do you get?

Now do .exit, and try it again running osqueryi with sudo:

$ sudo osqueryi
What is different? Why?
 SQL - Simple Queries
select * from users;

select * from users limit 5;

protip: (use limit 1 or .schema users to get column names)

select count(*) from users;

select uid, gid, username, description, directory

from users limit 5;
 SQL – ORDER BY
Let's take what we have here and "sort" the results

select uid, gid, username, description, directory

from users limit 5;

select uid, gid, username, description, directory

from users order by gid asc limit 5;  Protip – before LIMIT

How would you do this by username?

 SQL - WHERE and LIKE
If you want to get data from a specific row, you can get
ones that match data by using WHERE (protip – quotes!)

select uid, gid, username, description, directory from users

where username='systemd-timesync';

This gets you the one specific row. However, what if you
want all the “systemd” accounts?
 SQL - WHERE and LIKE
You can use the LIKE operator and wildcards before or
after a string to find partial matches

select uid, gid, username, description, directory from users

where username like 'system%';

Can you figure out how to get the same results using the
directory column?
 SQL - JOINing Table Data
Take a look at processes table

Processes table is very "noisy" – try just a few columns

select pid, name, cmdline from

processes limit 5;
 SQL - JOINing Table Data
Processes run as users.

You may want to see what the username is for a given process.

What columns do the users and processes table have in

common?
 SQL - JOINing Table Data
Users Columns Processes Columns

uid pid
gid name
uid_signed path
gid_signed cmdline
username state
description cwd
directory root
shell uid
uuid gid
 SQL - JOINing Table Data
Both tables have a "uid" column for the user ID number

Let's take the process data we need with user id, and then map
the corresponding user name from the users table.

select p.pid, p.name, u.uid, u.username

from processes p
join users u on u.uid=p.uid;
 “consider JOINing against the users table”
select * from shell_history;

W0617 21:41:10.583434  1534 virtual_table.cpp:557]

The shell_history table returns data based on the
current user by default, consider JOINing against
the users table

select * from shell_history WHERE shell_history.uid

IN (SELECT uid from users);
 Tables that May Need "join against users:"

account_policy_data firefox_addons
authorized_keys known_hosts
browser_plugins opera_extensions
crashes safari_extensions
chrome_extensions shell_history
 Date Functions
osquery> .mode line
select local_time from time;

> local_time = 1529608143 <-- in unix epoch time

select datetime(local_time, 'unixepoch', 'localtime')

as formatted_time from time;

> formatted_time = 2018-06-21 15:09:09

* https://www.sqlite.org/lang_datefunc.html
 Math
osquery> select path, type, blocks_available, blocks_size from mounts
where path = '/';
path = /
type = ext4
blocks_available = 22653804
blocks_size = 4096

osquery> select path, type, round((blocks_available * blocks_size *10e-

10),2) as gigs_free from mounts where path='/';
path = /
type = ext4
gigs_free = 92.79
Uptycs Intro to osquery

PART TWO, THE END

Uptycs Intro to osquery

PART THREE
 Special Tables
osquery_
_events

"File" tables
Extensions
 Special Tables - osquery_
tables that start with "osquery_" are diagnostic tables for osquery

osquery_
events – shows current event publishers and subscribers
extensions – show registered extensions
flags – show all recognized flags, and current status
info – status of current installation
packs – shows any registered query packs
registry – summary of components registered with osquery
schedule – scheduled queries from config & query packs
 Special Tables - _events
_events tables do not work like “normal tables”

select * from processes;

Show you all processes at a given point in time
Query later, shows you the delta between point 1 in time and point 2

select * from process_events;

Saves up events after first query at point 1 in time
Query at point 2 in time gets _ALL_ events since point 1, unless buffer has been
overwritten
 Special Tables - _events
_events tables use a pub/sub model
There are a few different event publishers per OS
There are standardized listeners in osquery

NOT consistent across all OS yet, unfortunately

Windows is specifically lacking consistency with others
* Catching Everything with osquery Events
https://www.youtube.com/watch?v=yFfWv9wAhyA
 Files, Monitoring, "Carving" and Compliance
osquery provides several different ways to get information about files
ALL are path-specific!

The file table give information about a file when you do the query

The file_events table gives you information about changes to specific files
and file paths

The carves table gives you the ability to carve files from an operating system
(but not trivial)
 Extensions
Extensions are code that runs alongside osquery, but is not a part
of the osquery code
Can be written in Python or Go (and possibly other languages)
Extensions can do all sorts of things – including things that
violate osquery design principles
Extensions can read things that are not in osquery core tables
Extensions can now _WRITE_ to the endpoint, changing configurations
Extensions can allow for the capture of additional forensic data
Uptycs Intro to osquery

PART THREE
Uptycs Intro to osquery

EXPLORING SPECIAL TABLES

 osquery_events Table
sudo osqueryi
osquery> select * from osquery_events;

What do you see in the "events" column?

How about the "active" column?

Events are turned on/off through config & flags.

osqueryi sees none of them by default.
How do we fix this?
 Configuring osqueryi to See Events
For this we need to use flags*:

--disable_events=false
--disable_audit=false
--audit_allow_config=true
These can be set from either command line or the flags file.
*https://osquery.readthedocs.io/en/stable/deployment/process-auditing/
 Configuring osqueryi to See Events
Take a look at these file:

/etc/osquery/osquery.conf
/etc/osquery/osquery.flags

what command flags are in the second file?

 Launching osqueryi to See Events
Run osqueryi as follows (all on one line):

sudo osqueryi --disable_events=false

--config_path=/etc/osquery/osquery.conf
--flagfile=/etc/osquery/osquery.flags

What flags are being sent via the flags file?

What flags are being sent via the command line?
 Querying Events
now, take a look at osquery_events again

osquery> select * from osquery_events;

What is different?
 Querying Events
let's refine the events we are looking at

select * from osquery_events where active=1;

What are we seeing here?

All publishers and subscribers are not equal.

file_events and yara_events require additional config.

 File Integrity Monitoring
open a new shell, your choice

navigate to /etc/osquery and examine the osquery.conf file

$ less /etc/osquery/osquery.conf

find the "file_paths" section

This is where File Event / File Integrity Monitoring (FIM) is configured

 File Integrity Monitoring
  "file_paths": {
    "monitor_this": [
      "/var/tmp/filetest/%%"
    ]
"webserver_fim": [
"/var/www/html/%%"

JSON snippet – specifies a label and a file path.

The file path can include % or %% wildcards.
They do NOT always work the way you expect*.
* https://www.uptycs.com/blog/wildcards-and-globbing-in-osquery
 File Integrity Monitoring
  "file_paths": {
    "monitor_this": [
      "/var/tmp/filetest/%%"
    ]
"webserver_fim": [
"/var/www/html/%%"

If you had a webserver, what would you expect this line

to do?
 Let's simulate some "bad behavior"
In a separate command prompt from osqueryi

Type ~/Documents/backdoor.sh

(NO PEEKING)
 Let's Check File Events
in the osquery terminal, look at osquery_events again
you should now see some file events as well.

osqueryi> select * from file_events;

will return events now.

What do they show you?

 More File Events
If you want, cd to /var/www/html/
and delete the files that are not index.html

What do you see in file_events now?

What is different?
 Let's Investigate Further
What else did that "script" do? (don't PEEK!)

Let's say it started a backdoor. How might we

look for it?
 Let's Investigate Further
Some tables to think about (for this instance):

processes
process_open_sockets
 Let's Investigate Further
What if I told you that almost no service on this server ran on
a high-order port?
(greater than 1024)

How could you combine processes and

process_open_sockets so that you can see what process is
listening on what port?
How about users to see what user a process ran as?
 Example socket query
SELECT pid,
local_address,
local_port,
remote_address,
remote_port
FROM process_open_sockets
WHERE local_port > 1024;
 Example Join
SELECT p.pid,
u.username,
p.name,
p.path,
po.local_address,
po.local_port,
po.remote_address,
po.remote_port
FROM processes p
JOIN process_open_sockets po on p.pid=po.pid
JOIN users u on p.uid=u.uid
WHERE po.local_port > 1024;
 Let's Investigate Further
What about process_events?

(take a look)
 Let's Investigate Further
What about process_events?

try SELECT pid, path, cmdline, uid FROM process_events;

It's almost cheating, isn't it?

What other things could we join with process events?

 Example Join
Try this to show user names for the events:

SELECT p.pid, u.username, p.path,

p.cmdline FROM process_events p JOIN
users u on p.uid=u.uid limit 1;

Remove the limit to see them all

Uptycs Intro to osquery

PART THREE – THE END

Uptycs Intro to osquery

PART FOUR
Uptycs Intro to osquery

AUGEAS AND PROMETHEUS

 Special Tables – Augeas*
Augeas – a separate open-source project*
Reads configuration files into key-value pairs
Used by osquery to make *nix config files parse-able by
osquery without having to write a unique table for each one
Lenses
What comes with osquery
Rolling your own
This is also an open-source project – contribute back!
* http://augeas.net/
 Special Tables – Prometheus*
Osquery is not that great for performance metrics
Prometheus is an open source metrics collection &
publishing project
Prometheus has a LOT of metrics it returns
With the Prometheus table, you can query a
Prometheus API and get results inside of osquery
* https://prometheus.io/
 Augeas
Augeas can load configuration files into key value pairs

Does this through the augeas table

The query MUST have a path to a config file to work efficiently

select * from augeas where

path='/etc/sudoers';
 Augeas
select label, value from augeas where
path='/etc/sudoers' and label not like '%comment%';

select label, value from augeas where

path='/etc/ssh/sshd_config' and label not like
'%comment%';

/etc/crontab – compare to 'select * from crontab'

/etc/hosts – compare to 'select * from hosts'
 Prometheus
Prometheus is an open-source monitoring solution

Prometheus exports counters and publishes them to a local API endpoint on a small server it runs.
You can query this API in a table built into osquery.

Let's start up the services

sudo systemctl start prometheus

sudo systemctl start node_exporter
 Prometheus
Prometheus initially just reports on its own Go application.

Node Exporter is a Prometheus module that gathers a large variety of metrics from the
computer in question and publishes them into Prometheus.

Prometheus publishes to
http://localhost:9100/metrics
http://localhost:9090/metrics

You can see the entries telling osquery about this in the
/etc/osquery/osquery.conf file.
 Prometheus
let's try to take a look at prometheus in osquery.

Prometheus can be _very_ noisy

select * from prometheus_metrics limit 10;

This is where LIKE statements will come in handy

 Prometheus
First you see statistics on the actual Go application.
Let's look for things from the node_exporter and limit columns

select metric_name, metric_value from

prometheus_metrics where metric_name like 'node_cpu%';

Try node_disk, node_filesystem, node_memory, node_netstat,

node_network to see some of the other sets of info available.
 Augeas and Prometheus – and much more
Augeas was one way of extending osquery to read config files

Prometheus is another – for metrics

other programs are supported in osquery tables

Extensions can do almost anything you can dream up

Uptycs Intro to osquery

PART FIVE
Uptycs Intro to osquery

SCALE AND COMMUNITY

 Osquery at Scale – the Challenges
osquery is really powerful

but there's a lot more to deploying it a scale beyond

the open-source project

You have to create an ecosystem, with osquery being

just one (important) component of it
Osquery at Scale – General Architecture
Putting Open-Source Components Together

*https://www.uptycs.com/blog/deploying-osquery-at-scale-a-comprehensive-list-of-open-source-tools
 Data over time
If you gather enough data over time using osquery, you can recreate
most of the state of a machine at different points in time.

Your data store and retrieval method has to take this into account.

Very useful for investigations and timelines.

However, with the same tool you can go back and query in real time.
 Osquery and Community
Can osquery become the “Apache for Endpoint?”

One of the top open-source projects for security (as per github)

Immense growth & activity

Adoption in “early adopter” as well as some more conservative

ones
 Who’s Using osquery Now?
DIY community
Large scale IT companies
Adoption in Silicon Valley
Facebook (obviously) Large financial institutions

Airbnb, Apple, Etsy, Google,

Even Security Companies
Netflix, Uber, Stripe and many
more . . . Carbon Black
 How You can Join and Contribute
Download a binary and try it out - https://osquery.io/downloads/
Download or clone from Github - https://github.com/facebook/osquery/
Join the osquery slack - https://osquery-slack.herokuapp.com/
Follow @osquery and contributors on Twitter

Facebook has designated some “starter” items for less experienced coders
Contribute a "query pack" (IOC) to find mac (or other) malware
Help improve documentation
Write a blog post about solving a problem w/ osquery and share it
 See Uptycs Live!
Want to see how Uptycs can deliver osquery
analytics at scale?
Click here to register for a Live Demo!