USER SESSION RECORDING

FOR THE ENTERPRISE

An Open-Source Effort by Red Hat

Nikolai Kondrashov
Software Engineer
Tech Talk 2017.12.04
 INTRODUCTION

Nikolai “spbnick” Kondrashov, a software engineer

● Working at Red Hat Common Logging team
● Focusing on the User Session Recording project
● Maintaining FreeRADIUS packages
● Founder and maintainer of the DIGImend project
● Flirting with embedded as a hobby

2 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
WHY?
 THERE IS A DEMAND

Customers have been telling us for a long time:

● We need to comply with government regulations
● We need to track what contractors do on our systems
● We need to know who broke our server and how

4 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 AND A DREAM

What people and governments want:

● Record everything users do
● Store that somewhere safe
● Let us find who did that thing
● Show us how they did it

5 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 THERE IS A SUPPLY

A great number of commercial offerings:

● From application-level proxies on dedicated hardware
● To user-space processes on the target system
● Recording keystrokes, display, commands, apps, URLs, etc.
● Integrated with identity management, and access control
● With central storage, searching, and playback

6 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 BUT NOT GOOD ENOUGH

Still people are not satisfied:

● Expensive
● Sometimes very expensive
● Can’t fix it yourself
● Can’t improve it yourself

7 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 WHAT CAN BE BETTER?

The customers want:

● Free (as in beer)
● Open-Source, so they can fix, or at least understand it better
● Yet still has support

8 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 WAIT, WE HAVE IT ALREADY!

Nope, not really:

● script(1) plus duct tape
○ popular, but not security-oriented, needs lots of DIY
● sudo(8) I/O logging
○ security-oriented, has searching, but not centralized
● TTY audit with auditd(8)
○ security-oriented, can be centralized, but only for input

9 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
WHAT?
 SO, WHAT DO WE NEED?

Hottest features requested:

● Record what the user enters, sees on the screen, executes, accesses
● Get it off the machine ASAP, and store centrally and securely
● Search, analyze, and correlate with other events
● Playback in real time, or later
● Control centrally

11 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 SOUNDS FAMILIAR!

Let’s do it with logs!

● Audit system records processes executed, files accessed
● Logging servers know how to deliver
● There’s a whole zoo of storing/searching/analyzing solutions

12 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 LEAN AND MEAN

Why it’s better:

● Reuses log plumbing
○ No separate infrastructure needed
○ Saves resources
○ Reduces maintenance load
● Allows easy correlation with all the other logs
○ Not just an isolated “video of the terminal”
○ Lets you see what was behind the scenes

13 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 FIRST...

What to take out of the store/search/analyze zoo?

● Open-Source
● Scalable
● Hip

14 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 YES, ELASTICSEARCH AND KIBANA!

Our ViaQ project is bringing them to Red

Hat product portfolio:
https://github.com/ViaQ
● Normalize all the logs
● Put them into Elasticsearch
● Provide dashboards and analytics
● Part of OpenShift, coming to
OpenStack and other Red Hat
products!

15 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 THEN...

How can we:

● Control centrally what, where and whom to record?
● Log what user enters and sees?
● Make sense of audit logs?
● Deliver to Elasticsearch?
● Play everything back?

16 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 CENTRAL CONTROL?

Naturally, FreeIPA and SSSD!

● Manage domains, hosts, groups,
users, and more
● Cache credentials and
authenticate offline
● Session Recording control
being designed

17 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 LOG INPUT AND OUTPUT?

We made a tool for that - tlog

http://scribery.github.io/tlog
● A shim between the terminal and the
shell, started at login
● Converts what passes in between to
searchable JSON
● Logs to syslog or journal
● Plays back recordings on a terminal

18 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 MAKE SENSE OF AUDIT LOGS?

We made a tool for that too - aushape

http://scribery.github.io/aushape/
● Listens for audit events
● Converts them to JSON or XML
● Both have official schemas
● Logs to syslog
● Developed with the help from auditd

19 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 DELIVER TO ELASTICSEARCH?

Any popular logging service:

RSYSLOG*
Or our coming solution:

ViaQ
* Distributed by Red Hat now

20 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 PLAY EVERYTHING BACK?

We’re building a Web UI

● Playback data from Elasticsearch
● See input, output, commands
executed and files accessed
● Search for input, output, commands
and files
● Reuse and integrate
● Starting in Cockpit with storage in
journal

21 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 ALL TOGETHER NOW!

Auditd Aushape
Fluentd
Tlog
Rsyslog
Logstash
WebUI
Elasticsearch
Kibana

22 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
DEMO!
 IN THIS DEMO...

● A recorded user logs in

● Playback of the session is started at the same time
● Some work is done on the terminal
● Terminal I/O and converted audit logs are seen in journal
● Logs in Elasticsearch are displayed by Kibana
● Guest appearance: recordings in Cockpit

24 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
HOW?
 HOW TLOG WORKS?
Console login example

1
Starting a console session:
1
PAM
1. User authenticates to login via PAM
login 2
2. NSS tells login: tlog is the shell
NSS
3. login starts tlog 3
6 6 syslog
4. Env/config tell tlog the actual shell
tlog
5. tlog starts the actual shell in a PTY journal
6. tlog logs everything passing between 56
its terminal and the PTY, via PTY
syslog(3) or sd-journal(3)
shell

26 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 CONTROL TLOG WITH SSSD
Console login example

When a recorded user logs in: 2

PAM
1. SSSD finds a match for the user in its pam_sss 2
configuration login
2. pam_sss stores the actual user shell NSS SSSD
in the PAM environment 4 1
3 nss_sss 3
3. nss_sss tells login: tlog is the shell
tlog conf
4. login starts tlog with PAM
environment 5
5. tlog starts the actual user shell
retrieved from environment
shell

27 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 CONTROL TLOG WITH FREEIPA
Plan so far

Which users to record on which hosts: PAM

FreeIPA
● Recording configurations are linked
NSS conf HBAC
to HBAC rules, like SELinux maps
When users login: rule
conf
● SSSD fetches applicable rules SSSD HBAC
● SSSD decides if recording is enabled conf rule
● Proceeds as on previous slide

28 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 EXTRA TLOG FEATURES

Also control:
● What to record: input/output/window resizes
● “You are being recorded” notice
● Where to write: sd-journal(3), syslog(3), or file
● Low latency vs. low overhead
Basic playback on the terminal:
● From Elasticsearch
● From Journal
● From file

29 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 TLOG SCHEMA

{
Optimized for streaming and searching: "ver" : "2.2",
"host" : "tlog-client.example.com",
● Chopped into messages for "rec" : "c8aa248c81264f5d98d1..."
streaming, which can be merged "user" : "user1",
"term" : "xterm",
● Input and output stored separately "session" : 23,
● All I/O preserved "id" : 1,
"pos" : 0,
● Invalid UTF-8 stored separately
"timing" : "=56x22+98>23",
● Timing separate, ms precision "in_txt" : "",
● Window resizes preserved "in_bin" : [ ],
"out_txt" : "[user1@tlog-client ~]$ ",
"out_bin" : [ ]
}

30 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 HOW AUSHAPE WORKS?

From the kernel to Elasticsearch: Kernel Elasticsearch

● Kernel sends messages to auditd netlink
JSON
● Auditd passes messages to audispd Auditd
● Audispd distributes them to plugins,
binary Fluentd
including aushape
Aushape formats JSON Audispd
●
Rsyslog
● Aushape logs it through syslog(3) audit log
● Fluentd/Rsyslog/Logstash deliver it Aushape Logstash
to Elasticsearch
JSON log

31 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 AUSHAPE SCHEMAS
Mimicking the audit log, XML and JSON are similar, raw log can be preserved

<log> [
<event serial="number" {
time="timestamp"> "serial": number,
<text> "time": "timestamp",
<line>log message</line> ... "text": [
</text> "log message", ...
<data> ],
<record> "data": {
<field i="value" r="value"/> ... "record": {
</record> ... "field": ["value", "value"], ...
</data> }, ...
</event> ... }
</log> }, ...
]

32 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 AUSHAPE EXAMPLES
A heavily-trimmed event

<event serial="880" {
time="2016-09-28T19:34:44.771+03:00"> "serial":880,
<data> "time":"2016-09-28T19:34:44.771+03:00",
<syscall> "data":{
<syscall i="execve" r="59"/> "syscall":{
<success i="yes"/> "syscall":["execve","59"],
</syscall> "success":["yes"]
<cwd> },
<cwd i="/home/user"/> "cwd":{
</cwd> "cwd":["/home/user"]
<execve> },
<a i="ps"/> "execve":[
</execve> "ps"
</data> ]
</event> }
}

33 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 HOW COCKPIT UI WORKS?
Code in development

Setup for recordings in Cockpit: host

● Tlog logs to Journal, adding a tty tlog-rec
recording ID field
● To list recordings, Cockpit looks for cockpit journal
tlog messages in Journal, groups by
recording ID
● Cockpit JavaScript-based player cockpit tlog-play.js
reads and plays back Journal browser
messages with recording ID.

34 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
CHALLENGES!
 TLOG CHALLENGES

Lots of fun problems:

● How not to record passwords
○ Detect “echo off” mode, or cooperate with TTY audit
● Detect graphical sessions and don’t record under them
○ Perhaps look at environment variables
● Support charset conversion
○ Use iconv, and keep original text
● Handle various terminal types
○ Use our own terminal type

36 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 AUSHAPE CHALLENGES

Some more fun (and not so fun) problems:

● Audit log is a mess
○ Can’t fix. Track all the cases, use what auditd knows
● Somehow generate coherent schemas
○ Keep schema simple, use auditd record/field dictionaries
● Convert character encodings
○ Iconv, and keep invalid text in base64 or discard

37 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 WEB UI CHALLENGES
We’re taking them to Cockpit Hackfest!

On the road to first release for Cockpit:

● Random positioning in playback
○ Key frames?
● Searching for input or output
○ Log fields directly to Journal?
● Correlation with audit logs
○ It’s about time

38 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
TRY IT!
 TRY TLOG!
https://github.com/Scribery/tlog

● Download and install a release RPM, or

● Build from source, dependencies:
○ json-c-devel / libjson-c-dev
○ libcurl-devel / libcurl4-*-dev
○ systemd-devel/libsystemd-journal-dev
● Log to and playback from file
○ Easiest, good for testing
● Log to and playback from Elasticsearch
● Instructions in README.md!
● Submit issues, suggestions and pull requests!

40 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 TRY AUSHAPE!
https://github.com/Scribery/aushape

● Download and install a release RPM, or

● Build from source
○ Only audit-libs-devel / libauparse-dev is required
● Convert your own /var/log/audit/audit.log single-shot
○ Try both JSON and XML
● Set up live forwarding to Elasticsearch
● Instructions in README.md!
● Submit issues, suggestions and pull requests!

41 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
 TRY COCKPIT UI!
https://github.com/Scribery/cockpit/tree/scribery

● Checkout our scribery branch

● Build and run from source
○ Read HACKING.md
● Install tlog
● Set writer to “journal” in /etc/tlog/tlog-rec-session.conf
● Create a user with shell set to /usr/bin/tlog-rec-session
● Login as that user and do some stuff
● Checkout “Session Recording” page at http://localhost:9090

42 User Session Recording for the Enterprise - Nikolai Kondrashov, Red Hat
THANK YOU
User Session Recording Project
http://scribery.github.io/
THANK YOU
plus.google.com/+RedHat facebook.com/redhatinc

linkedin.com/company/red-hat twitter.com/RedHatNews

youtube.com/user/RedHatVideos