Shooting the trouble down to the

Wireshark Lua Plugin

June 2019

Shakthi Kannan
Version 0.9
[email protected]
Motivation
Hello Production Support Engineer,
We are seeing client timeouts in our cluster,
Can you analyze the logs on the server,
And let us know how to proceed further?

”input from client unsupported proto version”

Is the message that we see for our operation,
What in the client can cause this assertion?
We need your help to provide us with a solution.

We are making batch read queries in production,

And receiving the values in a timely fashion,
Aerospike uses RIPEMD160 hash function,
But, can we also read the digests in conjunction?

2
Wireshark Lua

●
Dissectors
Decode packet data.
●
Chained Dissectors
Access to one dissector’s data.
●
Post-dissectors
Called after every other dissector
has been called.
●
Listeners
Called for every packet that
matches a filter or tap.

Source: https://wiki.wireshark.org/Lua/Dissectors

3
Tap Listener
-- simple_http.lua
-- implements a very simple tap in Lua

-- this is going to be our counter

http_packets = 0

-- this is going to be our tap

tap_http = nil

-- first we declare the tap called “http tap” with the filter it is going to use
tap_http = Listener.new(nil, “http”)

-- this function will get called at the end(3) of the capture to print the summary
function tap_http.draw()
debug(“http packets:” .. http_packets)
end

-- this function is going to be called once each time the filter of the tap matches
function tap_http.packet()
http_packets = http_packets + 1
end

-- this function will be called at the end of the capture run

function tap_http.reset()
http_packets = 0
end

4
 Wireshark User Interface
Main Menu
Main Toolbar
Filter
Toolbar

Packet
List

Packet
Details

Protocol
Dissection

Packet
Bytes

5
 Usage
Help -> About Wireshark

Free (Libre)
Open Source

$ wireshark -X lua_script:aerospike.lua capture.pcapng

$ tshark -X lua_script:aerospike.lua capture.pcapng

You can also place plugins in ~/.wireshark/plugins folder.

6
Hello World Lua!
$ lua -v
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio

$ lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
> print(“Hello, World!”)
Hello World

$ cat hello_world.lua
#!/usr/bin/lua

print(“Hello, World!”)

$ lua hello_world.lua
Hello, World!

7
Lua: Assignment and Operations
$ lua
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
> i, j = 1, 2
Category Operator Associativity
> -3
Unary not # - Right to left
-3
Concatenation .. Right to left
> k = i + j
> k Multiplicative */% Left to right
3
Additive +- Left to right
> j ^ k
Relational < > <= >= == ~= Left to right
8.0
> k % j Equality == ~= Left to right
1 Logical AND and Left to right
> not i
Logical OR or Left to right
false

8
Lua: Strings
> name = “Lua” Escape Use
> type(name) Sequence

string \a Bell
> print(name .. “ Language”) \b Backspace
Lua Language \f Form feed
> print(“99” + 1) \n New line
\r Carriage return
100.0
\t Tab
> print(“Value of k”, k)
\v Vertical tab
Value of k 3 \\ Backslash
> [[ This is also a string ]] \” Double quotes
This is also a string \[ Left square
bracket
> print(“Lua’s \”syntax\” is simple!”)
\] Right square
Lua’s “syntax” is simple! bracket

9
Lua: Tables
> work_days = {“Mon”, “Tue”, “Wed”, “Thu”, “Fri”}
> work_days
table: 0x9f10b0 table: 0x9f10b0
> work_days[0] 1 Mon
nil 2 Tue

> work_days[1] 3 Wed

4 Thu
Mon
5 Fri
> work_days[6] = “Sat”
6 Sat
> table.insert(work_days, “Sun”) 7 Sun
> work_days[7]
Sun

10
Lua: Functions
-- fact.lua
function fact (n)
module clock module http
local f = 1
for i = 2, n do
f = f * i
end
return f
end
module yaml module string
print(fact(5))

$ lua fact.lua
120

11
Lua: Conditions and Loops

if number < 10 then repeat

print(“Less than 10”) print(number)
else number = number + 1
print(“Greater than 10”) until number >= 10
end

for number = 0, 9, 1 do
while number < 10 do
print(number)
print(number)
number = number + 1
end

end

12
Literate Programming
"I believe that the time is ripe for significantly better
documentation of programs, and that we can best achieve this
by considering programs to be works of literature. Hence, my
title: “Literate Programming.”

Let us change our traditional attitude to the construction of

programs: Instead of imagining that our main task is to
instruct a computer what to do, let us concentrate rather on
explaining to human beings what we want a computer to do."
~ Prof. Donald E. Knuth, 1984

13
Markdown Structure
# Requires...
# Configuration...
## Common...
# Helper Functions...
# Statistics...
## Hot Key...
# GUI...
# Protocols...
## Info...
## Batch...
## Message...
### Aerospike Message: Header Section...
### Aerospike Message: Fields...
### Aerospike Message: Operations...
### Functions...
## Heartbeat...
# The Main...

14
lit2lua
## Heartbeat

Heartbeat protocol

> +--------------------------+----------------------------+
> | Message Header | Message Fields |
> +--------------------------+----------------------------+

Message Header

> +------------+-------------+
> | size | type |
> +------------+-------------+
> 0 4 6

Constants

local HB_HEADER_SZ_START = 0
local HB_HEADER_SZ_LENGTH = 4
local HB_HEADER_TYPE_START = 4
local HB_HEADER_TYPE_LENGTH = 2

15
lit2lua ...
Op

| Value | Name | Description |

|------:|:---------------------|:------------------------------------------------------------|
| 1 | AS_MSG_OP_READ | Read the value |
| 2 | AS_MSG_OP_WRITE | Write the value |
| 3 | AS_MSG_OP_CDT_READ | Prospective CDT top-level ops |
| 4 | AS_MSG_OP_CDT_MODIFY | Prospective CDT top-level ops |
| 5 | AS_MSG_OP_INCR | Add a value to an existing value (only on integers) |
| 6 | Unused | Reserved |
| 7 | Unused | Reserved |
| 8 | Unused | Reserved |
| 9 | AS_MSG_OP_APPEND | Append a value to an existing value (on strings and blobs ) |
| 10 | AS_MSG_OP_PREPEND | Prepend a value to an existing value (on strings and blobs) |
| 11 | AS_MSG_OP_TOUCH | Touch a value (will only increment the generation) |
| 129 | AS_MSG_OP_MC_INCR | Memcache-compatible version of the increment command |
| 130 | AS_MSG_OP_MC_APPEND | Append value to existing value (only on strings) |
| 131 | AS_MSG_OP_MC_PREPEND | Prepend a value to an existing value (only on strings) |
| 132 | AS_MSG_OP_MC_TOUCH | Memcache-compatible touch (does not change generation) |

To generate the Wireshark Lua plugin from documentation, use:

$ less -f “docs/aerospike.lua.md” | lit2lua > lua/aerospike.lua

16
lit2lua ...
Table Definition

local TYPES_OPS = {
[1] = “AS_MSG_OP_READ”,
[2] = “AS_MSG_OP_WRITE”,
[3] = “AS_MSG_AP_CDT_READ”,
[4] = “AS_MSG_OP_CDT_MODIFY”,
[5] = “AS_MSG_OP_INCR”,
[6] = “Unused”,
[7] = “Unused”,
[8] = “Unused”,
[9] = “AS_MSG_OP_APPEND”,
[10] = “AS_MSG_OP_PREPEND”,
[11] = “AS_MSG_OP_TOUCH”,
[129] = “AS_MSG_OP_MC_INCR”,
[130] = “AS_MSG_OP_MC_APPEND”,
[131] = “AS_MSG_OP_MC_PREPEND”,
[132] = “AS_MSG_OP_MC_TOUCH”,
}

17
Protocol Dissection Pattern
Constants
local PROTO_VERSION_START = 0
local PROTO_VERSION_LENGTH = 1

local PROTO_TYPE_START = 1
local PROTO_TYPE_LENGTH = 1

local PROTO_TYPE_INFO = 1
local PROTO_TYPE_MSG = 3

local INFO_SIZE_START = 2
local INFO_SIZE_LENGTH = 6
local INFO_DATA_START = 8

> +---------+-------------+----------------------------------+
> | version | type | size |
> +---------+-------------+----------------------------------+
> 0 1 2 8

18
Protocol Dissection Pattern ...
Create Proto objects

local aerospike_info_proto = Proto(“Aerospike”, “Aerospike Info Protocol”)

local aerospike_attribute = Proto(“AerospikeAttribute”, “Aerospike Attributes”)
local aerospike_attribute_value = Proto(“AerospikeAttributeValue”, “Aerospike Attribute Value pairs”)

Proto header fields

local header_fields = {
version = ProtoField.uint8 (“header.version”, “Version”, base.DEC),
type = ProtoField.uint8 (“header.type”, “Type”, base.DEC, TYPES_PROTO),
size = ProtoField.uint64 (“header.size”, “Size”, base.DEC),
}

local header_attributes = {
attribute = ProtoField.string(“header.attribute”, “Attribute”),
}

local header_attribute_values = {
attribute = ProtoField.string(“header_attribute_values.attribute”, “Attribute “),
value = ProtoField.string(“header_attribute_values.value”, “Value”),
}

Register the protocol fields

aeropike_info_proto.fields = header_fields
aerospike_attribute.fields = header_attributes
aerospike_attribute_value.fields = header_attribute_values

19
Protocol Dissection Pattern ...
Functions
local function dissect_aerospike_info (tvbuf, tree, size)
-- Separate the data by newline
local data_tvbr = tvbuf:range(INFO_DATA_START, tonumber(size))
local data_string = data_tvbr:string()

local data_start = INFO_DATA_START

for line in string.gmatch(data_string, "[^\n]+") do
local d = tvbuf:range(data_start, string.len(line))
local d_string = d:string()

-- if contains attribute-values
if string.find(d_string, "\t") then
local parts = split_tab(d_string)
...
end
data_start = data_start + string.len(line) + 1 -- for \n
end
end

20
Dissector Table
# Configuration
local default_settings = {
aerospike_port = 3000,
heartbeat_multicast_port = 9918,
heartbeat_mesh_port = 3002,
}
# Create Proto objects
local aerospike_proto = Proto(“AerospikeProtocol”, “Aerospike Protocol”)
local heartbeat_proto = Proto(“AerospikeHeartbeat”, “Aerospike Heartbeat”)

# The Main
local function enable_dissector()
DissectorTable.get(“tcp.port”):add(
default_settings.aerospike_port, aerospike_proto)
DissectorTable.get(“tcp.port”):add(
default_settings.heartbeat_mesh_port, heartbeat_proto)
DissectorTable.get(“udp.port”):add(
default_settings.heartbeat_multicast_port, heartbeat_proto)
end

enable_dissector()

21
Live Coding!

Source: https://www.xkcd.com/378/

22
Message Protocol

Production
Support

Bit-level
Dissection

| Value | Name | Description |

|------:|:----------------------------------|---------------------------------|
| 1 | AS_MSG_INFO1_READ | Contains a read operation |
| 2 | AS_MSG_INFO1_GET_ALL | Get all bins data |
| 4 | Unused | Unused |
| 8 | AS_MSG_INFO1_BATCH | New batch protocol |
| 16 | AS_MSG_INFO1_XDR | Operation is performed by XDR |
| 32 | AS_MSG_INFO1_GET_NO_BINS | Do not read the bin information |
| 64 | AS_MSG_INFO1_CONSISTENCY_LEVEL_B0 | Read consistency level – bit 0 |
| 128 | AS_MSG_INFO1_CONSISTENCY_LEVEL_B1 | Read consistency level – bit 1 |

23
Heartbeat Protocol
Heartbeat protocol

> +------------------+----------------+
> | Message Header | Message Fields |
> +------------------+----------------+

Message Header

> +------------+-------------+
Network > | size | type |
Analysis > +------------+-------------+
> 0 4 6

Header Type

| Value | Name |
|------:|:--------------------|
| 0 | M_TYPE_FABRIC |
| 1 | M_TYPE_HEARTBEAT_V2 |
| 2 | M_TYPE_PAXOS |
| 3 | M_TYPE_MIGRATE |
| 4 | M_TYPE_PROXY |
| 5 | M_TYPE_HEARTBEAT |
| 6 | M_TYPE_CLUSTERING |
| 7 | M_TYPE_RW |
| 8 | M_TYPE_INFO |
| 9 | M_TYPE_EXCHANGE |
| 11 | M_TYPE_XDR |
| 15 | M_TYPE_SMD |

24
CDT List Operations
Modifying list of lists

‘values’ bin:
[[1523474230000, 39.04],
[1523474231001, 39.78],
[1523474236006, 40.07],
[1523474235005, 41.18],
[1523474233003, 40.89],
[1523474234004, 40.93]]

client.list_append(
key,
‘values’,
[1623474234004, 41.23])

'values' bin:
[[1523474230000, 39.04],
[1523474231001, 39.78],
[1523474236006, 40.07],
[1523474235005, 41.18],
[1523474233003, 40.89],
[1523474234004, 40.93],
[1623474234004, 41.23]]

Source: https://www.aerospike.com/docs/guide/cdt-list-ops.html

25
Reassembled TCP Segments

Anomaly
Detection

dissect_tcp_pdus(tvb, tree, min_header_size, get_len_func, dissect_func)

Source: https://www.wireshark.org/docs/wsdg_html_chunked/lua_module_Proto.html

26
Hot Key Report

Report
Generation

27
Tests: Execution
aeropike-wireshark-plugin/tests $ make clean; make

Testing test-msg-write-response.pcapng.pdml ... [OK]

Testing test-msg-write-request.pcapng.pdml ... [OK]
Testing test-batch.pcapng.pdml ... [OK]
Testing test-payload-response-greater-than-1500.pcapng.pdml ... [OK]
Testing test-heartbeat-mesh.pcapng.pdml ... [OK]
Testing test-payload-request-greater-than-1500.pcapng.pdml ... [OK]
Testing test-msg-read-response.pcapng.pdml ... [OK]
Testing test-msg-read-request.pcapng.pdml ... [OK]
Testing test-heartbeat-multicast.pcapng.pdml ... [OK]
Testing test-info-response.pcapng.pdml ... [OK]
Testing test-info-request.pcapng.pdml ... [OK]

Generate report using luacov-console ...

luacov-console ../lua
luacov-console -s # --no-colored
==============================================================================
Summary
==============================================================================

File Hits Missed Coverage

-----------------------------------------
../lua/aerospike.lua 634 48 92.96%
-----------------------------------------
Total 634 48 92.96%

28
Debugging
Function Description

critical(text) Critical severity

warn(text) Warning local d = require ‘debug’

message(text) Normal print (d.traceback())

info(text) Informational
d.debug()
debug(text) Debugging

report_failure(text) Message box with

error icon

Utility Functions
https://wiki.wireshark.org/LuaAPI/Utils
Development Tips
https://wiki.wireshark.org/Development/Tips

29
Luacheck
●
Accessing undefined variable
●
Line contains only whitespace
●
Setting non-standard global variable
●
Unused variable
●
Unused argument
●
Unused loop variable i
●
Unused function
●
Line is too long
●
Trailing whitespace in a comment

$ luarocks install luacheck

$ luacheck lua/aerospike.lua
Total: 0 warnings / 0 errors in 1 file

30
Performance
●
Lua performs slower than implementing a plugin in C
●
Wireshark becomes slow for capture files greater than 100 MB
●
Display filter and save filtered results
●
Use TCP/Allow sub-dissectors to reassemble TCP streams
●
Use faster CPU and more physical RAM
●
Stop other programs on machine to reduce system load
●
Split/merge packet captures to analyze critical time intervals
$ editcap -r source.pcap target.pcap 0-15000 # 0-15000 packets
$ editcap -i 20 source.pcap 20starget.pcap # 20s
$ editcap -c 10000 source.pcap 10000target.pcap # 10000 packets
$ editcap -s 128 source.pcap 128btarget.pcap # 128 bytes of packet
$ mergecap -w output.pcap client.pcap server.pcap

Source: https://www.wireshark.org/docs/man-pages/editcap.html
https://www.wireshark.org/docs/man-pages/mergecap.html

31
Summary

Define
Constants

Create
Protocol
Objects

Write
Protocol
Fields

Implement
Functions

Wire up to
Wireshark
Lua

32
Future Work

●
Migration
●
Clustering
●
Proxy
●
RW (Replication)
●
Fabric
●
Info
●
Exchange
●
System Metadata
●
Security
●
Cross Datacentre Replication (XDR)

33
References
●
Lua: https://www.lua.org

●
Wireshark Lua API: https://wiki.wireshark.org/LuaAPI

●
Aerospike Wireshark Lua Plugin:
https://github.com/aerospike/aerospike-wireshark-plugin

●
Lua Examples: https://www.wireshark.org/Lua/Examples

●
“Changing Wireshark with Lua: Writing a Lua Plug-in to Create a Custom
Decoder” (~ 1h 20m)
https://www.youtube.com/watch?v=HTtVHxIh6ww

●
Lua style guide: http://lua-users.org/wiki/LuaStyleGuide

●
Lua Performance: https://wiki.wireshark.org/Performance

●
Peter Wu (“Lekensteyn” at #wireshark irc.freenode.net) Wireshark notes:
https://git.lekensteyn.nl/peter/wireshark-notes

●
Lua scripting in Wireshark:
https://sharkfestus.wireshark.org/sharkfest.09/DT06_Bjorlykke_Lua%20Scri
pting%20in%20Wireshark.pdf

34
Thank You

@shakthimaan

35