EtherNet/IP Book of Knowledge

Network and Device Troubleshooting

PUBLIC INFORMATION

Rev 5058-CO900E Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
 Agenda

SomeBasics
Some Basics

PortMirroring
Port Mirroring

TrafficCapture
Traffic Capture

AnalyzingTraces
Analyzing Traces

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 2
 Some Basics
Why do we do this?
 Errors in the I/O tree
 Poor performance
 Intermittence communication
 Timeouts

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 3
 Some Basics
Before Wireshark
Before asking for a capture……..first check

 Port settings
 Auto vs. forced
 Speed and duplex
 Diagnostics (web pages
 Media counters
 Missed I/O
 CPU

Packet capture may not be needed!

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 4
 Some Basics
Forced Vs. Autonegotiate
Which one should be used?
 Autonegotiate is the default setting on devices and switches
 Used in most applications
 Unmanaged switches use auto
 Uses handshaking to determine fastest supported settings
 May decrease to 10mb in noisy environments (and will stay there until
next powercycle/cable disconnect)

Autonegotiate is default setting

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 5
 Some Basics
Forced Vs. Autonegotiate
Which one should be used?
 Forced takes time and effort to program the ports. Some applications
require forced setting
 Quick Connect (to eliminate handshaking time)
 Noisy Environments (to stay at 100mb)
 Becomes an issue when replacing equipment if forced is not set
(since auto is the default)

Forced is manually set

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 6
 Some Basics
Match Port Settings For Each Link
 Make sure port settings on each end of an Link (Ethernet cable) match
 Otherwise a duplex mismatch will occur which may cause
communication errors
 Different ports on a switch (or a device) are not required to match and may
be set differently

 Avoid half duplex

unless required by
device, half duplex is
slower

Both ends of a cable must match!

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 7
 Some Basics
Duplex Mismatch
 Check settings and media counters
 Autonegotiate will go to Half Duplex when
connected to another port that is forced
 Media errors can happen on both of
these devices causing lost, delayed,
and retransmitted packets

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 8
 Some Basics
Diagnostics
 Diagnostics viewable in various areas
 Web pages of devices
 RSLinx
 Switches
 HMI – Ethernet Diagnostics Faceplate

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 9
 Some Basics
Bad Cables
Intermittent network issues are often caused by mismatched ports or bad
cables
 Sometimes switches can detect bad cables
 Bad cables cause lost and retransmitted packets
 Use a decent cable checker or try replacing cable

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 10
 Some Basics
Bad Cables
 Bad cables that cause frequent intermittent errors can break a DLR
network.
 Bad cable anywhere on the network may cause a DLR rapid fault
 After DLR rapid fault, the bad cable may cause devices to drop off
 Check media errors on all devices
 Capture may show missing packets
 Can be difficult to track down

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 11
 Some Basics
Lost Packets
 Check if CPU loading on devices is greater then 90%
 Spikes in traffic may overwhelm device
 For I/O, check missed packets
 If there are missed IO packets, start checking on media errors on
all points between devices

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 12
 Some Basics
Lost Packets
 Bridged connection may indicate which network paths to check
 Look at missing packets counters
 Which devices do they belong?
 Where on the network are those devices?

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 13
 Some Basics
Module Rejects

 “Rejects” means that the Logix Ethernet module hardware passed an I/O packet to the firmware
but the firmware looks at the packet and then dropped it. This increments the Reject diagnostic
and also counts toward CPU usage. Rejects will occur for the following reasons:
 Duplicate multicast streams
 The firmware enables module hardware to accept specific multicast groups. The hardware cannot
distinguish duplicates.
 Recommendation: Avoid duplicate multicast groups by having less than 1025 nodes on a network.
 The hash table (hardware) is not perfect
 Resulting in some multicast being past to the module firmware.
 Recommendation: None.
 A network event disrupts traffic. The consumer times out and closes the connection. The data
producer has not yet timed out. Then, the absent stream starts arriving at the consumer again
because the network is now working good. The restarted stream will be rejected by the
consumer because there no longer is a CIP connection open with that connection identifier.
 Recommendation: None.
 All three of the above are the similar in that the firmware can’t associate a received packet with
an active CIP connection ---- it's rejected .
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 14
 Some Basics
HTTrack
 HTTrack software allows gather all the web pages from a device and from
multiple devices quickly and easily
 Answer ID 67297 How to capture diagnostic webpages from Ethernet
devices

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 15
 Some Basics
HTTrack
 Web pages are gathered in a format easily viewed and navigated
 Does not work with 1734-AENT(R) or Stratix 5700/8000
 Folder can be easily zipped and emailed

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 16
 Some Basics
EtherNet/IP Diagnostics Faceplate
Quickly gather diagnostics in one location

Available on sample code website

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 17
 Some Basics
EtherNet/IP Diagnostics Faceplate

 Monitors up to 50 devices
 Automatically detects and shows supported diagnostics such as media
errors, missed I/O, and link status

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 18
 Some Basics
Summary
 Duplex mismatch or bad cable results in the following
 Slower communication
 Lost packets
 Lost connections
 Intermittent issues
 These show up in diagnostics as
 Media counters
 Missed I/O
 CIP connection timeouts

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 19
 Agenda

SomeBasics
Some Basics

PortMirroring
Port Mirroring

TrafficCapture
Traffic Capture

AnalyzingTraces
Analyzing Traces

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 20
 Port Mirror - Stratix 5700/8000/8300
Multicast

NOTE for Multicast!

 Some switches, like the Stratix 5700 and 8000 series and other Cisco
switches, do not mirror multicast control traffic (Leave and join Group
messages)
 This can complicate multicast connection troubleshooting and make it
difficult to find switch or device issues
 It is recommended when possible to use an 1783-ETAP or other capture
device to capture traffic if multicast IO connection loss is being
investigated

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 21
 Port Mirror - Stratix 5700/8000/8300
Multicast

NOTE for DLR!

 DLR versions of Stratix 5700 switches do not mirror the DLR control traffic.
 If mirroring DLR control traffic is desired, use an ETAP

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 22
 Port Mirror - Stratix 5700/8000/8300
Device Manager
 Can use web page (Device Manager) of Stratix to configure port mirror for
Stratix 5700/8000/8300
 Web page requires that the cable to be disconnected from the port to be
used as the mirror during configuration

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 23
 Port Mirror - Stratix 5700/8000/8300
Device Manager
 To configure:
 Select Smart Ports
 Select port to be used as mirror, and select Edit
 Select Source Interface
 If desired, select Ingress Vlan
 If using HMI or other communications software on the computer that is needed
at the same time the port mirror is being done. Otherwise, leave Ingress Vlan
as none

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 24
 Port Mirror - Stratix 5700/8000/8300
CLI
PuTTY or other telnet client for CLI (Command Line Interface)
 Port mirror can also be done via CLI
 PuTTY is a popular free CLI client
 Fill in:
 IP address
 Port 23
 Telnet

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 25
 Port Mirror - Stratix 5700/8000/8300
CLI
Some mirror command examples
 To mirror a port with no ingress
 monitor session 1 destination interface fa2/7
 monitor session 1 source interface fa2/3 both
 To mirror all Vlan traffic
 monitor session 1 destination interface fa2/7
 monitor session 1 source Vlan 192 both
 Note – for high traffic, check port utilization to make sure port is not at 100%
 To see how port mirror is configured
 show run | section monitor
 To stop mirror
 no monitor session 1
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 26
 Port Mirror - Stratix 5700/8000/8300
CLI

 Following two lines show commands to set up mirror. The ingress vlan is
optional
 monitor session 1 source interface fa1/1
 monitor session 1 dest interface fa1/8 ingress vlan 192
 Sessions 1 and 2 available

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 27
 Port Mirror - Stratix 5700/8000/8300
CLI

 Show monitor can also be used to see current mirror setup

 In non config mode, show monitor
 In config mode, do show monitor

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 28
 Port Mirror
1783-ETAP
 To configure an ETAP for mirroring, use RSLinx Device Configuration
 Check Enable Device Port Debugging Mode
 Uncheck all other options as shown below

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 29
 Port Mirror
Stratix 6000
 Select both IN and OUT a port in the Mirror From column
 Select the Capture to Port
 Select Enabled for the Mirror Configuration drop down

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 30
 Agenda

SomeBasics
Some Basics

PortMirroring
Port Mirroring

TrafficCapture
Traffic Capture

AnalyzingTraces
Analyzing Traces

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 31
 Traffic Capture
Multiple Network Interface Cards?
 Does the computer have multiple Network Interface Cards (NIC)?
 If so, know which one is being used since they will show up as options in
Wireshark
 Can disable other NICs via right click
 Control Panel -> Network and Sharing -> Change adapter settings

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 32
 Traffic Capture
Firewalls

NOTE for Firewalls!

 Make sure that if the computer being used to capture traffic, has the vpn /
firewall / security software disabled, or check with IT that it will not interfere
with the capture
 Otherwise, a trace may be missing important information and may not
be useful

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 33
 Traffic Capture
Using Wireshark

 To start a capture in Wireshark

 Select the Capture menu

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 34
 Traffic Capture
Using Wireshark

 In the capture menu:

 Select the NIC to use
 Select Options

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 35
 Traffic Capture
Using Wireshark
 In the Options window, make sure:
 Promiscuous is checked and Capture Filter is empty
 This is important to preserve all information!
 If a rolling log is desired for long captures, fill similar to that shown
 Display Options
are as checked

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 36
 Traffic Capture
Using Wireshark
 Leave Wireshark running until a minute or more after the event that is
being investigated
 Make sure to “Stop” the capture first
 Save the capture after stopping it

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 37
 Traffic Capture
Using Wireshark

Switches and
routers

Sometimes two simultaneous captures are needed!

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 38
 Traffic Capture
Coloring Rules
Coloring rules can help when looking traces
 Answer ID 491368 has coloring rule attachment.
 Normal CIP traffic in shades of blue or black
 Retransmission highlighted in yellow
 CIP errors highlighted in red
 ICMP errors and STP changes in red
 To import coloring rules
 -> View -> Coloring Rules
 -> Delete current rules
 -> Import new rules

Coloring rules can be used to visually point possible CIP issues

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 39
 Traffic Capture
Coloring Rules
Coloring rules can help show if CIP traffic was captured

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 40
 Traffic Capture
Coloring Rules
Filtering in Wireshark can also help point out if expected packets can be
found.
 “ENIP” for RSLinx browse and I/O
 “CIP” for RSLinx HMI comms and messaging
 Answer ID 491368 has more filters

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 41
 Traffic Capture
Is The Trace Useful?
Does the trace have useful information or is it empty?
 Empty traces have been taken (unknowingly)
 To avoid wasted effort due to empty traces, it helps to know what a trace
should look like

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 42
 Traffic Capture
Is The Trace Useful?
HMI and Message Traffic

 Filter on “ip.addr eq 192.168.1.76 and ip.addr eq 192.168.1.226”

 Use your IP addresses
 CIP Traffic should be flowing in both directions
 Wireshark does not decode all CIP commands and may show as
unknown service – this is ok
 Protocol can be CIP or CIP CM
 Series of CIP request followed by replies
 Replies are usually “Success”

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 43
 Traffic Capture
Is The Trace Useful?
 HMI and Message Traffic example picture
 Requests coming from HMI (226), and replies are being sent back
(from 76)
 CIP or CIP CM protocol in both directions

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 44
 Traffic Capture
Is The Trace Useful?
 General I/O traffic should have ENIP packets
 Filter on ENIP
 ENIP traffic from I/O module will be going to multicast address of
239.x.x.x if unicast not checked, or to the controller address
otherwise
 Controller will send packets directly to I/O module
 ENIP packets also have ID’s and sequence numbers

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 45
 Traffic Capture
Is The Trace Useful?
 I/O data from specific node
 ENIP data is flowing too and from the I/O node (91)
 May need to filter on ip.addr == 192.168.1.91 to hide other traffic

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 46
 Traffic Capture
Is The Trace Useful?
 This is a “good” trace of a device error.
 Show Forward open to device
 Device complains about a conflict
 May be shown as an error in the I/O tree
 RSLogix 5000 ->I/O tree -> module properties -> connections tab

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 47
 Traffic Capture
Is The Trace Useful?
 If we want a trace of node 192.168.1.10…… the below trace does not
show any traffic coming or going to that node, check port mirror setup
 This trace only shows broadcast type traffic
 No point to point or multicast traffic

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 48
 Traffic Capture
Is The Trace Useful?
 The below could be a trace of mirroring more then 1 port.
 Lots of duplicate packets indicated by Wireshark of two remote
devices – more difficult to analyze, but sometimes possible

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 49
 Traffic Capture
Is The Trace Useful?
 The below may be the result of mirroring more then 1 port.
 Lots of duplicate IO packets right next to each other
 More difficult to figure out the trace

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 50
 Traffic Capture
Is The Trace Useful?
 In this example, there is a lack of any traffic except multicast
 May be due to forgetting port mirror or mirroring wrong port
 This trace may not be useful
 Try ““ip.addr eq x.x.x.x and ip.addr eq y.y.y.y”
 to see if there is any traffic going back to the I/O module.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 51
 Agenda

SomeBasics
Some Basics

PortMirroring
Port Mirroring

TrafficCapture
Traffic Capture

LookingAtAtTraces
Looking Traces

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 52
 Looking At Traces
General Questions
Where can I find answers to general Ethernet questions?
 How does TCP work?
 What does a normal or standard Ethernet frame like?
 How is Ethernet traffic managed?
 What is an ARP table?
 How was Ethernet developed?
 What is a Vlan tagged packet?
 What are SYN packets?
 What is a Vlan?
 What is CAT6?
Internet searches are you friend! Don’t be afraid to use them!!!!

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 53
 Looking At Traces
ODVA CIP Spec
 When analyzing traces, It’s often necessary to have the licensed CIP
specification which describes the expected format of packets and
information flow

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 54
 Looking At Traces
ODVA CIP Spec
 Error codes listed in CIP Spec

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 55
 Looking At Traces
ODVA CIP Spec
 CIP Information that all devices depend on in CIP Vol 1
 Examples
 Generic I/O
 Forward Opens

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 56
 Looking At Traces
ODVA CIP Spec
 QOS, Forward open, etc., for Ethernet described in CIP Vol. 2
 CIP Safety specification in Vol. 5

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 57
 Looking At Traces
Data Access manual
 Logix5000 Data Access manual
 Describes public read/write methods
 Can sometimes be helpful for 3rd party HMI traces
 Located on Literature Library
 RSLinx (Classic and Enterprise) sometimes use undocumented methods

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 58
 Looking At Traces
Coloring Rules
Coloring rules can help when looking traces
 Answer ID 491368 has coloring rule attachment.
 Normal CIP traffic in shades of blue or black
 Retransmission highlighted in yellow
 CIP errors highlighted in red
 ICMP errors and STP changes in red
 To import coloring rules
 -> View -> Coloring Rules
 -> Delete current rules
 -> Import new rules

Coloring rules can be used to visually point possible CIP issues

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 59
 Looking At Traces
CIP uses TCP/IP and UDP/IP
TCP/IP
 TCP guarantees packet delivery and data order
 Packets contain sequence number
 TCP tries to keep connection open by resending data if it does not get
an ack (lost data) and will wait before sending additional data
 TCP/IP stands for
 Transmission Control Protocol / Internet Protocol

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 60
 Looking At Traces
CIP uses TCP/IP and UDP/IP
I/O data uses UDP/IP
 UDP does not guarantee delivery
 Up to application to check
 Sent once, not verified by sender
 UDP/IP stands for
 User Datagram Protocol / Internet Protocol

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 61
 Looking At Traces
CIP uses TCP/IP and UDP/IP
 Unicast
 UDP and TCP
 I/O data (port 2222)
 CIP (port 44818) - HMI, Controller messaging, programming, creating I/O
connections
 Multicast
 UDP only
 I/O data (port 2222)
 Needs switch management IGMP on large networks

 Broadcast
 UDP only
 RSLinx browse (port 44818)

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 62
 Looking At Traces
IEEE and ODVA
 List Identity replies
 Can usually determine product manufacturer from MAC address or
from the Vendor ID of the reply from Wireshark display
 IEEE has MAC ranges, ODVA had Vendor ID list, (search internet)

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 63
 Looking At Traces
ARP
 ARP stands for Address Resolution Protocol
 ARPs are a necessary part of traffic. It’s how devices associate MAC
addresses to IP Addresses to find each other
 Will show up periodically, this is normal
 Can be filtered with “arp” or “!arp”

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 64
 Looking At Traces
ARP
 ARPs are a layer 2 message that imbed IP addresses as data to find MAC
addresses of local devices
 Broadcast range is FF:FF:FF:FF:FF:FF
 ARPs can be device to device or broadcast
 Remote IPs use MAC address of router
 ARPs are only for local subnet

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 65
 Looking At Traces
ARP
 Module may ARP for a missing device (powered off for example) about
once a second
 Module may ARP for device it is communicating with 1 or 2 times a minute
 If a module is ARPing and IP address at a much greater frequency (10’s or
more a second), take a closer look to see if device issue
 If there is lack of communication, investigate the unanswered ARPS

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 66
 Looking At Traces
ARP
 What does IP Device tracking look like?
 See Answer ID 568750 for needed changes to switch config

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 67
 Looking At Traces
DLR
 DLR beacons and announce frames are a normal part of DLR ring traffic
mirrored by ETAP
 Can be filtered with “dlr” or “!dlr”
 Uses Vlan tags (802.1Q) which will usually get blocked by managed
switches if plugged directly into ring
 All beacon packets have same source and dest mac addresses

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 68
 Looking At Traces
VLAN tags
 Some NIC cards and Windows settings can strip VLAN tags from packets.
 ETAP does not strip the VLAN tags.
 Used in DLR beacons
 As well as some other protocols
 And switch Trunk traffic

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 69
 Looking At Traces
A Normal I/O Trace
 This is a normal I/O trace of one I/O device during power up
 Devices ARP, create a TCP connection, open a CIP connection, and
send CIP communications

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 70
 Looking At Traces
A Normal I/O Trace
 Multiple ARPs happen on power up
 “Who has….tell 0.0.0.0” is for duplicate IP detection
 Gratuitous ARPs announce that device is available

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 71
 Looking At Traces
A Normal I/O Trace
 Controller ARPs for I/O hardware address

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 72
 Looking At Traces
A Normal I/O Trace
 TCP opened
 SYN(synchoronize) request to port 44818
 SYN/ACK back from port 44818

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 73
 Looking At Traces
A Normal I/O Trace
 Register session
 Contains session handle, used in all CIP TCP traffic
 Req ID is ignored, Rsp ID should be non zero

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 74
 Looking At Traces
A Normal I/O Trace
 Forward Open and response
 Forward open contains serial number, Connection ID’s, RPIs, and
more.
 Serial number used when a forward close is issued (not the ID’s)

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 75
 Looking At Traces
A Normal I/O Trace
 I/O is sent from port 2222 to port 2222
 ID’s are unique within an device (a singular IP address)
 Sequence number starts at 0 or 1 and increments by 1

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 76
 Looking At Traces
I/O Timeouts
I/O data uses UDP/IP
 Timeout is indicated in the connection forward open packets
 Also shown on the bridge connections of EN2T
 I/O standard uses a binary multiplier starting at 4 (4,8,16,32,64,128) times
the RPI to define a timeout value over and closest too 100ms
 Example: an RPI of 15 ms * 8 = 120 ms timeout value
 Example: an RPI of 25 ms * 4 = 100 ms timeout value
 Example: an RPI of 3 ms * 64 = 192 ms timeout value
 ControlLogix Redundancy output timeouts may be double (over 200ms)
 I/O safety timeout is define by user application from the I/O tree

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 77
 Looking At Traces
Analyzing I/O packets
 Add the connection ID column as shown below.
 Make connection ID sorting and search easier
 Can use new column with right click functionality

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 78
 Looking At Traces
I/O Graph
 Wireshark I/O graph can be used to see what the capture packet rate is
while the capture is running or on a saved file
 Expected rate depends on traffic being captured
 Sometimes can be useful to find problem areas

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 79
 Looking At Traces
Conversation Filters
 Conversation filter filters for 2 specific devices
 MAC address
 TCP
 IP

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 80
 Looking At Traces
Run bit
 If forward open is in the trace, Wireshark might be able to parse out the
run bit from the output packets (from controller to I/O)
 Only some I/O devices use this format
 Device should only use data when run bit is on

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 81
 Looking At Traces
Multicast Recommendations
 With latest releases, unicast can be used for I/O and produce/consume
tags
 If multicast is used, select IGMP Snooping on all switches
 Enable IGMP querier on all switches that support v2 or later. With IGMP
version 2 or later, all switches will negotiate and the lowest IP address will
be the IGMP querier.
Note: IGMP v1 enabled queriers will not negotiate and should not be enabled
at the same time. If there are no v2 queriers, use only one v1 querier

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 82
 Looking At Traces
Multicast
 If multicast I/O connections drop, the Ethernet module will issue an IGMP
leave message
 Note: Some switches, like the Stratix 8000, do not mirror leaves and
joins (group). It is recommended to use ETAP to capture traffic. May
be needed to investigate the UDP multicast traffic flow
 If a module joins a group, it will send a group message
 When multicast I/O connections exist, the Ethernet module will issue an
IGMP “Membership Report group” message for each “Membership Query”

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 83
 Multicast - IGMP Leave happens when?

A Logix Ethernet module sends an IGMP Leave when all CIP connections
through that module are broken for the multicast address being consumed.

 Case 1: Startup
 When a consumer receives a successful Forward Open reply for a multicast group,
the consumer starts sending heartbeats and also sends an IGMP Join.
 If the first multicast is not received in 10 seconds, the consumer sends an IGMP
Leave.
 The consumer considers the CIP connection as timed out and stops sending unicast.
 Examples of timeouts after the first data:
 A 2ms RPI has a CIP connection timeout of 128ms. Then, X >128ms.
 A 100ms RPI has a CIP connection timeout of 400ms. Then, X > 400ms.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 84
 Multicast - IGMP Leave happens when?

 Case 2: Two controllers consuming same tag thru same ENxx

 At the consumer Logix chassis, if there are 2 consumer controllers for
the same tag and both are consuming thru the same ENBT module,
then that ENBT will send a Leave for that multicast group when both
Logix controllers no longer want to consume that group. A Logix
controller will not consume under the following conditions:
 Consumed tag is inhibited or deleted
 I/O connection is inhibited or deleted

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 85
 Multicast - IGMP Leave happens when?

 Case 3: Tag producer dies/disconnected/disappears

 At the consumer Logix chassis, if the tag producer is disconnected or
the infrastructure (switch) fails, the consumer Ethernet module no
longer detects the produced tag and the CIP connection will close.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 86
 Multicast - IGMP Leave happens when?

 Case 4: Duplicate multicast address

 Consider the case of 2 Logix controllers each consuming data from different
data producers. And, each data producer transmits its data using the same
multicast address. This is allowed in the EIP spec because each multicast
stream includes unique information that differentiates.
 If 2 or more multicast producers are using the same multicast address, the
consumer ENBT module will not send a Leave until all consumer Logix
controllers no longer want to receive that multicast group.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 87
 Looking At Traces
Multicast

 Here is picture with filter “IGMP” of a connection being broken and remade

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 88
 Looking At Traces
Finding Connection IDs

For analysis:
The two connection IDs associated with each CIP connection need to be identified.
These can be obtained from a EN2T web page. To make them available dynamically
in Wireshark, configure a web browser (ex. Internet Explorer) to read the EN2T page
shown below.

89

02/28/2012
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
 Looking At Traces
Finding Connection IDs

For analysis:
After a Wireshark capture is obtained, you need to learn the I/O connection IDs that were valid prior to
an I/O connection loss. To do this:
1. Set a filter in Wireshark
TCP and ip.src==IP_of_IE_pc and ip.dst==IPofTheEN2T and tcp.seq==0
Example: TCP and ip.src==192.168.1.250 and ip.dst==192.168.1.200 and tcp.seq==0
2. Examine the first TCP packet in the TCP/HTTP sequence prior to the loss.
Below, notice the 1 second interval between 1.663 and 2.763

90

02/28/2012
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
 Looking At Traces
Finding Connection IDs

 Now, with that packet select Analysis and Follow TCP Stream

 Then, select Raw and Save as an html file.

 Then, open the html file with a web browser such as Internet Explorer (see next page).

02/28/2012
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
 Looking At Traces
Finding Connection IDs

 These connection IDs can then be used in a Wireshark filter to monitor

packet flow from source to destination.

02/28/2012
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.
 Looking At Traces
Message Instruction
 Example of setting the CIP message inactivity timeout configuration. This
timeout is best used with unconnected MSGs. Screen capture below
copied from Rockwell knowledge base document 22644
 If a timeout occurs, the MSG will error (0x204) at the specified MSG timeout
 The Ethernet module does not inform the controller of the inactivity timeout.
This timer begins when MSG.EN = 1

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 93
 Looking At Traces
Message Instruction
 Sample capture of traffic with timeout set to 5 seconds
 TCP Reset happens after 5 seconds to close the TCP connection
 Reduces or eliminates keeps alive TCP traffic

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 94
 Looking At Traces
Message Instruction
 Sample capture of traffic with timeout set to 55 seconds
 Notice the keep alive traffic
 Timeout will happen after 120 seconds of not activity if it is not
specified

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 95
 Looking At Traces
Malformed packets
 Wireshark might show packets as malformed when they are not
 Packets tagged as bad checksum may be due to NIC hardware
offloading (Checksum changed or zeroed by hardware)

 If many packets of interest show this, then disable checksum checking.

 In Wireshark, click on Edit  Preferences  Protocols  TCP (or
UDP)  (Uncheck) Validate TCP (or UDP) checksum if possible
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 96
 Looking At Traces
Malformed packets
 Packets in Wireshark tagged as Malformed might not be
 The below picture shows a “Malformed” TCP packet.
 This is actually a CIP packet that is not being parsed correctly.
 Port 1090 is associated with ff-fms fieldbus and is incorrectly decoded

 For this example, to change this behavior, follow these steps:

 Click on Analyze  Enabled Protocols  Scroll down to FF (or whatever
protocol is indicated by the port)  Uncheck the box  Apply
 Wireshark should then decode based on the other port which in this case is
44818 which is what CIP uses
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 97
 Looking At Traces
Malformed packets
 Packets in Wireshark tagged as Malformed might not be
 The below picture shows a “Malformed” IO packet.
 This is actually an input only connection that does not have a run/idle 32bit
header, but Wireshark assumes connections have 32bit header

 For this example, to change this behavior (if desired), follow these steps:
 Click on Edit -> Preferences -> Protocols -> ENIP -> uncheck Dissect 32-bit
header in the O->T direction
 Wireshark will then not try to decode the packet and will not flag it with an
error
PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 98
 Looking At Traces
Window Size
 If a device stops sending data, check the window sizes
 If window size starts above 2k, but then is decreasing and stays below
2k, then the other side may not be reading it’s data. In this case,
investigate the device whose buffer is not recovering.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 99
 Looking At Traces
ICMP Packets
 ICMP error packets usually indicate a problem
 In this example, RSLinx was shut down, and computer replies that packet
cannot be processed since RSLinx shut down the port

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 100
 Looking At Traces
QOS
 Some devices do not support QOS
 Zero value in QOS field
 Supposed to accept packet if receiving QOS packet
 Some devices support QOS
 Non zero value in QOS field

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 101
 Looking At Traces
Duplicate packets?
 Duplicate packets can happen if:
 Device issues
 Device does not ack
 Network issues
 Packets get duplicated on network

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 102
 Looking At Traces
Duplicate packets?
 Duplicate packets could be cause by the way the mirror was done and are
not duplicate in the network and may actually be expected
 Example: Only packets coming from node 226 are getting duplicated. The
computer has RSLinx and Wireshark. RSLinx is communicating with device.

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 103
 Looking At Traces
Removing Duplicate Packets
 Can get rid of duplicate packets with editcap.exe
 Example:
 C:\Users\<user>\Desktop\Wireshark training\Traces>"c:\Program
Files\Wireshark\editcap.exe" -d input_file.pcapng output_file.pcapng

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 104
 Looking At Traces
Routing between Vlans
 VLANs and routers get in the middle if devices are not on the same subnet
 Packets get layer 2 mac address of VLANs
 Example shows router at address 192.168.1.5

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 105
 Misc Tips
Routing between Vlans
 View mac addresses learned on switch with CLI
 Show mac address-table
 View the mac address of the vlan or port with CLI
 Show int vlan # (or fa1/1)
 View the mac address with device web page if there is an “arp table”

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 106
 Misc Tips
Pinging
 Can you ping a device?
 At prompt “ping <ip address>”
 What devices are there?
 Ping all IP addresses in subnet in a Windows prompt
 C:\>for /l %lastoctect in (1,1,254) do @ping 192.168.1.%lastoctet -n 1 -w 100 | find "Reply"

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 107
 Misc Tips
Is Switch Dropping Packets?
 Stratix/Cisco switches will drop Vlan tagged packets (possibly miswired
DLR ring)
 “Show Interface <port>” to look for input errors
 These packets are dropped

PUBLIC INFORMATION Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved. 108
Ethernet Book of Knowledge Troubleshooting

PUBLIC INFORMATION

Follow ROKAutomation on Facebook & Twitter.

Connect with us on LinkedIn.

www.rockwellautomation.com

Rev 5058-CO900F Copyright © 2014 Rockwell Automation, Inc. All Rights Reserved.