Introduction

============

The goal of this debug feature is to provide a reliable, responsive,

accurate and secure debug capability to developers interested in
debugging MSM subsystem processor images without the use of a hardware
debugger.

The Debug Agent along with the Remote Debug Driver implements a shared
memory based transport mechanism that allows for a debugger (ex. GDB)
running on a host PC to communicate with a remote stub running on
peripheral subsystems such as the ADSP, MODEM etc.

The diagram below depicts end to end the components involved to

support remote debugging:

: :
: HOST (PC) : MSM
: ,--------, : ,-------,
: | | : | Debug | ,--------,
: |Debugger|<--:-->| Agent | | Remote |
: | | : | App | +----->| Debug |
: `--------` : |-------| ,--------, | | Stub |
: : | Remote| | |<---+ `--------`
: : | Debug |<-->|--------|
: : | Driver| | |<---+ ,--------,
: : `-------` `--------` | | Remote |
: : LA Shared +----->| Debug |
: : Memory | Stub |
: : `--------`
: : Peripheral Subsystems
: : (ADSP, MODEM, ...)

Debugger: Debugger application running on the host PC that

communicates with the remote stub.
Examples: GDB, LLDB

Debug Agent: Software that runs on the Linux Android platform

that provides connectivity from the MSM to the
host PC. This involves two portions:
1) User mode Debug Agent application that discovers
processes running on the subsystems and creates
TCP/IP sockets for the host to connect to. In addition
to this, it creates an info (or meta) port that
users can connect to discover the various
processes and their corresponding debug ports.

Remote Debug A character based driver that the Debug

Driver: Agent uses to transport the payload received from the
host to the debug stub running on the subsystem
processor over shared memory and vice versa.

Shared Memory: Shared memory from the SMEM pool that is accessible
from the Applications Processor (AP) and the
subsystem processors.

Remote Debug Privileged code that runs in the kernels of the

Stub: subsystem processors that receives debug commands
from the debugger running on the host and
acts on these commands. These commands include reading
and writing to registers and memory belonging to the
subsystem's address space, setting breakpoints,
single stepping etc.

Hardware description
====================

The Remote Debug Driver interfaces with the Remote Debug stubs
running on the subsystem processors and does not drive or
manage any hardware resources.

Software description
====================

The debugger and the remote stubs use Remote Serial Protocol (RSP)
to communicate with each other. This is widely used protocol by both
software and hardware debuggers. RSP is an ASCII based protocol
and used when it is not possible to run GDB server on the target under
debug.

The Debug Agent application along with the Remote Debug Driver
is responsible for establishing a bi-directional connection from
the debugger application running on the host to the remote debug
stub running on a subsystem. The Debug Agent establishes connectivity
to the host PC via TCP/IP sockets.

This feature uses ADB port forwarding to establish connectivity

between the debugger running on the host and the target under debug.

Please note the Debug Agent does not expose HLOS memory to the
remote subsystem processors.

Design
======

Here is the overall flow:

1) When the Debug Agent application starts up, it opens up a shared memory
based transport channel to the various subsystem processor images.

2) The Debug Agent application sends messages across to the remote stubs
to discover the various processes that are running on the subsystem and
creates debug sockets for each of them.

3) Whenever a process running on a subsystem exits, the Debug Agent

is notified by the stub so that the debug port and other resources
can be reclaimed.

4) The Debug Agent uses the services of the Remote Debug Driver to
transport payload from the host debugger to the remote stub and vice versa.

5) Communication between the Remote Debug Driver and the Remote Debug stub
running on the subsystem processor is done over shared memory (see figure).
SMEM services are used to allocate the shared memory that will
be readable and writeable by the AP and the subsystem image under debug.
A separate SMEM allocation takes place for each subsystem processor
involved in remote debugging. The remote stub running on each of the
subsystems allocates a SMEM buffer using a unique identifier so that both
the AP and subsystem get the same physical block of memory. It should be
noted that subsystem images can be restarted at any time.
However, when a subsystem comes back up, its stub uses the same unique
SMEM identifier to allocate the SMEM block. This would not result in a
new allocation rather the same block of memory in the first bootup instance
is provided back to the stub running on the subsystem.

An 8KB chunk of shared memory is allocated and used for communication

per subsystem. For multi-process capable subsystems, 16KB chunk of shared
memory is allocated to allow for simultaneous debugging of more than one
process running on a single subsystem.

The shared memory is used as a circular ring buffer in each direction.

Thus we have a bi-directional shared memory channel between the AP
and a subsystem. We call this SMQ. Each memory channel contains a header,
data and a control mechanism that is used to synchronize read and write
of data between the AP and the remote subsystem.

Overall SMQ memory view:

:
: +------------------------------------------------+
: | SMEM buffer |
: |-----------------------+------------------------|
: |Producer: LA | Producer: Remote |
: |Consumer: Remote | subsystem |
: | subsystem | Consumer: LA |
: | | |
: | Producer| Consumer|
: +-----------------------+------------------------+
: | |
: | |
: | +--------------------------------------+
: | |
: | |
: v v
: +--------------------------------------------------------------+
: | Header | Data | Control |
: +-----------+---+---+---+-----+----+--+--+-----+---+--+--+-----+
: | | b | b | b | | S |n |n | | S |n |n | |
: | Producer | l | l | l | | M |o |o | | M |o |o | |
: | Ver | o | o | o | | Q |d |d | | Q |d |d | |
: |-----------| c | c | c | ... | |e |e | ... | |e |e | ... |
: | | k | k | k | | O | | | | I | | | |
: | Consumer | | | | | u |0 |1 | | n |0 |1 | |
: | Ver | 0 | 1 | 2 | | t | | | | | | | |
: +-----------+---+---+---+-----+----+--+--+-----+---+--+--+-----+
: | |
: + |
: |
: +------------------------+
: |
: v
: +----+----+----+----+
: | SMQ Nodes |
: |----|----|----|----|
: Node # | 0 | 1 | 2 | ...|
: |----|----|----|----|
: Starting Block Index # | 0 | 3 | 8 | ...|
: |----|----|----|----|
: # of blocks | 3 | 5 | 1 | ...|
: +----+----+----+----+
:

Header: Contains version numbers for software compatibility to ensure

that both producers and consumers on the AP and subsystems know how to
read from and write to the queue.
Both the producer and consumer versions are 1.
: +---------+-------------------+
: | Size | Field |
: +---------+-------------------+
: | 1 byte | Producer Version |
: +---------+-------------------+
: | 1 byte | Consumer Version |
: +---------+-------------------+

Data: The data portion contains multiple blocks [0..N] of a fixed size.
The block size SM_BLOCKSIZE is fixed to 128 bytes for header version #1.
Payload sent from the debug agent app is split (if necessary) and placed
in these blocks. The first data block is placed at the next 8 byte aligned
address after the header.

The number of blocks for a given SMEM allocation is derived as follows:

Number of Blocks = ((Total Size - Alignment - Size of Header
- Size of SMQIn - Size of SMQOut)/(SM_BLOCKSIZE))

The producer maintains a private block map of each of these blocks to

determine which of these blocks in the queue is available and which are free.

Control:
The control portion contains a list of nodes [0..N] where N is number
of available data blocks. Each node identifies the data
block indexes that contain a particular debug message to be transferred,
and the number of blocks it took to hold the contents of the message.

Each node has the following structure:

: +---------+-------------------+
: | Size | Field |
: +---------+-------------------+
: | 2 bytes |Staring Block Index|
: +---------+-------------------+
: | 2 bytes |Number of Blocks |
: +---------+-------------------+

The producer and the consumer update different parts of the control channel
(SMQOut / SMQIn) respectively. Each of these control data structures contains
information about the last node that was written / read, and the actual nodes
that were written/read.

SMQOut Structure (R/W by producer, R by consumer):

: +---------+-------------------+
: | Size | Field |
: +---------+-------------------+
: | 4 bytes | Magic Init Number |
: +---------+-------------------+
: | 4 bytes | Reset |
: +---------+-------------------+
: | 4 bytes | Last Sent Index |
: +---------+-------------------+
: | 4 bytes | Index Free Read |
: +---------+-------------------+

SMQIn Structure (R/W by consumer, R by producer):

: +---------+-------------------+
: | Size | Field |
: +---------+-------------------+
: | 4 bytes | Magic Init Number |
: +---------+-------------------+
: | 4 bytes | Reset ACK |
: +---------+-------------------+
: | 4 bytes | Last Read Index |
: +---------+-------------------+
: | 4 bytes | Index Free Write |
: +---------+-------------------+

Magic Init Number:

Both SMQ Out and SMQ In initialize this field with a predefined magic
number so as to make sure that both the consumer and producer blocks
have fully initialized and have valid data in the shared memory control area.
Producer Magic #: 0xFF00FF01
Consumer Magic #: 0xFF00FF02

SMQ Out's Last Sent Index and Index Free Read:

Only a producer can write to these indexes and they are updated whenever
there is new payload to be inserted into the SMQ in order to be sent to a
consumer.

The number of blocks required for the SMQ allocation is determined as:
(payload size + SM_BLOCKSIZE - 1) / SM_BLOCKSIZE

The private block map is searched for a large enough continuous set of blocks
and the user data is copied into the data blocks.

The starting index of the free block(s) is updated in the SMQOut's Last Sent
Index. This update keeps track of which index was last written to and the
producer uses it to determine where the the next allocation could be done.

Every allocation, a producer updates the Index Free Read from its
collaborating consumer's Index Free Write field (if they are unequal).
This index value indicates that the consumer has read all blocks associated
with allocation on the SMQ and that the producer can reuse these blocks for
subsquent allocations since this is a circular queue.

At cold boot and restart, these indexes are initialized to zero and all
blocks are marked as available for allocation.

SMQ In's Last Read Index and Index Free Write:

These indexes are written to only by a consumer and are updated whenever
there is new payload to be read from the SMQ. The Last Read Index keeps
track of which index was last read by the consumer and using this, it
determines where the next read should be done.
After completing a read, Last Read Index is incremented to the
next block index. A consumer updates Index Free Write to the starting
index of an allocation whenever it has completed processing the blocks.
 This is an optimization that can be used to prevent an additional copy
of data from the queue into a client's data buffer and the data in the queue
itself can be used.
Once Index Free Write is updated, the collaborating producer (on the next
data allocation) reads the updated Index Free Write value and it then
updates its corresponding SMQ Out's Index Free Read and marks the blocks
associated with that index as available for allocation. At cold boot and
restart, these indexes are initialized to zero.

SMQ Out Reset# and SMQ In Reset ACK #:

Since subsystems can restart at anytime, the data blocks and control channel
can be in an inconsistent state when a producer or consumer comes up.
We use Reset and Reset ACK to manage this. At cold boot, the producer
initializes the Reset# to a known number ex. 1. Every other reset that the
producer undergoes, the Reset#1 is simply incremented by 1. All the producer
indexes are reset.
When the producer notifies the consumer of data availability, the consumer
reads the producers Reset # and copies that into its SMQ In Reset ACK#
field when they differ. When that occurs, the consumer resets its
indexes to 0.

6) Asynchronous notifications between a producer and consumer are

done using the SMP2P service which is interrupt based.

Power Management
================

None

SMP/multi-core
==============

The driver uses completion to wake up the Debug Agent client threads.

Security
========

From the perspective of the subsystem, the AP is untrusted. The remote

stubs consult the secure debug fuses to determine whether or not the
remote debugging will be enabled at the subsystem.

If the hardware debug fuses indicate that debugging is disabled, the

remote stubs will not be functional on the subsystem. Writes to the
queue will only be done if the driver sees that the remote stub has been
initialized on the subsystem.

Therefore even if any untrusted software running on the AP requests

the services of the Remote Debug Driver and inject RSP messages
into the shared memory buffer, these RSP messages will be discarded and
an appropriate error code will be sent up to the invoking application.

Performance
===========

During operation, the Remote Debug Driver copies RSP messages

asynchronously sent from the host debugger to the remote stub and vice
versa. The debug messages are ASCII based and relatively short
(<25 bytes) and may once in a while go up to a maximum 700 bytes
depending on the command the user requested. Thus we do not
anticipate any major performance impact. Moreover, in a typical
functional debug scenario performance should not be a concern.

Interface
=========

The Remote Debug Driver is a character based device that manages

a piece of shared memory that is used as a bi-directional
single producer/consumer circular queue using a next fit allocator.
Every subsystem, has its own shared memory buffer that is managed
like a separate device.

The driver distinguishes each subsystem processor's buffer by

registering a node with a different minor number.

For each subsystem that is supported, the driver exposes a user space
interface through the following node:
- /dev/rdbg-<subsystem>
Ex. /dev/rdbg-adsp (for the ADSP subsystem)

The standard open(), close(), read() and write() API set is

implemented.

The open() syscall will fail if a subsystem is not present or supported

by the driver or a shared memory buffer cannot be allocated for the
AP - subsystem communication. It will also fail if the subsytem has
not initialized the queue on its side. Here are the error codes returned
in case a call to open() fails:
ENODEV - memory was not yet allocated for the device
EEXIST - device is already opened
ENOMEM - SMEM allocation failed
ECOMM - Subsytem queue is not yet setup
ENOMEM - Failure to initialize SMQ

read() is a blocking call that will return with the number of bytes written
by the subsystem whenever the subsystem sends it some payload. Here are the
error codes returned in case a call to read() fails:
EINVAL - Invalid input
ENODEV - Device has not been opened yet
ERESTARTSYS - call to wait_for_completion_interruptible is interrupted
ENODATA - call to smq_receive failed

write() attempts to send user mode payload out to the subsystem. It can fail
if the SMQ is full. The number of bytes written is returned back to the user.
Here are the error codes returned in case a call to write() fails:
EINVAL - Invalid input
ECOMM - SMQ send failed

In the close() syscall, the control information state of the SMQ is

initialized to zero thereby preventing any further communication between
the AP and the subsystem. Here is the error code returned in case
a call to close() fails:
ENODEV - device wasn't opened/initialized

The Remote Debug driver uses SMP2P for bi-directional AP to subsystem

notification. Notifications are sent to indicate that there are new
debug messages available for processing. Each subsystem that is
supported will need to add a device tree entry per the usage
specification of SMP2P driver.
In case the remote stub becomes non operational or the security configuration
on the subsystem does not permit debugging, any messages put in the SMQ will
not be responded to. It is the responsibility of the Debug Agent app and the
host debugger application such as GDB to timeout and notify the user of the
non availability of remote debugging.

Driver parameters
=================

None

Config options
==============

The driver is configured with a device tree entry to map an SMP2P entry
to the device. The SMP2P entry name used is "rdbg". Please see
kernel\Documentation\arm\msm\msm_smp2p.txt for information about the
device tree entry required to configure SMP2P.

The driver uses the SMEM allocation type SMEM_LC_DEBUGGER to allocate memory
for the queue that is used to share data with the subsystems.

Dependencies
============

The Debug Agent driver requires services of SMEM to

allocate shared memory buffers.

SMP2P is used as a bi-directional notification

mechanism between the AP and a subsystem processor.

User space utilities

====================

This driver is meant to be used in conjunction with the user mode

Remote Debug Agent application.

Other
=====

None

Known issues
============
For targets with an external subsystem, we cannot use
shared memory for communication and would have to use the prevailing
transport mechanisms that exists between the AP and the external subsystem.

This driver cannot be leveraged for such targets.

To do
=====

None