Touch Projector

Alan Cheng
Hope Harrison

13 December 2017
Contents

1 Project Overview 2

2 Process: Acquire Frames – Alan 5

2.1 Depth Frame Acquisition (Software) . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Data Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 FT2232H Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Image Storage and Thresholding . . . . . . . . . . . . . . . . . . . . . 7

3 Process Frames 12
3.1 Reduce Noise – Hope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.2 Detect Finger Blobs – Hope . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Find Gestures – Alan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Send Mouse Info – Hope 16

4.1 Computing Mouse Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.2 Output Mouse Report to Pmod . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 Send Mouse Commands Over USB . . . . . . . . . . . . . . . . . . . . . . . 17

5 Conclusion and Future Work 18

6 Informal Remarks 19

7 Acknowledgements 21

A FT2232H Mini Module Setup 22

B System Diagram 24

C Verilog Code 26

D Nexys4 Constraint File (xdc) 46

E Kinect Software Code – Kinect 2.0 SDK 51

F Kinect Software Code – libfreenect2 55

G Arduino HID Mouse Code 59

1
Chapter 1

Project Overview

We present an unique HMI device: the touch projector (Fig. 1.1. This projector allows users
to interact with any projected image as if it were a touchscreen. Specifically, we present a
hardware-focused approach to creating such device, which utilizes a FPGA to provide a
fast and consistent operation. Our prototype consists of a projector, Kinect, and a FPGA.
The Kinect provides depth-level sensing using a time of flight camera, which can be used to
determine finger touchpoints. The FPGA provides fast image processing on the data from
the Kinect, as well as acting as a HID-compliant mouse. This allows for our device to be
used with any arbitrary computer, as the control is done at the OS level. Thus, users can
simply touch the surface they are projecting onto, and the computer receives a click at the
point they touched. Examples applications of this device are:

Figure 1.1: One of the authors, Alan Cheng, using the touch projector

2
 • Education - allows instructors to give interactive presentations

• Business - creating a giant collaborative workspace akin to a smart whiteboard

• Entertainment - combinations with AR and/or VR tech to enable more personal in-

teractive experiences

This project is divided into three technical main modules:

Figure 1.2: Process breakdown for project

1. Acquire Frames: Kinect acquires depth-image frames and sends them to the FPGA.

2. Process Frames: FPGA processes image frames to calculate the coordinates where
the user is touching the surface. Thresholding, noise reduction, and edge detection is
needed.

3. Send Mouse Info: Coordinates of the touch as well as whether it was just pressed or
just released are transformed to the relative size of the monitor and send as a HID-
compliant mouse signal.

Figure 1.3 gives a more descriptive overview of the system architecture, showing the
subdivision of each process into hardware. The Acquire Frames module deals with the
Kinect to PC connection, the Process Frames module deals with FPGA processing, and the
Send Mouse Info module deals with FPGA to Teensy to PC connection. We aim to focus our
project into FPGA processing, rather than convention software options, for faster runtimes.
The follow chapters deal with a more descriptive overview of each process.

3
Figure 1.3: System architecture level design of project

4
Chapter 2

Process: Acquire Frames – Alan

In order to detect finger contact with a surface, the distance of a finger is needed. Using
RGB data calibrated to detect fingers blobs of a certain size is a possible method, but can
be plagued by a multitude of image properties such as lighting. Using a depth image from a
depth-sensing camera reduces the impact of external image changes, and provides an easier
interface. Thus, we use the Kinect 2.0, with its time of flight depth camera, to determine
finger touchpoints.
There are no openly available drivers that allow for direct connection of the Kinect to the
Nexys4 board (or any other FPGAs for that matter). To provide a bridge between Kinect
and FPGA, we utilize a computer to handle the connection. Specifically, we use Kinect
PC drivers coupled with OpenCV to acquire the depth image stream, and we send each
frame at 30 frames per second(fps) to the Nexys via FTDI’s FT2232H Hi-Speed Dual USB
UART/FIFO IC.

2.1 Depth Frame Acquisition (Software)

Both Micosoft SDK 2.0 and libfreenect2 where used independently to acquire data stream.
Microsoft’s SDK is officially supported, and perhaps is the more polished library. However,
the SDK is limited to use only on the Visual Studio platform. Moreover, the depth image
acquired is not the raw depth image, as Microsoft applies both bilateral filtering and edge-
aware filtering to the image before it can be read from.
In comparison libfreenect2 is a 3rd party open source cross-platform driver. It isn’t as
polished, nor is the documentation as available, but it does allow more control to the Kinect
settings. Specifically for this project, the bilateral filtering and edge-aware filtering can be
disable, so the raw depth frame can be used capture.
In this project, both drivers were used independently. Equivalent code for can be found
in the Appendix: Microsoft’s SDK 2.0, libfreenect (with OpenCL). Both work at an approx-
imately equal frame rate.

5
2.2 Data Transmission
2.2.1 FT2232H Configuration
The Kinect 2.0 captures depth frames of resolution 512x424 at 30fps, with depth values in
the range of 0 to 4500 mm having high precision. To maintain the depth data without loss,
we must be able to supply a 104.2 Mbit/s data transmission rate. This is because we need
to send a data bus of 512 × 424 pixels, each of size 2 bytes to cover the range of 0 to 4500.
Options for direct connection between PC and the Nexys4 are either too slow (Serial Port) or
too complex (Ethernet). We opted for using the FT2232H to provide fast data transmission
(it supports up to 480 Mbit/s) yet reasonable difficulty in use. Using the FTDI D2XX driver
as well as OpenCV, in a x86 VC++ application we can send the depth data (see Appendix
for code).
The FT2232H supports several modes of operation. However, only the synchronous FIFO
mode supports up to the 480 Mbit/s transmission rate. Using and outputting a 60 MHz clock,
the FT2232H outputs 1 byte of data at the given clock rate. Using FTDI’s configuration
software FT Prog (Fig 2.1), and the FTDI driver D2XX (Fig 2.2), the FT2232H can be
configured to these settings.

Figure 2.1: FT Prog configuration settings for synchronous FIFO mode

FT Prog Configuration

• PortA/B: Hardware = 245 FIFO

• PortA/B: Driver = D2XX

6
UCHAR mask = 0 x f f ;
UCHAR mode ;

mode = 0 x00 ; // r e s e t mode

f t s t a t u s = FT SetBitMode ( f t h a n d l e , mask , mode ) ;

Sleep ( 1 0 ) ;

mode = 0 x40 ; // s y n c h r o n o u s FIFO mode

f t s t a t u s = FT SetBitMode ( f t h a n d l e , mask , mode ) ;

Figure 2.2: D2XX C code to configure FT2232H to synchronous FIFO mode

D2XX Configuration
The wiring of the FT2232H is shown in figure A. We must connect the corresponding
VCCIO pins to connect the board to USB power. Outputs labeled are then connected to
the Nexys4 via 2 PMODs. PMOD JB corresponds to the data handling signals: CLK, RXF,
WR, RD, and OE. PMOD JC corresponds to the sent byte of data AD[7:0]. It should
be noted that the resistors on the PMOD JC of the Nexys4 were removed to
support the high rate of data transfer. Moreover, the ground signals of the
PMODs should be connected to the same ground as the FT2232H. Short physical
connections should be done in order to reduce noise on these signals. For instance, with a
long wire, the clock signal becomes noisy and unusable. A physical protoboard was used for
the FT2232H (see Appendix A
• CLK – 60 MHz clock output
• RXF – Data received signal (active low: 1 = no data in FIFO, 0 = data in FIFO)
• WR – Write enable input (active low: 1 = disables write, 0 = write data to FIFO from
data pins)
• RD – Read enable input (active low: 1 = disables read, 0 = read data from data pins)
• OE – Output enable input (active low: 1 = disables output, 0 = transfer data in FIFO
to data pins)
RXF goes low when there is data sent over to the FIFO from the USB. In order to read
the data, we must first set OE low to move data from the FIFO to the output pins AD[7:0].
Then, one clock cycle later, we set RD low to read from those pins. FTDI’s documentation
depicts the process in Fig. 2.4. The module ”FT2232H Control” handles this functionality,
reading in signals CLK and RXF, while outputting signals WR, RD, and OE.

2.2.2 Image Storage and Thresholding

In order to detect finger touches on a surface, we first store a depth frame of the background
in memory. Then, with each live incoming frame, we can compare the depth values between

7
 Figure 2.3: Wiring for FT2232H Mini Module

the two. Any object touching, or close to touching the surface will have a depth value
around 0 to 20mm less than the stored background. In implementation, this means that we
can receive a binarized image by subtracting the current frame depth values from the stored
background, where pixels within a set difference range is set to white, and others are set to
black. Fig. 2.5 shows the output of this technique.
There are several points to notice in the image:

1. The noise in an approximately rectangular frame is from the loss in accuracy of the
background. In practice, this area is outside the project image and can be ignored.

2. The hand/arm outline is present. This outline effect is a result of the Kinect’s time of
flight depth camera, which reads a gradiant of depth values between the abrupt edge
of the hand and air. Filtering is applied to remove the outlines, and will be discussed
later.

In terms of FPGA memory, this means we have to have storage for both the background
frame and the current received depth frame. The Nexys has 4860 kbits of fast BRAM storage,
of which 3473 kbits are allocated to this background frame. We cannot store two of such
images in BRAM, so for the current frame we only store its binarized image which takes 217
kbits. This is handled by the module ”ram control”.
This module contains 3 modes of operation: IDLE, WRITING, and UPDATING.

8
 Figure 2.4: Wiring for FT2232H Mini Module

1. IDLE - There is no depth data sent by the PC to the FT2232H yet, so the module
stays idle

2. WRITING - There is new depth data, and there is a signal to write, so the depth frame
is stored as the background frame in BRAM

3. UPDATING - There is new depth data, but there is no signal to write, so the depth
frame is read and thresholded with the existing background frame, and the resulting
binarized frame is stored in BRAM.

A simple state machine for these modes of operations are described in Fig. 2.6. Three
variables are used for state transition: write, new frame, and ready. Write is connected to a
pushbutton (BTND) that acts as the write signal. Thus, when a user presses the button, the
module will stored the incoming frame as the background frame in BRAM. new frame signals
high where there exists a new depth frame to be read. Each new frame is first preceded by
a specific code of DDCCBBAA. Module detect start sets signal new frame high when the

9
 Figure 2.5: Thresholded frame of finger touching a surface

coding sequence is found. Finally, ready signifies when the incoming depth frame has fully
been written to either the background or binary frame BRAM.
Operation in states WRITING and UPDATING are almost identical, as both needs
to read a full depth frame from the FT2232H. These states are handled in the module
frame mem control. Recall that each pixel is encoded by 2 bytes of data. Since the FT2232H
can only output 1 byte of data per clock cycle, it takes 2 clock cycles to read a pixel value.
Thus, in order to reconstruct the full depth value, we store the previous read value as well,
combining both the current and previous value to form the depth value. Starting from
new frame signal, this module increments an index corresponding with the current pixel
whose values are being read. After two clock cycles of read data, the index increments.
Moreover, another register data state keeps track of whether we are reading the first or
second value of a depth value. If we are reading the second value (data state == HI), then
we can combine the current with the previous value to get our depth value. This depth
either gets written to a dual port BRAM for the background frame (in state WRITING), or
compared with the corresponding background frame value at the current index to return a
single bit with 1 representing white and 0 representing black. This single bit value is written
to a separately dual port BRAM for the current binarized frame. Finally, this module sets
ready to high when the entire frame is ready, that is when the index exceeds the last pixel
index of 512 × 424. Fig. 2.7 demonstrates the relation between index and read data.

10
 Figure 2.6: FSM describing the modes of operation of ram control

Figure 2.7: Signal propagation of index, data, and thresholded value

11
Chapter 3

Process Frames

3.1 Reduce Noise – Hope

After binarization, our image is represented as a series of pixels which are either 0 (black) or
1 (white). We then filter out any sequences of 1s which are too small to correspond to a finger
touch, and generally smooth the image. The filtering is done by the opening morphological
operator. first applying an erosion kernel where we set all the pixels to 0 unless every pixel
directly surrounding them (in a 3x3 square) is 1. This will make any small blobs disappear.
We then apply a dilation kernel where any pixel that has a direct neighbor that is 1 also
becomes 1.
We originally planned on storing the entire image as a 2D 512 × 424 register and then
applying the kernel directly for each pixel in the image. This worked well at first for small
images and it was easy to implement because we could read any pixel value at any time and
set each pixel value easily.
Unfortunately, this did not work on larger images, because of compile times. For synthesis
and routing, dealing with 512 × 424 different wires would take a excruciatingly long time to
compile. To make compilation times reasonable, we limit data storage (after reading from
BRAM) to a 512×3 sized register. This sized register is enough to perform kernel operations
on one row of image data. Thus, our current method is the following: We first populate
a 512 x 3 register with the top 3 pixels of our image. Then while we apply the erosion or
dilation kernel to these pixels, we populated a buffer of the next row down. After finishing
the first three rows, the second row is moved up into the first slot, the third row into the
second, and the next row into the third. This continues so that we always have a three row
window to look at.
Opening worked very well for the images we received from the Kinect. After binarization,
the image still shows a thin outline around the hand and arm, rather than just a finger.
Figure 3.1 shows an example of this. It is very undesirable because it makes it difficult
to differentiate just the fingers. But because the outlines are so thin, the dilation almost
completely erases them and does leave us with only the denser fingertip, as seen in figure
3.2, which is the same image as before, but with the erosion and dilation applied.

12
Figure 3.1: The fingertip and outline of hand after binarization, where the blue rectangle
signifies the projection surface. Note: this is an artistic replication of actual image data, as
there wasn’t an easy way to save live image frames from the FPGA

3.2 Detect Finger Blobs – Hope

Because the noise reduction through erosion and dilation cleaned the image so well, the blob
detection was not as difficult as it otherwise might be. The algorithm we use for blob finding
is: once we find a pixel with a value of 1, set the current blob bounds to the coordinates of
that pixel. For each other pixel in the image, check if it is within some set threshold of the
current blob bounds, and if so, expand the blob bounds to include this pixel. Continue until
we have the bounds of the entire blob of densely packed pixels. We then check if the bounds
for this blob are the right size for a finger tip, and if so set the blob as valid.
We have the capacity in our algorithm to set the current blob to 0 if it is not valid
(because it’s either too big or too small) and then continue looking for another blob in an
attempt to find the real fingertip, however in practice, this did not prove helpful or necessary.
There are always a lot of small blobs around the edge of the screen – searching through these

13
Figure 3.2: Resultant binary image after performing the opening morphological operation,
where the blue rectangle significes the projection surface. Note: this is the result of opening
done in software on the previous image

blobs would have just been a waste of time and would slow our system down. But in the
center of the image where the screen is, there are rarely any extraneous blobs other than
fingertips. So we instead confine our blob search to set coordinates within the middle of the
frame, and this works well for us.
Currently, the coordinates for the blob search must be set manually using switches on
the Nexys board. The user should set these coordinates to align with the bounds of the
projected image. In the future, an autocalibrate option could be added by looking at color
data rather than just depth data to find the edges of the screen, but we were not able to
implement that at this time.

14
3.3 Find Gestures – Alan
We planned to be able to detect multiple unique gestures such as pinch to zoom. However,
given time constraints, this was never achieved. However, it is important to determine finger
press, finger move, and finger release gestures that equate to mouse clicks, which is handled
in module gesture detect.
Finger press is defined as the first instance where a finger touch is recognized by the
blob detection as described in earlier sections. This translate to a mouse click on the PC.
Finger moved is defined as the instance when a finger is moving around on the surface after
the initial mouse click. Finger release is defined as the instance when a finger leaves the
surfaces. This translates to the release of a mouse click on the PC. However, because of
problems with noise and inconsistent framerates caused by the image processing modules,
it is possible for some finger touch to be not recognized while the finger is still physically
touching the surface. Thus, we implemented a buffer to allow frames with dropped finger
recognition. Given a set maximum frame timeout value (10 was used for demonstration),
the module will not signal mouse release until the number of frames counted sequentially
without a finger presence reaches the timeout value.

15
Chapter 4

Send Mouse Info – Hope

In order to control the computer by touching the projection, we must have our device appear
as an HID mouse to the computer. Since the Nexys 4 is only able to function as a USB
host (but not slave), we are not able to output directly from the Nexys 4 to the computer.
Therefore, we will use a Teensy as an intermediate.

4.1 Computing Mouse Report

Using the data from the gesture report of whether the user has pressed, released or moved,
and the coordinates of their finger, we run a state machine which sets these parameters one
after another to be sent over a serial connection to the Teensy. There are 15 bytes of data
which must be sent:

• X coordinate of press (2 bytes)

• Y coordinate of press (2 bytes)

• X minimum bound (2 bytes)

• Y minimum bound (2 bytes)

• X maximum bound (2 bytes)

• Y maximum bound (2 bytes)

• Is pressed? (really 1 bit, but sent as 1 byte for simplicity)

• Is released? (really 1 bit, but sent as 1 byte for simplicity)

• Is moved? (really 1 bit, but sent as 1 byte for simplicity)

This module makes sure that these numbers are sent in this order (1 byte at a time) so that
the Teensy can read them in the same order and know which is which. When the pmod
output module indicates that it has finished sending a number, the Mouse Report sets the
next number to be sent.

16
4.2 Output Mouse Report to Pmod

Figure 4.1: The Teensy with pin 9 wired to JA[0]

The pmod output module sends whatever parameter it is currently being fed to the
Teensy. We use a 115200 baud rate, and send on JA[0] which is connected to pin 9 on the
Teensy (fourth pin from the bottom on the left in figure 4.1). This module functions as a
normal serial sender, except that it also gives a pulse to indicate when it has finished sending
a 1 byte number so that the Mouse Report Module will know when to feed it a new number
to send.

4.3 Send Mouse Commands Over USB

The Teensy uses a built in serial driver to receive the data from the digital output of the
Nexys. As it receives the 15 bytes it uses the order to determine the values of all the
parameters in the mouse report. It uses the minimum and maximum bounds as well as the
absolute coordinates of the press and the resolution of the monitor of the computer it is
connected to (this must be pre-set) to calculate the coordinates to move the mouse to. The
reason the absolute coordinates and the bounds are sent to the Teensy rather than just the
relative position within the bounds is to avoid doing division on the FPGA. The is moved?
is pressed? and is released? parameters control whether the Teensy actually sends a move
command to those coordinates and whether that also includes a press and a release of the
left mouse button.

17
Chapter 5

Conclusion and Future Work

Through this project, we show a working prototype of a touch project working primarily on
FPGA hardware. We were able to stream high-speed depth data from the Kinect to the PC
to the FPGA using the FT2232H at around 30fps for a 512 by 424 resolution image. We
also shown image processing modules to remove noise in the resulting thresholding depth
image, and were able to detect key blobs in the processed image. Finally, we were able to
send commands for the Teensy to act as a HID compliant mouse for the computer. Because
of this interface, we were able to run a variety of different applications, such as Fresh Paint,
Solitare, Chess, and Osu!.
There are a plethora of problems with the prototype presented. Two key problems will
be discussed on.

1. Performance - although we transfer data at 30 fps, the image processing module takes
longer to complete. Therefore, the real framerate probably lies around 10-20 fps. We
can improve accuracy and latency of the device by reducing the runtime of the image
processing algorithm. One method would be to switch to a highly parallel system
consisting of many modules that each represent a pixel. If a module knows the value
of itself and it’s neighbors, we could do 3x3 kernel operations in one clock cycle,
significantly cutting down the runtime.

2. Accuracy - there is very low accuracy between finger touchpoint and the resulting
mouse location. This is because the fingertip itself its really found, but rather the
finger joint that protrudes out slightly farther from the surface background. Perhaps
it would be useful to consider the skeletal composition of the hand. Furthermore,
movement is rather jagged, and can be fixed with filters. Alan has previously worked
on a hardware based implementation of the Kalman Filter using Faddeev’s algorithm
that may be helpful.

18
Chapter 6

Informal Remarks

Alan:
Driver work and data transmission is hard. Do not underestimate the time it takes to
implement this, despite how boring or easy it looks to be. I anticipated completing this
module in 1-2 weeks, but in actuality it ended up taking all the way up to the last 4 days
before it was completely finished. And I guess what goes with that is to plan your timeline
well, and leave a lot of buffer space of mistakes. Also, make sure that your measuring tools
are plugged in properly. I have definitely wasted time trying to debug a scoping issue when
it was just that the probe was plugged backwards.
In regards to the project as a whole, I am glad that we were able to get a working
prototype. This touch projector idea has been a big part of my personal projects/interests
throughout the years, and I am always glad to let other people experience it. As for the
technical achievements, I think there is a lot to be desired. To be honest, I would have much
prefer to work on interesting applications or utilities of the touch projector rather than driver
work, but what must be done must be done.
Lastly, I am always look for people interested in working on this idea, or for people who
would perhaps like to see this spun into a start-up type idea. Please contact me if you are
interested!
Hope:
This project was challenging, yet also fun. It did however have a lot of frustrations. As
advice to future students of 6.111, we would recommend that if you have any issues that
you think might be a problem later on, try to bring them up and get ideas early on. For
example, with the image processing, I (Hope) knew it took a long time to compile and could
have realized earlier that we would not be able to hold the entire image in a register once it
got bigger than the small images I used to test. If I’d addressed this earlier, the last week
would not have been as stressful. Another good resource for debugging when you are at that
stage is making test benches. At first I usually just debugged by trying my code and hoping
it worked, or maybe outputting some signals to LEDs to look at them and try to see why
it’s not working. But when my partner Alan gave me the idea to make a testbench it made
the debugging easier because I could probe all the signals and see everything that was going
on. So it’s really not just something to use when required in the labs, it’s actually a tool
that will help you a lot! In the end, though, the most important is to have fun while working
hard and remember what you’ll accomplish. We were really happy to see our project work

19
and get to try playing solitaire on just an ordinary whiteboard with a projector!

20
Chapter 7

Acknowledgements

We would like to thank our project mentor, Joe Steinmeyer, who provided us with the
FT2232H, modified Nexys4 Board, and the multiple projectors. He went out of his way
to help our struggling project throughout the most stressful times. More importantly, he
provided us with invaluable advice that helped us make our way to the final prototype.
Alan: I would especially like to thank Joe for spending countless hours with me debug-
ging the FT2232H module throughout the past couple weeks, and for dealing with the many
lengthy emails that I sent. Without his help, this project would not be successful.

We would also like to thank Professor Gim Hom for making 6.111 a wonderful experience.
He also helped with our image processing module, and suggested that we cut down on the
register sizes in order to make the module compile in a relatively short amount of time. He
truly is the most enthusiastic person for these projects, and 6.111 just wouldn’t be the same
without him.

Alan: Finally, I would also like to thank Professor Marek Perkowski of Portland State
University. He provided initial guidance during the original project – a software implemen-
tation of the touch projector – back in 2013. Without his help during my highschool years,
the original idea would have never happened, and I would have not been the engineer that
I am now.

21
Appendix A

FT2232H Mini Module Setup

22
23
Appendix B

System Diagram

24
25
Appendix C

Verilog Code

26
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : TP_Top
3 // Author : Alan Cheng
4 // Description : Top level module
5 //
6 // Dependencies : everything
7 //
8 // Additional Comments :
9 //
10 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
11 module TP_Top (
12 input CLK100MHZ ,
13 input [ 15 : 0 ] SW ,
14 input BTNC , BTNU , BTNL , BTNR , BTND ,
15 output [ 3 : 0 ] VGA_R ,
16 output [ 3 : 0 ] VGA_B ,
17 output [ 3 : 0 ] VGA_G ,
18 output VGA_HS ,
19 output VGA_VS ,
20 output LED16_B , LED16_G , LED16_R ,
21 output LED17_B , LED17_G , LED17_R ,
22 output [ 15 : 0 ] LED ,
23 output [ 7 : 0 ] SEG , // segments A - G (0 -6) , DP (7)
24 output [ 7 : 0 ] AN , // Display 0 -7
25 output [ 7 : 0 ] JA , // Output PMOD to Teensy
26 input [ 7 : 0 ] JC , // FT2232_H input Data
27 input [ 1 : 0 ] JB_IN , // FT2232_H { RXF , clock_60_mhz }
28 output [ 2 : 0 ] JB_OUT // FT2232_H { OE , RD , WE }
29 );
30
31
32 // create 25 mhz system clock
33 wire clock_25mhz ;
34 c l o c k _ q u a r t e r _ d i v i d e r clockgen (. clk100_mhz ( CLK100MHZ ) , . clock_25mhz ( clock_25mhz ) ) ;
35
36 // instantiate 7 - segment display ;
37 wire [ 31 : 0 ] data ;
38 wire [ 6 : 0 ] segments ;
39 assign data = ( SW [ 15 ] ) ? {22 ’ d0 , SW [ 9 : 0 ] } : {11 ’ d0 , SW [ 9 : 5 ] , 11 ’ d0 , SW [ 4 : 0 ] };
40 display_8hex display (. clk ( clock_25mhz ) ,. data ( data ) , . seg ( segments ) , . strobe ( AN ) ) ;
41 assign SEG [ 6 : 0 ] = segments ;
42 assign SEG [ 7 ] = 1 ’ b1 ;
43
44 assign LED [ 15 : 2 ] = SW [ 15 : 2 ] ;
45 assign LED16_R = BTNL ; // left button -> red led
46 assign LED16_G = BTNC ; // center button -> green led
47 assign LED16_B = BTNR ; // right button -> blue led
48 assign LED17_R = BTNL ;
49 assign LED17_G = BTNC ;
50 assign LED17_B = BTNR ;
51
52 wire BTNC_clean , BTNU_clean , BTND_clean , BTNL_clean , BTNR_clean ;
53 debounce debounce_btnc (. reset ( global_reset ) , . clock ( clock_25mhz ) , . noisy ( BTNC ) , . clean ( BTNC_clean ) ) ;
54 debounce debounce_btnu (. reset ( global_reset ) , . clock ( clock_25mhz ) , . noisy ( BTNU ) , . clean ( BTNU_clean ) ) ;
55 debounce debounce_btnd (. reset ( global_reset ) , . clock ( clock_25mhz ) , . noisy ( BTND ) , . clean ( BTND_clean ) ) ;
56 debounce debounce_btnl (. reset ( global_reset ) , . clock ( clock_25mhz ) , . noisy ( BTNL ) , . clean ( BTNL_clean ) ) ;
57 debounce debounce_btnr (. reset ( global_reset ) , . clock ( clock_25mhz ) , . noisy ( BTNR ) , . clean ( BTNR_clean ) ) ;
58
59 wire [4:0] threshold_high_bound , t h r e s h o l d _ l o w _ b o u n d ;
60 wire [4:0] blob_min_size , blob_max_size ;
61 wire [9:0] edge_to_ignore_up_h ;
62 wire [9:0] edge_to_ignore_up_v ;
63 wire [9:0] edge_to_ignore_bot_h ;
64 wire [9:0] edge_to_ignore_bot_v ;
65
66 swit ch_cont rol s w i t c h _ p a r a m e t e r s (. clk ( clock_25mhz ) , . SW ( SW ) , . BTNC_clean ( BTNC_clean ) , . BTNL_clean ( BTNL_clean ) , .
,→ BTNR_clean ( BTNR_clean ) ,
67 . threshold_high_bound ( threshold_high_bound ),
68 . threshold_low_bound ( threshold_low_bound ),
69 . blob_min_size ( blob_min_size ) ,
70 . blob_max_size ( blob_max_size ) ,
71 . edge_to_ignore_up_h ( edge_to_ignore_up_h ),
72 . edge_to_ignore_up_v ( edge_to_ignore_up_v ),
73 . edge_to_ignore_bot_h ( edge_to_ignore_bot_h ),
74 . edge_to_ignore_bot_v ( edge_to_ignore_bot_v ));
75
76 parameter WIDTH = 512; // Width of image frame
77 parameter HEIGHT = 424; // Depth of image frame
78 parameter LOGSIZE = 18; // smallet LOGSIZE s . t . 2^ LOGSIZE > WIDTH * HEIGHT
79
80 wire WR , RD , OE ; // Write / Read / Output_Enable from FT2232H
81 wire img_ready ; // Signals when binarized frame is ready in RAM
82
83 assign JB_OUT [ 0 ] = WR ;
84 assign JB_OUT [ 1 ] = RD ;
85 assign JB_OUT [ 2 ] = OE ;
86
87 wire [ 7 : 0 ] ft_data ;
88
89 assign write _top_lef t = SW [ 12 ] ;
90 assign w r i t e _ b o t t o m _ r i g h t = SW [ 11 ] ;
91

27
 92 FT2232_Data PC_Data (. clock_60_mhz ( JB_IN [0 ] ) , . RXF ( JB_IN [ 1 ] ) , . WR ( WR ) , . RD ( RD ) , . OE ( OE ) ) ;
93
94 wire pixel ; // pixel value ([0 ,1]) at vga_index , or internal index while updating
95 wire [ LOGSIZE -1 : 0 ] index_out ;
96 wire [ LOGSIZE -1 : 0 ] index _img_pro c ;
97
98 ram_control #(. WIDTH ( WIDTH ) , . HEIGHT ( HEIGHT ) , . LOGSIZE ( LOGSIZE ) )
99 stream_handle (. reset ( BTNU_clean ) , . clk ( CLOCK100MHZ ) , . clock_60_mhz ( JB_IN [0 ] ) , . write_btn ( BTND_clean ) ,
100 . RXF (~ JB_IN [ 1 ] ) , . OE (~ OE ) , . RD (~ RD ) , . data ( JC ) , . ready ( img_ready ) , . read_index ( vga_index )
101 ,. wr_bin_mem ( wr_bin_mem ) , . data_bin ( wr_bin_val )
102 ,. index_out ( index_out ) ,
103 . low_bound ( t h r e s h o l d _ h i g h _ b o u n d ) , . high_bound ( t h r e s h o l d _ l o w _ b o u n d ) ) ;
104
105 assign LED [ 0 ] = img_ready ;
106 assign LED [ 1 ] = BTND_clean ;
107
108 // Binary BRAM to be used for image_process
109 wire wr_bin_mem , wr_bin_val ;
110 frame_bram #(. SIZE ( WIDTH * HEIGHT ) , . LOGSIZE ( LOGSIZE ) , . BITS (1 ’ b1 ) )
111 binary_bram (. wr_addr ( index_out ) ,. rd_addr ( index_im g_proc ) ,. wr_clk ( JB_IN [0 ] ) ,. rd_clk ( CLK100MHZ ) ,
112 . we ( wr_bin_mem ) ,. rd (1 ’ b1 ) ,. din ( wr_bin_val ) ,. dout ( pixel ) ) ;
113
114 // Binary BRAM to be used VGA display of thresholded image ( without image process )
115 wire [ 17 : 0 ] in d ex _i mg _ in pu t ;
116 wire pixel_input ;
117 assign i nd e x_ im g_ i np ut = ( vcount_in << 9) + hcount_in ;
118 frame_bram #(. SIZE ( WIDTH * HEIGHT ) , . LOGSIZE ( LOGSIZE ) , . BITS (1 ’ b1 ) )
119 b i n a r y _ i n p u t _ b r a m (. wr_addr ( index_out ) ,. rd_addr ( i nd e x_ im g_ i np ut ) ,. wr_clk ( JB_IN [0 ] ) ,. rd_clk ( clock_25mhz ) ,
120 . we ( wr_bin_mem ) ,. rd (1 ’ b1 ) ,. din ( wr_bin_val ) ,. dout ( pixel_input ) ) ;
121 wire [ 7 : 0 ] p ix el _ bi n_ in p ut ;
122 assign p ix e l_ bi n_ i np ut = ( a t _ d i s p l a y _ a r e a _ i n && pixel_input ) ? 255 : 0; // Map [1 ,0] to [255 ,0]
123 wire [ 9 : 0 ] hcount_in , vcount_in ;
124 wire a t _ d i s p l a y _ a r e a _ i n ;
125 vga #(. HSIZE ( WIDTH ) , . VSIZE ( HEIGHT ) ) v g a _ i n p u t _ d i s p l a y (. vga_clock ( clock_25mhz ) ,. hcount ( hcount_in ) ,. vcount ( vcount_in ) ,
,→ . at _d i sp la y_ a re a ( a t _ d i s p l a y _ a r e a _ i n ) ) ;
126
127 // 2. Process Frames
128 wire [ 7 : 0 ] pixel_out ;
129 wire [ 9 : 0 ] touch_h ; // y coordinate of user touch
130 wire [ 9 : 0 ] touch_v ; // x coordinate of user touch
131 wire touch ; // whether there is a valid touch
132 wire [ 9 : 0 ] top_left_x , top_left_y , bottom_right_x , b ottom_r ight_y ;
133 wire touch_ready ;
134 wire write_top_left , w r i t e _ b o t t o m _ r i g h t ;
135 wire in_blob_box , in_frame_box ; // whether the current pixel is on the edge of the box around the blob
136 image_process #(. HSIZE ( WIDTH ) ,. VSIZE ( HEIGHT ) ) i m ag e_ pr o ce ss _1
137 (. clock ( CLK100MHZ ) , . fr am e_ a va il ab l e ( img_ready ) ,
138 . bin_index ( i ndex_img _proc ) ,. pixel_out ( pixel_out ) , . pixel_in ( pixel ) ,
139 . blob_min_size ( blob_min_size ) , . blob_max_size ( blob_max_size ) ,
140 . edge_to_ignore_up_h ( edge_to_ignore_up_h ), . edge_to_ignore_up_v ( edge_to_ignore_up_v ),
141 . edge_to_ignore_bot_h ( edge_to_ignore_bot_h ), . edge_to_ignore_bot_v ( edge_to_ignore_bot_v ),
142 . hsync ( hsync ) ,. vsync ( vsync ) ,. in_blob_box ( in_blob_box ) ,. in_frame_box ( in_frame_box ) ,
143 . touch_h ( touch_h ) ,. touch_v ( touch_v ) ,. touch ( touch ) ,. touch_ready ( touch_ready ) ,
144 . top_left_h ( top_left_x ) , . top_left_v ( top_left_y ) , . bottom_r ight_h ( bottom _right_x ) , . botto m_right_ v (
,→ bott om_right _y ) ) ;
145
146 wire [ 7 : 0 ] pixel_vga ;
147 assign pixel_vga = SW [ 10 ] ? p ix el _b i n_ in pu t : pixel_out ;
148 assign VGA_R = in_blob_box ? 255 : pixel_vga ;
149 assign VGA_G = in_frame_box ? 255 : in_blob_box ? 0 : pixel_vga ;
150 assign VGA_B = in_frame_box ? 0 : pixel_vga ;
151 assign VGA_HS = ~ hsync ;
152 assign VGA_VS = ~ vsync ;
153
154 wire is_press , is_release , is_move , is_scroll ;
155 wire s e n d _ m o u s e _ e n a b l e ;
156
157 // 3. Send Mouse Info
158 // Sends the touch location to the Teensy over serial :
159 send_mouse s e n d _ m o u s e _ m o d u l e (. x ( touch_h ) , . y ( touch_v ) , . is_press ( is_press ) , . is_release ( is_release ) , . is_move ( is_move )
,→ ,
160 . top_left_x ( top_left_x ) , . top_left_y ( top_left_y ) , . b ottom_ri ght_x ( bottom_ right_x ) , .
,→ bott om_right _y ( bot tom_righ t_y ) ,
161 . reset ( BTNU_clean ) , . clock ( CLK100MHZ ) , . send_enable ( s e n d _ m o u s e _ e n a b l e ) ,
162 . pmod ( JA ) ) ;
163
164 gest ure_dete ct gesture_ module (. clock ( CLK100MHZ ) , . touch_h ( touch_h ) , . touch_v ( touch_v ) , . touch ( touch ) , . touch_ready (
,→ touch_ready ) ,
165 . is_press ( is_press ) , . is_release ( is_release ) , . is_move ( is_move ) , . is_scroll ( is_scroll ) ,
166 . send_mouse_enable ( send_mouse_enable ));
167 endmodule
168
169 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
170 // The following are virtually untouched from the labkit . Only parameters for the
171 // correct resolution was added into VGA
172 module c l o c k _ q u a r t e r _ d i v i d e r ( input clk100_mhz , output reg clock_25mhz = 0) ;
173 reg counter = 0;
174
175 // VERY BAD VERILOG
176 // VERY BAD VERILOG
177 // VERY BAD VERILOG
178 // But it ’ s a quick and dirty way to create a 25 Mhz clock

28
179 // Please use the IP Clock Wizard under FPGA Features / Clocking
180 //
181 // For 1 Hz pulse , it ’ s okay to use a counter to create the pulse as in
182 // assign onehz = ( counter == 100 _000_000 ) ;
183 // be sure to have the right number of bits .
184
185 always @ ( posedge clk100_mhz ) begin
186 counter <= counter + 1;
187 if ( counter == 0) begin
188 clock_25mhz <= ~ clock_25mhz ;
189 end
190 end
191 endmodule
192
193 module vga #( parameter HSIZE =512 ,
194 parameter VSIZE =424)
195 ( input vga_clock ,
196 output reg [ 9 : 0 ] hcount = 0 , // pixel number on current line
197 output reg [ 9 : 0 ] vcount = 0 , // line number
198 output reg vsync , hsync ,
199 output a t _d is pl a y_ ar ea ) ;
200
201 // Comments applies to XVGA 1024 x768 , left in for reference
202 // horizontal : 1344 pixels total
203 // display 1024 pixels per line
204 reg hblank , vblank ;
205 wire hsyncon , hsyncoff , hreset , hblankon ;
206 assign hblankon = ( hcount == 639) ; // active H 1023
207 assign hsyncon = ( hcount == 655) ; // active H + FP 1047
208 assign hsyncoff = ( hcount == 751) ; // active H + fp + sync 1183
209 assign hreset = ( hcount == 799) ; // active H + fp + sync + bp 1343
210
211 // vertical : 806 lines total
212 // display 768 lines
213 wire vsyncon , vsyncoff , vreset , vblankon ;
214 assign vblankon = hreset & ( vcount == 479) ; // active V 767
215 assign vsyncon = hreset & ( vcount ==490 ) ; // active V + fp 776
216 assign vsyncoff = hreset & ( vcount == 492) ; // active V + fp + sync 783
217 assign vreset = hreset & ( vcount == 523) ; // active V + fp + sync + bp 805
218
219 // sync and blanking
220 wire next_hblank , next_vblank ;
221 assign next_hblank = hreset ? 0 : hblankon ? 1 : hblank ;
222 assign next_vblank = vreset ? 0 : vblankon ? 1 : vblank ;
223 always @ ( posedge vga_clock ) begin
224 hcount <= hreset ? 0 : hcount + 1;
225 hblank <= next_hblank ;
226 hsync <= hsyncon ? 0 : hsyncoff ? 1 : hsync ; // active low
227
228 vcount <= hreset ? ( vreset ? 0 : vcount + 1) : vcount ;
229 vblank <= next_vblank ;
230 vsync <= vsyncon ? 0 : vsyncoff ? 1 : vsync ; // active low
231
232 end
233
234 assign a t_ d is pl ay _ ar ea = (( hcount >= 0) && ( hcount < 512) && ( vcount >= 0) && ( vcount < 424) ) ;
235
236 endmodule
237
238 module switc h_contro l (
239 input clk ,
240 input [ 15 : 0 ] SW ,
241 input BTNC_clean ,
242 input BTNL_clean ,
243 input BTNR_clean ,
244 output reg [ 4 : 0 ] threshold_high_bound ,
245 output reg [ 4 : 0 ] threshold_low_bound ,
246 output reg [ 4 : 0 ] blob_min_size ,
247 output reg [ 4 : 0 ] blob_max_size ,
248 output reg [ 9 : 0 ] edge_to_ignore_up_h ,
249 output reg [ 9 : 0 ] edge_to_ignore_up_v ,
250 output reg [ 9 : 0 ] edge_to_ignore_bot_h ,
251 output reg [ 9 : 0 ] e d g e _ t o _ i g n o r e _ b o t _ v ) ;
252
253 always @ ( posedge clk ) begin
254 if ( BTNC_clean ) begin
255 t h r e s h o l d _ h i g h _ b o u n d <= SW [4 : 0 ] ;
256 t h r e s h o l d _ l o w _ b o u n d <= SW [9 : 5 ] ;
257 end
258 if ( BTNL_clean ) begin
259 blob_min_size <= SW [4 : 0 ] ;
260 blob_max_size <= SW [9 : 5 ] ;
261 end
262 if ( BTNR_clean ) begin
263 case ( SW [ 14 : 13 ] )
264 2 ’ b00 : e d g e _ t o _ i g n o r e _ u p _ h <= SW [9 : 0 ] ;
265 2 ’ b01 : e d g e _ t o _ i g n o r e _ u p _ v <= SW [9 : 0 ] ;
266 2 ’ b10 : e d g e _ t o _ i g n o r e _ b o t _ h <= SW [9 : 0 ] ;
267 2 ’ b11 : e d g e _ t o _ i g n o r e _ b o t _ v <= SW [9 : 0 ] ;
268 default : e d g e _ t o _ i g n o r e _ u p _ h <= SW [9 : 0 ] ;
269 endcase
270 end

29
271 end
272 endmodule

30
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : FT2232_Data
3 // Author : Alan Cheng
4 // Description : Controls RD and OE signals to read from FT2232H properly
5 //
6 // Dependencies :
7 //
8 // Additional Comments : See FT2232H FIFO Documentation for timing properties
9 // Reacts async hronousl y with changes in RXF
10 //
11 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
12
13 module FT2232_Data (
14 input clock_60_mhz , // 60 MHz FT_2232H clock
15 input RXF , // RXF from FT_2232H : 0 if there is data to be read
16 output WR , // WR to FT_2232H : 0 if data needs to be written
17 output RD , // RD to FT_2232H : 0 to read from buffer
18 output OE // OE to FT_2232H : 0 to enable output for data
19 // to be moved from buffer . Must be low before RD
20 );
21
22 reg RD_buf , OE_buf ;
23 reg tmp = 1; // tmp buffer to delay signal
24
25 always@ ( posedge clock_60_mhz ) begin
26 if (! RXF ) begin
27 RD_buf <= OE_buf ;
28 OE_buf <= tmp ;
29 tmp <= RXF ;
30 end
31 else begin
32 RD_buf <= 1 ’ b1 ;
33 OE_buf <= 1 ’ b1 ;
34 tmp <= 1 ’ b1 ;
35 end
36 end
37
38 assign WR = 1 ’ b1 ; // We never write to PC
39
40 assign RD = RD_buf ;
41 assign OE = OE_buf ;
42
43 endmodule
44
45 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
46 // Module Name : ram_control
47 // Author : Alan Cheng
48 // Description : handles incoming image data from FT2232H and :
49 // 1) stores full frame in BRAM with button press
50 // 2) thresholds for binary image using stored frame in BRAM
51 //
52 // Dependencies : bram , background_mem_control , d e t e c t _ n e w _ f r a m e
53 //
54 // Additional Comments : read_index input allows to read from bram when possible
55 // this top module focuses on state control
56 //
57 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
58 module ram_control #( parameter WIDTH =512 , HEIGHT =424 , LOGSIZE =18 ,
59 DEPTH_SIZE =16 , BIN_SIZE =1)
60 ( input reset , // global reset
61 input clk , // 100 MHz FPGA system clock
62 input clock_60_mhz , // 60 MHz FT_2232H clock
63 input write_btn , // Write signal : 1 stores background frame in BRAM
64 input RXF , // RXF from FT_2232H
65 input OE , // Output_Enable from FT_2232H
66 input RD , // RD from FT_2232H
67 input [ 7 : 0 ] data , // Incoming byte of data from FT_2232H
68 input [ LOGSIZE -1 : 0 ] read_index , // Input index of Binary BRAM frame to read
69 input [ 4 : 0 ] low_bound , // lower bound for threshold
70 input [ 4 : 0 ] high_bound , // upper bound for threshold
71 output ready , // Ready signal : 1 if complete frame is stored ; 0 if updating
72 output bin_out , // Binary pixel value at read_index or index_bin if updating
73 output wr_bin_mem , // Write to binary memory singal
74 output data_bin , // binary value for pixel at index_bin
75 output [ LOGSIZE -1 : 0 ] index_out // index of output binary pixel value
76 );
77
78 // States :
79 parameter [ 1 : 0 ] UPDATING = 2 ’ b10 ; // Updating binary image
80 parameter [ 1 : 0 ] WRITING = 2 ’ b01 ; // Storing frame to background frame BRAM
81 parameter [ 1 : 0 ] IDLE = 2 ’ b00 ; // Idle ; all processes done
82 reg [ 1 : 0 ] state ;
83
84 wire [ LOGSIZE -1 : 0 ] index_bin , index_bin_mem ; // coresponding index to data_bin , index input to binary BRAM
85
86 wire wr_bin , new_frame ; // calculated write timing / write input for binary BRAM ( wr_bin / wr_bin_mem )
87 // new_frame flag
88
89 detect_start d e t e c t _ n e w _ f r am e (. reset ( reset ) , . clock_60_mhz ( clock_60_mhz ) , . RD ( RD ) , . data_in ( data ) , . ready ( new_frame ) ) ;
90
91 b a c k g r o u n d _ m e m _ c o n t r o l #(. WIDTH ( WIDTH ) , . HEIGHT ( HEIGHT ) , . LOGSIZE ( LOGSIZE ) , . DEPTH_SIZE ( DEPTH_SIZE ) )
92 strea m_contr ol (. reset ( reset || new_frame ) , . clk ( clk ) , . clock_60_mhz ( clock_60_mhz ) , . new_data ( RD ) ,

31
93 . RXF ( RXF ) , . write ( state [ 0 ] ) , . data_in ( data ) , . data_bin_wr ( data_bin ) , . WR_bin ( wr_bin ) ,
94 . index_bin ( index_bin ) , . ready ( ready ) , . low_bound ( low_bound ) , . high_bound ( high_bound ) ) ;
95
96 assign wr_bin_mem = ( state == UPDATING ) ? wr_bin : 0; // prohibit writes to binary BRAM when not
,→ updating
97 assign index_bin_mem = ( state == IDLE ) ? read_index : index_bin ; // when IDLE , take index from input ; else use
,→ signal from mem_control
98
99 // State Machine Model
100 always@ ( posedge clock_60_mhz ) begin
101 // Global reset
102 if ( reset )
103 state <= IDLE ;
104 else begin
105 debug <= new_frame ;
106 if ( state == IDLE ) begin
107 if ( write_btn && new_frame )
108 state <= WRITING ;
109 else if ( new_frame )
110 state <= UPDATING ;
111 end
112 else if ( state == UPDATING ) begin
113 if ( ready && ! new_frame )
114 state <= IDLE ;
115 end
116 else if ( state == WRITING ) begin
117 if ( ready && ! new_frame )
118 state <= IDLE ;
119 end
120 end
121 end
122
123 endmodule
124
125 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
126 // Module Name : b a c k g r o u n d _ m e m _ c o n t r o l
127 // Author : Alan Cheng
128 // Description : handles incoming image data from FT2232H and :
129 // 1) stores full frame in BRAM with button press
130 // 2) thresholds for binary image using stored frame in BRAM
131 //
132 // Dependencies : bram
133 //
134 // Additional Comments : adding some debug code for thresholding non - reliant
135 // on background frame BRAM
136 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
137 module b a c k g r o u n d _ m e m _ c o n t r o l #( parameter WIDTH =512 , // Depth of image resolution (512 x424 )
138 HEIGHT =424 , // Width of image resolution
139 LOGSIZE =18 , // Smallet LOGSIZE s . t . 2^ LOGSIZE > WIDTH * HEIGHT
140 DEPTH_SIZE =16) // Size of each pixel : 2 bytes = 16 bits
141 ( input reset , // global reset
142 input clk , // 100 MHz FPGA system clock
143 input clock_60_mhz , // 60 MHz FT_2232H clock
144 input new_data , // 1 indicates there is new data to process
145 input write , // active - high write signal
146 input RXF , // RXF from FT_2232H
147 input [ 7 : 0 ] data_in , // data read from FT2232H
148 input [ 4 : 0 ] low_bound , // lower bound for threshold
149 input [ 4 : 0 ] high_bound , // upper bound for threshold
150 output reg data_bin_wr , // data for binary image
151 output reg WR_bin , // write enable for binary image ram
152 output reg [ LOGSIZE -1 : 0 ] index_bin , // index corresponding to data_bin_wr
153 output reg ready // 1 when image frame is done processing
154 );
155
156 // Determines white area in binary image if within threshold margins
157 wire [ 4 : 0 ] BA CK _T H RE SH _M I N ; // mem - 30 = lower bound
158 wire [ 4 : 0 ] BA CK _T H RE SH _M A X ; // mem - 2 = upper bound
159
160 assign B AC K _T HR ES H _M IN = high_bound ;
161 assign B AC K _T HR ES H _M AX = low_bound ;
162
163 // Fixed range limitations
164 parameter BACK _DEPTH_M IN = 500;
165 parameter BACK _DEPTH_M AX = 2000;
166 parameter NUM_PIXELS = WIDTH * HEIGHT -1; // Max number of pixels ; used to flag ready
167
168 // state of inc data : 1=2 nd byte , 0=1 st byte
169 parameter HI = 1;
170 parameter LO = 0;
171 reg data_state ;
172
173 reg prev_RXF ; // previous RXF
174 reg [ 7 : 0 ] prev_data ; // previous data
175
176 reg [ LOGSIZE -1 : 0 ] index ; // index of current pixel
177
178 // 2 bytes make full distance values
179 wire [ 15 : 0 ] data_back_wr ; // {2 nd byte , 1 st byte }\
180 assign data_back_wr = { prev_data , data_in };
181
182 wire [ 15 : 0 ] ram_data ; // 2 byte at index from background BRAM

32
183
184 // BRAM to store image
185 frame_bram #(. SIZE ( WIDTH * HEIGHT ) , . LOGSIZE ( LOGSIZE ) , . BITS ( DEPTH_SIZE ) )
186 b ac kg ro u nd _b ra m (. wr_addr ( index ) ,. rd_addr ( index ) ,. wr_clk ( clock_60_mhz ) ,. rd_clk ( clock_60_mhz ) ,
187 . we ( write && new_data ) ,. rd (1 ’ b1 ) ,. din ( data_back_wr ) ,. dout ( ram_data ) ) ;
188
189 // Threshold based on set range parameters
190 wire bin ;
191 assign bin = (( data_back_wr > ( ram_data - BA CK _T H RE SH _M I N ) ) &&
192 ( data_back_wr < ( ram_data - BA CK _ TH RE SH _ MA X ) ) &&
193 ( data_back_wr > BACK _DEPTH_ MIN ) &&
194 ( ram_data > BAC K_DEPTH _MIN ) &&
195 ( data_back_wr < BACK _DEPTH_ MAX ) &&
196 ( ram_data < BAC K_DEPTH _MAX ) ) ;
197
198
199 // DEBUG signals
200 parameter FLAT_THRES = 1500; // 16 ’ h05DC
201
202 always@ ( posedge clock_60_mhz ) begin
203 // Global reset
204 if ( reset ) begin
205 WR_bin <= 0;
206 index <= 0;
207 index_bin <= 0;
208 data_bin_wr <= 0;
209 ready <= 0;
210 prev_data <= 0;
211 data_state <= LO ;
212 end
213
214 // if we have new data to proces ...
215 else if ( new_data ) begin
216 // we have both components of a distance value
217 if ( data_state == HI ) begin
218 // If we are in UPDATE
219 // ... create binary image from thresholds
220 if ( write == 0) begin
221 data_bin_wr <= bin ;
222 index_bin <= index ;
223 end
224 // Else we are in WRITING
225 // ... and we prohibit wrting to binary BRAM
226 else
227 data_bin_wr <= 0;
228
229 // Done throughout all cases :
230 WR_bin <= 1; // write to background BRAM
231
232 // increment index
233 if ( index >= NUM_PIXELS )
234 ready <= 1;
235 else if (~( RXF && ~ prev_RXF ) ) begin
236 ready <= 0;
237 index <= index + 1;
238 end
239
240 data_state <= LO ; // Toggle state of input
241 end
242
243 // We only have the first component of a distance value
244 else begin
245 WR_bin <= 0;
246 data_state <= HI ;
247 end
248
249 // Store the previous value
250 prev_data <= data_in ;
251 prev_RXF <= RXF ;
252 end
253
254 end
255
256 endmodule
257
258 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
259 // Module Name : detect_start
260 // Author : Alan Cheng
261 // Description : Detect starting sequence for new frame { DD , CC , BB , AA }
262 //
263 // Dependencies :
264 //
265 // Additional Comments :
266 //
267 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
268 module detect_start (
269 input reset , // global reset
270 input clock_60_mhz , // 60 MHz FT_2232H clock
271 input RD , // RD to FT_2232H : 0 to read from buffer
272 input [ 7 : 0 ] data_in , // data read from FT2232H
273 output ready , // flags when starting sequence is detected
274 output [ 7 : 0 ] prev_out

33
275 );
276
277 // storage containers for past states
278 reg [ 7 : 0 ] data_prev1 , data_prev2 , data_prev3 ;
279
280 // sequence to be detected
281 parameter [ 7 : 0 ] START0 = 8 ’ hAA ;
282 parameter [ 7 : 0 ] START1 = 8 ’ hBB ;
283 parameter [ 7 : 0 ] START2 = 8 ’ hCC ;
284 parameter [ 7 : 0 ] START3 = 8 ’ hDD ;
285
286 always@ ( posedge clock_60_mhz ) begin
287 if ( reset ) begin
288 data_prev1 <= 0;
289 data_prev2 <= 0;
290 data_prev3 <= 0;
291 end
292 else if ( RD ) begin
293 data_prev1 <= data_in ;
294 data_prev2 <= data_prev1 ;
295 data_prev3 <= data_prev2 ;
296 end
297 end
298
299 assign ready = (( data_in == START0 ) && ( data_prev1 == START1 )
300 && ( data_prev2 == START2 ) && ( data_prev3 == START3 ) ) ;
301
302 assign prev_out = data_in ;
303 endmodule

34
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : bram
3 // Author : Alan Cheng
4 // Description : Single port BRAM
5 //
6 // Dependencies :
7 //
8 // Additional Comments :
9 //
10 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
11 module bram #( parameter SIZE =512*424 , LOGSIZE =18 , BITS =1)
12 ( input wire [ LOGSIZE -1 : 0 ] addr ,
13 input wire clk ,
14 input wire [ BITS -1 : 0 ] din ,
15 output reg [ BITS -1 : 0 ] dout ,
16 input wire we ) ;
17 // let the tools infer the right number of BRAMs
18 (* ram_style = " block " *)
19 reg [ BITS -1 : 0 ] mem [ SIZE -1 : 0 ] ;
20 always @ ( posedge clk ) begin
21 if ( we ) mem [ addr ] <= din ;
22 dout <= mem [ addr ] ;
23 end
24 endmodule
25
26 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
27 // Module Name : frame_bram
28 // Author : Alan Cheng
29 // Description : Dual port BRAM
30 //
31 // Dependencies :
32 //
33 // Additional Comments :
34 //
35 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
36 module frame_bram #( parameter SIZE =512*424 , LOGSIZE =18 , BITS =1)
37 ( input wire [ LOGSIZE -1 : 0 ] wr_addr ,
38 input wire [ LOGSIZE -1 : 0 ] rd_addr ,
39 input wire wr_clk ,
40 input wire rd_clk ,
41 input wire [ BITS -1 : 0 ] din ,
42 output reg [ BITS -1 : 0 ] dout ,
43 input wire we ,
44 input wire rd ) ;
45 // let the tools infer the right number of BRAMs
46 (* ram_style = " block " *)
47 reg [ BITS -1 : 0 ] mem [ SIZE -1 : 0 ] ;
48 always @ ( posedge wr_clk )
49 if ( we ) mem [ wr_addr ] <= din ;
50 always @ ( posedge rd_clk )
51 if ( rd ) dout <= mem [ rd_addr ] ;
52 endmodule

1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : f r a m e _ i n d e x _ c o n t r o l
3 // Author : Hope Harrison , with conversion to 512 x3 window by Alan Cheng
4 // Description : Processes an image in order to get the coordinates of a fingertip in the image .
5 //
6 // Dependencies : f r a m e _ i n d e x _ c o n t r o l
7 //
8 // Additional Comments :
9 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
10
11 module image_process #( parameter HSIZE = 512 , parameter VSIZE =424)
12 ( input clock ,
13 input reset ,
14 input frame_available , // Indicates that new data is available in frame , should be a pulse
15 input pixel_in , // the current pixel of the binarized image
16 input [ 4 : 0 ] blob_min_size , blob_max_size , // parameters for which sized blobs will be detected as valid
17 input [ 9 : 0 ] edge_to_ignore_up_h , // sets the image area to look for blobs in
18 input [ 9 : 0 ] edge_to_ignore_up_v ,
19 input [ 9 : 0 ] edge_to_ignore_bot_h ,
20 input [ 9 : 0 ] edge_to_ignore_bot_v ,
21 output [ 17 : 0 ] bin_index , // the current index to look at in the binarized image ( corresponds with
,→ pixel_in )
22 output [ 7 : 0 ] pixel_out , // the current pixel of the processed image
23 output hsync , vsync , // signals to send to VGA
24 output in_blob_box , in_frame_box , // whether the current pixel is on the edge of the box around the blob
25 output [ 9 : 0 ] touch_h , touch_v , // coordinates of user touch
26 output reg touch , touch_ready // whether there is a valid touch
27 );
28
29
30 // Create 25 mhz clock - used for VGA display ( the monitor complains if you use a faster clock )
31 wire clock_25mhz ;
32 c l o c k _ q u a r t e r _ d i v i d e r clockgen (. clk100_mhz ( clock ) , . clock_25mhz ( clock_25mhz ) ) ;
33
34 // These are the possible states in image processing
35 parameter IDLE = 0;
36 parameter GET_IMAGE = 1;
37 parameter ERODE = 2;

35
38 parameter DILATE = 3;
39 parameter BLOB = 4;
40 parameter PREP_DILATE = 5;
41 parameter WRITE_FINAL = 6;
42 reg [ 3 : 0 ] state ;
43
44 // These parameters can be changed to control how the image processing works
45 parameter BLOB_DIST = 3;
46 parameter MAX_B LOB_COU NT = 4; // the maximum number of blobs that we will look at in an attempt to find
,→ a finger touch ( takes 1 cycle per pixel per blob )
47
48 wire [ 9 : 0 ] hcount , vcount ;
49 wire [ 9 : 0 ] hcount_disp , vcount_disp ; // these are the hcount and vcount for display on the monitor , as opposed
,→ to for our processing which is faster
50 wire a t_ di sp l ay _a re a ;
51 wire pixel_on ; // whether the current pixel should be on
52 reg [ 9 : 0 ] start_h , end_h , start_v , end_v ; // the bounds of the box around current blob
53 reg frame [ HSIZE -1 : 0 ] [ 2 : 0 ] ;
54 reg eroded [ HSIZE -1 : 0 ] [ 2 : 0 ] ; // the eroded version of the image
55 reg next_row [ HSIZE -1 : 0 ] ;
56 reg b e g i n _ p r o c e s s i n g ; // is triggered when a new frame comes in
57 reg started_blob ; // whether we have found a blob yet
58 reg [ 5 : 0 ] blob_count ; // the number of blobs perviously looked at ( that didn ’ t meet criteria )
59 reg l a s t _ f r a m e _ a v a i l a b l e ;
60 reg new_frame ;
61 wire [ 9 : 0 ] top_left_h , top_left_v , bottom_right_h , b ottom_r ight_v ;
62
63 // Make an hcount and vcount for displaying and also a faster one for cycling through pixels in image processing
64 vga #(. HSIZE ( HSIZE ) , . VSIZE ( VSIZE ) ) vga_display (. vga_clock ( clock_25mhz ) ,. hcount ( hcount_disp ) ,. vcount ( vcount_disp ) ,.
,→ hsync ( hsync ) ,. vsync ( vsync ) ,. a t_ di sp l ay _a re a ( a t _d is pl a y_ ar ea ) ) ;
65
66 reg wr_erosion ;
67 wire erosion_pixel ;
68 wire erosion_out ;
69 bram #(. SIZE ( HSIZE * VSIZE ) , . LOGSIZE (18) , . BITS (1) )
70 erosion_bram (. addr ( bin_index ) ,. clk ( clock ) ,. we ( wr_erosion ) ,. din ( erosion_pixel ) ,. dout ( erosion_out ) ) ;
71
72 reg wr_dilation ;
73 wire d ilation _pixel ;
74 wire process_out ;
75
76 reg reset_count , hard_reset ;
77
78 reg wr_blob ;
79 reg [ 9 : 0 ] h_process , v_process ;
80 wire process_pixel ;
81 wire [ 17 : 0 ] p r o c e s s _ b r a m _ i n d e x ;
82 wire [ 17 : 0 ] process_index ;
83 wire [ 17 : 0 ] disp_index ;
84
85 wire write_process ;
86 assign write_process = wr_dilation | wr_erosion | wr_blob ;
87
88 assign process_index = ( v_process * HSIZE ) + h_process ;
89 assign p r o c e s s _ b r a m _ i n d e x = bin_index ; // ( wr_blob ) ? process_index : bin_index ;
90 assign process_pixel = ( wr_blob ) ? 0 : dilation _pixel ;
91
92 frame_bram #(. SIZE ( HSIZE * VSIZE ) , . LOGSIZE (18) , . BITS (1 ’ b1 ) )
93 process_bram (. wr_addr ( p r o c e s s _ b r a m _ i n d e x ) ,. rd_addr ( bin_index ) ,. wr_clk ( clock ) ,. rd_clk ( clock ) ,
94 . we ( write_process ) ,. rd (1 ’ b1 ) ,. din ( process_pixel ) ,. dout ( process_out ) ) ;
95
96 wire write_final ;
97 assign write_final = ( state == WRITE_FINAL ) ;
98 frame_bram #(. SIZE ( HSIZE * VSIZE ) , . LOGSIZE (18) , . BITS (1 ’ b1 ) )
99 final_bram (. wr_addr ( bin_index ) ,. rd_addr ( disp_index ) ,. wr_clk ( clock ) ,. rd_clk ( clock_25mhz ) ,
100 . we ( write_final ) ,. rd (1 ’ b1 ) ,. din ( process_out ) ,. dout ( pixel_on ) ) ;
101
102 f r a m e _ i n d e x _ c o n t r o l #(. HSIZE ( HSIZE ) ,. VSIZE ( VSIZE ) ) frame_ division (. reset ( reset || hard_reset ) , . reset_count (
,→ reset_count ) , . clk ( clock ) , . vcount ( vcount ) , . hcount ( hcount ) ) ;
103
104 assign bin_index = ( vcount * HSIZE ) + hcount ;
105 assign disp_index = ( vcount_disp * HSIZE ) + hcount_disp ;
106
107 // Erosion kernel
108 assign erosion_pixel = frame [ hcount -1 ] [ 0 ] &
109 frame [ hcount -1 ] [ 1 ] &
110 frame [ hcount -1 ] [ 2 ] &
111 frame [ hcount ] [ 0 ] &
112 frame [ hcount ] [ 2 ] &
113 frame [ hcount +1 ] [ 0 ] &
114 frame [ hcount +1 ] [ 1 ] &
115 frame [ hcount +1 ] [ 2 ] ;
116
117 // dilation kernel
118 assign dilat ion_pixe l = eroded [ hcount -1 ] [ 0 ] |
119 eroded [ hcount -1 ] [ 1 ] |
120 eroded [ hcount -1 ] [ 2 ] |
121 eroded [ hcount ] [ 0 ] |
122 eroded [ hcount ] [ 2 ] |
123 eroded [ hcount +1 ] [ 0 ] |
124 eroded [ hcount +1 ] [ 1 ] |
125 eroded [ hcount +1 ] [ 2 ] ;

36
126
127
128 wire look_for_blob ;
129 assign look_for_blob = ( hcount > edge_to_ignore_up_h
130 & hcount < HSIZE - e d g e _ t o _ i g n o r e _ b o t _ h
131 & vcount > edge_to_ignore_up_v
132 & vcount < VSIZE - e d g e _ t o _ i g n o r e _ b o t _ v ) ;
133
134 integer index ;
135 assign top_left_h = e d g e _ t o _ i g n o r e _ u p _ h ;
136 assign top_left_v = e d g e _ t o _ i g n o r e _ u p _ v ;
137 assign bottom _right_ h = ( HSIZE - e d g e _ t o _ i g n o r e _ b o t _ h ) ;
138 assign bottom _right_ v = ( VSIZE - e d g e _ t o _ i g n o r e _ b o t _ v ) ;
139
140
141 always @ ( posedge clock ) begin
142
143 l a s t _ f r a m e _ a v a i l a b l e <= f ra m e_ av ai l ab le ;
144 new_frame <= (~ l a s t _ f r a m e _ a v a i l a b l e & f ra m e_ av ai l ab le ) ;
145
146 case ( state )
147 IDLE : begin
148 touch_ready <= 0;
149 if ( new_frame ) begin
150 state <= GET_IMAGE ;
151 started_blob <= 0;
152 start_h <= 0;
153 end_h <= 0;
154 start_v <= 0;
155 end_v <= 0;
156 blob_count <= 0;
157 touch <= 0;
158
159 reset_count <= 1;
160 wr_erosion <= 0;
161 hard_reset <= 1;
162 end
163 else begin
164 reset_count <= 0;
165 wr_erosion <= 0;
166 hard_reset <= 0;
167 end
168 end
169 GET_IMAGE : begin
170 hard_reset <= 0;
171 if (( hcount == HSIZE -2) && ( vcount == 2) ) begin
172 state <= ERODE ;
173 reset_count <= 1;
174 wr_erosion <= 1;
175 end
176 else begin
177 reset_count <= 0;
178 wr_erosion <= 0;
179 end
180 frame [ hcount ] [ vcount ] <= pixel_in ;
181 end
182 ERODE : begin
183 reset_count <= 0;
184
185 if (( hcount == HSIZE -2) && ( vcount == VSIZE -3) ) begin
186 state <= PREP_DILATE ;
187 hard_reset <= 1;
188 wr_erosion <= 0;
189 end
190 else if ( hcount == HSIZE -2) begin
191 for ( index =0; index < HSIZE -1; index = index +1) begin
192 frame [ index ] [ 0 ] <= frame [ index ] [ 1 ] ;
193 frame [ index ] [ 1 ] <= frame [ index ] [ 2 ] ;
194 frame [ index ] [ 2 ] <= next_row [ index ] ;
195 end
196 end
197 next_row [ hcount ] <= pixel_in ;
198 end
199 PREP_DILATE : begin
200 hard_reset <= 0;
201 if (( hcount == HSIZE -2) && ( vcount == 2) ) begin
202 state <= DILATE ;
203 reset_count <= 1;
204 wr_dilation <= 1;
205 end
206 eroded [ hcount ] [ vcount ] <= erosion_out ;
207 end
208 DILATE : begin
209 reset_count <= 0;
210
211 if (( hcount == HSIZE -2) && ( vcount == VSIZE -3) ) begin
212 state <= WRITE_FINAL ;
213 reset_count <= 1;
214 wr_dilation <= 0;
215 end
216 else if ( hcount == HSIZE -2) begin
217 for ( index =0; index < HSIZE -1; index = index +1) begin

37
218 eroded [ index ] [ 0 ] <= eroded [ index ] [ 1 ] ;
219 eroded [ index ] [ 1 ] <= eroded [ index ] [ 2 ] ;
220 eroded [ index ] [ 2 ] <= next_row [ index ] ;
221 end
222 end
223 next_row [ hcount ] <= erosion_out ;
224
225 end
226 WRITE_FINAL : begin
227 reset_count <= 0;
228 if (( hcount == HSIZE -2) && ( vcount == VSIZE -3) ) state <= BLOB ;
229 end
230 BLOB : begin
231 reset_count <= 0;
232
233 if ( hcount >= HSIZE -2 & vcount >= VSIZE -3) begin
234 started_blob <= 0;
235 if ( end_h - start_h >= blob_min_size & end_h - start_h <= blob_max_size
236 & end_v - start_v >= blob_min_size & end_v - start_v <= blob_max_size ) begin
237 // blob is valid
238 state <= IDLE ;
239 touch <= 1;
240 touch_ready <= 1;
241 wr_blob <= 0;
242 end
243 else begin
244 // blob is not valid or no blob found
245 touch <= 0;
246 if ( blob_count < MAX_BL OB_COUN T ) blob_count <= blob_count + 1;
247 else begin
248 touch <= 0;
249 touch_ready <= 1;
250 wr_blob <= 0;
251 state <= IDLE ;
252 started_blob <= 0;
253 end
254 end
255 end
256 else if ( look_for_blob & ~ started_blob & process_out ) begin
257 start_h <= hcount +1;
258 end_h <= hcount +1;
259 start_v <= vcount ;
260 end_v <= vcount ;
261 started_blob <= 1;
262 h_process <= hcount ;
263 v_process <= vcount ;
264 wr_blob <= 1;
265 end
266 else if ( look_for_blob & started_blob & process_out & hcount +1 >= start_h - BLOB_DIST &
267 hcount +1 <= end_h + BLOB_DIST &
268 vcount >= start_v - BLOB_DIST &
269 vcount <= end_v + BLOB_DIST ) begin
270 // pixel is part of blob
271 if ( hcount +1 < start_h ) start_h <= hcount +1;
272 if ( hcount +1 > end_h ) end_h <= hcount +1;
273 if ( vcount < start_v ) start_v <= vcount ;
274 if ( vcount > end_v ) end_v <= vcount ;
275 h_process <= hcount ;
276 v_process <= vcount ;
277 wr_blob <= 1;
278 end
279 else begin
280 wr_blob <= 0;
281 end
282 end
283 endcase
284 end
285
286 assign in_blob_box = ( hcount_disp > start_h ) & ( vcount_disp > start_v ) & ( hcount_disp < end_h ) & ( vcount_disp < end_v )
,→ ;
287
288 assign in_frame_box = ( hcount_disp == top_left_h & vcount_disp >= top_left_v & vcount_disp <= bott om_right _v ) |
289 ( hcount_disp == bott om_right _h & vcount_disp >= top_left_v & vcount_disp <=
,→ botto m_right_ v ) |
290 ( vcount_disp == top_left_v & hcount_disp >= top_left_h & hcount_disp <= b ottom_ri ght_h )
,→ |
291 ( vcount_disp == bott om_right _v & hcount_disp >= top_left_h & hcount_disp <=
,→ botto m_right_ h ) ;
292
293 assign pixel_out = a t _d is p la y_ ar e a ? ( pixel_on ? 255 : 0) : 0;
294 assign touch_h = start_h + ( end_h - start_h ) /2;
295 assign touch_v = start_v + ( end_v - start_v ) /2;
296
297 endmodule
298
299 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
300 // Module Name : f r a m e _ i n d e x _ c o n t r o l
301 // Author : Alan Cheng
302 // Description : Controls horizontal and vertical index increments for image_process
303 //
304 // Dependencies :
305 //

38
306 // Additional Comments : reset = global reset or hard reset ; sets it to (1 ,0)
307 // reset_count = soft_rest ; sets it to (1 ,1)
308 //
309 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
310 module f r a m e _ i n d e x _ c o n t r o l #( parameter HSIZE =512 , VSIZE =424)
311 ( input reset , // hard reset
312 input reset_count , // soft reset
313 input clk , // 100 MHz Nexys4 clock
314 input pause_v , // halt counting
315 output reg [ 9 : 0 ] vcount , // vertical index
316 output reg [ 9 : 0 ] hcount ) ; // horizontal index
317
318 always@ ( posedge clk ) begin
319 if ( reset ) begin
320 hcount <= 1;
321 vcount <= 0;
322 end
323 else if ( reset_count ) begin
324 hcount <= 1;
325 vcount <= 1;
326 end
327 else begin
328 // If we have reached the very last pixel to process
329 if (( hcount >= HSIZE -2) && ( vcount >= VSIZE -3) ) begin
330 hcount <= 1;
331 vcount <= 1;
332 end
333 // If we have reached the end of a line
334 else if ( hcount >= HSIZE -2) begin
335 hcount <= 1;
336 vcount <= vcount + 1;
337 end
338 else hcount <= hcount + 1;
339 end
340 end
341 endmodule

39
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : gesture _detect
3 // Author : Hope Harrison , with timeout buffer written by Alan Cheng
4 // Description : Calculates whether a user has pressed or released based on past inputs .
5 //
6 // Dependencies :
7 //
8 // Additional Comments : is_scroll is unused
9 //
10 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
11
12 module gestu re_detec t (
13 input clock ,
14 input [ 9 : 0 ] touch_h , touch_v , // coordinates of user ’ s fingertip
15 input touch , // whether the fingertip is touching
16 input touch_ready , // a pulse indicating calculations finished and the touch_h , touch_v , and touch
,→ readings are valid
17 output reg is_press , is_release , is_move , is_scroll , // mouse actions
18 output reg s e n d _ m o u s e _ e n a b l e // whether to send any command to the Teensy in order to control the mouse
19 );
20
21 reg touch_active ;
22
23 parameter TIMEOUT_TOUCH = 10;
24 reg [ 5 : 0 ] miss_count ;
25
26 always @ ( posedge clock ) begin
27 if ( touch_ready ) begin
28 if ( touch & ~ touch_active ) begin
29 is_press <= 1;
30 is_release <= 0;
31 is_move <= 1;
32 is_scroll <= 0;
33 s e n d _ m o u s e _ e n a b l e <= 1;
34
35 miss_count <= TIMEOUT_TOUCH ;
36 touch_active <= 1;
37 end
38 else if ( touch ) begin
39 is_press <= 0;
40 is_release <= 0;
41 is_move <= 1;
42 is_scroll <= 0;
43 s e n d _ m o u s e _ e n a b l e <= 1;
44
45 miss_count <= TIMEOUT_TOUCH ;
46 touch_active <= 1;
47 end
48 else if (~ touch & touch_active ) begin
49 if ( miss_count == 0) begin
50 is_press <= 0;
51 is_release <= 1;
52 is_move <= 0;
53 is_scroll <= 0;
54 s e n d _ m o u s e _ e n a b l e <= 1;
55
56 touch_active <= 0;
57 end
58 else begin
59 is_press <= 0;
60 is_release <= 0;
61 is_move <= 1;
62 is_scroll <= 0;
63 s e n d _ m o u s e _ e n a b l e <= 1;
64
65 miss_count <= miss_count - 1;
66 touch_active <= 1;
67 end
68 end
69 else if (~ touch ) begin
70 is_press <= 0;
71 is_release <= 0;
72 is_move <= 0;
73 is_scroll <= 0;
74 s e n d _ m o u s e _ e n a b l e <= 1;
75
76 touch_active <= 0;
77 end
78 end
79
80 else begin
81 s e n d _ m o u s e _ e n a b l e <= 0;
82 end
83 end
84
85 endmodule

40
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : send_mouse
3 // Author : Hope Harrison
4 // Description : Assigns the byte to be sent over serial in the correct order for the mouse report .
5 //
6 // Dependencies :
7 //
8 // Additional Comments :
9 //
10 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
11
12 module send_mouse (
13 input [ 15 : 0 ] x , y , // coordinates of mouse
14 input [ 7 : 0 ] is_press , is_release , is_move , // type of mouse action
15 input [ 15 : 0 ] top_left_x , top_left_y , bottom_right_x , bottom_right_y , // bounds of screen
16 input send_enable , // whether or not to send to the mouse ( should be true only after having processed new data )
17 input reset , clock ,
18 output [ 7 : 0 ] pmod
19 );
20
21 reg tx_enable ;
22 reg [ 7 : 0 ] data_to_send ;
23 wire xmit_done ;
24 wire serial_stream ;
25 wire xmit_clk ;
26
27 reg state ;
28 parameter WAITING = 0;
29 parameter SENDING = 1;
30 parameter SPEED = 1;
31 reg [ 3 : 0 ] p a r a m e t e r _ t o _ s e n d ;
32 parameter X1 = 0;
33 parameter X2 = 1;
34 parameter Y1 = 2;
35 parameter Y2 = 3;
36 parameter TOP_LEFT_X1 = 4;
37 parameter TOP_LEFT_X2 = 5;
38 parameter TOP_LEFT_Y1 = 6;
39 parameter TOP_LEFT_Y2 = 7;
40 parameter B O TT OM _R I GH T_ X1 = 8;
41 parameter B O TT OM _R I GH T_ X2 = 9;
42 parameter B O TT OM _R I GH T_ Y1 = 10;
43 parameter B O TT OM _R I GH T_ Y2 = 11;
44 parameter IS_PRESS = 12;
45 parameter IS_RELEASE = 13;
46 parameter IS_MOVE = 14;
47 parameter NONE = 15;
48
49 reg d eb ug _ tx _e na b le ;
50 reg d eb ug _ xm it _d o ne ;
51 reg [ 2 : 0 ] count ;
52 reg [ 3 : 0 ] send_count ;
53 reg debug_x mit_clk ;
54
55 serial_send sender (. clk ( clock ) , . data ( data_to_send ) , . start_send ( tx_enable ) , . xmit_data ( serial_stream ) , . xmit_clk (
,→ xmit_clk ) , . xmit_done ( xmit_done ) ) ;
56
57 assign pmod [ 7 : 1 ] = 7 ’ b0 ;
58 assign pmod [ 0 ] = serial_stream ;
59
60 always @ ( posedge clock ) begin
61 case ( state )
62 WAITING : begin
63 if ( send_enable ) begin
64 state <= SENDING ;
65 p a r a m e t e r _ t o _ s e n d <= X1 ;
66 data_to_send <= x [ 7 : 0 ] ;
67 tx_enable <= 1;
68 end
69 else begin
70 tx_enable <= 0;
71 end
72 end
73 SENDING : begin
74 if ( xmit_done ) begin
75 p a r a m e t e r _ t o _ s e n d <= p a r a m e t e r _ t o _ s e n d + 1;
76 case ( p a r a m e t e r _ t o _ s e n d + 1)
77 X1 : begin
78 data_to_send <= x [ 7 : 0 ] ;
79 tx_enable <= 1;
80 end
81 X2 : begin
82 data_to_send <= x [ 15 : 8 ] ;
83 tx_enable <= 1;
84 end
85 Y1 : begin
86 data_to_send <= y [ 7 : 0 ] ;
87 tx_enable <= 1;
88 end
89 Y2 : begin
90 data_to_send <= y [ 15 : 8 ] ;
91 tx_enable <= 1;

41
 92 end
93 TOP_LEFT_X1 : begin
94 data_to_send <= top_left_x [7 : 0 ] ;
95 tx_enable <= 1;
96 end
97 TOP_LEFT_X2 : begin
98 data_to_send <= top_left_x [ 15 : 8 ] ;
99 tx_enable <= 1;
100 end
101 TOP_LEFT_Y1 : begin
102 data_to_send <= top_left_y [7 : 0 ] ;
103 tx_enable <= 1;
104 end
105 TOP_LEFT_Y2 : begin
106 data_to_send <= top_left_y [ 15 : 8 ] ;
107 tx_enable <= 1;
108 end
109 B OT TO M_ R IG HT _X 1 : begin
110 data_to_send <= bott om_right _x [7 : 0 ] ;
111 tx_enable <= 1;
112 end
113 B OT TO M_ R IG HT _X 2 : begin
114 data_to_send <= bott om_right _x [ 15 : 8 ] ;
115 tx_enable <= 1;
116 end
117 B OT TO M_ R IG HT _Y 1 : begin
118 data_to_send <= bott om_right _y [7 : 0 ] ;
119 tx_enable <= 1;
120 end
121 B OT TO M_ R IG HT _Y 2 : begin
122 data_to_send <= bott om_right _y [ 15 : 8 ] ;
123 tx_enable <= 1;
124 end
125 IS_PRESS : begin
126 data_to_send <= is_press ;
127 tx_enable <= 1;
128 end
129 IS_RELEASE : begin
130 data_to_send <= is_release ;
131 tx_enable <= 1;
132 end
133 IS_MOVE : begin
134 data_to_send <= is_move ;
135 tx_enable <= 1;
136 end
137 NONE : begin
138 tx_enable <= 0;
139 state <= WAITING ;
140 end
141 endcase
142 end
143 else begin
144 tx_enable <= 0;
145 end
146 end
147 endcase
148 end
149
150 endmodule

42
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Module Name : serial_send
3 // Author : Hope Harrison
4 // Description : Sends the data input over serial on xmit_data .
5 //
6 // Dependencies :
7 //
8 // Additional Comments :
9 //
10 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
11
12 module serial_send (
13 input clk ,
14 input [ 7 : 0 ] data , // 8 bit ascii data
15 input start_send , // one clock pulse width , enter pressed , data available
16 output reg xmit_data , // serial data sent to RS232 output driver
17 output reg xmit_clk , // baud rate ; sent to logic analyzer for debugging
18 output reg xmit_done // indicates that transmission of 1 byte is finished
19 );
20
21 // this section sets up the clk ;
22 parameter DIVISOR = 868; // create 115 ,200 baud rate clock , not exact , but should work .
23 parameter MARK = 1 ’ b1 ;
24 parameter STOP_BIT = 1 ’ b1 ;
25 parameter START_BIT =1 ’ b0 ;
26
27 parameter WAIT = 0;
28 parameter SEND = 1;
29
30 reg [ 31 : 0 ] count ;
31 reg [ 9 : 0 ] data_buffer ;
32 reg state ;
33 reg [ 3 : 0 ] sent_bits ;
34
35 always @ ( posedge clk )
36 begin
37 count <= xmit_clk ? 0 : count +1;
38 xmit_clk <= count == DIVISOR -1;
39
40 case ( state )
41 WAIT : begin
42 xmit_done <= 0;
43 if ( start_send ) begin
44 data_buffer <= { STOP_BIT , data , START_BIT };
45 sent_bits <= 0;
46 state <= SEND ;
47 end
48 end
49 SEND : begin
50 if ( sent_bits >= 10) begin
51 state <= WAIT ;
52 xmit_done <= 1;
53 end
54 else if ( xmit_clk ) begin
55 sent_bits <= sent_bits + 1;
56 data_buffer <= { STOP_BIT , data_buffer [9 : 1 ] };
57 xmit_data <= data_buffer [0 ] ;
58 xmit_done <= 0;
59 end
60
61 end
62 endcase
63
64 end
65
66 endmodule

1 ‘timescale 1 ns / 1 ps
2 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
3 // Company : g . p . hom
4 // Engineer :
5 //
6 // Create Date : 18:18:59 04/21/2013
7 // Module Name : display_8hex
8
9 // Description : Display 8 hex numbers on 7 segment display
10 //
11 // Revision :
12 // Revision 0.01 - File Created
13 // Additional Comments :
14 //
15 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
16 module display_8hex (
17 input clk , // system clock
18 input [ 31 : 0 ] data , // 8 hex numbers , msb first
19 output reg [ 6 : 0 ] seg , // seven segment display output
20 output reg [ 7 : 0 ] strobe // digit strobe
21 );
22
23 localparam bits = 13;
24

43
25 reg [ bits : 0 ] counter = 0; // clear on power up
26
27 wire [ 6 : 0 ] segments [ 15 : 0 ] ; // 16 7 bit memorys
28 assign segments [ 0 ] = 7 ’ b100_0000 ;
29 assign segments [ 1 ] = 7 ’ b111_1001 ;
30 assign segments [ 2 ] = 7 ’ b010_0100 ;
31 assign segments [ 3 ] = 7 ’ b011_0000 ;
32 assign segments [ 4 ] = 7 ’ b001_1001 ;
33 assign segments [ 5 ] = 7 ’ b001_0010 ;
34 assign segments [ 6 ] = 7 ’ b000_0010 ;
35 assign segments [ 7 ] = 7 ’ b111_1000 ;
36 assign segments [ 8 ] = 7 ’ b000_0000 ;
37 assign segments [ 9 ] = 7 ’ b001_1000 ;
38 assign segments [ 10 ] = 7 ’ b000_1000 ;
39 assign segments [ 11 ] = 7 ’ b000_0011 ;
40 assign segments [ 12 ] = 7 ’ b010_0111 ;
41 assign segments [ 13 ] = 7 ’ b010_0001 ;
42 assign segments [ 14 ] = 7 ’ b000_0110 ;
43 assign segments [ 15 ] = 7 ’ b000_1110 ;
44
45 always @ ( posedge clk ) begin
46 counter <= counter + 1;
47 case ( counter [ bits : bits -2 ] )
48 3 ’ b000 : begin
49 seg <= segments [ data [ 31 : 28 ] ] ;
50 strobe <= 8 ’ b0111_1111 ;
51 end
52
53 3 ’ b001 : begin
54 seg <= segments [ data [ 27 : 24 ] ] ;
55 strobe <= 8 ’ b1011_1111 ;
56 end
57
58 3 ’ b010 : begin
59 seg <= segments [ data [ 23 : 20 ] ] ;
60 strobe <= 8 ’ b1101_1111 ;
61 end
62 3 ’ b011 : begin
63 seg <= segments [ data [ 19 : 16 ] ] ;
64 strobe <= 8 ’ b1110_1111 ;
65 end
66 3 ’ b100 : begin
67 seg <= segments [ data [ 15 : 12 ] ] ;
68 strobe <= 8 ’ b1111_0111 ;
69 end
70
71 3 ’ b101 : begin
72 seg <= segments [ data [ 11 : 8 ] ] ;
73 strobe <= 8 ’ b1111_1011 ;
74 end
75
76 3 ’ b110 : begin
77 seg <= segments [ data [ 7 : 4 ] ] ;
78 strobe <= 8 ’ b1111_1101 ;
79 end
80 3 ’ b111 : begin
81 seg <= segments [ data [ 3 : 0 ] ] ;
82 strobe <= 8 ’ b1111_1110 ;
83 end
84
85
86
87
88 endcase
89 end
90
91 endmodule

44
 1 // Switch Debounce Module
2 // use your system clock for the clock input
3 // to produce a synchronous , debounced output
4 module debounce #( parameter DELAY =1000000) // .01 sec with a 100 Mhz clock
5 ( input reset , clock , noisy ,
6 output reg clean ) ;
7
8 reg [ 19 : 0 ] count ;
9 reg new ;
10
11 always @ ( posedge clock )
12 if ( reset )
13 begin
14 count <= 0;
15 new <= noisy ;
16 clean <= noisy ;
17 end
18 else if ( noisy != new )
19 begin
20 new <= noisy ;
21 count <= 0;
22 end
23 else if ( count == DELAY )
24 clean <= new ;
25 else
26 count <= count +1;
27
28 endmodule

45
Appendix D

Nexys4 Constraint File (xdc)

46
 1 ## This file is a general . xdc for the Nexys4 DDR Rev . C
2 ## To use it in a project :
3 ## - uncomment the lines corresponding to used pins
4 ## - rename the used ports ( in each line , after get_ports ) according to the top level signal names in the project
5
6 ## Clock signal
7 set_property - dict { PACKAGE_PIN E3 IOSTANDARD LVCMOS33 } [ get_ports { CLK100MHZ }] ; # I O _ L 1 2 P _ T 1 _ M R C C _ 3 5 Sch = clk100mhz
8 create_clock - add - name sys_clk_pin - period 10.00 - waveform {0 5} [ get_ports { CLK100MHZ }] ;
9
10
11 ## Switches
12
13 set_property - dict { PACKAGE_PIN J15 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 0 ] } ] ; # I O _ L 2 4 N _ T 3 _ R S 0 _ 1 5 Sch = sw [0 ]
14 set_property - dict { PACKAGE_PIN L16 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 1 ] } ] ; # I O _ L 3 N _ T 0 _ D Q S _ E M C C L K _ 1 4 Sch = sw [1 ]
15 set_property - dict { PACKAGE_PIN M13 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 2 ] } ] ; # I O _ L 6 N _ T 0 _ D 0 8 _ V R E F _ 1 4 Sch = sw [2 ]
16 set_property - dict { PACKAGE_PIN R15 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 3 ] } ] ; # I O _ L 1 3 N _ T 2 _ M R C C _ 1 4 Sch = sw [3 ]
17 set_property - dict { PACKAGE_PIN R17 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 4 ] } ] ; # I O _ L 1 2 N _ T 1 _ M R C C _ 1 4 Sch = sw [4 ]
18 set_property - dict { PACKAGE_PIN T18 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 5 ] } ] ; # I O _ L 7 N _ T 1 _ D 1 0 _ 1 4 Sch = sw [5 ]
19 set_property - dict { PACKAGE_PIN U18 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 6 ] } ] ; # I O _ L 1 7 N _ T 2 _ A 1 3 _ D 2 9 _ 1 4 Sch = sw [6 ]
20 set_property - dict { PACKAGE_PIN R13 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 7 ] } ] ; # I O _ L 5 N _ T 0 _ D 0 7 _ 1 4 Sch = sw [7 ]
21 set_property - dict { PACKAGE_PIN T8 IOSTANDARD LVCMOS18 } [ get_ports { SW [ 8 ] } ] ; # IO_L24N_T3_34 Sch = sw [8 ]
22 set_property - dict { PACKAGE_PIN U8 IOSTANDARD LVCMOS18 } [ get_ports { SW [ 9 ] } ] ; # IO_25_34 Sch = sw [9 ]
23 set_property - dict { PACKAGE_PIN R16 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 10 ] } ] ; # I O _ L 1 5 P _ T 2 _ D Q S _ R D W R _ B _ 1 4 Sch = sw [ 10 ]
24 set_property - dict { PACKAGE_PIN T13 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 11 ] } ] ; # I O _ L 2 3 P _ T 3 _ A 0 3 _ D 1 9 _ 1 4 Sch = sw [ 11 ]
25 set_property - dict { PACKAGE_PIN H6 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 12 ] } ] ; # IO_L24P_T3_35 Sch = sw [ 12 ]
26 set_property - dict { PACKAGE_PIN U12 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 13 ] } ] ; # I O _ L 2 0 P _ T 3 _ A 0 8 _ D 2 4 _ 1 4 Sch = sw [ 13 ]
27 set_property - dict { PACKAGE_PIN U11 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 14 ] } ] ; # I O _ L 1 9 N _ T 3 _ A 0 9 _ D 2 5 _ V R E F _ 1 4 Sch = sw [ 14
,→ ]
28 set_property - dict { PACKAGE_PIN V10 IOSTANDARD LVCMOS33 } [ get_ports { SW [ 15 ] } ] ; # I O _ L 2 1 P _ T 3 _ D Q S _ 1 4 Sch = sw [ 15 ]
29
30
31 ## LEDs
32
33 set_property - dict { PACKAGE_PIN H17 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 0 ] } ] ; # I O _ L 1 8 P _ T 2 _ A 2 4 _ 1 5 Sch = led [0 ]
34 set_property - dict { PACKAGE_PIN K15 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 1 ] } ] ; # I O _ L 2 4 P _ T 3 _ R S 1 _ 1 5 Sch = led [1 ]
35 set_property - dict { PACKAGE_PIN J13 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 2 ] } ] ; # I O _ L 1 7 N _ T 2 _ A 2 5 _ 1 5 Sch = led [2 ]
36 set_property - dict { PACKAGE_PIN N14 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 3 ] } ] ; # I O _ L 8 P _ T 1 _ D 1 1 _ 1 4 Sch = led [3 ]
37 set_property - dict { PACKAGE_PIN R18 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 4 ] } ] ; # I O _ L 7 P _ T 1 _ D 0 9 _ 1 4 Sch = led [4 ]
38 set_property - dict { PACKAGE_PIN V17 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 5 ] } ] ; # I O _ L 1 8 N _ T 2 _ A 1 1 _ D 2 7 _ 1 4 Sch = led [5 ]
39 set_property - dict { PACKAGE_PIN U17 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 6 ] } ] ; # I O _ L 1 7 P _ T 2 _ A 1 4 _ D 3 0 _ 1 4 Sch = led [6 ]
40 set_property - dict { PACKAGE_PIN U16 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 7 ] } ] ; # I O _ L 1 8 P _ T 2 _ A 1 2 _ D 2 8 _ 1 4 Sch = led [7 ]
41 set_property - dict { PACKAGE_PIN V16 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 8 ] } ] ; # I O _ L 1 6 N _ T 2 _ A 1 5 _ D 3 1 _ 1 4 Sch = led [8 ]
42 set_property - dict { PACKAGE_PIN T15 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 9 ] } ] ; # I O _ L 1 4 N _ T 2 _ S R C C _ 1 4 Sch = led [9 ]
43 set_property - dict { PACKAGE_PIN U14 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 10 ] } ] ; # I O _ L 2 2 P _ T 3 _ A 0 5 _ D 2 1 _ 1 4 Sch = led [ 10 ]
44 set_property - dict { PACKAGE_PIN T16 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 11 ] } ] ; # I O _ L 1 5 N _ T 2 _ D Q S _ D O U T _ C S O _ B _ 1 4 Sch =
,→ led [ 11 ]
45 set_property - dict { PACKAGE_PIN V15 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 12 ] }]; # I O _ L 1 6 P _ T 2 _ C S I _ B _ 1 4 Sch = led [ 12 ]
46 set_property - dict { PACKAGE_PIN V14 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 13 ] }]; # I O _ L 2 2 N _ T 3 _ A 0 4 _ D 2 0 _ 1 4 Sch = led [ 13 ]
47 set_property - dict { PACKAGE_PIN V12 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 14 ] }]; # I O _ L 2 0 N _ T 3 _ A 0 7 _ D 2 3 _ 1 4 Sch = led [ 14 ]
48 set_property - dict { PACKAGE_PIN V11 IOSTANDARD LVCMOS33 } [ get_ports { LED [ 15 ] }]; # I O _ L 2 1 N _ T 3 _ D Q S _ A 0 6 _ D 2 2 _ 1 4 Sch = led [
,→ 15 ]
49
50 set_property - dict { PACKAGE_PIN R12 IOSTANDARD LVCMOS33 } [ get_ports { LED16_B }]; # I O _ L 5P _ T 0 _ D 0 6 _ 1 4 Sch = led16_b
51 set_property - dict { PACKAGE_PIN M16 IOSTANDARD LVCMOS33 } [ get_ports { LED16_G }]; # I O _ L 1 0 P _ T 1 _ D 1 4 _ 1 4 Sch = led16_g
52 set_property - dict { PACKAGE_PIN N15 IOSTANDARD LVCMOS33 } [ get_ports { LED16_R }]; # I O _ L 1 1 P _ T 1 _ S R C C _ 1 4 Sch = led16_r
53 set_property - dict { PACKAGE_PIN G14 IOSTANDARD LVCMOS33 } [ get_ports { LED17_B }]; # I O _ L 1 5 N _ T 2 _ D Q S _ A D V _ B _ 1 5 Sch = led17_b
54 set_property - dict { PACKAGE_PIN R11 IOSTANDARD LVCMOS33 } [ get_ports { LED17_G }]; # IO_0_14 Sch = led17_g
55 set_property - dict { PACKAGE_PIN N16 IOSTANDARD LVCMOS33 } [ get_ports { LED17_R }]; # I O _ L 1 1 N _ T 1 _ S R C C _ 1 4 Sch = led17_r
56
57
58 ##7 segment display
59
60 set_property - dict { PACKAGE_PIN T10 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 0 ] }]; # I O _ L 2 4 N _ T 3 _ A 0 0 _ D 1 6 _ 1 4 Sch = ca
61 set_property - dict { PACKAGE_PIN R10 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 1 ] }]; # IO_25_14 Sch = cb
62 set_property - dict { PACKAGE_PIN K16 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 2 ] }]; # IO_25_15 Sch = cc
63 set_property - dict { PACKAGE_PIN K13 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 3 ] }]; # I O _ L 1 7 P _ T 2 _ A 2 6 _ 1 5 Sch = cd
64 set_property - dict { PACKAGE_PIN P15 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 4 ] }]; # I O _ L 1 3 P _ T 2 _ M R C C _ 1 4 Sch = ce
65 set_property - dict { PACKAGE_PIN T11 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 5 ] }]; # I O _ L 1 9 P _ T 3 _ A 1 0 _ D 2 6 _ 1 4 Sch = cf
66 set_property - dict { PACKAGE_PIN L18 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 6 ] }]; # I O _ L 4 P _ T 0 _ D 0 4 _ 1 4 Sch = cg
67
68 set_property - dict { PACKAGE_PIN H15 IOSTANDARD LVCMOS33 } [ get_ports { SEG [ 7 ] } ] ; # I O _ L 1 9 N _ T 3 _ A 2 1 _ V R E F _ 1 5 Sch = dp
69
70 set_property - dict { PACKAGE_PIN J17 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 0 ] }]; # I O _ L 2 3 P _ T 3 _ F O E _ B _ 1 5 Sch = an [0 ]
71 set_property - dict { PACKAGE_PIN J18 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 1 ] }]; # I O _ L 2 3 N _ T 3 _ F W E _ B _ 1 5 Sch = an [1 ]
72 set_property - dict { PACKAGE_PIN T9 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 2 ] }]; # I O _ L 2 4 P _ T 3 _ A 0 1 _ D 1 7 _ 1 4 Sch = an [2 ]
73 set_property - dict { PACKAGE_PIN J14 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 3 ] }]; # I O _ L 1 9 P _ T 3 _ A 2 2 _ 1 5 Sch = an [3 ]
74 set_property - dict { PACKAGE_PIN P14 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 4 ] }]; # I O _ L 8 N _ T 1 _ D 1 2 _ 1 4 Sch = an [4 ]
75 set_property - dict { PACKAGE_PIN T14 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 5 ] }]; # I O _ L 1 4 P _ T 2 _ S R C C _ 1 4 Sch = an [5 ]
76 set_property - dict { PACKAGE_PIN K2 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 6 ] }]; # IO_L23P_T3_35 Sch = an [6 ]
77 set_property - dict { PACKAGE_PIN U13 IOSTANDARD LVCMOS33 } [ get_ports { AN [ 7 ] }]; # I O _ L 2 3 N _ T 3 _ A 0 2 _ D 1 8 _ 1 4 Sch = an [7 ]
78
79
80 ## Buttons
81
82 # set_property - dict { PACKAGE_PIN C12 IOSTANDARD LVCMOS33 } [ get_ports { CPU_RESETN }] ; # I O _ L 3 P _ T 0 _ D Q S _ A D 1 P _ 1 5 Sch =
,→ cpu_resetn
83
84 set_property - dict { PACKAGE_PIN N17 IOSTANDARD LVCMOS33 } [ get_ports { BTNC }]; # I O _ L 9 P _ T 1 _ D Q S _ 14 Sch = btnc
85 set_property - dict { PACKAGE_PIN M18 IOSTANDARD LVCMOS33 } [ get_ports { BTNU }]; # I O _ L 4 N _ T 0 _ D 0 5 _ 14 Sch = btnu
86 set_property - dict { PACKAGE_PIN P17 IOSTANDARD LVCMOS33 } [ get_ports { BTNL }]; # I O _ L 1 2 P _ T 1 _ M R C C _ 1 4 Sch = btnl
87 set_property - dict { PACKAGE_PIN M17 IOSTANDARD LVCMOS33 } [ get_ports { BTNR }]; # I O _ L 1 0 N _ T 1 _ D 1 5 _ 1 4 Sch = btnr
88 set_property - dict { PACKAGE_PIN P18 IOSTANDARD LVCMOS33 } [ get_ports { BTND }]; # I O _ L 9 N _ T 1 _ D Q S _ D 1 3 _ 1 4 Sch = btnd

47
 89
90
91 ## Pmod Headers
92
93
94 ## Pmod Header JA
95
96 set_property - dict { PACKAGE_PIN C17 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 0 ] }]; # I O _ L 2 0 N _ T 3 _ A 1 9 _ 1 5 Sch = ja [1 ]
97 set_property - dict { PACKAGE_PIN D18 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 1 ] }]; # I O _ L 2 1 N _ T 3 _ D Q S _ A 1 8 _ 1 5 Sch = ja [2 ]
98 set_property - dict { PACKAGE_PIN E18 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 2 ] }]; # I O _ L 2 1 P _ T 3 _ D Q S _ 1 5 Sch = ja [3 ]
99 set_property - dict { PACKAGE_PIN G17 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 3 ] }]; # I O _ L 1 8 N _ T 2 _ A 2 3 _ 1 5 Sch = ja [4 ]
100 set_property - dict { PACKAGE_PIN D17 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 4 ] }]; # I O _ L 1 6 N _ T 2 _ A 2 7 _ 1 5 Sch = ja [7 ]
101 set_property - dict { PACKAGE_PIN E17 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 5 ] }]; # I O _ L 1 6 P _ T 2 _ A 2 8 _ 1 5 Sch = ja [8 ]
102 set_property - dict { PACKAGE_PIN F18 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 6 ] }]; # I O _ L 2 2 N _ T 3 _ A 1 6 _ 1 5 Sch = ja [9 ]
103 set_property - dict { PACKAGE_PIN G18 IOSTANDARD LVCMOS33 } [ get_ports { JA [ 7 ] }]; # I O _ L 2 2 P _ T 3 _ A 1 7 _ 1 5 Sch = ja [ 10 ]
104
105
106 ## Pmod Header JB
107
108 # set_property - dict { PACKAGE_PIN D14 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 0 ] }]; # I O _ L 1 P _ T 0 _ A D 0 P _ 1 5 Sch = jb [1 ]
109 # set_property - dict { PACKAGE_PIN F16 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 1 ] }]; # I O _ L 1 4 N _ T 2 _ S R C C _ 1 5 Sch = jb [2 ]
110 # set_property - dict { PACKAGE_PIN G16 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 2 ] }]; # I O _ L 1 3 N _ T 2 _ M R C C _ 1 5 Sch = jb [3 ]
111 # set_property - dict { PACKAGE_PIN H14 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 3 ] }]; # I O _ L 1 5 P _ T 2 _ D Q S _ 1 5 Sch = jb [4 ]
112 # set_property - dict { PACKAGE_PIN E16 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 4 ] }]; # I O _ L 1 1 N _ T 1 _ S R C C _ 1 5 Sch = jb [7 ]
113 # set_property - dict { PACKAGE_PIN F13 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 5 ] }]; # I O _ L 5 P _ T 0 _ A D 9 P _ 1 5 Sch = jb [8 ]
114 # set_property - dict { PACKAGE_PIN G13 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 6 ] }]; # IO_0_15 Sch = jb [ 9 ]
115 # set_property - dict { PACKAGE_PIN H16 IOSTANDARD LVCMOS33 } [ get_ports { JB [ 7 ] }]; # I O _ L 1 3 P _ T 2 _ M R C C _ 1 5 Sch = jb [ 10 ]
116
117 set_property - dict { PACKAGE_PIN F16 IOSTANDARD LVCMOS33 } [ get_ports { JB_IN [ 1 ] } ] ; # I O _ L 1 4 N _ T 2 _ S R C C _ 1 5 Sch = jb [2 ]
118 set_property - dict { PACKAGE_PIN G16 IOSTANDARD LVCMOS33 } [ get_ports { JB_OUT [ 0 ] } ] ; # I O _ L 1 3 N _ T 2 _ M R C C _ 1 5 Sch = jb [3 ]
119 set_property - dict { PACKAGE_PIN H14 IOSTANDARD LVCMOS33 } [ get_ports { JB_OUT [ 1 ] } ] ; # I O _ L 1 5 P _ T 2 _ D Q S _ 1 5 Sch = jb [4 ]
120 set_property - dict { PACKAGE_PIN E16 IOSTANDARD LVCMOS33 } [ get_ports { JB_OUT [ 2 ] } ] ; # I O _ L 1 1 N _ T 1 _ S R C C _ 1 5 Sch = jb [7 ]
121 set_property - dict { PACKAGE_PIN H16 IOSTANDARD LVCMOS33 } [ get_ports { JB_IN [ 0 ] } ] ; # I O _ L 1 3 P _ T 2 _ M R C C _ 1 5 Sch = jb [ 10 ]
122
123 set_property C L O C K _ D E D I C A T E D _ R O U T E FALSE [ get_nets { JB_IN_IBUF [ 0 ] } ]
124 ## Pmod Header JC
125
126 set_property - dict { PACKAGE_PIN K1 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 0 ] }]; # IO_L23N_T3_35 Sch = jc [1 ]
127 set_property - dict { PACKAGE_PIN F6 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 1 ] }]; # I O _ L 1 9 N _ T 3 _ V R E F _ 3 5 Sch = jc [2 ]
128 set_property - dict { PACKAGE_PIN J2 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 2 ] }]; # IO_L22N_T3_35 Sch = jc [3 ]
129 set_property - dict { PACKAGE_PIN G6 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 3 ] }]; # IO_L19P_T3_35 Sch = jc [4 ]
130 set_property - dict { PACKAGE_PIN E7 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 4 ] }]; # IO_L6P_T0_35 Sch = jc [7 ]
131 set_property - dict { PACKAGE_PIN J3 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 5 ] }]; # IO_L22P_T3_35 Sch = jc [8 ]
132 set_property - dict { PACKAGE_PIN J4 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 6 ] }]; # I O _ L 2 1 P _ T 3 _ D Q S _ 3 5 Sch = jc [9 ]
133 set_property - dict { PACKAGE_PIN E6 IOSTANDARD LVCMOS33 } [ get_ports { JC [ 7 ] }]; # I O _ L 5 P _ T 0 _ A D 1 3 P _ 3 5 Sch = jc [ 10 ]
134
135
136 ## Pmod Header JD
137
138 set_property - dict { PACKAGE_PIN H4 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 0 ] }]; # I O _ L 2 1 N _ T 3 _ D Q S _ 3 5 Sch = jd [1 ]
139 set_property - dict { PACKAGE_PIN H1 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 1 ] }]; # IO_L17P_T2_35 Sch = jd [2 ]
140 set_property - dict { PACKAGE_PIN G1 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 2 ] }]; # IO_L17N_T2_35 Sch = jd [3 ]
141 set_property - dict { PACKAGE_PIN G3 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 3 ] }]; # IO_L20N_T3_35 Sch = jd [4 ]
142 set_property - dict { PACKAGE_PIN H2 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 4 ] }]; # I O _ L 1 5 P _ T 2 _ D Q S _ 3 5 Sch = jd [7 ]
143 set_property - dict { PACKAGE_PIN G4 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 5 ] }]; # IO_L20P_T3_35 Sch = jd [8 ]
144 set_property - dict { PACKAGE_PIN G2 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 6 ] }]; # I O _ L 1 5 N _ T 2 _ D Q S _ 3 5 Sch = jd [9 ]
145 set_property - dict { PACKAGE_PIN F3 IOSTANDARD LVCMOS33 } [ get_ports { JD [ 7 ] }]; # I O _ L 1 3 N _ T 2 _ M R C C _ 3 5 Sch = jd [ 10 ]
146
147 # set_property C L O C K _ D E D I C A T E D _ R O U T E FALSE [ get_nets { JD_IBUF [ 7 ] } ]
148
149 # set_property - dict { PACKAGE_PIN F3 IOSTANDARD LVCMOS33 } [ get_ports { JD_IN [ 0 ] } ] ; # I O _ L 1 3 N _ T 2 _ M R C C _ 3 5 Sch = jd [ 10 ]
150 # set_property C L O C K _ D E D I C A T E D _ R O U T E FALSE [ get_nets { JD_IN_IBUF [ 0 ] } ]
151 ## set_property - dict { PACKAGE_PIN H4 IOSTANDARD LVCMOS33 } [ get_ports { JD_IN [ 0 ] } ] ; # I O _ L 2 1 N _ T 3 _ D Q S _ 3 5 Sch = jd [1 ]
152 # set_property - dict { PACKAGE_PIN H1 IOSTANDARD LVCMOS33 } [ get_ports { JD_IN [ 1 ] } ] ; # IO_L17P_T2_35 Sch = jd [2 ]
153
154 # set_property - dict { PACKAGE_PIN G1 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 0 ] } ] ; # IO_L17N_T2_35 Sch = jd [3 ]
155
156 # set_property - dict { PACKAGE_PIN G3 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 1 ] } ] ; # IO_L20N_T3_35 Sch = jd [4 ]
157 # set_property - dict { PACKAGE_PIN H2 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 2 ] } ] ; # I O _ L 1 5 P _ T 2 _ D Q S _ 3 5 Sch = jd [7 ]
158 ## set_property - dict { PACKAGE_PIN G4 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 3 ] } ] ; # IO_L20P_T3_35 Sch = jd [8 ]
159 ## set_property - dict { PACKAGE_PIN G2 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 4 ] } ] ; # I O _ L 1 5 N _ T 2 _ D Q S _ 3 5 Sch = jd [9 ]
160 ## set_property - dict { PACKAGE_PIN F3 IOSTANDARD LVCMOS33 } [ get_ports { JD_OUT [ 5 ] } ] ; # I O _ L 1 3 N _ T 2 _ M R C C _ 3 5 Sch = jd [ 10 ]
161
162
163 ## Pmod Header JXADC
164
165 # set_property - dict { PACKAGE_PIN A14 IOSTANDARD LVDS } [ get_ports { XA_N [ 1 ] }]; # I O _ L 9 N _ T 1 _ D Q S _ A D 3 N _ 1 5 Sch = xa_n [1 ]
166 # set_property - dict { PACKAGE_PIN A13 IOSTANDARD LVDS } [ get_ports { XA_P [ 1 ] }]; # I O _ L 9 P _ T 1 _ D Q S _ A D 3 P _ 1 5 Sch = xa_p [1 ]
167 # set_property - dict { PACKAGE_PIN A16 IOSTANDARD LVDS } [ get_ports { XA_N [ 2 ] }]; # I O _ L 8 N _ T 1 _ A D 1 0 N _ 1 5 Sch = xa_n [2 ]
168 # set_property - dict { PACKAGE_PIN A15 IOSTANDARD LVDS } [ get_ports { XA_P [ 2 ] }]; # I O _ L 8 P _ T 1 _ A D 1 0 P _ 1 5 Sch = xa_p [2 ]
169 # set_property - dict { PACKAGE_PIN B17 IOSTANDARD LVDS } [ get_ports { XA_N [ 3 ] }]; # I O _ L 7 N _ T 1 _ A D 2 N _ 1 5 Sch = xa_n [3 ]
170 # set_property - dict { PACKAGE_PIN B16 IOSTANDARD LVDS } [ get_ports { XA_P [ 3 ] }]; # I O _ L 7 P _ T 1 _ A D 2 P _ 1 5 Sch = xa_p [3 ]
171 # set_property - dict { PACKAGE_PIN A18 IOSTANDARD LVDS } [ get_ports { XA_N [ 4 ] }]; # I O _ L 1 0 N _ T 1 _ A D 1 1 N _ 1 5 Sch = xa_n [4 ]
172 # set_property - dict { PACKAGE_PIN B18 IOSTANDARD LVDS } [ get_ports { XA_P [ 4 ] }]; # I O _ L 1 0 P _ T 1 _ A D 1 1 P _ 1 5 Sch = xa_p [4 ]
173
174
175 ## VGA Connector
176
177 set_property - dict { PACKAGE_PIN A3 IOSTANDARD LVCMOS33 } [ get_ports { VGA_R [ 0 ] }]; # I O _ L 8 N _ T 1 _ A D 1 4 N _ 3 5 Sch = vga_r [0 ]
178 set_property - dict { PACKAGE_PIN B4 IOSTANDARD LVCMOS33 } [ get_ports { VGA_R [ 1 ] }]; # I O _ L 7 N _ T 1 _ A D 6 N _ 3 5 Sch = vga_r [1 ]
179 set_property - dict { PACKAGE_PIN C5 IOSTANDARD LVCMOS33 } [ get_ports { VGA_R [ 2 ] }]; # I O _ L 1 N _ T 0 _ A D 4 N _ 3 5 Sch = vga_r [2 ]
180 set_property - dict { PACKAGE_PIN A4 IOSTANDARD LVCMOS33 } [ get_ports { VGA_R [ 3 ] }]; # I O _ L 8 P _ T 1 _ A D 1 4 P _ 3 5 Sch = vga_r [3 ]

48
181
182 set_property - dict { PACKAGE_PIN C6 IOSTANDARD LVCMOS33 } [ get_ports { VGA_G [ 0 ] }]; # I O _ L 1 P _ T 0 _ A D 4 P _ 3 5 Sch = vga_g [0 ]
183 set_property - dict { PACKAGE_PIN A5 IOSTANDARD LVCMOS33 } [ get_ports { VGA_G [ 1 ] }]; # I O _ L 3 N _ T 0 _ D Q S _ A D 5 N _ 3 5 Sch = vga_g [1 ]
184 set_property - dict { PACKAGE_PIN B6 IOSTANDARD LVCMOS33 } [ get_ports { VGA_G [ 2 ] }]; # I O _ L 2 N _ T 0 _ A D 1 2 N _ 3 5 Sch = vga_g [2 ]
185 set_property - dict { PACKAGE_PIN A6 IOSTANDARD LVCMOS33 } [ get_ports { VGA_G [ 3 ] }]; # I O _ L 3 P _ T 0 _ D Q S _ A D 5 P _ 3 5 Sch = vga_g [3 ]
186
187 set_property - dict { PACKAGE_PIN B7 IOSTANDARD LVCMOS33 } [ get_ports { VGA_B [ 0 ] }]; # I O _ L 2 P _ T 0 _ A D 1 2 P _ 3 5 Sch = vga_b [0 ]
188 set_property - dict { PACKAGE_PIN C7 IOSTANDARD LVCMOS33 } [ get_ports { VGA_B [ 1 ] }]; # IO_L4N_T0_35 Sch = vga_b [1 ]
189 set_property - dict { PACKAGE_PIN D7 IOSTANDARD LVCMOS33 } [ get_ports { VGA_B [ 2 ] }]; # I O _ L 6 N _ T 0 _ V R E F _ 3 5 Sch = vga_b [2 ]
190 set_property - dict { PACKAGE_PIN D8 IOSTANDARD LVCMOS33 } [ get_ports { VGA_B [ 3 ] }]; # IO_L4P_T0_35 Sch = vga_b [3 ]
191
192 set_property - dict { PACKAGE_PIN B11 IOSTANDARD LVCMOS33 } [ get_ports { VGA_HS }] ; # IO_L4P_T0_15 Sch = vga_hs
193 set_property - dict { PACKAGE_PIN B12 IOSTANDARD LVCMOS33 } [ get_ports { VGA_VS }] ; # I O _ L 3 N _ T 0 _ D Q S _ A D 1 N _ 1 5 Sch = vga_vs
194
195 ## Micro SD Connector
196
197 # set_property - dict { PACKAGE_PIN E2 IOSTANDARD LVCMOS33 } [ get_ports { SD_RESET }] ; # I O _ L 1 4 P _ T 2 _ S R C C _ 3 5 Sch = sd_reset
198 # set_property - dict { PACKAGE_PIN A1 IOSTANDARD LVCMOS33 } [ get_ports { SD_CD } ] ; # I O _ L 9 N _ T 1 _ D Q S _ A D 7 N _ 3 5 Sch = sd_cd
199 # set_property - dict { PACKAGE_PIN B1 IOSTANDARD LVCMOS33 } [ get_ports { SD_SCK }] ; # I O _ L 9 P _ T 1 _ D Q S _ A D 7 P _ 3 5 Sch = sd_sck
200 # set_property - dict { PACKAGE_PIN C1 IOSTANDARD LVCMOS33 } [ get_ports { SD_CMD }] ; # IO_L16N_T2_35 Sch = sd_cmd
201 # set_property - dict { PACKAGE_PIN C2 IOSTANDARD LVCMOS33 } [ get_ports { SD_DAT [ 0 ] } ] ; # IO_L16P_T2_35 Sch = sd_dat [0 ]
202 # set_property - dict { PACKAGE_PIN E1 IOSTANDARD LVCMOS33 } [ get_ports { SD_DAT [ 1 ] } ] ; # IO_L18N_T2_35 Sch = sd_dat [1 ]
203 # set_property - dict { PACKAGE_PIN F1 IOSTANDARD LVCMOS33 } [ get_ports { SD_DAT [ 2 ] } ] ; # IO_L18P_T2_35 Sch = sd_dat [2 ]
204 # set_property - dict { PACKAGE_PIN D2 IOSTANDARD LVCMOS33 } [ get_ports { SD_DAT [ 3 ] } ] ; # I O _ L 1 4 N _ T 2 _ S R C C _ 3 5 Sch = sd_dat [3 ]
205
206
207 ## Accelerometer
208
209 # set_property - dict { PACKAGE_PIN E15 IOSTANDARD LVCMOS33 } [ get_ports { ACL_MISO }] ; # I O _ L 1 1 P _ T 1 _ S R C C _ 1 5 Sch = acl_miso
210 # set_property - dict { PACKAGE_PIN F14 IOSTANDARD LVCMOS33 } [ get_ports { ACL_MOSI }] ; # I O _ L 5 N _ T 0 _ A D 9 N _ 1 5 Sch = acl_mosi
211 # set_property - dict { PACKAGE_PIN F15 IOSTANDARD LVCMOS33 } [ get_ports { ACL_SCLK }] ; # I O _ L 1 4 P _ T 2 _ S R C C _ 1 5 Sch = acl_sclk
212 # set_property - dict { PACKAGE_PIN D15 IOSTANDARD LVCMOS33 } [ get_ports { ACL_CSN }] ; # I O _ L 1 2 P _ T 1 _ M R C C _ 1 5 Sch = acl_csn
213 # set_property - dict { PACKAGE_PIN B13 IOSTANDARD LVCMOS33 } [ get_ports { ACL_INT [ 1 ] } ] ; # I O _ L 2 P _ T 0 _ A D 8 P _ 1 5 Sch = acl_int [1
,→ ]
214 # set_property - dict { PACKAGE_PIN C16 IOSTANDARD LVCMOS33 } [ get_ports { ACL_INT [ 2 ] } ] ; # I O _ L 2 0 P _ T 3 _ A 2 0 _ 1 5 Sch = acl_int [2
,→ ]
215
216
217 ## Temperature Sensor
218
219 # set_property - dict { PACKAGE_PIN C14 IOSTANDARD LVCMOS33 } [ get_ports { TMP_SCL }] ; # I O _ L 1 N _ T 0 _ A D 0 N _ 1 5 Sch = tmp_scl
220 # set_property - dict { PACKAGE_PIN C15 IOSTANDARD LVCMOS33 } [ get_ports { TMP_SDA }] ; # I O _ L 1 2 N _ T 1 _ M R C C _ 1 5 Sch = tmp_sda
221 # set_property - dict { PACKAGE_PIN D13 IOSTANDARD LVCMOS33 } [ get_ports { TMP_INT }] ; # I O _ L 6 N _ T 0 _ V R E F _ 1 5 Sch = tmp_int
222 # set_property - dict { PACKAGE_PIN B14 IOSTANDARD LVCMOS33 } [ get_ports { TMP_CT } ] ; # I O _ L 2 N _ T 0 _ A D 8 N _ 1 5 Sch = tmp_ct
223
224 ## O mn id i re ct io n al Microphone
225
226 # set_property - dict { PACKAGE_PIN J5 IOSTANDARD LVCMOS33 } [ get_ports { M_CLK } ] ; # IO_25_35 Sch = m_clk
227 # set_property - dict { PACKAGE_PIN H5 IOSTANDARD LVCMOS33 } [ get_ports { M_DATA }] ; # IO_L24N_T3_35 Sch = m_data
228 # set_property - dict { PACKAGE_PIN F5 IOSTANDARD LVCMOS33 } [ get_ports { M_LRSEL }] ; # IO_0_35 Sch = m_lrsel
229
230
231 ## PWM Audio Amplifier
232
233 # set_property - dict { PACKAGE_PIN A11 IOSTANDARD LVCMOS33 } [ get_ports { AUD_PWM }] ; # IO_L4N_T0_15 Sch = aud_pwm
234 # set_property - dict { PACKAGE_PIN D12 IOSTANDARD LVCMOS33 } [ get_ports { AUD_SD } ] ; # IO_L6P_T0_15 Sch = aud_sd
235
236
237 ## USB - RS232 Interface
238
239 # set_property - dict { PACKAGE_PIN C4 IOSTANDARD LVCMOS33 } [ get_ports { UART_TXD_IN }] ; # I O _ L 7 P _ T 1 _ A D 6 P _ 3 5 Sch =
,→ uart_txd_in
240 # set_property - dict { PACKAGE_PIN D4 IOSTANDARD LVCMOS33 } [ get_ports { UART_RXD_OUT }] ; # I O _ L 1 1 N _ T 1 _ S R C C _ 3 5 Sch =
,→ uart_rxd_out
241 # set_property - dict { PACKAGE_PIN D3 IOSTANDARD LVCMOS33 } [ get_ports { UART_CTS }] ; # I O _ L 1 2 N _ T 1 _ M R C C _ 3 5 Sch = uart_cts
242 # set_property - dict { PACKAGE_PIN E5 IOSTANDARD LVCMOS33 } [ get_ports { UART_RTS }] ; # I O _ L 5 N _ T 0 _ A D 1 3 N _ 3 5 Sch = uart_rts
243
244 ## USB HID ( PS /2)
245
246 # set_property - dict { PACKAGE_PIN F4 IOSTANDARD LVCMOS33 } [ get_ports { PS2_CLK }] ; # I O _ L 1 3 P _ T 2 _ M R C C _ 3 5 Sch = ps2_clk
247 # set_property - dict { PACKAGE_PIN B2 IOSTANDARD LVCMOS33 } [ get_ports { PS2_DATA }] ; # I O _ L 1 0 N _ T 1 _ A D 1 5 N _ 3 5 Sch = ps2_data
248
249
250 ## SMSC Ethernet PHY
251
252 # set_property - dict { PACKAGE_PIN C9 IOSTANDARD LVCMOS33 } [ get_ports { ETH_MDC }] ; # I O _ L 1 1 P _ T 1 _ S R C C _ 1 6 Sch = eth_mdc
253 # set_property - dict { PACKAGE_PIN A9 IOSTANDARD LVCMOS33 } [ get_ports { ETH_MDIO }] ; # I O _ L 1 4 N _ T 2 _ S R C C _ 1 6 Sch = eth_mdio
254 # set_property - dict { PACKAGE_PIN B3 IOSTANDARD LVCMOS33 } [ get_ports { ETH_RSTN }] ; # I O _ L 1 0 P _ T 1 _ A D 1 5 P _ 3 5 Sch = eth_rstn
255 # set_property - dict { PACKAGE_PIN D9 IOSTANDARD LVCMOS33 } [ get_ports { ETH_CRSDV } ] ; # I O _ L 6 N _ T 0 _ V R E F _ 1 6 Sch = eth_crsdv
256 # set_property - dict { PACKAGE_PIN C10 IOSTANDARD LVCMOS33 } [ get_ports { ETH_RXERR }] ; # I O _ L 1 3 N _ T 2 _ M R C C _ 1 6 Sch = eth_rxerr
257 # set_property - dict { PACKAGE_PIN C11 IOSTANDARD LVCMOS33 } [ get_ports { ETH_RXD [ 0 ] } ] ; # I O _ L 1 3 P _ T 2 _ M R C C _ 1 6 Sch = eth_rxd [
,→ 0 ]
258 # set_property - dict { PACKAGE_PIN D10 IOSTANDARD LVCMOS33 } [ get_ports { ETH_RXD [ 1 ] } ] ; # I O _ L 1 9 N _ T 3 _ V R E F _ 1 6 Sch = eth_rxd [
,→ 1 ]
259 # set_property - dict { PACKAGE_PIN B9 IOSTANDARD LVCMOS33 } [ get_ports { ETH_TXEN }] ; # I O _ L 1 1 N _ T 1 _ S R C C _ 1 6 Sch = eth_txen
260 # set_property - dict { PACKAGE_PIN A10 IOSTANDARD LVCMOS33 } [ get_ports { ETH_TXD [ 0 ] } ] ; # I O _ L 1 4 P _ T 2 _ S R C C _ 1 6 Sch = eth_txd [
,→ 0 ]
261 # set_property - dict { PACKAGE_PIN A8 IOSTANDARD LVCMOS33 } [ get_ports { ETH_TXD [ 1 ] } ] ; # I O _ L 1 2 N _ T 1 _ M R C C _ 1 6 Sch = eth_txd [
,→ 1 ]
262 # set_property - dict { PACKAGE_PIN D5 IOSTANDARD LVCMOS33 } [ get_ports { ETH_REFCLK }] ; # I O _ L 1 1 P _ T 1 _ S R C C _ 3 5 Sch =
,→ eth_refclk
263 # set_property - dict { PACKAGE_PIN B8 IOSTANDARD LVCMOS33 } [ get_ports { ETH_INTN }] ; # I O _ L 1 2 P _ T 1 _ M R C C _ 1 6 Sch = eth_intn

49
264
265
266 ## Quad SPI Flash
267
268 # set_property - dict { PACKAGE_PIN K17 IOSTANDARD LVCMOS33 } [ get_ports { QSPI_DQ [ 0 ] } ] ; # I O _ L 1 P _ T 0 _ D 0 0 _ M O S I _ 1 4 Sch =
,→ qspi_dq [ 0 ]
269 # set_property - dict { PACKAGE_PIN K18 IOSTANDARD LVCMOS33 } [ get_ports { QSPI_DQ [ 1 ] } ] ; # I O _ L 1 N _ T 0 _ D 0 1 _ D I N _ 1 4 Sch =
,→ qspi_dq [ 1 ]
270 # set_property - dict { PACKAGE_PIN L14 IOSTANDARD LVCMOS33 } [ get_ports { QSPI_DQ [ 2 ] } ] ; # I O _ L 2 P _ T 0 _ D 0 2 _ 1 4 Sch = qspi_dq [2 ]
271 # set_property - dict { PACKAGE_PIN M14 IOSTANDARD LVCMOS33 } [ get_ports { QSPI_DQ [ 3 ] } ] ; # I O _ L 2 N _ T 0 _ D 0 3 _ 1 4 Sch = qspi_dq [3 ]
272 # set_property - dict { PACKAGE_PIN L13 IOSTANDARD LVCMOS33 } [ get_ports { QSPI_CSN }] ; # I O _ L 6 P _ T 0 _ F C S _ B _ 1 4 Sch = qspi_csn

50
Appendix E

Kinect Software Code – Kinect 2.0

SDK

51
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Program Name : k i n e c t D e p t h S t r e a m
3 // Author : Alan Cheng
4 // Description : Streams depth frame to FT2232H using Kinect SDK 2.0 drivers
5 //
6 // Additional Comments : very unpolished ; basic off example D2XX / Kinect code
7 //
8 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
9 # include " stdafx . h "
10 # include < opencv2 / core / core . hpp >
11 # include < opencv2 / highgui / highgui . hpp >
12 # include < opencv2 / imgproc / imgproc . hpp >
13 # include < math .h >
14 # include < Windows .h >
15 # include < Kinect .h >
16 # include < iostream >
17 # include < fstream >
18 # include < string >
19 # include " ftd2xx . h "
20
21 using namespace cv ;
22 using namespace std ;
23
24 IKinectSensor * pSensor ;
25 HRESULT hResult ;
26 I C o l o r F r a m e S o u r c e * pColorSource ;
27 I C o l o r F r a m e R e a d e r * pColorReader ;
28 IFrameDescription * pColorDescription ;
29 I D e p t h F r a m e S o u r c e * pDepthSource ;
30 I D e p t h F r a m e R e a d e r * pDepthReader ;
31 I F r a m e D e s c r i p t i o n * pDescription ;
32
33 template < class Interface >
34 inline void SafeRelease ( Interface *& p I n t e r f a c e T o R e l e a s e )
35 {
36 if ( p I n t e r f a c e T o R e l e a s e != NULL ) {
37 pInterfaceToRelease - > Release () ;
38 p I n t e r f a c e T o R e l e a s e = NULL ;
39 }
40 }
41
42 inline int KinectInit ()
43 {
44 cv :: se tU s eO pt im i ze d ( true ) ;
45
46 // Sensor
47 hResult = S_OK ;
48 hResult = G e t D e f a u l t K i n e c t S e n s o r (& pSensor ) ;
49 if ( FAILED ( hResult ) ) {
50 std :: cerr << " Error : G e t D e f a u l t K i n e c t S e n s o r " << std :: endl ;
51 return -1;
52 }
53
54 hResult = pSensor - > Open () ;
55 if ( FAILED ( hResult ) ) {
56 std :: cerr << " Error : IKinectSensor :: Open () " << std :: endl ;
57 return -1;
58 }
59
60 // - - - - - - - - - - - - - - - - - - - -[ Depth ] - - - - - - - - - - - - - - - - - - - - - - - - - - -
61 // Source
62 hResult = pSensor - > g e t _ D e p t h F r a m e S o u r c e (& pDepthSource ) ;
63 if ( FAILED ( hResult ) ) {
64 std :: cerr << " Error : IKinectSensor :: g e t _ D e p t h F r a m e S o u r c e () " << std :: endl ;
65 return -1;
66 }
67
68 // Reader
69 hResult = pDepthSource - > OpenReader (& pDepthReader ) ;
70 if ( FAILED ( hResult ) ) {
71 std :: cerr << " Error : I D e p t h F r a m e S o u r c e :: OpenReader () " << std :: endl ;
72 return -1;
73 }
74
75 // Description
76 hResult = pDepthSource - > g e t _ F r a m e D e s c r i p t i o n (& pDescription ) ;
77 if ( FAILED ( hResult ) ) {
78 std :: cerr << " Error : I D e p t h F r a m e S o u r c e :: g e t _ F r a m e D e s c r i p t i o n () " << std :: endl ;
79 return -1;
80 }
81
82 }
83
84 int main ()
85 {
86 KinectInit () ;
87
88 // - - - - - - - - - - - - - - - - - - - - - - - - -[ Depth Initi alizatio n ] - - - - - - - - - - - - - - - - - - - - - - - - - -
89 int dWidth = 0;
90 int dHeight = 0;
91 pDescription - > get_Width (& dWidth ) ; // 512
92 pDescription - > get_Height (& dHeight ) ; // 424

52
 93 unsigned int bufferSize = dWidth * dHeight * sizeof ( unsigned short ) ;
94
95 // Range ( Range of Depth is 500 -8000[ mm ] , Range of Detection is 500 -4500[ mm ] )
96 unsigned short min = 0;
97 unsigned short max = 0;
98 pDepthSource - > g e t _ D e p t h M i n R e l i a b l e D i s t a n c e (& min ) ; // 500
99 pDepthSource - > g e t _ D e p t h M a x R e l i a b l e D i s t a n c e (& max ) ; // 4500
100 std :: cout << " Range : " << min << " - " << max << std :: endl ;
101
102 // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
103
104 cv :: Mat backgroundMat ( dHeight , dWidth , CV_16UC1 ) ;
105 cv :: Mat bufferMat ( dHeight , dWidth , CV_16UC1 ) ;
106 cv :: Mat binaryMat ( dHeight , dWidth , CV_8UC1 ) ;
107 cv :: Mat depthMat ( dHeight , dWidth , CV_8UC1 ) ;
108
109 // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
110
111 // cv :: namedWindow (" Color ") ;
112 cv :: namedWindow ( " Depth " ) ;
113
114 // IColorFrame * pColorFrame = nullptr ;
115 IDepthFrame * pDepthFrame = nullptr ;
116
117 UINT16 * backBuffer = nullptr ;
118 cv :: Mat backFrame ( dHeight , dWidth , CV_16UC1 ) ;
119
120 // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
121
122 FT_HANDLE fthandle ;
123 FT_STATUS ftstatus ;
124
125 UCHAR LatencyTimer = 2; // our default setting is 16
126
127 /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
128 // Configure the FT2232HL to Synchronous FIFO
129 /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
130 ftstatus = FT_Open (0 , & fthandle ) ;
131
132 if ( ftstatus != FT_OK ) {
133 printf ( " opening failed ! with error % d \ n " , ftstatus ) ;
134 return 1;
135 }
136
137 DWORD InT ransferS ize = 64000;
138 ftstatus = F T _ S e t U S B P a r a m e t e r s ( fthandle , InTransferSize , 0) ;
139 if ( ftstatus != FT_OK ) {
140 printf ( " changing buffer size failed ! with error % d \ n " , ftstatus ) ;
141 return 1;
142 }
143
144 UCHAR mask = 0 xff ;
145 UCHAR mode ;
146
147 mode = 0 x00 ; // reset mode
148 ftstatus = FT_SetBitMode ( fthandle , mask , mode ) ;
149
150 Sleep (10) ;
151
152 mode = 0 x40 ; // synchronous FIFO mode
153 ftstatus = FT_SetBitMode ( fthandle , mask , mode ) ;
154 FT_Purge ( fthandle , FT_PURGE_RX ) ;
155
156 if ( ftstatus != FT_OK ) {
157 printf ( " configuration failed ! with error % d \ n " , ftstatus ) ;
158 return 1;
159 }
160 else {
161 ftstatus = F T _ S e t L a t e n c y T i m e r ( fthandle , LatencyTimer ) ;
162 ftstatus = F T _ S e t U S B P a r a m e t e r s ( fthandle , 0 x10000 , 0 x10000 ) ;
163 ftstatus = F T _ S e t F l o w C o n t r o l ( fthandle , FT_FLOW_RTS_CTS , 0 x0 , 0 x0 ) ;
164 ftstatus = FT_Set Timeouts ( fthandle , 5000 , 5000) ;
165 }
166 printf ( " Synchronous FIFO mode set !\ n ") ;
167
168 DWORD EventDWord ;
169 DWORD RxBytes ;
170 DWORD TxBytes ;
171 DWORD BytesWritten ;
172
173 UCHAR data_buf_init [8] = { 0 x00 , 0 x00 , 0 x00 , 0 x00 , 0 xDD , 0 xCC , 0 xBB , 0 xAA };
174
175 // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -[ Get Background Loop ] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
176 while (1) {
177 # define send BufferSi ze 640000
178 unsigned char ImgBuffer [640000];
179 pDepthFrame = nullptr ;
180
181 hResult = pDepthReader - > A c q u i r e L a t e s t F r a m e (& pDepthFrame ) ;
182 if ( SUCCEEDED ( hResult ) )
183 {
184 hResult = pDepthFrame - > A c c e s s U n d e r l y i n g B u f f e r (& bufferSize , reinterpret_cast < UINT16 ** >(& bufferMat . data ) ) ;

53
185 if ( SUCCEEDED ( hResult ) )
186 {
187 pDepthFrame - > A c c e s s U n d e r l y i n g B u f f e r (& bufferSize , & backBuffer ) ;
188 for ( int i = 0; i < 15; i ++)
189 ImgBuffer [ i ] = data_buf_init [ i ];
190 int index = 16;
191 for ( int i = 0; i < dHeight ; i ++)
192 {
193 for ( int j = 0; j < dWidth ; j ++)
194 {
195 UINT16 value = backBuffer [ i * dWidth + j ];
196 char lo = value & 0 xFF ;
197 char hi = value >> 8;
198
199 ImgBuffer [ index ] = hi ;
200 ImgBuffer [ index + 1] = lo ;
201
202 index += 2;
203 }
204 }
205 bufferMat . convertTo ( depthMat , CV_8U , -255.0 f / 8000.0 f , 255.0 f ) ;
206 }
207 }
208
209 /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
210 // Write to FT2232HL
211 /* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
212 ftstatus = FT_GetStatus ( fthandle , & RxBytes , & TxBytes , & EventDWord ) ;
213
214 if ( ftstatus == FT_OK ) {
215 ftstatus = FT_Write ( fthandle , ImgBuffer , sizeof ( ImgBuffer ) , & BytesWritten ) ;
216 }
217
218 SafeRelease ( pDepthFrame ) ;
219
220 cv :: imshow ( " Depth " , depthMat ) ;
221
222 if ( cv :: waitKey (30) == VK_ESCAPE ) {
223 break ;
224 }
225 }
226
227 cv :: d e s t r o y A l l W i n d o w s () ;
228
229 SafeRelease ( pColorSource ) ;
230 SafeRelease ( pColorReader ) ;
231 SafeRelease ( p C o l o r D e s c r i p t i o n ) ;
232
233 SafeRelease ( pDepthSource ) ;
234 SafeRelease ( pDepthReader ) ;
235 SafeRelease ( pDescription ) ;
236 if ( pSensor ) {
237 pSensor - > Close () ;
238 }
239 SafeRelease ( pSensor ) ;
240 cv :: d e s t r o y A l l W i n d o w s () ;
241
242 return 0;
243 }

54
Appendix F

Kinect Software Code – libfreenect2

55
 1 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
2 // Program Name : l i b f r e e n e c t 2 S t r e a m
3 // Author : Alan Cheng
4 // Description : Streams depth frame to FT2232H using libfreenect2 drivers
5 //
6 // Additional Comments : very unpolished ; basic off example libfreenect2 code
7 //
8 // / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
9
10 // ! [ headers ]
11 # include < iostream >
12 # include < stdio .h >
13 # include < iomanip >
14 # include < time .h >
15 # include < signal .h >
16
17 // ! [ OpenCV library ]
18 # include < opencv2 / core / core . hpp >
19 # include < opencv2 / opencv . hpp >
20 # include < opencv2 / imgproc / imgproc . hpp >
21
22 # include < math .h >
23 # include < Windows .h >
24
25 // ! [ libfreenect2 library ]
26 # include < libfreenect2 / libfreenect2 . hpp >
27 # include < libfreenect2 / f r a m e _ l i s t e n e r _ i m p l .h >
28 # include < libfreenect2 / registration .h >
29 # include < libfreenect2 / pa ck e t_ pi pe l in e .h >
30 # include < libfreenect2 / logger .h >
31
32 # include " ftd2xx . h "
33 // ! [ headers ]
34
35 using namespace std ;
36 using namespace cv ;
37
38 bool p r o t o n e c t _ s h u t d o w n = false ; // Whether the running application should shut down .
39
40 void s igint_h andler ( int s )
41 {
42 p r o t o n e c t _ s h u t d o w n = true ;
43 }
44
45 int main ()
46 {
47 std :: cout << " Streaming from Kinect One sensor ! " << std :: endl ;
48
49 // ! [ context ]
50 libfreenect2 :: Freenect2 freenect2 ;
51 libfreenect2 :: F r ee ne c t2 De vi c e * dev = 0;
52 libfreenect2 :: Pack etPipeli ne * pipeline = 0;
53 // ! [ context ]
54
55 // ! [ discovery ]
56 if ( freenect2 . e n u m e r a t e D e v i c e s () == 0)
57 {
58 std :: cout << " no device connected ! " << std :: endl ;
59 return -1;
60 }
61
62 string serial = freenect2 . g e t D e f a u l t D e v i c e S e r i a l N u m b e r () ;
63
64 std :: cout << " SERIAL : " << serial << std :: endl ;
65
66 pipeline = new libfreenect2 :: O p e n C L P a c k e t P i p e l i n e () ;
67
68 if ( pipeline )
69 {
70 // ! [ open ]
71 dev = freenect2 . openDevice ( serial , pipeline ) ;
72 // ! [ open ]
73 }
74 else {
75 dev = freenect2 . openDevice ( serial ) ;
76 }
77
78 if ( dev == 0)
79 {
80 std :: cout << " failure opening device ! " << std :: endl ;
81 return -1;
82 }
83
84 signal ( SIGINT , sigint_ handler ) ;
85 p r o t o n e c t _ s h u t d o w n = false ;
86
87 libfreenect2 :: F r ee ne c t2 De vi c e :: Config config ;
88 config . E n a b l e B i l a t e r a l F i l t e r = true ;
89 config . E n a b l e E d g e A w a r e F i l t e r = true ;
90 dev - > s e t C o n f i g u r a t i o n ( config ) ;
91
92 // ! [ listeners ]

56
 93 libfreenect2 :: S y n c M u l t i F r a m e L i s t e n e r listener ( libfreenect2 :: Frame :: Depth ) ;
94 libfreenect2 :: FrameMap frames ;
95
96 // dev - > s e t C o l o r F r a m e L i s t e n e r (& listener ) ;
97 dev - > s e t I r A n d D e p t h F r a m e L i s t e n e r (& listener ) ;
98 // ! [ listeners ]
99
100 // ! [ start ]
101 dev - > start () ;
102
103 std :: cout << " device serial : " << dev - > g e tS er ia l Nu mb er () << std :: endl ;
104 std :: cout << " device firmware : " << dev - > g e t F i r m w a r e V e r s i o n () << std :: endl ;
105 // ! [ start ]
106
107 // ! [ registration setup ]
108 libfreenect2 :: Registration * registration = new libfreenect2 :: Registration ( dev - > g e t I r C a m e r a P a r a m s () , dev - >
,→ g e t C o l o r C a m e r a P a r a m s () ) ;
109
110 Mat depthmat ;
111
112 // ! [ ft2232h setup ]
113 FT_HANDLE fthandle ;
114 FT_STATUS ftstatus ;
115
116 UCHAR LatencyTimer = 2; // our default setting is 16
117
118 // Configure the FT2232HL to Synchronous FIFO
119 ftstatus = FT_Open (0 , & fthandle ) ;
120
121 if ( ftstatus != FT_OK ) {
122 printf ( " opening failed ! with error % d \ n " , ftstatus ) ;
123 return 1;
124 }
125
126 DWORD InT ransferS ize = 64000;
127 ftstatus = F T _ S e t U S B P a r a m e t e r s ( fthandle , InTransferSize , 0) ;
128 if ( ftstatus != FT_OK ) {
129 printf ( " changing buffer size failed ! with error % d \ n " , ftstatus ) ;
130 return 1;
131 }
132
133 UCHAR mask = 0 xff ;
134 UCHAR mode ;
135
136 mode = 0 x00 ; // reset mode
137 ftstatus = FT_SetBitMode ( fthandle , mask , mode ) ;
138
139 Sleep (10) ;
140
141 mode = 0 x40 ; // synchronous FIFO mode
142 ftstatus = FT_SetBitMode ( fthandle , mask , mode ) ;
143 FT_Purge ( fthandle , FT_PURGE_RX ) ;
144
145 if ( ftstatus != FT_OK ) {
146 printf ( " configuration failed ! with error % d \ n " , ftstatus ) ;
147 return 1;
148 }
149 else {
150 ftstatus = F T _ S e t L a t e n c y T i m e r ( fthandle , LatencyTimer ) ;
151 ftstatus = F T _ S e t U S B P a r a m e t e r s ( fthandle , 0 x10000 , 0 x10000 ) ;
152 ftstatus = F T _ S e t F l o w C o n t r o l ( fthandle , FT_FLOW_RTS_CTS , 0 x0 , 0 x0 ) ;
153 ftstatus = FT_Set Timeouts ( fthandle , 5000 , 5000) ;
154 }
155 printf ( " Synchronous FIFO mode set !\ n ") ;
156
157 DWORD EventDWord ;
158 DWORD RxBytes ;
159 DWORD TxBytes ;
160 DWORD BytesWritten ;
161
162 int dWidth = 512;
163 int dHeight = 424;
164
165 // New frame starting sequence : DDCCBBAA
166 UCHAR data_buf_init [8] = { 0 x00 , 0 x00 , 0 x00 , 0 x00 , 0 xDD , 0 xCC , 0 xBB , 0 xAA };
167
168 // ! [ Depth image stream loop ]
169 while (! p r o t o n e c t _ s h u t d o w n )
170 {
171 unsigned char ImgBuffer [640000]; // Buffer to store data to send to FT2232H
172
173 // Capture new depth frame
174 listener . w ai tF o rN ew Fr a me ( frames ) ;
175 libfreenect2 :: Frame * depth = frames [ libfreenect2 :: Frame :: Depth ];
176
177 cv :: Mat ( depth - > height , depth - > width , CV_32FC1 , depth - > data ) . copyTo ( depthmat ) ;
178
179 // Fill beginning of buffer with starting sequence
180 for ( int n = 0; n < 15; n ++)
181 ImgBuffer [ n ] = data_buf_init [ n ];
182 int index = 16;
183

57
184 // Convert 2 D OpenCV mat to buffer data that can be sent
185 for ( int i = 0; i < dHeight ; i ++) {
186 for ( int j = 0; j < dWidth ; j ++) {
187 UINT16 value = depthmat . at < float >( i , j ) ; // Read value at (i , j )
188 char lo = value & 0 xFF ; // Split value into two bytes
189 char hi = value >> 8;
190
191 ImgBuffer [ index ] = hi ; // Store the 2 bytes into buffer
192 ImgBuffer [ index + 1] = lo ;
193
194 index += 2;
195 }
196 }
197
198 ftstatus = FT_GetStatus ( fthandle , & RxBytes , & TxBytes , & EventDWord ) ; // Get status of FT2232H
199
200 // Only send data when ready
201 if ( ftstatus == FT_OK ) {
202 // Function to send data to FT2232_H
203 ftstatus = FT_Write ( fthandle , ImgBuffer , sizeof ( ImgBuffer ) , & BytesWritten ) ;
204 }
205
206 int key = cv :: waitKey (1) ;
207 p r o t o n e c t _ s h u t d o w n = p r o t o n e c t _ s h u t d o w n || ( key > 0 && (( key & 0 xFF ) == 27) ) ; // shutdown on escape
208
209 // ! [ loop end ]
210 listener . release ( frames ) ;
211 }
212 // ! [ loop end ]
213
214 // ! [ stop ]
215 dev - > stop () ;
216 dev - > close () ;
217 // ! [ stop ]
218
219 delete registration ;
220
221 std :: cout << " Streaming Ends ! " << std :: endl ;
222 return 0;
223 }

58
Appendix G

Arduino HID Mouse Code

59
 1 // Author : Hope Harrison
2 // Description : Teensy as HID mouse from FPGA PMOD data
3
4 // set this to the hardware serial port you wish to use
5 # define HWSERIAL Serial2
6
7 int x_1 = 0;
8 int x_2 = 0;
9 int y_1 = 0;
10 int y_2 = 0;
11 int x = 0;
12 int y = 0;
13 int top_left_x_1 = 0;
14 int top_left_x_2 = 0;
15 int top_left_x = 0;
16 int top_left_y_1 = 0;
17 int top_left_y_2 = 0;
18 int top_left_y = 0;
19 int b o t t o m _ r i g h t _ x _ 1 = 0;
20 int b o t t o m _ r i g h t _ x _ 2 = 0;
21 int bottom_ right_x = 0;
22 int b o t t o m _ r i g h t _ y _ 1 = 0;
23 int b o t t o m _ r i g h t _ y _ 2 = 0;
24 int bottom_ right_y = 0;
25 int relative_x = 0;
26 int relative_y = 0;
27 int is_press = 0;
28 int is_release = 0;
29 int is_move = 0;
30
31 const int MONITOR_X = 3840;
32 const int MONITOR_Y = 2160;
33
34 const int OFFSET_X = 200;
35 const int OFFSET_Y = -200;
36
37 void setup () {
38 Serial . begin (9600) ;
39 HWSERIAL . begin (115200) ;
40 Mouse . screenSize ( MONITOR_X , MONITOR_Y ) ;
41 }
42
43 void loop () {
44 int incomingByte ;
45
46 if ( HWSERIAL . available () >= 15) {
47 x_1 = HWSERIAL . read () ;
48 x_2 = HWSERIAL . read () ;
49 y_1 = HWSERIAL . read () ;
50 y_2 = HWSERIAL . read () ;
51 top_left_x_1 = HWSERIAL . read () ;
52 top_left_x_2 = HWSERIAL . read () ;
53 top_left_y_1 = HWSERIAL . read () ;
54 top_left_y_2 = HWSERIAL . read () ;
55 b o t t o m _ r i g h t _ x _ 1 = HWSERIAL . read () ;
56 b o t t o m _ r i g h t _ x _ 2 = HWSERIAL . read () ;
57 b o t t o m _ r i g h t _ y _ 1 = HWSERIAL . read () ;
58 b o t t o m _ r i g h t _ y _ 2 = HWSERIAL . read () ;
59 is_press = HWSERIAL . read () ;
60 is_release = HWSERIAL . read () ;
61 is_move = HWSERIAL . read () ;
62
63 x = x_2 *256 + x_1 ;
64 y = y_2 *256 + y_1 ;
65 top_left_x = top_left_x_2 *256 + top_left_x_1 ;
66 top_left_y = top_left_y_2 *256 + top_left_y_1 ;
67 bott om_righ t_x = b o t t o m _ r i g h t _ x _ 2 *256 + b o t t om _ r i g h t _ x _ 1 ;
68 bott om_righ t_y = b o t t o m _ r i g h t _ y _ 2 *256 + b o t t om _ r i g h t _ y _ 1 ;
69
70 relative_x = MONITOR_X - float ( x - top_left_x ) / float ( bottom _right_x - top_left_x ) * MONITOR_X + OFFSET_X ;
71 relative_y = float ( y - top_left_y ) / float ( bottom_r ight_y - top_left_y ) * MONITOR_Y + OFFSET_Y ;
72
73 if ( is_move ) {
74 if ((0 < x ) && ( x < MONITOR_X ) && (0 < y ) && ( y < MONITOR_Y ) )
75 Mouse . moveTo ( relative_x , relative_y ) ;
76 }
77 if ( is_press ) {
78 Mouse . press ( MOUSE_LEFT ) ;
79 }
80 else if ( is_release ) {
81 Mouse . release ( MOUSE_LEFT ) ;
82 }
83
84 // Use this to scroll :
85 // Mouse . scroll ( scroll_up - scroll_down , 0) ;
86
87 // For debugging
88 Serial . print ( " Absolute touch point (x , y ) = ( ") ;
89 Serial . print (x , DEC ) ;
90 Serial . print ( " , " ) ;
91 Serial . print (y , DEC ) ;
92 Serial . println ( " ) " ) ;

60
 93
94 Serial . print ( " Top left (x , y ) = ( ") ;
95 Serial . print ( top_left_x , DEC ) ;
96 Serial . print ( " , " ) ;
97 Serial . print ( top_left_y , DEC ) ;
98 Serial . println ( " ) " ) ;
99
100 Serial . print ( " Bottom right (x , y ) = ( ") ;
101 Serial . print ( bottom_right_x , DEC ) ;
102 Serial . print ( " , " ) ;
103 Serial . print ( bottom_right_y , DEC ) ;
104 Serial . println ( " ) " ) ;
105
106 Serial . print ( " Move mouse to (x , y ) = ( ") ;
107 Serial . print ( relative_x , DEC ) ;
108 Serial . print ( " , " ) ;
109 Serial . print ( relative_y , DEC ) ;
110 Serial . println ( " ) " ) ;
111
112 Serial . print ( " Press ? " ) ;
113 Serial . println ( is_press , BIN ) ;
114 Serial . print ( " Release ? " ) ;
115 Serial . println ( is_release , BIN ) ;
116 Serial . print ( " Move ? " ) ;
117 Serial . println ( is_move , BIN ) ;
118
119 // Serial . print (" Scroll = ") ;
120 // Serial . println ( scroll_up - scroll_down ) ;
121 }
122 }

61