US008504994B2

(12) Ulllted States Patent

Golender et a].
(54) SYSTEM AND METHOD FOR SOFTWARE

(10) Patent N0.:

(45) Date of Patent:
4,598,364 A

US 8,504,994 B2
Aug. 6, 2013

7/ 1986 Gum et al.

DIAGNOSTICS USINGA COMBINATION OF

, ,

2
5,021,949 A 5,121,489 A

$41013? 4131
was 1 e .

VISUAL AND DYNAMIC TRACING

6/1991 Molten et 31' 6/1992 Andrews

(75)

Inventors: Valery Golender, Kfar Saba (IL); Ido

Ben Moshe, HerZlia (IL); Shlomo

5,193,180 A
5,265,254 A
5,335,344 A

3/ 1993 Hastings
11/1993 Blasciak et al.

wygodnys Ramut Hasharon (1L)

5,297,274 A

3/ 1994 Jackson

8/1994 Hastings

(73) Assignee: Identify Software, Ltd., Petach-Tikva

5347649 A

9/1994 Alderson

(1L)
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 W0 W0 96'05556

(Continued)
FOREIGN PATENT DOCUMENTS
2/1996

U.S.C. 154(b) by 811 days.

(21) Appl' NO: 12/575357

OTHER PUBLICATIONS

Abramson, D., et al., A Debugging Tool for Software Evolution,

(22) Filed:
(65)

Oct 7, 2009

EASE-95., 7th International Workshop on Computer-Aided Software

ngrneerlng, Jul. 1995.

Prior Publication Data US 2010/0088683 A1 Apr. 8, 2010

Related US. Application Data

(Continued)
Prlmary Exammer i Tuan A Vu

(63)
(60)

Continuation of application No. 09/799,338, ?led on Mar. 5, 2001, now abandoned.

Provisional application N0~ 60/186,636, ?led 011 Mar3, 2000Int- Cl-

(57)

ABSTRACT

A software system is disclosed that provides remote trouble

shooting and tracing of the execution of computer programs.

The software system includes an information- gathering mod ule that gathers run-time information about program execu tion, program interaction with the operating system and the system resources, monitors user actions and captures screen
Output, and passes the gathered infonnation to an informa

(51)
(52)

G06F 9/44
U-s- Cl-

(2006-01)

USPC ........................................................ .. 717/128

(58) Field of Classi?cation Search

USPC ............... .. 7l7/l24il28, 129; 709/203, 217,

tion-display module, the gathered information allowing a

support technician (e.g., a software developer, a help desk

709/221, 224; 707/10; 7l4/28i35, 38, 45,

person, etc.) to see the user interactions with the program and

714/47, 50, 46; 710/6; 715/700 See application ?le for complete search history.
(56) References Cited
U.S. PATENT DOCUMENTS
4,503,495 A 3/1985 Boudreau

corresponding reactions of the system, and/or to remotely view environment variables, ?le access operations, system
interactions, and user interactions that occur on the users

computer and locate failed operations that cause execution

problems.
4,511,960 A 4/1985 Boudreau
SUPPORT S115

20 Claims, 18 Drawing Sheets

[12w
NT WWH ER SWE

SEND VISUAL PQOELEM MoNwoR APPLEVEL TC =15 TO HE CJS

CUSTOMER 5 TE
APPucAnoN IRACNG u SING
wsuAt PROBLEM MoWoR AGENT

ANALYZE FiOELEMS USING SOLRCE CODE EHGTRAPPER ANAJZER

APPUCAI ON SOJRCE CODE

US 8,504,994 B2
Page 2
U.S. PATENT DOCUMENTS
5,386,565 5,394,544 5,408,650 5,410,685 5,421,009
5,386,522 A

OTHER PUBLICATIONS
Advisory Action dated Oct. 20, 2008 from Related U.S. Appl. No. 10/80&760~ Advisory Action dated Oct. 7, 2005 from Related U.S. Appl. No. 09/799,338.
-

A A A A A

V1995 2/1995 4/ 1995 4/1995 5/1995

1/1995

Tanaka et a1 Motoyama et al. Arsenault Banda et a1.

Evans

5,430,586 A
5,446,876 A
5,450,586 A

Platt 7/1995 K09

8/1995 Levlne et a1~

Appelbe et al., Integrating tools for debugging and developing multitasking programs, ACM, 1993, pp. 78-88.
Bates, Peter C., Debugging Heterogeneous Distributed Systems
Using Event-Based Models of Behavior, ACM Transactions on

9/1995 Kuzara et 31'

2
5483468 A
5,513,317 A
4/1996

Qgggliy
V1996 Chen et a1
Borchardt ' et al.

Computer Systems, vol. 13, No. 1, Feb. 1995, pp. 1-31.

Bruegge, C.P. et al., A Framework for Dynamic Program Analyz
ers, OOPSLA, 1993, pp. 62-85.
_ _

5 526 485 A

6/l996 Brodsky

Forrest, S., et al., A sense ofselffor UIllX processes, Proceedings of

535333192 A
5,551,037 5,574,897 5,581,697 5,590,354 A A A A

7/1996 Hawley et a1
8/1996 11/1996 12/1996 12/1996 Fowler et a1. Hermsmeier et al. Gramlich et al. Klapproth et 81.

1996 IEEE Symposium on Computer Security and Privacy, 1996, pp.

120-128 Geer, C.P. et al., Instruction Stream Trace, IBM Technical Disclo sure Bulletin, vol. 26, No. 11, Apr. 1984, pp. 6217-6220. GoldsZmidt et al, Interactive blackbox debugging for concurrent

5,612,898 A
5,615,331 A 5,632,032 A
5,642,478 A

3/1997 Huckins
3/1997 Toonans et 31' 5/1997 Au et a1~
6/1997 Chen et al.
3/1998
4/1998

language, ACM pp 271_2g2, 1990

Harward, L.D. Jr., PL/1 Trace Program, IBM Technical Disclosure Bulletin, vol. 13, N0. 4, Sep. 1970, pp. 855-857.
H t TA G

5,657,438 A
5,732,210 A

8/l997 Wygodny et a1
BuZbee
Ahmad et a1.

1 En, .1 .is IeInerga J racf97a3c1ity,2446 24ers ica 1sc osure u e 1, V0 '

' '

1T

-1-

,, IBMT 1111'_
_

1D-

B 1

~
_

a?

PP

'
_

5,740,355 A
5,745,748 A

4/l998 Watanabe et a1
@1998 Harper 7/1998 Parker et a1 714/381 1

James R., Ef?cient Program Tracing, IEEE, May 1993, pp.

Lee et al., Learning Patterns from Unix Process Execution Traces for Intrusion Detection, AAAI Technical Report 1997 (retrieved on

5 771 385 A 537813720 A *

5,848,274 5,867,643 5,870,606 5,896,535 5,903,718 5,928,369

A A A A A A

12/1998 2/1999 2/1999 4/1999 5/1999

Hamby et 31 Sutton Lindsey Ronstrom Marik

7/1999 Keyser et 31,

8/1999 John, Jr. et a1.

Jun. 11, 2009) Retrieved from the Internet: URL: https://www.aaai. org/Papers/Workshops/l997/WS-97-07/WS97-07-010.pdf. Malony, Allen D. et al., Traceview: A Trace Visualization Tool, IEEE, SPP- 1991, PP 19-28~ Martonosi, Margaret et al., Effectiveness of Trace Sampling for Performance Debugging Tools, ACM SIGMETRICS, 1993, pp.
248-259.

5,938,778 A

5,940,618 5,960,198 5,983,366 6,003,143

6,026,433 6,026,438 6,047,124 6,065,043 6,108,330

A A A A
A A A A A

8/1999 9/1999 11/1999 12/ 1999

2/2000 2/ 2000 4/2000 5/2000 8/2000

Blandy et al. Roediger et al. King Kim et al.

DArlach et al.

Meier, Michael S., et al., Experiences with Building Distributed Debuggers, SPDT, 1996, pp. 70-79. Mukherjea, Sougata et al., Applying Algorithm Animation Tech niques for Program Tracing, Debugging, and Understanding, IEEE,
1993, pp. 456-465. Netzer, H.B., et al., Optimal Tracing and Incremental Reexecution for Debugging Long-Running Programs, ACM SIGPLAN 1994 Conference on Programming Language Design and Implementation, Jun. 2024, Orlando, Florida, pp. 313-325.
NetZer, Robert H.B. eta1., O Ptimal Tracin g and Re P lay for Debu 6 -

PiaZZa et al. M?fsl?nd Domenikos et a1~ Bhatia er 91

6,173,340 B1 *

1/2001 Gready et a1~ ~~~~~~~~~~~~~~~~~~ ~~ 710/6

6,202,199 B1 6,219,826 B1

3/2001 Wygodny et a1~ 4/2001 De Pauw et 31'

ging Message-Passing Parallel Programs, IEEE, 1992, pp. 502-511. NetZer, Robert H.B., Optimal Tracing and Replay for Debugging

2752;; 6263456 Bl
7 7

Enameluck let 31' 70001 B0255 it :1

'

Shared-Memory Parallel Programs, AClVUONR Workshop on Par ?llel and Distributed Debugging, May 17-18, 1993, San Diego, Cali
ornia, pp. 1-12.

g3;

1392333113, et 31'
4/2002 ODonnell
7/2002 10/2002

Plattner, Bernhard et al., Monitoring Program Execution: A Survey,

IEEE, Nov. 1981, pp. 76-93.

6360331 B2
6 374 369 B1
6,415,394 B1 6 467 052 B1*

3/2002 Ven et a1

Redell, Experience with TopaZ teledebugging, ACM SIGPLAN &

,, Reiss, Steven P., Trace-Based Debugging, Automated and Algo . . .

. Fruehling et al. Kaler et al 714/39 ' """""""""" "

SIGOPS, pp. 35-44, 1990.

rithmic Debugging Workshop, May 3-5, 1993.

6490696 Bl
6,587,969 B1 * 6593940 B1 *
6 865 508 B2

12/2002 Wood et al'

Rosenberg Jonathan B How Debuggers Work' Algorithms Data

gss?glilyetai'l
7/2003 Weinberg et 7/2003 Petersen et a1
300% Ueki Anderson et a1 et 31

Structures, and Architecture, John Wiley & Sons, Inc., 1996.

714/46 Schieber, Colleen et al., RATCHET: Real-time Address Trace Com pression Hardware for Extended Traces, Performance Evaluation
Review, Soule, K., V01. Algorithm 21, NOS. 3 for and Tracing 4, Execution Paths to a Given Loca

7:058:928 7,089,536 7,114,150 7,386,839

2002/0087949 2003/0005414 2003/00888 54 2004/0060043 2006/0150162 2006/0242627 2008/ 0244534

B2 B2 B2 B1
A1 A1 A1 A1 A1 A1 A1

6/2006 8/2006 9/2006 @2008

Wygodny et a1 Ueki et al. Dimpsey et a1 Golender et 31

tion in a Program, IBM Technical Disclosure Bulletin, vol. 14, No. 4, _$eP~_1971; PP 1016-1019 _ _ Spinellis, Diomidis, Trace: A Tool for Logging Operating System Call Transactions, Operating Systems Review Publication, vol. 28,
N9 4, Oct. 1994, PP 56-61 Timmerman, F Gielen et al., High Level Tools for the Debugging of Real-Time Multiprocessor Systems, AClVUONR Workshop on Par

7/2002 Golender et al. 1/2003 Elliott et a1, 5/2003 Wygodny et a1,

3/2004 Frysinger et a1, 7/2006 Mongkolsmai et al. 10/2006 Wygodny et al.

10/ 2008 Golender et al.

allel and Distributed Debugging, May 17-18, 1993, San Diego, Cali fornia, pp. 151-158. Tsai, Jeffrey JP. et al., A Noninvasive Architecture to Monitor
Real-Time Distributed Systems, IEEE, Mar. 1990, pp. 11-23.

US 8,504,994 B2
Page 3
Ueki, K., et al., A Probe Debugging Method, Technical Report of Ieice, Jul. 2000. Wilner, David, WindView: A Tool for Understanding Real-time Embedded Software through System Visualization, ACM
SIGPLAN Notices, vol. 30, No. 11, Nov. 1995, pp. 117-123.

Past, present, and future of?ce actions, amendments, arguments, and

other relevant documents or materials in the ?le history of U.S.

Letters Patent No. 7,386,839, issued Jun. 10, 2008, entitled System and Method for Troubleshooting Software Con?guration Problems

Using Application Tracing.

Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 12/135,678, ?led Jun. 9, 2008, entitled System and Method for

Wismuller, Debugging of globally optimized programs using data

?ow analysis, ACM SIGPLAN, Apr. 1994, pp. 278-289. Of?ce Action dated Apr. 11, 2004 from Related U.S. Appl. No.

09/799,338.
Of?ce Action dated Aug. 6, 2008 from Related U.S. Appl. No.

Troubleshooting Software Con?guration Problems Using Applica tion Tracing. Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 10/808,760, ?led Mar. 25, 2004, entitled System and Method for

10/808,760.
Of?ce Action dated Dec. 31, 2007 from Related U.S. Appl. No.

10/ 808,760.
Of?ce Action dated Jan. 2, 2009 from Related U.S. Appl. No.

Troubleshooting Runtime Software Problems Using Application

Learning.
Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 11/165,442, ?led Jun. 23, 2005, entitled System and Method for

10/ 808,760.
Of?ce Action dated Jan. 23, 2006 from Related U.S. Appl. No.

09/799,338.
Of?ce Action dated Jan. 9, 2007 from Related U.S. Appl. No.

09/799,338.
Of?ce Action dated Jun. 22, 2009 from Related U.S. Appl. No.

Automated Tuning of Program Execution Tracing. Jeffrey K. Hollingsworth and Barton P Miller, An Adaptive Cost System for Parallel Program InstrumentationiProceedings of the
Second International Euro-Par Conference on Parallel Processingi

10/ 808,760.
Of?ce Action dated Jun. 6, 2005 from Related U.S. Appl. No.

09/799,338.
Of?ce Action dated Sep. 1, 2006 from Related U.S. Appl. No.

09/799,338.
Of?ce Action dated Sep. 27, 2007 from Related U.S. Appl. No.

vol. 1, 1996, 10 pages. Paolo Bellavista, Antonio Corradi, and Cesare Stefanelli, Java for On-Line Distributed Monitoring of Heterogeneous Systems and Ser vicesiThe Computer Journal, vol. 45, No. 6, 2002, 13 pages. Martin Shulz, John May, and John Gyllenhaal, DynTG: A Tool for

09/799,338.
Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S.

Interactive, Dynamic InstrumentationiICCS 2005, LNCS 3515,

2005, 9 pages. Daniel Reed et al. An Overview of the Pablo Performance Analysis

Letters Patent No. 6,202,199, issued Mar. 13, 2001, entitled System and Method for Remotely Analyzing the Execution of Computer

EnvironmentiUniversity of Illinois Department of Computer Sci

ence, Nov. 7, 1992, 45 pages. Celine Boutros Saab, Xavier Bonnaire, and Bertil Folliot Phoenix: A

Programs.
Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S.

Letters Patent No. 6,282,701, issued Aug. 28, 2001, entitled System and Method for Monitoring and Analyzing the Execution of Com

Self Adaptable Monitoring Platform for Cluster Managementi Cluster Computing 5, 2002, 11 pages. Weiming Gu et al. Falcon: On-line monitoring and Steering of
Large-Scale Parallel Programs , Proceedings of the Fifth Sympo sium on the Frontiers of Massively Parallel Computation, 1995, 38
pages.

puter Programs.
Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 09/748,752, ?led Dec. 26, 2000, entitled System and Method for

Final Of?ce Action received for U.S. Appl. No. 10/ 808,760, mailed
on Mar. 3, 2010, 19 pages.

System and Method for Conditional Tracing of Computer Programs,

now abandoned.

Past, present, and future of?ce actions, amendments, arguments, and

other relevant documents or materials in the ?le history of U.S. Appl.

No. 09/923,972, ?led Aug. 7, 2001, entitled System and Method for System and Method for Conditional Tracing of Computer Programs,
now abandoned.

Non Final Of?ce Action received for U.S. Appl. No. 10/ 808,760, mailed on Aug. 20, 2010, 20 pages. Notice ofAllowance received for U.S. Appl. No. 10/808,760, mailed on Feb. 7, 2011, 11 pages. Bowring, James F., et al., Active Learning for Automatic Classi? cation of Software Behavior, Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis, (2004), pp. 195-205.

Past, present, and future of?ce actions, amendments, arguments, and

other relevant documents or materials in the ?le history of U.S.

Hofmeyr, et al., Intrusion Detection using Sequences of System Calls, Journal of Computer Security, Retrieved on [Aug. 13, 2010],
Retrieved from the Internet <http://www.cs.unm.edu/-steveah/jcs

Letters Patent No. 7,058,928, issued Jun. 6, 2006, entitled System and Method for System and Method for Conditional Tracing of

accepted.pdf>, (1998), pp. 151-180.

Of?ce Action received for U. S. Appl. No. 10/ 808,760, mailed on Mar.

Computer Programs.
Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 11/447,579, ?led Jun. 6, 2006, entitled System and Method for

3, 2010, 19 pages.
Of?ce Action received for U.S. Appl. No. 10/808,760, mailed on

System and Method for Conditional Tracing of Computer Programs. Past, present, and future of?ce actions, amendments, arguments, and
other relevant documents or materials in the ?le history of U.S. Appl. No. 09/799,338, ?led Mar. 5, 2001, entitled System and Method for Software Diagnostics Using a Combination of Visual and Dynamic Tracing, now abandoned.

Aug. 20, 2010, 20 pages. Bowring et al.,Active Learning for Automatic Classi?cation of Soft
ware Behavior, Proceedings of the 2004 ACM SIGSOFT interna

tional symposium on Software testing and analysis, vol. 29 No.4, Jul. 11-14, 2004, pp. 195-205.

* cited by examiner

US. Patent

Aug. 6, 2013

Sheet 1 0f 18

US 8,504,994 B2

AT THE DEVELOPER SITE

m0

//Z

/05 \
BUGTRAPPER ANALYZER

TI
DEVELOPER

f/z/
DEBUG INFORMATION

AT THE CUSTOMER SITE

12a

USER
m4 m2

\
BUCTRARRER

TRACED USER
ARRLICATION

ACENT
DUCTRARRER
TRACE LIBRARY

//Z4
COMMANDS

(CLIENT)
(e g
. . START

STOP

TRACE

>> BUGTRARPER
TRACE LIBRARY

NOTIFICATTONS (e.g.

(AGENT SIDE)

_ SUBPROCESS CREATED)

(CUENT SIDE)
l

TRACE BUFFER

(SHARED MEMORY)

14/6? [15

US. Patent

Aug. 6, 2013

Sheet 2 0f 18

US 8,504,994 B2

BACK AT THE DEVELORERS SITE

/06

7/2

X
BDCTRARRER

00

f.
I 9 A I?
DEVELOPER I
3 5;

ANALYZER
TRACE
TREE

/J0

II

/Z2
TRACE LOG FILE

f/Z/
DEBUG INFORMATION

//2

(A
g

BDCTRARRER IN ONLINE MODE

00

DEVELOPER

/06\
BDCTRARRER ANALYZER COMMANDS (6.9. START/STOP TRACE)
BOOTRARRER w

//02
TRACED USER APPLICATION

(CLIENT)
BLICTRARRER

TRACE LIBRARY

NOTIFICATIONS (evg-

TRACE LIBRARY

(ACENT SIDE)

_ SUBPROCESS CREATED)

(CLIENT SIDE)
l

TRACE
TREE
/05

\
725

i/ZZ\

TRACE BUFFER (SHARED MEMORY)

TRACE

US. Patent

Aug. 6, 2013

Sheet 3 0f 18

US 8,504,994 B2

:SIag8?s;

_ @DU xL2m3u6q51o2ims

EA, % E@ @
m?2E39/8tw5m%h

l/ Q kw

US. Patent

Aug. 6, 2013

Sheet 5 0f 18

US 8,504,994 B2

W400
405

lgrocess List
/

IXII

. . _L__ Applications Processes

Nome
Exploring - C:\BugTropper\fire\Rele..

PIDI
115

Fire Simulation

113

CE

Concei

Refresh

[/61 4

US. Patent

Aug. 6, 2013

Sheet 7 0f 18

US 8,504,994 B2

602

i161

US. Patent

Aug. 6, 2013

Sheet 9 0f 18

US 8,504,994 B2

f 802
I E] File Class Tobe Nome Process = E C: \Progrom Fi|es\I\/luTek\BugTrc1pper\F
E gE E? E EETI E Threod Thread Thread Thread 57 65 125 69

FIG (9

US. Patent

Aug. 6, 2013

Sheet 10 0f 18

US 8,504,994 B2

K900
BugTrapper Agent Attach to running processes
Check enabled processes that you want to trace.

g?

Disabled processes may be started and traced from

the next dialog box.
The Finish button means that all the processes in memory will be traced.

List of processes:

iBl6gJSbounce\Debu

< Back

Finish

Cancel

906

902/ 904/

FIG: ,9

US. Patent

Aug. 6, 2013

Sheet 11 0f 18

US 8,504,994 B2

K7000
BugTrapper Agent Start processes
To start and then trace an enabled process in the list, check the process and fill in the fields below or use

El

the default settings,

List of processes:

Current process Full Path Name

C: \BugTrapper\bounce\Debug\bounce

Parameters

Working Directory C: \BugTrapper\bounce\Debug

\\/ 0/ 6

< Back

i
7006/

Finish
i

Cancel
7004

7002

[/61 [0

US. Patent

Aug. 6, 2013

Sheet 15 0f 18

US 8,504,994 B2

7402

CREATE CLIENT PROCESS IN A

CREATE_SUSPENDED STATE

I
ALLOCATE MEMORY AND LOAD THE TRACE RUNTIME INITIALIZATION ROUTINE

7404
7406

I
OVERWRITE ENTRY POINT OF CLIENT WITH A JUMP TO TRACE RUNTIME INITIALIZATION

l
START CLIENT PROCESS (WHICH EXECUTES JUMP TO RUNTIME INITIALIZATION)

7408
7470

I
RESTORE ORICIONAL ENTRY POINT CODE AND SUSPEND CLIENT PROCESS

l
CALL INST_ATTACH ROUTINE AND LOAD INSTRUMENTAION DLL

7472

14/61 14

US. Patent

Aug. 6, 2013

Sheet 16 0f 18

US 8,504,994 B2

M2675

E|_M<I3mOW>E 0KOCZ4E MIU

NQmK

%\\ %\

US. Patent

Aug. 6, 2013

Sheet 17 0f 18

US 8,504,994 B2

VZ%A_M@
:22% 2; EM 823 m

@W E