Programming Language
Processing
Machine Learning
for
Source Code Understanding and Generation
Saikat Chakraborty
Ph.D. Candidate
Columbia University, New York
https://thenewstack.io/how-much-time-do-developers-spend-actually-writing-code/
2
My Research Interest
Code Summarization Code Editing/ Repair
Bug Detection Code Search
Code Generation / Synthesis Other Source Code Analysis
3
Identify Bugs
x = parameter(free)
Y = trace_in_dataflow(x)
if x or y is modified:
WARNING!!
[e.g., Alloy model checking, KLEE, SAGE, Cute,
AFL Fuzzers, Java Path Finder, LLVM Passes]
4
Automated Edit / Program Repair
l = location(gets)
v = parameters(gets)
s = memory_size(v)
new_stmt = “fgets (v, s, stdin)”
Update l with new_stmt
[1] Simfix Jiang et.al. 2018
[2] Elixir Saha et.al.2017]
5
Pattern/Rule Based Automation
Too Many
Different
Patterns
Low [4]
tolerance l = location(gets)
for v = parameters(gets)
s = memory_size(v)
Deviation
x = parameter(free)
Y = trace_in_dataflow(x)
new_stmt = “fgets (v, s, stdin)”
Update l with new_stmt
if[3]
x or y is modified:
WARNING!!
[3] Johnson et. al. 2013
[4] Saha et. al. 2017
6
Data-Driven Automation
x = parameter(free)
Y = trace_in_dataflow(x)
if x or y is modified:
WARNING!!
l = location(gets)
v = parameters(gets)
s = memory_size(v)
new_stmt = “fgets (v, s, stdin)”
Update l with new_stmt
7
Data-Driven Automation – Specification Mining
l = location(gets)
x = parameter(free)
v = parameters(gets)
Y = trace_in_dataflow(x)
s = memory_size(v)
if x or y is modified:
new_stmt = “fgets (v, s, stdin)”
WARNING!!
Update l with new_stmt
[5] Pradel et. al. 2012
[6] Le et. al. 2018
[7] Meng et. al. 2017
8
Data-Driven Automation – Automated Analyzer
9
Data-Driven Automation – Automated Analyzer
10
Artificial Intelligence for Software Engineering
Curate and
Preprocess Design Training AI-based
Data Model Automation Tool
11
Challenges of AI for SE
Understanding Generating
Source Code Source Code
Understanding Structure Ensuring the Syntactic and
And Functionality of Semantic Correctness for
Source Code Generating Source Code
12
Programming Language Processing
Program Analysis Machine Learning
13
Code Understanding Task - Example
Understand Source Code
Vulnerability Detection
14
Code Generation Tasks - Example
Code Translation
Generate Source Code
Code Editing/
Program Repair
15
Understanding Source Code - Requirement
Control
Dependency
Syntax
Structure
Data
Dependency
16
Generating Source Code - Requirement
1. Syntactic correctness. 2. Semantic correctness.
Syntactically Incorrect Semantically Incorrect
17
Programming Language Processing
Syntax
Encoding
Semantics
18
Encoding Syntax and Semantics
Syntax
Code
Modeling
Explicit Implicit
Encoding Encoding
Ways of Natural Channel
Encoding Mutation
Formal Channel
Mutation
Semantics
19
Explicit Encoding
Code
Modeling
Explicit Implicit
Encoding Encoding
Natural Channel
Mutation
Formal Channel
[8] Learning to Represent Program as Graphs
Mutation Allamanis et. al. 2017
[9] Learning to Represent Edits
Yin et. al. 2019
[10] HOPPITY – Dinella et. al. 2020
20
CODIT: Code Editing with Tree Based Neural Models
(TSE’20)
Explicit Encoding of Code Structure for Code Edit
21
Automated Code Editing
Example Code Edits
Code Before Edit Code After Edit
22
CODIT: Code Editing With Tree Based Models
Code Before Edit Code After Edit
23
CODIT Step 1 : Tree Translation
24
CODIT Step 1 :Tree Translation
25
CODIT Step 1 :Tree Translation
Rules sequence of Rules sequence of
Syntax Tree before edit Syntax Tree after edit
26
CODIT Step 1 :Tree Translation
27
CODIT Step 1 :Tree Translation
28
CODIT Step 2 : Token Translation
Token Translation
29
CODIT Step 2 : Token Translation
Reachable Variables in
Edit Location:
{inst, object, tmp}
Reachability Analysis in Edit Location
30
CODIT: Code Editing With Tree Based Models
# of # of Edit Code Fragment Size
Data Set
Projects Examples # of Tokens # of Nodes
Generic Code Edit Max - 38 Max - 47
48 32,473
from Github Avg - 15 Avg - 20
Pull Request Edits Max - 34 Max - 47
3 5546
[Tufano et. al. 2019] Avg - 17 Avg - 23
31
CODIT: Code Editing With Tree Based Models
Method Generic Code Edits Pull Request Edit
LSTM-Seq2Seq 3.77% 11.26%
Sequence
Tufano et. al. 6.57% 23.65%
Based
SequenceR 9.76% 26.43%
Tree2Seq 11.04% 23.49%
Tree
Based CODIT 15.94% 28.87%
32
CODIT – Results
CODIT fixes 15 bugs completely and 10 bugs partially, out of 80 bugs in Defects4j.
Closure
Compiler:
Bug-3
33
Explicit Encoding PL knowledge into model
Reachability Analysis Context Free
Grammar
34
ReVeal (TSE’21)
Explicit Encoding of Code Structure for Vulnerability
Detection
35
ReVeal
Training
Data
Node GGNN
Code Property
Graph Features
Trained ReVeal
Graph Embedding
36
ReVeal - Results
ReVeal
F1 score
Token Graph Token Graph
Based Based Based Based
[10] [10]
Chromium & Debian FFMpeg & Qemu
[10] Devign Zhou et. al. 2019
37
Explicit Encoding PL knowledge into model
Code Property Graph
38
Explicit Encoding – Take aways
Code
Modeling
• Precision
• Guarantee Pros
Explicit Implicit
Encoding Encoding • Explainable
Natural Channel
Mutation
• Model design overhead
Cons
Formal Channel • Transferability
Mutation
39
Implicit Encoding
Code
Modeling
Explicit Implicit Large Data Corpus
Encoding Encoding
Natural Channel
Mutation
Mutated Input De-Mutation
Formal Channel
Mutation
[11] CodeBERT – Feng et. al. 2019
[12] GraphCodeBERT – Guo et. al. 2020
[13] CodeX – Chen et. al. 2021
[14] CodeT5 – Wang et. al. 2021
40
Dual Channel Hypothesis for Source Code [15]
Natural Channel [16] Formal Channel
[15] Casalanuovo et. al. 2020
[16] Karampatsis et. al. 2020
41
Implicit Encoding
Code
Modeling
Explicit Implicit
Encoding Encoding
Natural Channel
Mutator
Mutation
Formal Channel
Mutation
42
PLBART: Unified pretraining Program “Understanding”
and “Generation”. (NAACL’21)
Implicit Encoding by mutating the natural channel
43
PLBART – What Is It?
Encoder Decoder
Pretraining – Unsupervised
way of learning code patterns.
44
PLBART – Components
1. Read Code.
2. Understands Code.
Encoder 3. Reason about any errors in code.
4. Learns robust representation
Decoder 1. Generate Code.
2. Learns Coding Patterns.
45
PLBART – Pretraining
Noise Injector
Correct Code Noisy Code
Decoder Encoder
PLBART
Token Masking Token Deletion Token Infilling
46
PLBART – Jointly Learning to Understand and
Generate
Token Masking
PLBART
47
PLBART – Jointly Learning to Understand and
Generate
Token Deletion
PLBART
48
PLBART – Jointly Learning to Understand and
Generate
Token Infilling
PLBART
49
PLBART – Pretraining
• Noise Properties • Training Objectives
• Mutate Natural Chennel • Generate the whole code.
• Likely to Break Syntax • Learns syntax implicitly
• Learn Coding Patterns.
• Multi-lingual Training
• Java
• Python
• NL from Stack overflow
50
PLBART – Experiment (Applications)
51
PLBART – Results – Code Generation from NL
Dataset: Concode
Metric: EM (exact match %), BLEU-4(max 100), CodeBLEU(max 100)
CodeBLEU =
0.25 * token_match +
0.25 * keywords_match +
0.25 * syntax_match +
0.25 * dataflow_match
52
PLBART – Search Augmented Code Generation
(EMNLP’21)
PLBART
✓ Up to 21% CodeBLEU Improvement Over GraphCodeBERT.
✓ Up to 12% improvement due to Search Augmentation.
53
PLBART – Multi Modal Code Edit (ASE’21)
PLBART
✓ Edit Location (Code to be edited), Context, and a Summary of the Edit.
✓ Up to 34% Improvement Over CodeBERT, and 30% Improvement over CodeGPT.
✓ Up to 16% improvement because of Multi-Modality.
54
PLBART – Results – Code Translation
Dataset: CodeXGLUE [Lu et. al. 2021]
Metric: EM (exact match %), BLEU-4(max 100), CodeBLEU(max 100)
CodeBLEU =
0.25 * token_match +
0.25 * keywords_match +
0.25 * syntax_match +
0.25 * dataflow_match
55
An interesting example of Code Translation
Input C# Code PLBART generated Better Code
56
Implicit Encoding
Code
Modeling
Explicit Implicit
Encoding Encoding
Natural Channel
Mutator
Mutation
Formal Channel
Mutation
59
NatGen:
Implicit Encoding by mutating the formal channel
60
NatGen: Generative pre-training by
“Naturalizing” source code
Write Semantic / Functional Equivalent Code in
“More Natural” way
61
NatGen: Generative pre-training by
“Naturalizing” source code
De-Naturalizing
Transformation
62
NatGen: De-Naturalizing Transformations
Confusing Statements [17]
Dead Code Insertion
Operand Swap
[17] Gopstein et. al. 2020
63
NatGen: Generative pre-training by
“Naturalizing” source code
Encoder Decoder
NatGen
64
NatGen: Some Initial Results – Few Shot Learning
65
Explicit Encoding – Take aways
Code
Modeling
• Little overhead
• Unsupervised Pros
Explicit Implicit
Encoding Encoding • Transferable
Natural Channel
Mutation • No Guarantee
• Potential Bias from Cons
Formal Channel
Mutation
Mutation
66
My Research Applications
Programming Language Processing
Code Summarization
NeuralCodeSum (ACL’20), PLBART (NAACL’21)
Vulnerability Detection
ReVeal(TSE’21), DISCO(ACL’22)
Code Editing
PLP
CODIT (TSE’20), MODIT(ASE’21), DiffBERT (Facebook)
Code Generation and Translation
PLBART (NAACL’21), SNG-PLBART(NAACL’22 - under
review), DataTypeLM ForCode (ACL’18).
Code Search and Synthesis
RedCoder (EMNLP’21), CodePanda (W.I.P)
67
What’s Next? (Short Term Goal)
➢ API driven Program Synthesis
➢ Improving Semantic Code Search with RL
Synthesizer
68
What’s Next? (Short Term Goal)
➢ Representing Code Context as Dynamic
Graph
➢ Learning Code Syntax and Semantics with
Reinforcement Learning (RL)
➢ Other Information Modalities
1. Other Software Metadata – Comments, Commits, Code Evolution metadata.
2. Analysis of Program Binaries.
3. Dynamic Analysis of Program – based on execution behavior.
69
What’s Next? (Long Term Goal)
➢ Code Generation
Formal Analysis Probabilistic models
Guarantee for the Analysis Scalable and Transferrable
Noise Intolerant No theoretical Guarantee
➢ Developer Feedback Oriented Automation
70
Baishakhi Ray
Adviser
Wasi Ahmad Miltos Allanmanis Kai-Wei Chang Yanju Chen Prem Devanbu
UCLA / Amazon MSR UCLA UCSB UC Davis
Yangruibo Ding Yu Feng Rahul Krishna Toufiq Parag Rizwan Parvez
Columbia UCSB Columbia / IBM UC Davis UCLA 71
72
