ML Project

Our goal is to automate the front-end web development process by taking a design as input and generating the equivalent HTML code output. This would streamline the design workflow for businesses. We are inspired by image captioning models which connect images and text. Our proposed model uses a CNN to extract features from the input design, GRUs to encode the code tokens, and a decoder GRU to generate the HTML code output one token at a time. We will use tools like Jupyter Notebook, TensorFlow, Keras and evaluate models like RNNs, GRUs and LSTMs.

Uploaded by

RAGURAM SHANMUGAM

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPTX, PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

95 views13 pages

ML Project

Uploaded by

RAGURAM SHANMUGAM

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPTX, PDF, TXT or read online on Scribd

You are on page 1/ 13

CS 6301

MACHINE LEARNING PROJECT

UI DEVELOPMENT
AUTOMATION
Raguram S - 2018503052
Barath S - 2018503510
Jaivishnu S - 2018503524
Sadheesh N - 2018503548
• ABSTRACT
• MOTIVATION
• TODAY’S WORKFLOW
• INSPIRATION
• PROPOSED WORK
• PROPOSED WORK EXAMPLE
• PROPOSED MODEL
• PROPOSED MODEL EXAMPLE
• TOOLS USED
• REFERENCES
Agenda /
Topics
The automation of the front-end development process is a
relatively new research area which has gained more focus in
the recent years.

Writing HTML code for each and every small page is a time
consuming process.

ABSTRACT This can be a potential candidate for automation.

Our work involves getting a design as input which spits an

equivalent looking webpage.

Our goal at insight was to use natural language processing

algorithms to significantly streamline the design workflow and
empower any business to quickly create and test web pages.
MOTIVATION

• Creating intuitive and engaging experiences for users is a critical goal for
companies of all sizes, and it’s a process driven by quick cycles of
prototyping, designing, and user testing.
• Large corporations like Facebook have the bandwidth to dedicate entire
teams to the design process, which can take several weeks and involve
multiple stakeholders; small businesses don’t have these resources, and their
user interfaces may suffer as a result.
• Our goal at Insight was to use modern deep learning algorithms to
significantly streamline the design workflow and empower any business to
quickly create and test web pages.
• This provides us a motivation to automate the development process.
TODAY’S DEVELOPMENT WORKFLOW

• A typical design workflow might look like the following:

• Product managers perform user research to generate a list of
specifications
• Designers take those requirements and explore low-fidelity
prototypes, eventually creating high-fidelity mockups
• Engineers implement those designs into code and finally ship the
product to users
• The length of the development cycle can quickly turn into a
bottleneck, and companies like Airbnb have started to use
machine learning to make this process more efficient.
INSPIRATION

• The problem we are solving falls under a broader umbrella of tasks known as
program synthesis, the automated generation of working source code.
• Though much of program synthesis deals with code generated from natural
language specifications or execution traces, in my case I could leverage the fact
that I had a source image (the hand-drawn wireframe) to start with.
• There’s a well-studied domain in machine learning called image captioning that
seeks to learn models that tie together images and text, specifically to
generate descriptions of the contents of a source image.
IMAGE CAPTIONING
(Inspiration for Program Synthesis)
PROPOSED WORK

WRITING HTML CODE FOR OUR WORK INVOLVES OUR GOAL AT INSIGHT WE DECIDED TO REFRAME THE PROBLEM WE ARE
EACH AND EVERY SMALL GETTING A DESIGN AS WAS TO USE NATURAL MY TASK INTO ONE OF SOLVING FALLS UNDER A
PAGE IS A TIME INPUT WHICH SPITS AN LANGUAGE PROCESSING IMAGE CAPTIONING, WITH BROADER UMBRELLA OF
CONSUMING PROCESS. EQUIVALENT LOOKING ALGORITHMS TO THE DRAWN WEBSITE TASKS KNOWN AS
THIS CAN BE A POTENTIAL WEBPAGE. SIGNIFICANTLY WIREFRAME AS THE INPUT PROGRAM SYNTHESIS, THE
CANDIDATE FOR STREAMLINE THE DESIGN IMAGE AND ITS AUTOMATED GENERATION
AUTOMATION. WORKFLOW AND CORRESPONDING HTML OF WORKING SOURCE
EMPOWER ANY BUSINESS CODE AS ITS OUTPUT CODE USING AN
TO QUICKLY CREATE AND TEXT. ENCODER-DECODER
TEST WEB PAGES. ARCHITECTURE.
PROPOSED WORK EXAMPLE
PROPOSED MODEL
• The model architecture used in image captioning that consists of three major
parts:
• A computer vision model that uses a Convolutional Neural Network (CNN) to
extract image features from the source images
• A language model consisting of a Gated Recurrent Unit (GRU) that encodes
sequences of source code tokens
• A decoder model (also a GRU), which takes in the output from the previous two
steps as its input, and predicts the next token in the sequence
• We could use different encoder and decoder models to gain better precision
and accuracy and to tune the hyperparameters. Such as
• Naive - RNN
• GRU
• LSTM
PROPOSED MODEL DIAGRAM
TOOLS & LIBRARIES USED

TOOLS USED: LIBRARIES USED:

• Jupyter Notebook • OpenCV
• Google Colab • PIL library
• Web Browser • Tensorflow
• Keras
• IDE
• PyTorch
• GitHub
• Numpy
• SciPy
REFERENCES

Papers: Blogs:

https://www.researchgate.net/publication/32 https://webandcrafts.com/blog/ai-machine-le
5920827_pix2code_Generating_Code_from_a_ arning-impact-front-end-development/
Graphical_User_Interface_Screenshot

https://ieeexplore.ieee.org/iel7/9023468/903 https://medium.com/@Jessicawlm/how-ai-an
3528/09033956.pdf d-machine-learning-are-transforming-front-en
d-development-db74523fd46b