0% found this document useful (0 votes)

21 views8 pages

Design2Code: Open-Source AI Matching Commercial Giants in Front-End Development

Dive into our latest article exploring Design2Code, an AI model developed by Microsoft and Google DeepMind. Learn how it’s revolutionizing front-end engineering by transforming visual designs into functional code, and matching the abilities of commercial models like Gemini Pro Vision.

Uploaded by

My Social

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

21 views8 pages

Design2Code: Open-Source AI Matching Commercial Giants in Front-End Development

Uploaded by

My Social

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 8

To read more such articles, please visit our blog https://socialviews81.blogspot.

com/

Design2Code: Open-Source AI Matching Commercial Giants

in Front-End Development

Introduction

Front-end engineering is the process of creating user interfaces for web

applications. It involves designing, coding, testing, and deploying web
pages that are responsive, interactive, and user-friendly. Front-end
engineering is a complex and time-consuming task that requires a lot of

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

skills and creativity. However, what if we could automate some or all of

these steps using artificial intelligence?

This is the vision behind Design2Code. Design2Code is a pioneering

system developed by a team of researchers from Stanford University,
Georgia Tech, Microsoft, and Google DeepMind. The project is a
testament to the rapid advancements in Generative AI, achieving
unprecedented capabilities in multimodal understanding and code
generation. The primary motivation behind the development of this
model was to enable a new paradigm of front-end development, where
multimodal Large Language Models (LLMs) could directly convert visual
designs into code implementations. The development of this model is an
attempt to explore the possibility of automating front-end engineering
and to bridge the gap between design and code.

What is Design2Code?

Design2Code is an innovative AI model and an open-source project that

has the capability to transform a given design image into HTML and CSS
code. The model takes an image of a web page design as input and
outputs the corresponding HTML and CSS code that can render the
same design in a browser. It can handle various design elements, such
as text, images, buttons, icons, layouts, colors, fonts, and styles.
Moreover, the model can generate responsive code that can adapt to
different screen sizes and devices.

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

Source - https://salt-nlp.github.io/Design2Code/

Key Features of Design2Code

Design2Code is a unique and powerful model with several key features

that make it stand out from other models that can generate code from
design images. Some of these features are:

● Code Generation from Design Images: One of the most

impressive features of Design2Code is its ability to automatically
generate HTML and CSS code from design images in a snap1.
The system takes care of the coding process, saving a lot of time
and effort for web developers and designers who want to create
web pages from their design sketches or mockups. Design2Code
can generate code that is faithful to the design image, as well as
responsive and functional.
● Compatibility with Various Design Formats: Another feature of
Design2Code is its compatibility with different design formats,
allowing users to upload their design files in various formats or
simply drag and drop them onto the platform2. This flexibility
makes it a versatile and handy tool for web developers and

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

designers who work with different design tools, such as

Photoshop, Sketch, Figma, or Adobe XD. Design2Code can
handle various design elements, such as text, images, buttons,
icons, layouts, colors, fonts, and styles.

Capabilities/Use Case of Design2Code

Design2Code has several capabilities and potential future use cases that
demonstrate its value for automating front-end engineering. Some of
these are:

● Democratizing Front-End Development: Design2Code has the

potential to democratize the development of front-end web
applications. It allows non-experts to build applications easily and
quickly. By converting design drafts into front-end code, it greatly
reduces the workload of developers. This tool is particularly useful
for those who have concrete ideas for what to build or design but
lack the sophisticated skills required for implementing visual
designs of websites into functional code.
● Rapid Prototyping: Design2Code could possibly help web
developers and designers to quickly create prototypes of web
pages from their design sketches or mockups. This can save a lot
of time and effort that would otherwise be spent on coding the web
pages manually. Design2Code can also help to validate the
feasibility and functionality of the design ideas and to get feedback
from the users or clients.
● Code Learning: Design2Code could also potentially help novice
web developers and learners to understand and learn how to code
web pages from design images. Design2Code can provide a visual
and interactive way of learning HTML and CSS by showing the
correspondence between the design elements and the code
snippets. Design2Code can also help to improve the coding skills

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

and knowledge of the learners by providing examples and

explanations of the generated code.
● Code Optimization: Design2Code could also potentially help to
optimize the code quality and performance of the web pages.
Design2Code can generate clean and concise code that follows
the best practices and standards of web development.
Design2Code can also generate responsive code that can adapt to
different screen sizes and devices, which can enhance the user
experience and accessibility of the web pages.

How does Design2Code work?

Design2Code uses multimodal Large Language Models (LLMs) to

transform visual designs into code implementations. It employs a set of
multimodal prompting methods, which have proven their effectiveness
on GPT-4V and Gemini Pro Vision.

The system features an open-source model, Design2Code-18B, that

rivals the performance of Gemini Pro Vision. The Design2Code-18B
model is designed to tackle the Design2Code task, which consists of
transforming visual designs of webpages into functional code
implementations. The model is built on the CogAgent-18B architecture
and is fine-tuned with synthetically generated Design2Code data. It
supports high-resolution input and is pretrained on extensive text-image
pairs, synthetic documents, LaTeX papers, and a small amount of
website data. The training data for the model is derived from the
Huggingface WebSight dataset, which contains website screenshot and
code implementation pairs. The model is fine-tuned using LoRA
modules, with a batch size of 32 and a learning rate of 1e-5. During
inference, the model uses a temperature of 0.5 and a repetition penalty
of 1.1. The model’s performance is measured based on a
comprehensive set of automatic metrics, including high-level visual

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

similarity and low-level element matching, to evaluate its ability to

precisely generate code implementations that render into the given
reference webpages.

Performance Comparison with Other Models

The performance comparison of the Design2Code model, especially the

open-source Design2Code-18B, is a crucial aspect of understanding its
abilities in automating front-end engineering tasks. The model has been
thoroughly compared with other multimodal language models (LLMs)
such as GPT-4V and Gemini Pro Vision.

source - https://arxiv.org/pdf/2403.03163.pdf

The comparison involved the creation of a comprehensive set of

automatic comparison metrics that capture both high-level visual
similarity and low-level element matching, supplemented by human
comparisons. The outcomes of the comparison indicate that the
Design2Code-18B model has shown competitive performance, matching
the abilities of commercial models such as Gemini Pro Vision. This
implies the potential of specialized “small” open models and skill

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

acquisition from synthetic data, as shown by the model’s performance on

the benchmark.

Design2Code has been benchmarked against a collection of 484 diverse

real-world webpages. Both human comparison and automatic metrics
show that GPT-4V performs the best on this task compared to other
models. In fact, in 49% of cases, annotators thought GPT-4V generated
webpages could replace the original reference webpages in terms of
visual appearance and content. This performance comparison highlights
the effectiveness of Design2Code-18B in the task of transforming visual
designs into code implementations.

How to Access and Use this Model?

Design2Code is open-source and accessible on GitHub. To use

Design2Code, users need to have an OpenAI account and enter their
API key (specifically GPT4 vision access) in the settings dialog. The
project details are displayed on the project’s webpage, and the dataset
used for the project can be located on Hugging Face.

If you are interested to learn more about this model, all relevant links are
provided under the 'source' section at the end of this article.

Limitations And Future Work

While Design2Code has made significant strides in automating front-end

engineering, there are areas where it can improve. For instance,
open-source models like Design2Code-18B mostly lag in recalling visual
elements from the input webpages and in generating correct layout
designs. However, aspects like text content and coloring can be
drastically improved with proper finetuning.

Looking ahead, Design2Code can serve as a useful benchmark to power

many future research directions. Some of these include:

To read more such articles, please visit our blog https://socialviews81.blogspot.com/

● Better Prompting Techniques: There’s room for improvement in

the prompting techniques for multimodal LLMs, especially in
handling complex webpages. For example, incrementally
generating different parts of the webpage could be a potential
approach.
● Training Open Multimodal LLMs with Real-World Webpages:
Preliminary experiments showed the difficulty of directly training on
real web pages since they are too long and noisy. Future work
could explore data cleaning pipelines to make such training stable.
● Extending Beyond Screenshot Inputs: There’s potential to
extend beyond screenshot inputs, for example, to collect Figma
frames or sketch designs from front-end designers as the test
input. Such extension also requires careful re-design of the
evaluation paradigm.
● Including Dynamic Webpages: Extending from static webpages
to also include dynamic webpages is another potential area of
improvement. This also requires the evaluation to consider
interactive functions, beyond just visual similarity.

Conclusion

Design2Code represents a significant step forward in the field of

front-end development. By leveraging the power of AI, it has the
potential to democratize web application development, making it
accessible to non-experts. While there are areas for improvement, the
system’s current capabilities are impressive, and it holds great promise
for the future.

Source
research paper: https://arxiv.org/abs/2403.03163
project details: https://salt-nlp.github.io/Design2Code/
Github Repo: https://github.com/NoviScl/Design2Code
Dataset: https://huggingface.co/datasets/SALT-NLP/Design2Code