2024 First IDE Workshop (IDE)

A New Generation of Intelligent Development Environments

Mark Marron
University of Kentucky
Lexington, USA
[email protected]

ABSTRACT 1 INTRODUCTION
The practice of programming is undergoing a revolution with the The last 5 years have seen a revolution in the practice of program-
introduction of AI assisted development (copilots) and the creation ming. The introduction of AI assisted programming (copilots) [4]
of new programming languages that are designed explicitly for has changed the way that developers work by providing a new way
tooling, analysis, and automation. Integrated Development Envi- to generate code. The rise of easy to use package ecosystems [10]
ronments (IDEs) as they are currently conceptualized have not yet has made it easier than ever to reuse existing code and to integrate
responded to these changes. They are still designed around the idea it into new applications. The introduction of new programming
of a human programmer typing textual code into an editor window languages designed explicitly for tooling, analysis, and automa-
with the IDE providing assistance via the integration of various tion [8] has made the concept of transparent program validation
tools for syntax highlighting, compilation, debugging, and (maybe) and analysis a real possibility. The rise of distributed cloud pro-
code version control. This paper presents a vision for transforming gramming, with serverless and REST based microservices, has has
the IDE from an Integrated Development Environment to an Intel- made it easier than ever to deploy and scale applications but created
ligent Development Environment. The new IDE will be designed a whole new ecosystem of monitoring and observability tools.
around the idea of a human programmer as the manager or curator These changes have fundamentally altered the way that devel-
of a software project who, rather than manually typing in code opers work. However, the Integrated Development Environments
to implement a solution, will instead use the IDE to direct AI pro- (IDEs) that developers use have not kept pace with these changes.
gramming agents and/or automated tools to combine existing APIs, They are still designed around the idea of a human programmer
packages, and new code to implement the needed features. In this typing textual code into an editor window with the IDE provid-
new model, the fundamental roles of the IDE are to 1) facilitate the ing some assistance via the integration of various tools for syntax
communication between the human programmer and the AI agents highlighting, compilation, debugging, and (maybe) code version
and automated tools and 2) organize the workflow tasks needed to control. This paper presents experience from working on these
go from requirements gathering to the final tested and validated de- technologies and a vision for transforming the IDE from an Inte-
ployed feature. This paper presents a vision for the new Intelligent grated Development Environment to an Intelligent Development
Development Environment based on a range of proof-of-concept Environment.
high-value scenarios we have experimented with and discusses the This new IDE will be designed around the idea of a human pro-
challenges that remain to realizing these in a cohesive intelligent grammer as the manager or curator of a software project who will
development experience. use the development environment to gather requirements directly
in a live and semantically rich way then to direct AI programming
agents and/or automated tools to combine existing APIs, packages,
and new code to implement the needed features. This environment
also supports new workflows for validating and resolving potential
CCS CONCEPTS issues in the code, before it is even integrated, ensuring that the
• Software and its engineering → General programming lan- final implementation satisfies key correctness, security, and perfor-
guages. mance requirements. Finally, in this new model, the development
environment can fully close the lifecycle loop by working as an
information broker between the developer and the deployed appli-
KEYWORDS
cation. This allows the developer to quickly understand the current
Interactivity, Development Environment, AI Assisted Programming health of the application from a holistic perspective, to act on issues
without requiring context switching between tools, and re-enter
ACM Reference Format: the requirements gather process seamlessly as new needs arise.
Mark Marron. 2024. A New Generation of Intelligent Development Environ-
ments. In 2024 First IDE Workshop (IDE ’24), April 20, 2024, Lisbon, Portugal.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3643796.3648452

This work licensed under Creative Commons Attribution International 4.0 License.

IDE ’24, April 20, 2024, Lisbon, Portugal

© 2024 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0580-9/24/04
https://doi.org/10.1145/3643796.3648452

43
IDE ’24, April 20, 2024, Lisbon, Portugal Mark Marron

Figure 1: Requirements Gathering Workflow

This paper presents a vision for the new Intelligent Development the developer and customer can interactively walk through them
Environment based on a range of proof-of-concept high-value sce- with different data scenarios and experiment with changes to the
narios we have experimented with and outlines the work needed UI and/or planned computation components!
to realize these features/scenarios in a cohesive development expe-
rience.
3 AI ASSISTED PROGRAMMING
Once the requirements have been gathered and the data has been
2 REQUIREMENTS GATHERING described the next step is to use AI agents to generate the code
A key task for developers in the new world of AI assisted program- needed to realize the requirements. This is done by using the data
ming is to gather requirements and translate them into actionable description to generate a set of data-specifications that describe the
tasks for the AI agents and automated tools. Our experiments with APIs and data that each step operates on – in our scenarios using
this task have identified a workflow based around Prorogued Pro- TypeSpec [12] or Bosque [3]. These specifications are then consum-
gramming [1], live UI mockups (using React [11]), and rich data able by various AI agents, as a Mixture of Experts (MoE) that can
description languages [3, 12]. generate code to implement the needed functionality. A first agent
In this workflow the developer interactively walks through sce- works to convert the flow diagram and data specifications into a set
narios with the customer to identify the overall workflows that of high-level APIs that can be used to implement the functionality.
should be supported using live UI mockups, written declaratively Individual coding agents can then be used to implement each of
in React, and fills in the details of the data that is read/written the APIs, with the MoE agent providing guidance and feedback to
in each step with mock values written in a rich data description the developer as they work.
language (TypeSpec [12]) as they are needed at each step. A key aspect of this workflow is that the developer is not writing
Figure 1 shows an example of this workflow. In this scenario a code, but instead is managing the AI agents and providing feedback
developer is working to implement a new Todo application. The to refine and correct as the code is generated. This is a key shift in
first action the customer wants on startup is to show the current the role of the developer from a programmer to a manager/curator
todo tasks in the database. The developer can add the edge to a and this shift in roles must be supported in the development envi-
flow diagram, and add some free form text about the requirement ronment as well!
+ where to find the database and what login credentials should be A first step in this direction is to support the automatic conver-
used. The developer can then ask the customer what the data should sion of the sample data gathered as part of the requirements into
look like and fill in the mock information, ideally with extensive unit tests that can reject obviously incorrect code. Another step is
autocompletion support if the format is available using techniques to use the API and data specifications to generate a set of sanity
like [3, 12]. From this data a quick React UI component can be fuzz inputs to catch and correct any hard failures as the AI agents
written and the customer can interactively walk through the flow are working. However, the most important step is to provide direct
to see if it matches their expectations. Further, as these flows are support for the developer when they need to step in and provide
now captured in live-executable flow diagrams and UI components feedback in situations where there is ambiguity in the requirements

44
A New Generation of Intelligent Development Environments IDE ’24, April 20, 2024, Lisbon, Portugal

Figure 3: Example of automatically identifying possible run-

time errors.

Figure 2: Disambiguation Support

and the AI agents have identified multiple possible solutions. In

these cases simply asking the developer to pick the correct solution be able to automatically generate this information – and ideally
by code inspection is not a viable option as there may be many automatically validate higher level issues as well including, if an
possible solutions, with 100’s of lines each, and small differences API is used correctly or if there are possible runtime errors [8] or
between versions! Instead, the IDE can provide features [9] that if SemVer [6, 7] information is being updated consistently with a
show outputs which differ between the solutions, highlight key code change.
algorithmic differences in solutions, or that paraphrase key infor- Figure 3 shows the running example code that has been gener-
mation back in natural language. These enable the developer to ated by the AI agents, and after disambiguation from other options,
quickly identify the ambiguity and to provide feedback to the AI selected by the developer. In this case the developer has asked the
agents to resolve it. IDE to check if there are any possible runtime errors in the code.
An example of this is shown in Figure 2. In this example the As the code is written in Bosqe [5, 8] the IDE can use a static
developer has asked the AI agents to generate a set of programs validator to solve for possible failing inputs, or prove that none
that implement the functionality described by the natural language exist, for code like this. In this case there is a possible runtime error
“find the max pair for the lists”, the signature given by the API as the argMax function may be called with an empty list and the
description, and a single example/test case [[1, 2], [2, 3]] => checker can automatically find this case for the developer!
[2, 3]. From this set of requirements an AI agent might generate However, it is critical that the development environment provide
two plausible solutions, one that is the max of the individual lists more than just red squiggles to indicate possible errors. Instead, we
[max(x), max(y)] and that is the max sum of the index-wise pairs want to be able to provide the full input that causes the failure, key
zip(x, y).argMax(fn(v) => v.0 + v.1). points in the code that are related, and ideally a direct way to jump
Even for this simple code, manual inspection requires some into a debugger experience [2] to fully investigate the issue. In this
effort to identify the differences between the two solutions and case the IDE can show the input that causes the failure, the line
think about which one might be the intended one. However, the of code that fails, and even suggest fixes to the spec to resolve the
IDE can provide support to make this easier. In this case the IDE issue.
can show an input on which the two solutions differ and the output Figure 4 shows the case where we can automatically identify
generated by each solution. In this case the developer, or even a and suggest refinements to the specifications to resolve the issue.
non-expert customer, quickly identify which result is the desired In this example there are two options which the developer may
one, and thus, which is the correct solution. Having IDE support in want to consider, one adds a requirement that the inputs cannot
the form of overlays, side-by-side views with highlighting, or other be empty lists, requires !x.empty() && !y.empty(), while the
UX affordances to highlight the key differences between solutions other relaxes the output type to include a none value for this special
can make this process even easier. case. With support for the development environment to show these
choices, and ideally the impacts they have on other parts of the
4 DEEP TOOLING SUPPORT codebase and semantic versioning information, the developer can
A key aspect of the new Intelligent Development Environment is quickly make an informed decision about which option to select.
that it is designed to support the developer as a manager/curator of While these examples have focused on correctness and runtime
the AI agents and automated tools. In this role a developer needs to errors, the same approach can be used to provide support for other
quickly understand the behavior of AI generated code or imported aspects of the development process. A key issue in many organiza-
packages and validate that they are being used correctly. This re- tions is discoverability of APIs and packages. Using many of the
quires a new generation of deep tooling support that can provide same techniques, and richer specification languages [3, 12], the IDE
the developer with the information they need. Instead of simply can provide support for semantically searching package databases
providing a list of methods, types, and use/def sites that a devel- for relevant APIs, types, and services. Being able to easily surface
oper must manually investigate to understand the requirements of these into the development environment can make it easier for
an API call or what properties a type guarantees the IDE should developers to reuse and leverage existing code and packages.

45
IDE ’24, April 20, 2024, Lisbon, Portugal Mark Marron

non-code, or non-traditional code, artifact such as natural

language, flow diagrams, partial UI components, history of
disambiguated code options, etc. This requires the IDE to be
able to store and organize these artifacts in a first-class way
– not just treat them as random text files or blobs.
• Programming Language Support – Many of the automa-
tion features described in this work require a new generation
of programming languages, such as the Bosque work [3, 8] or
TypeSpec [12], that are designed to support tooling, analysis,
and automation. As a result there is a feedback loop between
development environment and programming language de-
Figure 4: Example of automatically suggesting revisions to sign that needs to be explored.
the specifications. • Rich Overlay or Specialized Modes – The new Intelligent
Development Environment will need to support a range
5 INFORMATION BROKER of new workflows. Our experience with various proof-of-
At this point in out hypothetical application (or feature) develop- concept workflows has shown that these workflows do not
ment process we can imagine that the developer has finally de- fit well into the traditional model of a tree of text files with a
ployed the feature. Today, this is where the development process single editor window. Instead, we need to explore new ways
ends and the operations process begins. However, in the new world to organize and present information to the developer. This
of an Intelligent Development Environment we envision that the may involve rich overlays, specialized modes, and even new
IDE will be a one-stop-shop for all information about the applica- UI paradigms.
tion that is currently spread across logs, APM monitoring tools, In this future an Intelligent Development Environment will be
SCM activities, and even telemetry and crash reports! more than just a combination of a text editor for a developer to type
This information allows a developer to quickly understand the code into and some integration of various external, build, test, and
current health of the application in a holistic way and act on issues debugging tools. Instead, it will be a new kind of development expe-
without requiring context switching between tools. For example, rience that is designed around the idea of a human programmer as
if a developer is notified of a crash report they can quickly jump the manager/curator of a software project who use the system to or-
to the code line where the error is reported, see information on ganize, explore, orchestrate, and validate the software development
the number of times it has occurred, know about recent changes process.
relevant code, and even jump directly into a debuggable recorded These areas present a rich set of research challenges for the com-
trace (if Time-Travel-Tracing is supported [2]) to investigate the munity to explore. However, we believe that the potential benefits
issue. of this work are substantial and will result in a new kind of devel-
Closing the loop back to the requirements gathering phase this opment experience that is more productive, more accessible, and
integration provides a natural way to gather feedback from the more enjoyable for developers and customers alike!
customer and identify new requirements, to fix bugs, improve per-
formance, or link open feature requests into the development envi- REFERENCES
ronment. In our opening Todo example (Figure 1) we can imagine [1] Mehrdad Afshari, Earl T. Barr, and Zhendong Su. Liberating the programmer
that the customer has reported a bug where the application is not with prorogued programming. In Onward!, 2012.
[2] Earl T. Barr, Mark Marron, Ed Maurer, Dan Moseley, and Gaurav Seth. Time-
showing the correct data when there is a failure to connect to the Travel Debugging for Javascript/Node.js. In FSE, 2016.
database. The developer can quickly jump to the code that is re- [3] Bosque object notation api. https://github.com/BosqueLanguage/BSQON, 2023.
[4] Copilot. https://github.com/features/copilot, 2023.
sponsible for this behavior, see the number of times it has occurred, [5] Stephen Goldbaum, Attila Mihaly, Tosha Ellison, Earl T. Barr, and Mark Marron.
triage the issue, and start modifying the requirements information High assurance software for financial regulation and business platforms. In
to add error handling code. This then feeds into a new loop of AI VMCAI, 2022.
[6] Patrick Lam, Jens Dietrich, and David J. Pearce. Putting the semantics into
assisted programming, validations, and then redeployment! semantic versioning. In Onward!, 2020.
[7] Mark Marron. Agile and dependable service development with Bosque and
6 ONWARD! Morphir. Linux Foundation Open Source Summit, 2021.
[8] Mark Marron. Toward programming languages for reasoning: Humans, symbolic
Our experience with the scenarios and piece-wise implementations systems, and AI agents. In Onward!, 2023.
of the new Intelligent Development Environment has shown that [9] Mikaël Mayer, Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron, Oleksandr
Polozov, Rishabh Singh, Benjamin Zorn, and Sumit Gulwani. User interaction
there is great potential for this new model of development. How- models for disambiguation in programming by example. In UIST, 2015.
ever, there is substantial work to be done to realize this vision. In [10] Npm. https://www.npmjs.com/, 2023.
particular, many of the steps require: [11] React. https://microsoft.github.io/typespec/, 2023.
[12] Typespec. https://microsoft.github.io/typespec/, 2023.
• Extending the Environment Beyond Code – Many of
the activities described in this work involve working with

46