Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce universal compilation host to execution framework #15960

Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Introduce universal compilation host to execution framework #15960

wants to merge 5 commits into from

Conversation

erak
Copy link
Collaborator

@erak erak commented Mar 20, 2025

Description

This PR implements a universal compiler host for running tests with various compilers (etc. the internal compiler aka. CompilerStack or standard JSON compiler).

Details

It introduces some common types for the compiler inputs and its output. A compilation module that wraps the linked / internal compiler was added. It can be used in a newly added universal compiler host. This host is then used inside the test execution framework.

Outlook

The input / output types introduced by this PR were inspired by the SolidityCompilationFramework. In a future PR the universal compiler host could be used there as well.

Ultimately, this PR makes it possible to implement a compilation module for the standard JSON compiler as well and use that in the pipeline.

@github-actions GitHub Actions
Copy link

Thank you for your contribution to the Solidity compiler! A team member will follow up shortly.

If you haven't read our contributing guidelines and our review checklist before, please do it now, this makes the reviewing process and accepting your contribution smoother.

If you have any questions or need our help, feel free to post them in the PR or talk to us directly on the #solidity-dev channel on Matrix.

@erak erak force-pushed the test-compiler-host branch from 08ce0db to 9387714 Compare March 20, 2025 10:02
@erak erak force-pushed the test-compiler-host branch 3 times, most recently from 6a47a21 to 9e50f8c Compare March 25, 2025 11:10
@stackenbotten
Copy link

There was an error when running chk_coding_style for commit f79fbaed6453a28a25f6d1cd338074525c4359cf:

Error: Trailing whitespace found:
test/libsolidity/util/compiler/InternalCompiler.cpp:93:        
test/libsolidity/util/compiler/InternalCompiler.cpp:94:    
test/libsolidity/util/compiler/InternalCompiler.cpp:95:    
test/libsolidity/util/compiler/InternalCompiler.cpp:98:    //     std::cout << "sourceName: " << sourceName << std::endl; 
test/libsolidity/util/compiler/InternalCompiler.cpp:104:

Please check that your changes are working as intended.

@erak erak force-pushed the test-compiler-host branch from f79fbae to 3b47946 Compare March 26, 2025 09:30
@erak erak force-pushed the test-compiler-host branch 3 times, most recently from 9e5df30 to eb50f43 Compare March 26, 2025 16:03
@erak erak marked this pull request as ready for review March 26, 2025 18:39
@erak erak changed the title [WIP] Introduce universal compilation host to execution framework Introduce universal compilation host to execution framework Mar 26, 2025
@erak erak force-pushed the test-compiler-host branch from eb50f43 to 132c6f4 Compare March 26, 2025 23:44
clonker
clonker reviewed Apr 3, 2025
View reviewed changes
Copy link
Member

@clonker clonker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some things need to be ironed out still, on a small scale you should make a pass over the PR and fix all the autos to use auto const& and auto const* where appropriate. Or respective mutable reference/pointer types. Or nothing, if you explicitly want to make a copy :)
Also (but optionally), I personally like to make as much const as possible. For ex., if you have a bool shouldDoSomething = m_compiler.shouldDoSomething() will the sole purpose to pass it on and not modify it further, I declare it as bool const shouldDoSomething = ....
On the bigger scale of things, I am not a big fan of these optionals everywhere to capture experimental solidity in which some data members aren't available. It leads to a lot of bloat. Perhaps a variant with two clean distinct structs would be better. But not sure, really.

libsolidity/interface/CompilerStack.cpp
) const
{
std::vector<ContractDefinition const*> contracts;
if (auto source = m_sources.find(_sourceName); source != m_sources.end())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (auto source = m_sources.find(_sourceName); source != m_sources.end())
if (
auto const& source = m_sources.find(_sourceName);
source != m_sources.end()
)

aside from that, do you think it would make sense to assert that there actually is a contract definition with the source name instead of silently returning an empty vector?

test/CMakeLists.txt
Comment on lines +120 to +125
libsolidity/util/compiler/Compiler.h
libsolidity/util/compiler/Compiler.cpp
libsolidity/util/compiler/InternalCompiler.h
libsolidity/util/compiler/InternalCompiler.cpp
libsolidity/util/compiler/CompilerHost.h
libsolidity/util/compiler/CompilerHost.cpp
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you order this alphabetically? :)

test/libsolidity/GasCosts.cpp
@@ -38,7 +38,8 @@ namespace solidity::frontend::test
#define CHECK_DEPLOY_GAS(_gasNoOpt, _gasOpt, _evmVersion) \
do \
{ \
u256 metaCost = GasMeter::dataGas(m_compiler.cborMetadata(m_compiler.lastContractName()), true, _evmVersion); \
auto compiledContract = m_compiler.output().contract().value(); \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this unnecessarily copies around compile contract instances. I see you implemented a signature

std::optional<CompiledContract> contract(ContractName const& _name = {}) const;

Instead, I would implement it as CompiledContract const* contract(ContractName const& _name = {}) const;. Ultimately the compiled contract comes from the m_sourceUnits map so all you have to do is point to it if it exists, return nullptr otherwise. Then you can also write the following

Suggested change
auto compiledContract = m_compiler.output().contract().value(); \
auto const* compiledContract = m_compiler.output().contract().value(); \

test/libsolidity/GasCosts.cpp
@@ -202,8 +203,9 @@ BOOST_AUTO_TEST_CASE(single_callvaluecheck)
}
)";
compileAndRun(sourceCode);
size_t bytecodeSizeNonpayable = m_compiler.object("Nonpayable").bytecode.size();
size_t bytecodeSizePayable = m_compiler.object("Payable").bytecode.size();
auto compilerOutput = m_compiler.output();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
auto compilerOutput = m_compiler.output();
auto const& compilerOutput = m_compiler.output();

the way you did it will make a copy of the output

test/libsolidity/GasCosts.cpp
Comment on lines +207 to +208
size_t bytecodeSizeNonpayable = compilerOutput.contract("Nonpayable").value().object.size();
size_t bytecodeSizePayable = compilerOutput.contract("Payable").value().object.size();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not assert that "Nonpayable" and "Payable" even exist as far as i can tell

test/libsolidity/util/compiler/InternalCompiler.cpp
for (auto sourceName: m_stack.sourceNames())
{
std::vector<CompiledContract> compiled;
for (auto contract: m_stack.contractDefinitions(sourceName))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for (auto contract: m_stack.contractDefinitions(sourceName))
for (auto const& contract: m_stack.contractDefinitions(sourceName))

test/libsolidity/util/compiler/InternalCompiler.cpp
std::string contractName = contract->fullyQualifiedName();
LinkerObject object = m_stack.object(contractName);
LinkerObject runtimeObject = m_stack.runtimeObject(contractName);
bool hasUnlinkedReferences = !object.linkReferences.empty();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is unused

test/libsolidity/util/compiler/InternalCompiler.cpp
for (auto const& error: m_stack.errors())
{
auto formatted = SourceReferenceFormatter::formatErrorInformation(
*error.get(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
*error.get(),
*error,

test/libsolidity/util/compiler/InternalCompiler.cpp
std::vector<AnnotatedEventSignature> signatures;
ContractDefinition const& contract = m_stack.contractDefinition(_contractName);

for (EventDefinition const* event: contract.events() + contract.usedInterfaceEvents())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use ranges::views::concat here to avoid creating an entirely new vector

test/libsolidity/util/compiler/InternalCompiler.h
#pragma once

#include <liblangutil/SourceReferenceFormatter.h>
#include <libsolutil/Common.h>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this use anything from this header?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clonker clonker clonker left review comments

@stackenbotten stackenbotten stackenbotten left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
external contribution ⭐
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@erak @stackenbotten @clonker