Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compilation to NodeJS module #111

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
5ac67bd
Initial wasm support
smiasojed Aug 29, 2024
88a888d
Merge remote-tracking branch 'origin/main' into resolc.js
smiasojed Nov 6, 2024
9b23e19
Fix Cargo.toml
smiasojed Nov 7, 2024
4f6debc
Make native version to compaile again
smiasojed Nov 7, 2024
d260472
Fmt
smiasojed Nov 7, 2024
b7b28ef
Update solc compiler version check
smiasojed Nov 7, 2024
b6baf6c
Make wasm version to compile
smiasojed Nov 8, 2024
a934ec2
Add temoprary wasm compilation output
smiasojed Nov 8, 2024
94445ba
Fix compilation for wasm target
smiasojed Nov 8, 2024
c51d50b
Add GHA for wasm target
smiasojed Nov 8, 2024
c0a82ce
Update wasm GHA
smiasojed Nov 8, 2024
8c7d18a
Update GHA for wasm build
smiasojed Nov 8, 2024
007b79e
Use bash in GHA
smiasojed Nov 8, 2024
1837643
Fix GHA emsdk path
smiasojed Nov 8, 2024
7f3d0ce
Update cmake versions in GHA
smiasojed Nov 8, 2024
9a8003a
Install llvm in GHA
smiasojed Nov 8, 2024
2551769
Remove old llvm-15 from GHA
smiasojed Nov 8, 2024
90423ff
Remove llvm-15 env from GHA
smiasojed Nov 8, 2024
677aedc
Switch GHA to ubuntu
smiasojed Nov 8, 2024
8a22587
Fix deps in GHA
smiasojed Nov 8, 2024
93d2f3b
Log LLVM version in GHA
smiasojed Nov 12, 2024
6e6fe20
Add Missing dep to GHA
smiasojed Nov 12, 2024
f59b47d
Add ltinfo dep to GHA
smiasojed Nov 12, 2024
64fefe7
Removed not needed libs from linking process
smiasojed Nov 12, 2024
6d16790
Update GHA
smiasojed Nov 12, 2024
14991f4
Fix CI for wasm path
smiasojed Nov 13, 2024
ce8bf3d
Apply suggestions from previous review
smiasojed Nov 14, 2024
563864d
Fix CI
smiasojed Nov 14, 2024
010a2ed
Fmt
smiasojed Nov 14, 2024
140545e
Fix CI
smiasojed Nov 14, 2024
881b883
Rename compiler to solc
smiasojed Nov 14, 2024
f57ab96
Revert changes in llvm build
smiasojed Nov 15, 2024
cece20d
Cleanup
smiasojed Nov 18, 2024
87dd77b
Remove libsolc crate
smiasojed Nov 18, 2024
d88ddb2
Refactor soljson compiler
smiasojed Nov 18, 2024
3072c03
Fix parallel feature
smiasojed Nov 18, 2024
63da721
Add worker code to resolc.js
smiasojed Nov 18, 2024
39e5047
Add test with standard json args
smiasojed Nov 18, 2024
f36d62c
Add stdin support
smiasojed Nov 20, 2024
cdebf69
Fix stdin support
smiasojed Nov 20, 2024
7740626
Improve description
smiasojed Nov 20, 2024
fcbe00f
Merge branch 'main' into resolc.js
smiasojed Nov 20, 2024
4918507
Update readme file
smiasojed Nov 20, 2024
0a82add
Add directory arg support to clone-llvm
smiasojed Nov 20, 2024
a772ff1
Fix build path
smiasojed Nov 21, 2024
32d4b23
Fix build script
smiasojed Nov 21, 2024
8990b2a
Update emscripten-build-llvm.sh
smiasojed Nov 21, 2024
05925f2
Update comment
smiasojed Nov 21, 2024
892c9b5
Update clone-llvm.sh
smiasojed Nov 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/build-revive-wasm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Build revive-wasm
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
workflow_dispatch:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the manual dispatch even though we build for every PR and master commit?

Copy link
Collaborator Author

@smiasojed smiasojed Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it uses 1day retention period, I left it in case someone would like to recreate artefacts.


env:
CARGO_TERM_COLOR: always
REVIVE_WASM_INSTALL_DIR: ${{ github.workspace }}/target/wasm32-unknown-emscripten/release
EMSCRIPTEN_VERSION: 3.1.64

jobs:
build-revive-wasm:
runs-on: ubuntu-latest
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v4

- name: Cache LLVM build
id: cache-llvm
uses: actions/cache@v3
with:
path: |
llvm18.0-emscripten
# Use a unique key based on LLVM version or configuration files to avoid cache invalidation
key: llvm-build-${{ runner.os }}-${{ hashFiles('clone-llvm.sh', 'emscripten-build-llvm.sh') }}

- name: Install Dependencies
run: |
sudo apt-get update && sudo apt-get install -y cmake ninja-build libncurses5
rustup target add wasm32-unknown-emscripten
# Install LLVM required for the compiler runtime, runtime-api and stdlib
curl -sSL --output llvm.tar.xz https://github.com/llvm/llvm-project/releases/download/llvmorg-18.1.4/clang+llvm-18.1.4-x86_64-linux-gnu-ubuntu-18.04.tar.xz
tar Jxf llvm.tar.xz
mv clang+llvm-18.1.4-x86_64-linux-gnu-ubuntu-18.04 llvm18/
echo "$(pwd)/llvm18/bin" >> $GITHUB_PATH
athei marked this conversation as resolved.
Show resolved Hide resolved
# Install Emscripten
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install ${{ env.EMSCRIPTEN_VERSION }}
./emsdk activate ${{ env.EMSCRIPTEN_VERSION }}
Comment on lines +41 to +45
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check out a specific tag here. Otherwise we build against a different version every time.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are using the version defined in EMSCRIPTEN_VERSION env which is set now to the rev fd61bacaf40131f74987e649a135f1dd559aff60


- run: |
rustup show
cargo --version
rustup +nightly show
cargo +nightly --version
cmake --version
bash --version
llvm-config --version

- name: Build LLVM
if: steps.cache-llvm.outputs.cache-hit != 'true'
run: |
export EMSDK_ROOT=${PWD}/emsdk
./emscripten-build-llvm.sh

- name: Use Cached LLVM
if: steps.cache-llvm.outputs.cache-hit == 'true'
run: |
echo "Using cached LLVM"

- name: Build revive
run: |
export LLVM_LINK_PREFIX=${PWD}/llvm18.0-emscripten
source ./emsdk/emsdk_env.sh
make install-wasm

- uses: actions/upload-artifact@v4
with:
name: revive-wasm
path: |
${{ env.REVIVE_WASM_INSTALL_DIR }}/resolc.js
${{ env.REVIVE_WASM_INSTALL_DIR }}/resolc.wasm
Comment on lines +76 to +78
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are those all the files that are needed? What about the hand written javascript files in /js? Or are you planning to do the full npm package release work flow as follow up?

Copy link
Collaborator Author

@smiasojed smiasojed Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add npm package with its workflow in next PR. Now we need these 2 files only.

retention-days: 1
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,10 @@
/*.s
/llvm-project
/llvm18.0
/llvm18.0-emscripten
node_modules
artifacts
tmp
package-lock.json
/*.html
/build
/js/resolc.*
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,10 +1,25 @@
.PHONY: install format test test-solidity test-cli test-integration test-workspace clean docs docs-build

RUSTFLAGS_EMSCRIPTEN := \
-Clink-arg=-sEXPORTED_FUNCTIONS=_main,_free,_malloc \
-Clink-arg=-sNO_INVOKE_RUN \
-Clink-arg=-sEXIT_RUNTIME \
-Clink-arg=-sINITIAL_MEMORY=64MB \
-Clink-arg=-sTOTAL_MEMORY=3GB \
-Clink-arg=-sALLOW_MEMORY_GROWTH \
-Clink-arg=-sEXPORTED_RUNTIME_METHODS=FS,callMain,stringToNewUTF8,cwrap \
-Clink-arg=-sMODULARIZE \
-Clink-arg=-sEXPORT_ES6 \
-Clink-arg=--js-library=js/soljson_interface.js

install: install-bin install-npm

install-bin:
cargo install --path crates/solidity

install-wasm:
RUSTFLAGS='$(RUSTFLAGS_EMSCRIPTEN)' cargo install --target wasm32-unknown-emscripten --path crates/solidity

install-npm:
npm install && npm fund

Expand Down
20 changes: 19 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,25 @@ resolc --version

### LLVM

`revive` requires a build of LLVM 18.1.4 or later including `compiler-rt`. Use the provided [build-llvm.sh](build-llvm.sh) build script to compile a compatible LLVM build locally in `$PWD/llvm18.0` (don't forget to add that to `$PATH` afterwards).
`revive` requires a build of LLVM 18.1.4 or later including `compiler-rt`. Use the provided [build-llvm.sh](build-llvm.sh) build script to compile a compatible LLVM build locally in `$PWD/llvm18.0` (don't forget to add that to `$PATH` afterwards).

### Cross-compilation to WASM

Cross-compiles the Revive recompiler to WASM to enable it to run in a Node.js or browser environment.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Cross-compiles the Revive recompiler to WASM to enable it to run in a Node.js or browser environment.
Cross-compiles the revive compiler to Wasm for running it in a Node.js or browser environment.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


Install [emscripten](https://emscripten.org/docs/getting_started/downloads.html). Tested on version 3.1.64.
To build resolc.js execute:

```bash
bash build-llvm.sh
export PATH=${PWD}/llvm18.0/bin:$PATH
export EMSDK_ROOT=<PATH_TO_EMSCRIPTEN_SDK>
bash emscripten-build-llvm.sh
source $EMSDK_ROOT/emsdk_env.sh
export LLVM_LINK_PREFIX=${PWD}/llvm18.0-emscripten
export PATH=$PATH:$PWD/llvm18.0-emscripten/bin/
make install-wasm
```

### Development

Expand Down
11 changes: 3 additions & 8 deletions build-llvm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,12 @@ set -euo pipefail
INSTALL_DIR="${PWD}/llvm18.0"
mkdir -p ${INSTALL_DIR}


# Clone LLVM 18 (any revision after commit bd32aaa is supposed to work)
if [ ! -d "llvm-project" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should clone into $LLVM_SRC_PREFIX instead of a hardcoded path (if it goes into a dedicated script it should be an argument instead)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

git clone --depth 1 --branch release/18.x https://github.com/llvm/llvm-project.git
fi

./clone-llvm.sh

# Build LLVM, clang
LLVM_SRC_PREFIX=${PWD}/llvm-project
LLVM_SRC_DIR=${LLVM_SRC_PREFIX}/llvm
LLVM_BUILD_DIR=${PWD}/build/llvm
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this changed? Before the target build directory was outside the llvm-project source directory

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have aligned it to this what i have in emscripten build:
LLVM_NATIVE=$LLVM_SRC/build-native
LLVM_WASM=$LLVM_SRC/build-wasm
We can reorganise it if you want

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please I think it is a standard practice to keep the build outside the source, i.e. leave it as it is now.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

LLVM_BUILD_DIR=${LLVM_SRC_PREFIX}/build/llvm
if [ ! -d ${LLVM_BUILD_DIR} ] ; then
mkdir -p ${LLVM_BUILD_DIR}
fi
Expand All @@ -38,7 +33,7 @@ cmake --install ${LLVM_BUILD_DIR}

# Build compiler builtins
COMPILER_RT_SRC_DIR=${LLVM_SRC_PREFIX}/compiler-rt
COMPILER_RT_BUILD_DIR=${PWD}/build/compiler-rt
COMPILER_RT_BUILD_DIR=${LLVM_SRC_PREFIX}/build/compiler-rt
if [ ! -d ${COMPILER_RT_BUILD_DIR} ] ; then
mkdir -p ${COMPILER_RT_BUILD_DIR}
fi
Expand Down
6 changes: 6 additions & 0 deletions clone-llvm.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/bin/bash
smiasojed marked this conversation as resolved.
Show resolved Hide resolved

# Clone LLVM 18 (any revision after commit bd32aaa is supposed to work)
if [ ! -d "llvm-project" ]; then
git clone --depth 1 --branch release/18.x https://github.com/llvm/llvm-project.git
fi
80 changes: 72 additions & 8 deletions crates/lld-sys/build.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,19 @@
fn llvm_config(arg: &str) -> String {
let output = std::process::Command::new("llvm-config")
use std::{
env,
path::{Path, PathBuf},
};

const LLVM_LINK_PREFIX: &str = "LLVM_LINK_PREFIX";

fn locate_llvm_config() -> PathBuf {
let prefix = env::var_os(LLVM_LINK_PREFIX)
.map(|p| PathBuf::from(p).join("bin"))
.unwrap_or_default();
prefix.join("llvm-config")
}

fn llvm_config(llvm_config_path: &Path, arg: &str) -> String {
let output = std::process::Command::new(llvm_config_path)
.args([arg])
.output()
.unwrap_or_else(|_| panic!("`llvm-config {arg}` failed"));
Expand All @@ -8,8 +22,11 @@ fn llvm_config(arg: &str) -> String {
.unwrap_or_else(|_| panic!("output of `llvm-config {arg}` should be utf8"))
}

fn set_rustc_link_flags() {
println!("cargo:rustc-link-search=native={}", llvm_config("--libdir"));
fn set_rustc_link_flags(llvm_config_path: &Path) {
println!(
"cargo:rustc-link-search=native={}",
llvm_config(llvm_config_path, "--libdir")
);

for lib in [
"lldELF",
Expand All @@ -22,27 +39,74 @@ fn set_rustc_link_flags() {
"LLVMTargetParser",
"LLVMBinaryFormat",
"LLVMDemangle",
// Required by `llvm-sys`
"LLVMRISCVDisassembler",
xermicus marked this conversation as resolved.
Show resolved Hide resolved
"LLVMRISCVAsmParser",
"LLVMRISCVCodeGen",
"LLVMRISCVDesc",
"LLVMRISCVInfo",
"LLVMExecutionEngine",
"LLVMOption",
"LLVMMCDisassembler",
"LLVMPasses",
"LLVMHipStdPar",
"LLVMCFGuard",
"LLVMCoroutines",
"LLVMipo",
"LLVMVectorize",
"LLVMInstrumentation",
"LLVMFrontendOpenMP",
"LLVMFrontendOffloading",
"LLVMGlobalISel",
"LLVMAsmPrinter",
"LLVMSelectionDAG",
"LLVMCodeGen",
"LLVMTarget",
"LLVMObjCARCOpts",
"LLVMCodeGenTypes",
"LLVMIRPrinter",
"LLVMScalarOpts",
"LLVMInstCombine",
"LLVMAggressiveInstCombine",
"LLVMTransformUtils",
"LLVMBitWriter",
"LLVMAnalysis",
"LLVMProfileData",
"LLVMDebugInfoDWARF",
"LLVMObject",
"LLVMMCParser",
"LLVMIRReader",
"LLVMAsmParser",
"LLVMMC",
"LLVMDebugInfoCodeView",
"LLVMBitReader",
"LLVMRemarks",
"LLVMBitstreamReader",
] {
println!("cargo:rustc-link-lib=static={lib}");
}

#[cfg(target_os = "linux")]
{
let target_os = std::env::var("CARGO_CFG_TARGET_OS").unwrap_or_default();
if target_os == "linux" {
xermicus marked this conversation as resolved.
Show resolved Hide resolved
println!("cargo:rustc-link-lib=dylib=stdc++");
println!("cargo:rustc-link-lib=tinfo");
}
}

fn main() {
llvm_config("--cxxflags")
println!("cargo:rerun-if-env-changed={}", LLVM_LINK_PREFIX);

xermicus marked this conversation as resolved.
Show resolved Hide resolved
let llvm_config_path = locate_llvm_config();

llvm_config(&llvm_config_path, "--cxxflags")
.split_whitespace()
.fold(&mut cc::Build::new(), |builder, flag| builder.flag(flag))
.flag("-Wno-unused-parameter")
.cpp(true)
.file("src/linker.cpp")
.compile("liblinker.a");

set_rustc_link_flags();
set_rustc_link_flags(&llvm_config_path);

println!("cargo:rerun-if-changed=build.rs");
}
15 changes: 13 additions & 2 deletions crates/solidity/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ thiserror = { workspace = true }
anyhow = { workspace = true }
which = { workspace = true }
path-slash = { workspace = true }
rayon = { workspace = true }
rayon = { workspace = true, optional = true }

serde = { workspace = true }
serde_json = { workspace = true }
Expand All @@ -41,6 +41,17 @@ inkwell = { workspace = true }
revive-common = { workspace = true }
revive-llvm-context = { workspace = true }


[target.'cfg(target_env = "musl")'.dependencies]
mimalloc = { version = "*", default-features = false }

[target.'cfg(target_os = "emscripten")'.dependencies]
libc = { workspace = true }
inkwell = { workspace = true, features = ["target-riscv", "llvm18-0-no-llvm-linking"]}

[features]
default = []
parallel = ["rayon"]

# Enable parallel by default only for non-emscripten targets
[target.'cfg(not(target_os = "emscripten"))'.features]
default = ["parallel"]
Loading
Loading