Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement support for running MetaCall from python.exe and node.exe. #533

Merged
merged 69 commits into from
Feb 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ac3c3f6
Initial commit, add base for NodeJS and constructor for MetaCall, sti…
viferga Nov 21, 2024
3ed5d23
Implemented cwd and options map.
viferga Nov 21, 2024
8e8fe8b
Add functions for detecting host from loader side.
viferga Nov 22, 2024
355c12f
Improve dynlink, trying to hook dlsym.
viferga Nov 27, 2024
cbbf4a4
Solve issues from detours, plugin manager, fork safety and added supp…
viferga Nov 28, 2024
1927c62
Unified all detours modules, add a destructor in metacall module for …
viferga Nov 28, 2024
d9604dd
Implemented metacall link table.
viferga Nov 28, 2024
83de553
Initialization working properly on NodeJS.
viferga Dec 4, 2024
7779200
Add delayed execution paths in NodeJS.
viferga Dec 4, 2024
55a8553
Add base for destroy in node.
viferga Dec 5, 2024
851cfee
Merge branch 'develop' into feature/run-outside-metacall-cli
viferga Dec 19, 2024
9bbc9a6
Host destruction working properly with nodejs.
viferga Dec 19, 2024
10f72e1
Add preload of sanitizers for node tests with node executable.
viferga Dec 19, 2024
ad08aec
Change metacall_destroy signature, improved bugs with atexit.
viferga Dec 19, 2024
0fe2295
Add NodeJS support for load_from_package and execution_path, add test…
viferga Jan 14, 2025
50c27f2
Add loader config with dependencies.
viferga Jan 29, 2025
3f53257
Use always Python3.
viferga Jan 30, 2025
911e7a3
Solve issues Python 3.13.
viferga Jan 30, 2025
aa13d6c
Implement dependencies for loaders.
viferga Jan 30, 2025
1f48816
Change link options.
viferga Feb 3, 2025
ab80538
Add base for supporting weak symbols on node loader.
viferga Feb 3, 2025
35f656a
Merge branch 'develop' into feature/run-outside-metacall-cli
viferga Feb 3, 2025
cd1cb34
Solve issue of NodeJS with MSVC.
viferga Feb 3, 2025
a6c0144
Add thread safety for metacall link.
viferga Feb 5, 2025
64a6150
Add proper link flags for node_loader on macos.
viferga Feb 5, 2025
f678919
Add basic atexit support in portability.
viferga Feb 12, 2025
9ba8828
Revert "Bump store2 in /source/scripts/node/gram/source/gram (#541)"
viferga Feb 12, 2025
037296f
Add init of atexit in logs.
viferga Feb 12, 2025
fc940c7
Solve issue with windows lib names.
viferga Feb 13, 2025
d4fbdb2
Improve node port types.
viferga Feb 20, 2025
7587d8a
Solve issues with funchook and rapid_json.
viferga Feb 20, 2025
12c3077
Add Workflow for Multi-Architecture Docker Image Build, Push, and Man…
Bcoderx6 Feb 21, 2025
d89b966
Trying to solve issues macos.
viferga Feb 24, 2025
d73b40a
Solve issues in platform delete tag.
viferga Feb 24, 2025
0493d41
Merge branch 'develop' into feature/run-outside-metacall-cli
viferga Feb 25, 2025
dfeb559
Trying to test macos.
viferga Feb 25, 2025
2a92f10
Trying to solve delete tag in multiplatform builds.
viferga Feb 25, 2025
0a40c7e
Set up backtrace plugin for Guix.
viferga Feb 25, 2025
72599a0
MacOS with prebuilt node.
viferga Feb 25, 2025
7121254
Add debug for makefile.
viferga Feb 25, 2025
bd1b230
Solve npm issues macos.
viferga Feb 25, 2025
b1fca38
Solve more npn issues.
viferga Feb 25, 2025
62c181c
Trying to solve macos.
viferga Feb 25, 2025
2baa0f4
Breaks make builds.
viferga Feb 25, 2025
c97a026
Solving issues with libnode.
viferga Feb 25, 2025
662cb9b
Trying to make node_loader work on macos.
viferga Feb 25, 2025
67b2b5b
Trying to make node_loader work on macos 2.
viferga Feb 25, 2025
7735599
Trying to make node_loader work on macos 3.
viferga Feb 25, 2025
3f737b0
Trying to make node_loader work on macos 4.
viferga Feb 25, 2025
18e4aec
Trying to make node_loader work on macos 5.
viferga Feb 25, 2025
7fbcce0
Solve issues with macos link.
viferga Feb 25, 2025
0a19cc7
Remove googletest.
viferga Feb 25, 2025
30ac8a9
Add debug info for build.
viferga Feb 25, 2025
a36aa91
Remove gtest.
viferga Feb 25, 2025
3f8ecf5
Add more debug macos.
viferga Feb 25, 2025
fbb2aa7
Solve issues with libfunchook.
viferga Feb 25, 2025
04220ad
Solve more issues funchook.
viferga Feb 25, 2025
8370332
Solving issues funchook.
viferga Feb 25, 2025
49fb30a
Solving funchook.
viferga Feb 26, 2025
bebf51e
Disable install.
viferga Feb 26, 2025
74899d3
Disable install 2.
viferga Feb 26, 2025
1b88116
Trying something.
viferga Feb 26, 2025
a07199a
Solving issues funchook.
viferga Feb 26, 2025
7cb91d1
Testing platform ci.
viferga Feb 26, 2025
cd13179
Enabled github platform.
viferga Feb 26, 2025
53620af
Add dynamic detection of asan paths.
viferga Feb 26, 2025
f8c7b4c
Solve issues on funchook windows.
viferga Feb 26, 2025
06d5481
Solve issues in funchook windows.
viferga Feb 27, 2025
120a383
Solve issues in github platform.
viferga Feb 27, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
60 changes: 37 additions & 23 deletions .github/workflows/docker-hub-platform.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ jobs:
manifest:
name: Create and Push Manifest Lists
needs: build
# Only run when master or when tagging a version
if: (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/')) && github.event_name != 'pull_request'
runs-on: ubuntu-latest
steps:
- name: Login to Docker Hub
Expand All @@ -104,10 +106,15 @@ jobs:

- name: Create and Push Manifest Lists
run: |
for tag in "deps" "dev" "runtime" "cli"; do
tags=("deps" "dev" "runtime" "cli")
platforms=("linux/amd64" "linux/386" "linux/arm64" "linux/riscv64" "linux/ppc64le" "linux/s390x" "linux/arm/v7" "linux/arm/v6")

echo "Create all the tags by platform"

for tag in "${tags[@]}"; do
echo "Creating manifest for tag: $tag"
platform_tags=""
for platform in "linux/amd64" "linux/386" "linux/arm64" "linux/riscv64" "linux/ppc64le" "linux/s390x" "linux/arm/v7" "linux/arm/v6"; do
for platform in "${platforms[@]}"; do
platform_tag=$(echo "${platform}" | tr '/' '-')
platform_tags="${platform_tags} ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${tag}-${platform_tag}"
done
Expand All @@ -116,28 +123,32 @@ jobs:
docker manifest push ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${tag}
done

- name: Create Version Specific Tags
if: startsWith(github.ref, 'refs/tags/')
run: |
VERSION=${GITHUB_REF#refs/tags/v}
tags=("deps" "dev" "runtime" "cli")
platforms=("linux/amd64" "linux/386" "linux/arm64" "linux/riscv64" "linux/ppc64le" "linux/s390x" "linux/arm/v7" "linux/arm/v6")
for tag in "${tags[@]}"; do
platform_tags=""
for platform in "${platforms[@]}"; do
platform_tags="${platform_tags} ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${tag}-${platform}"
done
docker manifest create ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION}-${tag} ${platform_tags} --amend
docker manifest push ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION}-${tag}
done
echo "Create the latest tag"

cli_platform_tags=""
for platform in ${{ matrix.platform }}; do
cli_platform_tags="${cli_platform_tags} ${DOCKER_USERNAME}/${IMAGE_NAME}:cli-${platform}"
for platform in ${platforms[@]}"; do
platform_tag=$(echo "${platform}" | tr '/' '-')
cli_platform_tags="${cli_platform_tags} ${DOCKER_USERNAME}/${IMAGE_NAME}:cli-${platform_tag}"
done
docker manifest create ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:latest ${cli_platform_tags} --amend
docker manifest push ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:latest

if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" = true ]]; then
VERSION=${GITHUB_REF#refs/tags/v}

echo "Create the version ${VERSION} tag"

for tag in "${tags[@]}"; do
platform_tags=""
for platform in "${platforms[@]}"; do
platform_tag=$(echo "${platform}" | tr '/' '-')
platform_tags="${platform_tags} ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${tag}-${platform_tag}"
done
docker manifest create ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION}-${tag} ${platform_tags} --amend
docker manifest push ${DOCKER_REGISTRY}/${DOCKER_USERNAME}/${IMAGE_NAME}:${VERSION}-${tag}
done
fi

cleanup:
name: Cleanup Platform Specific Tags
needs: [build, manifest]
Expand All @@ -146,16 +157,19 @@ jobs:
steps:
- name: Remove Platform-Specific Tags
run: |
platforms=("linux-amd64" "linux-386" "linux-arm64" "linux-riscv64" "linux-ppc64le" "linux-s390x" "linux-arm-v7" "linux-arm-v6")
tags=("deps" "dev" "runtime" "cli")
platforms=("linux/amd64" "linux/386" "linux/arm64" "linux/riscv64" "linux/ppc64le" "linux/s390x" "linux/arm/v7" "linux/arm/v6")

for tag in "${tags[@]}"; do
for platform in "${platforms[@]}"; do
platform_tag=$(echo "${platform}" | tr '/' '-')
tag_to_delete="${tag}-${platform_tag}"

for platform in "${platforms[@]}"; do
for tag in "${tags[@]}"; do
tag_to_delete="${tag}-${platform}"
echo "Deleting tag: ${tag_to_delete}"
echo "https://hub.docker.com/v2/repositories/${DOCKER_USERNAME}/${IMAGE_NAME}/tags/${tag_to_delete}/"

curl -X DELETE \
-H "Authorization: Bearer ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}" \
-H "Authorization: Bearer ${{ secrets.DOCKER_HUB_ACCESS_TOKEN_DELETE }}" \
"https://hub.docker.com/v2/repositories/${DOCKER_USERNAME}/${IMAGE_NAME}/tags/${tag_to_delete}/"
done
done
5 changes: 4 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
#

# CMake version
cmake_minimum_required(VERSION 3.14 FATAL_ERROR)
cmake_minimum_required(VERSION 3.15 FATAL_ERROR)

# Include cmake modules

Expand Down Expand Up @@ -241,6 +241,9 @@ else()
endif()
endif()

# Export compile commands
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

#
# CTest configuration
#
Expand Down
66 changes: 66 additions & 0 deletions cmake/CompileOptions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,72 @@ else()
set(SANITIZER_COMPILE_DEFINITIONS)
endif()

function(find_sanitizer NAME LINK_OPTION)
string(TOUPPER "${NAME}" NAME_UPPER)
set(SANITIZER_PROGRAM_CODE "int main() {return 0;}")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/sanitizer_locate.cpp" "${SANITIZER_PROGRAM_CODE}")

try_compile(
STATUS
${PROJECT_OUTPUT_DIR}
${CMAKE_CURRENT_BINARY_DIR}/sanitizer_locate.cpp
OUTPUT_VARIABLE SANITIZER_COMPILER_OUTPUT
LINK_OPTIONS ${LINK_OPTION}
COPY_FILE ${CMAKE_CURRENT_BINARY_DIR}/sanitizer_locate
)

if(NOT STATUS)
message(FATAL_ERROR "Could not find location for lib${NAME}: ${SANITIZER_COMPILER_OUTPUT}")
return()
endif()

file(GET_RUNTIME_DEPENDENCIES
EXECUTABLES ${CMAKE_CURRENT_BINARY_DIR}/sanitizer_locate
RESOLVED_DEPENDENCIES_VAR SANITIZER_PROGRAM_LIBRARIES
)

foreach(DEPENDENCY IN LISTS SANITIZER_PROGRAM_LIBRARIES)
string(FIND "${DEPENDENCY}" "${NAME}" POSITION)
if(POSITION GREATER -1)
set(LIB${NAME_UPPER}_PATH "${DEPENDENCY}" PARENT_SCOPE)
return()
endif()
endforeach()
endfunction()

if("${CMAKE_C_COMPILER_ID}" STREQUAL "GNU" OR "${CMAKE_C_COMPILER_ID}" STREQUAL "Clang" OR "${CMAKE_C_COMPILER_ID}" STREQUAL "AppleClang")
if(OPTION_BUILD_THREAD_SANITIZER)
find_sanitizer(tsan -fsanitize=thread)
set(SANITIZER_LIBRARIES_PATH
"${LIBTSAN_PATH}"
)
elseif(OPTION_BUILD_MEMORY_SANITIZER)
set(SANITIZER_LIBRARIES_PATH) # TODO
elseif(OPTION_BUILD_ADDRESS_SANITIZER)
find_sanitizer(asan -fsanitize=address)
find_sanitizer(ubsan -fsanitize=undefined)
set(SANITIZER_LIBRARIES_PATH
"${LIBASAN_PATH}"
"${LIBUBSAN_PATH}"
)
endif()
endif()

if(SANITIZER_LIBRARIES_PATH)
if(PROJECT_OS_LINUX)
list(JOIN SANITIZER_LIBRARIES_PATH " " SANITIZER_LIBRARIES_PATH_JOINED)
set(TESTS_SANITIZER_PRELOAD_ENVIRONMENT_VARIABLES
"LD_PRELOAD=${SANITIZER_LIBRARIES_PATH_JOINED}"
)
elseif(PROJECT_OS_FAMILY MATCHES "macos")
list(JOIN SANITIZER_LIBRARIES_PATH ":" SANITIZER_LIBRARIES_PATH_JOINED)
set(TESTS_SANITIZER_PRELOAD_ENVIRONMENT_VARIABLES
"DYLD_INSERT_LIBRARIES=${SANITIZER_LIBRARIES_PATH_JOINED}"
"DYLD_FORCE_FLAT_NAMESPACE=1"
)
endif()
endif()

if((PROJECT_OS_WIN AND MSVC) OR "${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
# MSVC and Clang do not require to link manually the sanitizer libraries
set(SANITIZER_LIBRARIES)
Expand Down
18 changes: 8 additions & 10 deletions cmake/InstallRapidJSON.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,17 @@ if(NOT RAPIDJSON_FOUND OR USE_BUNDLED_RAPIDJSON)
endif()

ExternalProject_Add(rapid-json-depends
GIT_REPOSITORY "https://github.com/Tencent/rapidjson.git"
GIT_TAG "${RAPIDJSON_VERSION}"
CMAKE_ARGS
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DRAPIDJSON_BUILD_DOC=Off
-DRAPIDJSON_BUILD_EXAMPLES=Off
-DRAPIDJSON_BUILD_TESTS=Off
TEST_COMMAND ""
GIT_REPOSITORY "https://github.com/Tencent/rapidjson.git"
GIT_TAG "${RAPIDJSON_VERSION}"
BUILD_COMMAND ""
CONFIGURE_COMMAND ""
INSTALL_COMMAND ""
TEST_COMMAND ""
)

ExternalProject_Get_Property(rapid-json-depends INSTALL_DIR)
ExternalProject_Get_Property(rapid-json-depends SOURCE_DIR)

set(RAPIDJSON_ROOT_DIR ${INSTALL_DIR})
set(RAPIDJSON_ROOT_DIR ${SOURCE_DIR})
set(RAPIDJSON_INCLUDE_DIRS ${RAPIDJSON_ROOT_DIR}/include)
set(RAPIDJSON_FOUND TRUE)

Expand Down
2 changes: 1 addition & 1 deletion source/adt/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE

PUBLIC
Expand Down
2 changes: 1 addition & 1 deletion source/benchmarks/log_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_cs_call_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,5 +186,7 @@ int main(int argc, char **argv)

::benchmark::RunSpecifiedBenchmarks();

return metacall_destroy();
metacall_destroy();

return 0;
}
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_node_call_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -292,5 +292,7 @@ int main(int argc, char **argv)
}
#endif /* OPTION_BUILD_LOADERS_NODE */

return metacall_destroy();
metacall_destroy();

return 0;
}
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_py_c_api_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_py_call_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -178,10 +178,7 @@ int main(int argc, char *argv[])
::benchmark::RunSpecifiedBenchmarks();
::benchmark::Shutdown();

if (metacall_destroy() != 0)
{
return 4;
}
metacall_destroy();

return 0;
}
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_py_init_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,10 +130,7 @@ BENCHMARK_DEFINE_F(metacall_py_init_bench, destroy)
/* Python */
#if defined(OPTION_BUILD_LOADERS_PY)
{
if (metacall_destroy() != 0)
{
state.SkipWithError("Error destroying MetaCall");
}
metacall_destroy();
}
#endif /* OPTION_BUILD_LOADERS_PY */
}
Expand Down
2 changes: 1 addition & 1 deletion source/benchmarks/metacall_rb_call_bench/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,7 @@ int main(int argc, char *argv[])
::benchmark::RunSpecifiedBenchmarks();
::benchmark::Shutdown();

if (metacall_destroy() != 0)
{
return 4;
}
metacall_destroy();

return 0;
}
2 changes: 1 addition & 1 deletion source/cli/metacallcli/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ target_compile_features(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE
${DEFAULT_LINKER_OPTIONS}
)
Expand Down
7 changes: 1 addition & 6 deletions source/cli/metacallcli/source/application.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -299,12 +299,7 @@ application::application(int argc, char *argv[]) :

application::~application()
{
int result = metacall_destroy();

if (result != 0)
{
std::cout << "Error while destroying MetaCall, exit code: " << result << std::endl;
}
metacall_destroy();
}

void application::arguments_parse(std::vector<std::string> &arguments)
Expand Down
2 changes: 1 addition & 1 deletion source/cli/plugins/cli_core_plugin/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ target_compile_options(${target}
# Linker options
#

target_link_libraries(${target}
target_link_options(${target}
PRIVATE

PUBLIC
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,12 @@

#include <cli_core_plugin/cli_core_plugin_api.h>

#include <dynlink/dynlink.h>

#ifdef __cplusplus
extern "C" {
#endif

CLI_CORE_PLUGIN_API int cli_core_plugin(void *loader, void *handle);

DYNLINK_SYMBOL_EXPORT(cli_core_plugin);

#ifdef __cplusplus
}
#endif
Expand Down
Loading
Loading