Unable to find locally installed cosmocc toolchain

[IMXEren]

I had installed cosmocc manually (added to PATH) and wanted to integrate it with xmake. The current implementation of find_cosmocc.lua fails to find the cosmocc program which triggers the install for the toolchain (add_requires("cosmocc")).

So, I tried to improve the find_cosmocc.lua and its variants (cxx, ar) ...

--!A cross-platform build utility based on Lua
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-- Copyright (C) 2015-present, TBOOX Open Source Group.
--
-- @author      ruki
-- @file        find_cosmocc.lua
--

-- imports
import("lib.detect.find_tool")
import("lib.detect.find_program")
import("lib.detect.find_programver")

-- find cosmocc
--
-- @param opt   the argument options, e.g. {version = true}
--
-- @return      program, version
--
-- @code
--
-- local cosmocc = find_cosmocc()
--
-- @endcode
--
function main(opt)
    opt = opt or {}
    opt.shell = true

    local cosmocc
    local fprg = opt.program or "cosmocc"
    local cosmocc_prg = find_program(fprg, opt)
    if cosmocc_prg then
        cosmocc = cosmocc_prg
    else
        local sh = find_tool("sh") or find_tool("bash") or find_tool("zsh")
        if sh then
            local find
            if os.is_host("windows") then
                find = format("where %s || echo", fprg)
            else
                find = format("command -v %s || echo", fprg)
            end
            cosmocc, _ = try { function() return os.iorunv(sh.program, { "-c", find }) end }
            local first_line = string.gmatch(cosmocc, "[^\n]+")()
            if first_line then
                cosmocc = first_line:trim()
            end
        end
    end

    local program = nil
    local version = nil
    if cosmocc then
        program = path.translate(cosmocc)
        if program and is_host("windows") then
            program = program:gsub("\\", "/")
        end
        if program and opt and opt.version then
            -- the version will be of the underlying
            -- tool being used by cosmocc and not of the
            -- cosmocc toolchain package
            version = find_programver(program, opt)
        end
    end
    return program, version
end

which helps find_tool to discover it. Also, added the on_fetch snippet for the cosmocc package:

on_fetch(function(package, opt)
        if opt.system then
            return package:find_tool("cosmocc")
        end
end)

---

The problem would still persist for to find the sdkdir and bindir in the toolchain config and hence, raises an error: @programdir\core\main.lua:329: @programdir\core\tool\toolchain.lua:280: @programdir\toolchains\cosmocc\check.lua:63: cosmocc toolchain not found!

Modifying the toolchains\cosmocc\check.lua with this ...

--!A cross-toolchain build utility based on Lua
--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
-- Copyright (C) 2015-present, TBOOX Open Source Group.
--
-- @author      ruki
-- @file        check.lua
--

-- imports
import("core.project.config")
import("lib.detect.find_path")
import("lib.detect.find_tool")
import("detect.sdks.find_cross_toolchain")

-- check the cross toolchain
function main(toolchain)
    -- get sdk directory
    local sdkdir = toolchain:sdkdir()
    local bindir = toolchain:bindir()

    -- find and locate sdk directory
    if not sdkdir then
        local tool = find_tool("cosmocc")
        if tool and tool.program then
            sdkdir = path.directory(path.directory((tool.program)))
        end
    end

    -- find cross toolchain from external envirnoment
    local cross_toolchain = find_cross_toolchain(sdkdir, { bindir = bindir })
    if not cross_toolchain then
        -- find it from packages
        for _, package in ipairs(toolchain:packages()) do
            local installdir = package:installdir()
            if installdir and os.isdir(installdir) then
                cross_toolchain = find_cross_toolchain(installdir)
                if cross_toolchain then
                    break
                end
            end
        end
    end
    if cross_toolchain then
        toolchain:config_set("cross", cross_toolchain.cross)
        toolchain:config_set("bindir", cross_toolchain.bindir)
        toolchain:config_set("sdkdir", cross_toolchain.sdkdir)
        toolchain:configs_save()
    else
        raise("cosmocc toolchain not found!")
    end
    return cross_toolchain
end

works but what if the local toolchain is somehow gets corrupt and I'm willing to use the remote toolchain package, I think the local one would still prevail. Is there something that could force it? Maybe some kind of opt.system = false ?

---

I'm willing to contribute to both this and xmake-repo after the conclusion of this discussion. Thanks!

[waruqi]

find_program will call where/which to find it, you need not add it.

xmake/xmake/core/sandbox/modules/import/lib/detect/find_program.lua

Line 255 in dba39b4

local ok, wherepaths = os.iorunv("where.exe", {program_name})

[waruqi]

which helps find_tool to discover it. Also, added the on_fetch snippet for the cosmocc package:

and xmake will call find_tool when fetching package, you need also not call it in on_fetch

[IMXEren]

I tested it again it fails on _do_check method ...

xmake b -vD
checking for gcc ... ok
checkinfo: cannot runv(cosmocc --version), No such file or directory
checking for cosmocc ... no


build cache stats:
cache directory: build\.build_cache
cache hit rate: 0%
cache hit: 0
cache hit total time: 0.000s
cache miss: 0
cache miss total time: 0.000s
new cached files: 0
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 0
compile total time: 0.000s

checking for gcc ... ok
checking for the c compiler (cc) ... gcc
checking for flags (-fPIC) ... ok
> gcc "-fPIC" "-m64"
checking for flags (-O3) ... ok
> gcc "-O3" "-m64"
checking for flags (-DNDEBUG) ... ok
> gcc "-DNDEBUG" "-m64"
create ok!
compile_commands.json updated!
[100%]: build ok, spent 0.296s

while it works on directly invoking print(os.iorunv("where.exe", {"cosmocc"}))

xmake b -vD
checking for gcc ... ok
G:\cosmo\bin\cosmocc


build cache stats:
cache directory: build\.build_cache
cache hit rate: 0%
cache hit: 0
cache hit total time: 0.000s
cache miss: 0
cache miss total time: 0.000s
new cached files: 0
remote cache hit: 0
remote new cached files: 0
preprocess failed: 0
compile fallback count: 0
compile total time: 0.000s

checking for gcc ... ok
checking for the c compiler (cc) ... gcc
checking for flags (-fPIC) ... ok
> gcc "-fPIC" "-m64"
checking for flags (-O3) ... ok
> gcc "-O3" "-m64"
checking for flags (-DNDEBUG) ... ok
> gcc "-DNDEBUG" "-m64"
create ok!
compile_commands.json updated!
[100%]: build ok, spent 0.359s

I'm currently on zsh on ucrt64 env of msys2. Although, both work on wsl2 env. Could you tell the reason?

[IMXEren]

Also, cosmocc --version should work:

cosmocc --version   
cosmocc (GCC) 12.3.0
Copyright (c) 2024 Justine Alexandra Roberts Tunney
Cosmopolitan Libc and LLVM libcxx/compiler-rt are subject to non-GPL
notice licenses, e.g. ISC, MIT, etc. Your compiled programs must embed
our copyright notices. This toolchain is configured to do so by default.
Cosmopolitan comes with absolutely NO WARRANTY of any kind.
For more information, see the Cosmopolitan LICENSE files.
Copyright (C) 2022 Free Software Foundation, Inc.
This launches GNU GCC/Binutils subprocesses, which is free software; see
cosmocc's LICENSE files for source code and copying conditions. There is
NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[waruqi]

you can debug https://github.com/xmake-io/xmake/blob/dev/xmake/core/sandbox/modules/import/lib/detect/find_program.lua
https://github.com/xmake-io/xmake/blob/dev/xmake/modules/detect/tools/find_cosmocc.lua

[IMXEren]

It's strange that adding opt.paths = "$(env PATH)" in find_cosmocc.lua fixed it easily! Working for all variants cxx and ar.

Still, setting it as toolchain errors in not found. (xmake.lua from the example)

xmake f -cvD && xmake b -vD
checking for platform ... msys
checking for architecture ... x86_64
checking for gcc ... ok
checking for unzip ... ok
checking for git ... ok
checking for gzip ... ok
checking for tar ... ok
git rev-parse HEAD
checking for cosmocc ... no
checking for cosmocc ... G:\cosmo\bin\cosmocc
error: @programdir\core\main.lua:329: @programdir\core\tool\toolchain.lua:280: @programdir\toolchains\cosmocc\check.lua:53: cosmocc toolchain not found!
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:973]:
    [@programdir\toolchains\cosmocc\check.lua:53]:
    [C]: in function 'xpcall'
    [@programdir\core\base\utils.lua:275]:
    [@programdir\core\tool\toolchain.lua:276]: in function 'check'
    [@programdir\actions\config\main.lua:144]: in function '_check_target_toolchains'
    [@programdir\actions\config\main.lua:403]:
    [C]: in function 'xpcall'
    [@programdir\core\base\utils.lua:275]:
    [@programdir\core\base\task.lua:491]: in function 'run'
    [@programdir\core\main.lua:327]: in function 'cotask'
    [@programdir\core\base\scheduler.lua:406]:

stack traceback:
        [C]: in function 'error'
        @programdir\core\base\os.lua:973: in function 'base/os.raiselevel'
        (...tail calls...)
        @programdir\core\main.lua:329: in upvalue 'cotask'
        @programdir\core\base\scheduler.lua:406: in function <@programdir\core\base\scheduler.lua:399>

Although this would fix that but ...

-- find and locate sdk directory
if not sdkdir then
    local tool = find_tool("cosmocc")
    if tool and tool.program then
        sdkdir = path.directory(path.directory((tool.program)))
    end
end

I'm not sure that this would be a good alternative for those who still want to use the remote toolchain (referring to the first post). Is there a interface for toolchain that specifies what type can be used? Maybe set_toolchains("@!cosmocc") can assert that to prefer remote > local (if remote is found). What do you think about this? Explicit definition of --sdk or Implicit search for --sdk.

[waruqi]

It's strange that adding opt.paths = "$(env PATH)" in find_cosmocc.lua fixed it easily! Working for all variants cxx and ar.

Because I disabled to find them from $PATH by default.

Still, setting it as toolchain errors in not found. (xmake.lua from the example)

The current cosmocc toolchain only supports finding it from the xmake-repo/cosmocc package, finding it from the PATH and other system paths is not supported.

[waruqi]

try this patch #5281

I'm not sure that this would be a good alternative for those who still want to use the remote toolchain (referring to the first post). Is there a interface for toolchain that specifies what type can be used? Maybe set_toolchains("@!cosmocc") can assert that to prefer remote > local (if remote is found). What do you think about this? Explicit definition of --sdk or Implicit search for --sdk.

add_requires("cosmocc", {system = true})

[IMXEren]

Because I disabled to find them from $PATH by default.

Understood. I had no idea that env.PATH was mutated that didn't contain the whole of $PATH.

try this patch #5281

Thanks for the quick patch, it's working fine now! Also, would you apply the opt.envs = { PATH = os.getenv("PATH") } change across find_cosmocxx.lua and find_cosmoar.lua just so they could be detected.