[bug] Various commands don't work while conan's (python's?) stdout is piped

[MikhailShostak]

Describe the bug

When I try to use conan from my app to get information about packages it returns an error.
conan list and conan search commands are affected by this bug, maybe something else.

OS: Windows 11 23H2
Python: 3.12.4
Compiler: Visual Studio 2022 17.10.3 (MSVC 19.40.33811)
Conan Version: 2.4.1.
Note: I didn't have this on 2.0.9, but recent update in Visual Studio versioning forced me to update conan.

How to reproduce it

The bug can be reproduced with simple app like this:

#include <boost/process.hpp>
#include <iostream>

int main(int argc, char *argv[])
{
    if (argc < 2)
    {
        return -1;
    }

    std::string cmd = argv[1];
    for (size_t i = 2; i < argc; ++i)
    {
        cmd += " ";
        cmd += argv[i];
    }

    boost::process::ipstream pipe_stream;
    boost::process::child c(cmd, boost::process::std_out > pipe_stream);
    
    std::string line;
    while (pipe_stream && std::getline(pipe_stream, line) && !line.empty())
    {
        std::cout << line << std::endl;
    }

    c.wait();
    return 0;
}

Output:

>Wrapper conan list boost --format=json
{
    "Local Cache": {
        "error": "'NoneType' object has no attribute 'write'"
    }
}

Repro on GitHub Actions:
https://github.com/MikhailShostak/conan-stdout-bug/actions/runs/9792674178/job/27039011852#step:7:9

[perseoGI]

Hi @MikhailShostak, thank you for your detailed review of your issue and for creating a repository to easily setup the environment!

I've just compiled it and tested it locally in a MacBook Pro M3 Max with this environment:

• Python 3.12.3
• Conan 2.4.1
• boost/1.85.0

And I'm not being able to reproduce your issue. This are some of the tests I've done:

$ ./Wrapper conan list boost --format=json
Found 1 pkg/version recipes matching boost in local cache
{
    "Local Cache": {
        "boost/1.85.0": {}
    }
}

$ ./Wrapper conan search "boost/[*]" --remote=conancenter
Found 17 pkg/version recipes matching boost/* in conancenter
conancenter
  boost
    boost/1.69.0
    boost/1.70.0
    boost/1.71.0
    boost/1.72.0
    boost/1.73.0
    boost/1.74.0
    boost/1.75.0
    boost/1.76.0
    boost/1.77.0
    boost/1.78.0
    boost/1.79.0
    boost/1.80.0
    boost/1.81.0
    boost/1.82.0
    boost/1.83.0
    boost/1.84.0
    boost/1.85.0

$ ./Wrapper conan search "boost/[*]" --remote=conancenter --format=json
Found 17 pkg/version recipes matching boost/* in conancenter
{
    "conancenter": {
        "boost/1.69.0": {},
        "boost/1.70.0": {},
        "boost/1.71.0": {},
        "boost/1.72.0": {},
        "boost/1.73.0": {},
        "boost/1.74.0": {},
        "boost/1.75.0": {},
        "boost/1.76.0": {},
        "boost/1.77.0": {},
        "boost/1.78.0": {},
        "boost/1.79.0": {},
        "boost/1.80.0": {},
        "boost/1.81.0": {},
        "boost/1.82.0": {},
        "boost/1.83.0": {},
        "boost/1.84.0": {},
        "boost/1.85.0": {}
    }
}

$ ./Wrapper conan install --requires="zlib/[*]" --build=missing

======== Input profiles ========
Profile host:
[settings]
arch=armv8
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=15
os=Macos

Profile build:
[settings]
arch=armv8
build_type=Release
compiler=apple-clang
compiler.cppstd=gnu17
compiler.libcxx=libc++
compiler.version=15
os=Macos


======== Computing dependency graph ========
Graph root
    cli
Requirements
    zlib/1.3.1#f52e03ae3d251dec704634230cd806a2 - Cache
Resolved version ranges
    zlib/[*]: zlib/1.3.1

======== Computing necessary packages ========
Requirements
    zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:2ee39e692ca4177b4b689b15bc1f2cfdf8f83706#a93f226c82100d7d48e2809e534845c4 - Cache

======== Installing packages ========
zlib/1.3.1: Already installed! (1 of 1)
WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X:
WARN: deprecated:     'cpp_info.names' used in: zlib/1.3.1

======== Finalizing install (deploy, generators) ========
cli: Generating aggregated env files
cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully

Did you have the chance to test it with a different OS?

[MikhailShostak]

hey @perseoGI, I guess it's Windows only issue. Tried on Ubuntu 22.04.3 LTS under WSL. it works fine there.

[perseoGI]

I see... @memsharded could you check it on Windows?

[memsharded]

It doesn't seem a problem in Conan, but Conan execution is basically getting sys.stderr = None, and this wouldn't be valid.
It seems the wrapper is somehow providing only stdout to the wrapped command, but not stderr.

Changing the wrapper code to:

boost::process::child c(cmd, boost::process::std_out > pipe_stream, boost::process::std_err > stderr);

fixes it.

[MikhailShostak]

@memsharded, yeah, just came up to the similar conclusion. reproduced this issue with WinAPI implementation as well.
providing a pipe handle for stderr fixes both cases.

https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/ns-processthreadsapi-startupinfoa
not sure why it's not redirected to the console buffer if hStdError is null.

I think the workaround with setting up another pipe works for me.
so, we can close this ticket if you are not going to handle the case when stderr is unavailable.

[memsharded]

so, we can close this ticket if you are not going to handle the case when stderr is unavailable.

No, I don't think so, most of the Conan informational messages, very necessary to understand what is happening, goes to stderr, I would expect it not defined to be an error, not a valid condition to suppress all the Conan messages. If that is intended, it should be explicitly redirected to null or something like that.

Probably not worth doing anything else, it is the first time in year that I see this use case of a wrapper not providing stderr at all (and in fact the behavior in other platforms rather than Windows is different, this sounds like an inconsistency of the boost:process library to me). So closing the ticket, thanks for the feedback!

[memsharded]

#17507 adds a new --output-file to enforce writing the result in that file irrespective of the stdout redirections or other possible issues. It will be in next Conan 2.12. Thanks for the feedback!