Idea/suggestion: Import future patches/features from Illumos/BSDs?

[mamccollum]

Hey there, I had an idea I wanted to share. I understand that future contributions are supposed to be in the Zlib License, & I know that Illumos & the BSDs are NOT under the Zlib License, however I was wondering if on new commands or commands already licensed under CDDL or existing compatible licenses, that we could potentially import commands, features, & more from Illumos and/or the BSDs such as OpenBSD.

Does anyone else believe this has potential? I could work on ensuring it uses libcommon, etc.
I understand that we already have some work to do with OpenBSD compatibility, but I believe this could help push some innovation and expansion of the project, albeit at the cost of maintaining more.

Thoughts? (PS: if we do go through with this, should we try to import the git history for the specific files from Illumos, etc.?)

[takusuman]

With exception with importing directly from Illumos, I think it could be a good idea to implement/port missing commands. I mean, my idea is to try to make this package not-so-CDDL'd, since it could cause annoyance later on to folks trying to use this in an embedded environment.
I'm on my way to port write(1) from UNIX v7, so it will pair up with mesg(1), but I haven't done a lot of work on it yet.
Don't misunderstand me, please, but I think it would be more helpful (from a user point of view) to have default utilities with the new POSIX 2008 standard implemented. We have started doing it when we fixed that bug in default rm.
But, anyway, what tools do you propose to port? Maybe there's something that I'm missing out and that could make a difference.

[takusuman]

By the way, about porting from OpenBSD, I had an idea of creating an "ucblib" package that could be used as a libbsd alternative. Not inside Heirloom NG, though.

[mamccollum]

(...) my idea is to try to make this package not-so-CDDL'd, since it could cause annoyance later on to folks trying to use this in an embedded environment.
Don't misunderstand me, please, but I think it would be more helpful (...) to have default utilities with the new POSIX 2008 standard implemented.

Alright, I understand! I'm not offended at all, and I understand the licensing issues & the need for updating standards. Is there any explicit tasks that should be done on that side? I know there's the docs for 2008 and 2017 and I could check for the behavior between expected in the standard and the results of Heirloom.

But, anyway, what tools do you propose to port?

I was double-checking with the tools in the GNU coreutils (not saying at all to port from there, but I was looking for compatibility with it) and I could have sworn there was more that I wanted to port, but all I could find that was actually missing that wasn't 100% a GNUism was seq and rmdir, which are relatively easy to develop.

Also on the topic of standards like POSIX -- is there any test suite I could/should use to see how compatible Heirloom NG's utils are in comparison to other stuff? I know there's the Open POSIX test suite, but it looks to not have been updated in at least a decade, if not another 5 years after that. I've also heard of the GNU coreutils test suite that the folks over there use to test their software.

[takusuman]

Is there any explicit tasks that should be done on that side?

Nothing that I can say at a first overview, besides the rm fix that I've said about before.

I know there's the docs for 2008 and 2017 and I could check for the behavior between expected in the standard and the results of Heirloom.

These will do the trick. 😄

that was actually missing that wasn't 100% a GNUism was seq and rmdir, which are relatively easy to develop.

Well, I could argue that one may use {1..X} or a C-style for loop instead of seq 1 X for counting, but that only applies in Korn Shell 93, GNU Bourne-Again and Z-Shell, and that there are many people who still writing Shell scripts for POSIX-only environments and depend on seq.

seq itself, except for some GNU extensions, is just a matter of taking argv[0] and argv[1] and counting between them.
rmdir shall be just a system call-kiosk command (like readlink, that I've implemented), so it also shouldn't be hard to implement.

Definitively on the list. 👍🏽

Another two commands that I was thinking about implementing were watch and timeout, but my code haven't work so well and I didn't tried to go further since then.

[takusuman]

Also on the topic of standards like POSIX -- is there any test suite I could/should use to see how compatible Heirloom NG's utils are in comparison to other stuff?

I was thinking about that recently and... no, there's nothing to test all the utilities at once.
However, sed has its own set of tests and grep has some tests described at the NOTES file.
Maybe a sane option would be to port the tests from toybox, since its public domain and it won't have any licence problems with this current source tree.
I was also thinking about testing in GitHub's VMs if it builds on other flavors of BSD and MacOS, it would be useful to this information both here as a table and as a badge on the website.

[mamccollum]

I was also thinking about testing in GitHub's VMs if it builds on other flavors of BSD and MacOS (...)

vmactions has workflows for building on the BSDs and Solaris even though they're not officially supported (I believe it uses the macOS VM with a VM inside of that to do the trick).

P.S. I once tried building on macOS and there were... several big issues. From a lot of missing headers (can be fixed with macbrew, but it's quite annoying) to similar issues with deprecated functions that OpenBSD had, and even things as basic as APFS not being case-sensitive (unless you specifically configured your Mac's FS otherwise in a rescue CD). I didn't want to create an issue at the time because it's a massive series of issues and I think it's better that we focus on primarily Linux & secondarily the BSDs for now. (Though I'm not the leader, so just take this as friendly advice)

[takusuman]

vmactions has workflows for building on the BSDs and Solaris even though they're not officially supported (I believe it uses the macOS VM with a VM inside of that to do the trick).

Yeah, I've been thinking about it! Although it is needed, maybe it's not our priority for now comparing to implementing those new tools.

P.S. I once tried building on macOS and there were... several big issues.

I've imagined...

I didn't want to create an issue at the time because it's a massive series of issues and I think it's better that we focus on primarily Linux & secondarily the BSDs for now.

Sure, I agree with you.

[mamccollum]

Also, I made a commit in my fork of the repository that changes the readlink makefile to where it will clean up the UCB binaries as make mrproper was leaving them behind. Should I just leave that commit there and make a PR later when I make more changes to my fork?

[takusuman]

Also, I made a commit in my fork of the repository that changes the readlink makefile to where it will clean up the UCB binaries as make mrproper was leaving them behind.

Damn it, how could I have missed that out?

Should I just leave that commit there and make a PR later when I make more changes to my fork?

For me it seems O.k.

[mamccollum]

Understood. I'll look into the test suites from toybox and get back to you. 😄

[takusuman]

Understood. I'll look into the test suites from toybox and get back to you. smile

Good! And I'll be implementing the seq command.
Merci!¹

Footnotes

1. I'm not sure if it's still being used in English, but I've learnt that "Merci" could be used as a thank in English. ↩

[mamccollum]

One more thing -- chgrp is also missing. Forgot to mention that, sorry.

[takusuman]

One more thing -- chgrp is also missing. Forgot to mention that, sorry.

Actually, it's not.
Many commands on Heirloom are supplied by symbolic/hard links that change argv[0].
For instance, chgrp is a link to chown, dfspace is a link to df etc, take a look at the manual pages on the website. The commands that don't have a description and/or an own directory are supplied per symbolic/hard links.

[mamccollum]

Oh wow, I didn't notice that. Thanks for informing me!

[takusuman]

O.k., the initial implementation wasn't standard, so I implemented seq according to the standard and got some new funky bugs.
I don't really know where to go now.
https://github.com/Projeto-Pindorama/heirloom-ng/tree/seq-impl

EDIT: Kind of fixed at #29

[takusuman]

Hi, Molly (@mamccollum), good night. How you are?
Just an update: now seq works just fine --- at least here, not sure if it won't be breaking or misbehaving in other host yet --- and rmdir already got implemented by Gunnar Ritter back in July 2002, and it also has its own directory here in the source tree.

seq was harder than I thought, not because of the algorithm, but because of the implementation standard combined with my "just do it"/compact style of programming --- while many got it in more than 90 or 100 lines, I got it in 65 lines of code, counting spaces. There were some hours of debugging in the last day in which I programmed it, along with some segmentation faults.

[takusuman]

Another two commands that I was thinking about implementing were watch and timeout, but my code haven't work so well and I didn't tried to go further since then.

watch(1) implementation first made at #30, improved at #31.
On my way to timeout(1).

[mamccollum]

Hey, that's good to hear. I think a while back (probably around 3 weeks ago now) I was working on a feature here, but I sadly forgot what it was. My fork ended up just turning into a messy disaster and I had to re-start. Is there anything I should assist in working on?

[takusuman]

Hey, that's good to hear. I think a while back (probably around 3 weeks ago now) I was working on a feature here, but I sadly forgot what it was. My fork ended up just turning into a messy disaster and I had to re-start. Is there anything I should assist in working on?

Well, just testing and completely porting to OpenBSD (although I think having contributions from more OpenBSD folks would help too).
watch(1) works surprisingly well for its "100-liner" size --- I risk to say that it's even better, proportionally comparing, than procps-ng's watch(1) ---, but the title/information header size being smaller than the terminal maximum line width kind of annoys me a little.
About timeout(1), I couldn't rewrite it yet, more because of it being complex. I'm afraid that, in the end, I end up sourcing it from OpenBSD's source tree and doing some modifications so it fits on Heirloom NG --- like using SVR4-like error reporting via pfmt()/prerror() instead of err()-like functions, etc.

[takusuman]

Just realized that du doesn't return anything if called with just one file.

[takusuman]

Just realized that du doesn't return anything if called with just one file.

I was busy to correct myself, but nevermind, it's part of the standard.
If I'm not mistaken, only /usr/5bin/posix/du prints individual files without -h or -s per default.
For ones who want to write portable scripts, use always du -s or even du -hs.

[takusuman]

@mamccollum I was messing around with Heirloom tar (since it's on my PATH) and I think I've found a new glitch.

I usually copy folders using tar -cvf and tar -xvf in a pipeline, like Plan 9 does, and I've found out that it doesn't extract for some reason, printing "tar: 1 file(s) not extracted".

I'll take a deeper look on it later but, for now, I'll be using Schily's tar as always.

[xplshn]

chimera-utils has made a lot of the groundwork for using fBSD coreutils in Linux. We could take some of the patches from there. Also, what about the code from SBASE and UBASE? by suckless.org, its all MIT licensed.

[takusuman]

chimera-utils has made a lot of the groundwork for using fBSD coreutils in Linux. We could take some of the patches from there. Also, what about the code from SBASE and UBASE? by suckless.org, its all MIT licensed.

First of all, sorry for the delay on the response.
I think that taking a small part of chimera-utils may be useful for some utilities --- such as write, that, although having to be compliant with its UNIX v7 version to be par with mesg, or something to help finish the timeout implementation.

About SBASE/UBASE, I have already taken a look at some of the code, I think that it would be equivalent to copying code examples. It may be useful as a reference, but I think that we can implement more complete utilities. But that's something to consider too.

[xplshn]

sbase is POSIX and minimalist. I thought that heirloom strived for that too. Is compatibility with GNU coreutils a wanted feature for the project?

[takusuman]

sbase is POSIX and minimalist. I thought that heirloom strived for that too. Is compatibility with GNU coreutils a wanted feature for the project?

You got it a little bit off I what I meant.
Heirloom is meant to be POSIX, if one wants to put it this way, but in the mid-path betwixt simplicity and convenience --- like you would find in some UNIX-compatible system cited at Heirloom's intro.
Suckless' sbase are more like an example --- similar to what you could find at OpenGroup.org when looking at the POSIX specification --- of what a command should be instead of what it could be.
I don't want Heirloom to be like GNU Coreutilities in this fork, but I also don't want to call just a handful of lines a "complete utility" when it lacks features that are useful for the end user --- and, at the same time, not redundant. A good example about what I mean would be a comparison betwixt procps-ng's watch, Heirloom NG's and Suckless' ubase one.
procps-ng's is overly complicated, Heirloom NG does its job flawlessly in less than 1/4 of L.O.C. that procps-ng has and Suckless' ubase works as good as a shell script hack done in 2 minutes and a half.

[takusuman]

For a matter of honesty, I must say that Suckless {u,s}base may work as well as POSIX specification for enlightening the way to go --- but I wouldn't plain copy the code into Heirloom's source tree just because it's under a compatible licence, at most fork it and enhance it.

[takusuman]

I'm quite busy lately, so I can't burn daylight working on #36 for now, even having references instead of writing completely from scratch as you suggested, nor discussing this matter further.

[xplshn]

Okay, I think I get it. Then what about taking inspiration/work/code from Toybox? 0BSD licensed, by the same guy that started Busybox, its currently used in Android phones, it also works on MCU-less devices, and its pretty lightweight yet convenient.

[takusuman]

Then what about taking inspiration/work/code from Toybox? 0BSD licensed, by the same guy that started Busybox, its currently used in Android phones, it also works on MCU-less devices, and its pretty lightweight yet convenient.

Yeah, that is somewhat the goal: a sane and yet convenient environment. Toybox uses too many internal functions, so porting code directly from it is more difficult than doing a clean-room implementing.
NetBSD/OpenBSD's userland is also an inspiration, but we avoid to take code directly from it and try to get how to implement utilities by ourselves.

I have taken your idea of basing some utilities on OpenBSD code when fixing/"filling" Heirloom NG's timeout implementation.

[takusuman]

Released 240220 today, I would like a feedback from some of you.

@arthurbacci pointed out that my method for converting the float back to a string is redundant and could lose precision, I have already taken this into consideration to fix in the next release.
seq is ridiculously incomplete yet, I think it could at least mimic Research UNIX v8 implementation and also have its "separator" condition fixed for some cases, but I think I can get around with this.

[takusuman]

Released 240220 today, I would like a feedback from some of you.

Many fixes are now being addressed at #41.

[takusuman]

I would like to complement this issue with the fact that tar is somehow broken.
For instance, while I was testing Copacabana Linux build system, I noticed that Heirloom's tar isn't extracting tar balls passed for it per a pipeline, responding with this error:

tar: 2 file(s) not extracted

Maybe this can be fixed after some debugging, just taking notes here if someone got to it before me.

[arthurbacci]

Please open an issue for tar

[takusuman]

Please open an issue for tar

That's going to be fun.

[takusuman]

Opened a specific issue for tar at #44.
cc.: @arthurbacci

[takusuman]

@arthurbacci I was thinking, could we borrow the "libutf-8" from Plan 9 for libcommon? Or some other implementation of UTF-8 for C.

I found this page which lists some implementations (including Plan 9's) and drawbacks:
https://www.linuxdoc.org/HOWTO/Unicode-HOWTO-6.html

There's also utf8proc by the Julia Programming Language development team, it looks good, and it's also small.
https://github.com/JuliaLang/utf8proc

This would be a grand improvement on Heirloom, since we could make other programs UTF-8 compliant too.

I'm saying this mostly because of #52, but we could apply this to ls, more/pg, everything (almost).

[arthurbacci]

@arthurbacci I was thinking, could we borrow the "libutf-8" from Plan 9 for libcommon? Or some other implementation of UTF-8 for C.

I found this page which lists some implementations (including Plan 9's) and drawbacks: https://www.linuxdoc.org/HOWTO/Unicode-HOWTO-6.html

There's also utf8proc by the Julia Programming Language development team, it looks good, and it's also small. https://github.com/JuliaLang/utf8proc

This would be a grand improvement on Heirloom, since we could make other programs UTF-8 compliant too.

I'm saying this mostly because of #52, but we could apply this to ls, more/pg, everything (almost).

Maybe https://libs.suckless.org/libgrapheme/ ?

[takusuman]

Maybe https://libs.suckless.org/libgrapheme/ ?

Any way to drop-in in this project?
I would've surrender to wchar if I could implement it without getting cryptic memory faults when running fgetwc().

[xplshn]

@arthurbacci I was thinking, could we borrow the "libutf-8" from Plan 9 for libcommon? Or some other implementation of UTF-8 for C.

I found this page which lists some implementations (including Plan 9's) and drawbacks: https://www.linuxdoc.org/HOWTO/Unicode-HOWTO-6.html

There's also utf8proc by the Julia Programming Language development team, it looks good, and it's also small. https://github.com/JuliaLang/utf8proc

This would be a grand improvement on Heirloom, since we could make other programs UTF-8 compliant too.

I'm saying this mostly because of #52, but we could apply this to ls, more/pg, everything (almost).

The LibUTF from Sbase is also really good and tidy: https://git.suckless.org/sbase/files.html
But I feel like libgrapheme would be better in the long term, its very maintainable

[takusuman]

The LibUTF from Sbase is also really good and tidy: https://git.suckless.org/sbase/files.html
But I feel like libgrapheme would be better in the long term, its very maintainable

That's a good suggestion too, but maybe we will just stick to libgrapheme.
I hope it doesn't change much, so we could just embed it in the code and add some directions at the build system to link it.

[xplshn]

@takusuman Is importing/porting code from https://github.com/illumos/illumos-gate/tree/master/usr/src/cmd acceptable? I don't know much C but I've been trying to create and port utilities from oBSD/Illumos to Linux, if I have any degree of success with it, would you be interested in adding them? Also, how do you feel about non-solaris code? Is Heirloom-ng the correct place to add these utilities or should there be another repo for additions that don't come from Heirloom nor are implementations of POSIX commands?

[takusuman]

@takusuman Is importing/porting code from https://github.com/illumos/illumos-gate/tree/master/usr/src/cmd acceptable? I don't know much C but I've been trying to create and port utilities from oBSD/Illumos to Linux, if I have any degree of success with it, would you be interested in adding them? Also, how do you feel about non-solaris code? Is Heirloom-ng the correct place to add these utilities or should there be another repo for additions that don't come from Heirloom nor are implementations of POSIX commands?

G'day, pardon for the delay on the response --- I'm trying to configure a new domain to this project and it's been complicated because of, well, Brazilian internet bureaucracy probably inherited from the last days of the Military Regime.

Is importing/porting code from https://github.com/illumos/illumos-gate/tree/master/usr/src/cmd acceptable?

About Illumos, the only problem to me is the licence --- I'm not against anything "quasi-copylefted", for instance, Copacabana's default shell (ksh93) is under something similar to CDDL ---, I'm aiming to have Heirloom NG in a permissive licence because I can see it running on embedded devices someday (under Copacabana or not), so I would like to keep it with the only restrictions being that our names as contributors being cited.
To be frank, I haven't studied the CDDL yet, so it may permit sublicencing to ZLib at some degree of modification.
Nonetheless, many programs such as echo or mvdir are under CDDL, so it's not forbidden to add CDDL code, but it would be preferred to have more copycenter code (under ZLib).
For licencing reasons, if desired to port real AT&T UNIX code, it would be preferred to port from UNIX 32v/v7 --- v8 and v10 are a little bit shady --- than from Solaris/illumos.

I've been trying to create and port utilities from oBSD/Illumos to Linux, if I have any degree of success with it, would you be interested in adding them?

In this case, no problem. :^)
Just remember to talk about it before, so we can decide how it can be done and what could be fixed/reviewed.

Also, how do you feel about non-solaris code?

Fine. Most things here, although being based on Solaris/SunOS payloads, are pure reimplementations and/or have heavy modifications to be portable --- since the Sun crew, with their own obvious reasons, didn't made the code portable in many parts.

Is Heirloom-ng the correct place to add these utilities or should there be another repo for additions that don't come from Heirloom nor are implementations of POSIX commands?

Even though this sounds contradictory, Heirloom is a set of UNIX utilities, but not strictly POSIX.
So yes, we can add extra utilities that are not in the POSIX standard --- like apply and pick, which was recently implemented by me ---, but I think that some utilities (such as Linux's fstype and fallocate, for instance) should be added to a new repository with not-so-portable tools. I think we should also aim to avoid feature-creep here, avoding to bring programs such as hexdump since we already have XENIX's hd.

By the way, I'm currently trying to implement wall (#46) and write (#47), which come from UNIX v7. Help on these would be appreciated since the change from utmp to utmpx broke a couple of things. I mean, a lot of things.
I think it would be better to not install these per default since it wouldn't made sense in all environments --- I mean, why would you need write in your single-user UNIX-compatible (Linux, *BSD, etc) machine? ---, but this could be changed at the top makefile later anyway.

[xplshn]

I too think avoiding feature creep is crucial, and that copylefted licenses should be avoided. One last question: Are you okay with Go code? Not necessarily in this repo, but in Copabanaca, I'd like to contribute but I am not proficient nor understand enough C, I may get by porting some commands using things like libbsd, but that's it.

[takusuman]

One last question: Are you okay with Go code? Not necessarily in this repo, but in Copabanaca

Go? We're all for it!
It's part of the "stack" that I've been planning for years.
For instance, the pacote package manager --- a reimplementation of the original Solaris SVR4-style package manager --- is being made in Go and some other programs such as L.E.mount will be remade in Go.
There's a lot of planning to be done, but I will be doing it until the end of this week.

[takusuman]

By the way, just a note: pacote isn't a direct competitor to bigdl nor other sophisticated package managers, it's the bare minimum to install packages reliably in an UNIX-compatible system.

[takusuman]

I'd like to contribute but I am not proficient nor understand enough C, I may get by porting some commands using things like libbsd, but that's it.

I can help with porting since I've done it before with another programs, no problem with that. By the way, C isn't as nearly as difficult if you're already programming in Go from a long time.
The biggest problem when porting are API differences, such as utmp vs utmpx like I've spoken before.

[xplshn]

The biggest problem when porting are API differences, such as utmp vs utmpx like I've spoken before.

I'm currently trying to port various utils to https://github.com/xplshn/a-utils, I've a local version of the source code of SystemV and UnixV32, I am re-implementing things in Go, using bits and pieces taken from https://github.com/u-root/u-root. I like Go because if you do everything using the language's std, your programs/projects are almost guaranteed to run even in places you don't expect them to, like plan9. Also, if I finish adapting this lua build script to compile various userland tools of OpenBSD in Linux to SH, I'll notify you, because I'd really love to see OpenBSD utils running in (mainstream) Linux. See https://github.com/oasislinux/oasis/tree/master/pkg/openbsd

[xplshn]

I realized that what I was doing was totally wrong, I should have first checked if OpenBSD to POSIX ports already existed, and, apparently, this one which hasn't received updates in 5 years isn't really far behind, I ran a diff -Naru on each directory, the majority of changes are on the build system itself, removing the use of pledge() and etc. https://github.com/ibara/baseutils

But you can do a quick and dirty trick, use the pledge() implementation of Cosmopolitan libc using -I/path/to/cosmopolitanSrc/libc/calls: https://github.com/jart/cosmopolitan/tree/master/libc/calls
There's also the command line tool called pledge which they have, its great.

[takusuman]

I like Go because if you do everything using the language's std, your programs/projects are almost guaranteed to run even in places you don't expect them to, like plan9.

I know, it was one of the things that made me think about Go back in 2021.
I think that in the future we could even work in a framework to make easier to build graphical applications (something like WindowsForms) and ship with the development stage of Copacabana, but that's not even on paper, only on thoughts and dreams.

Also, if I finish adapting this lua build script to compile various userland tools of OpenBSD in Linux to SH, I'll notify you, because I'd really love to see OpenBSD utils running in (mainstream) Linux. See https://github.com/oasislinux/oasis/tree/master/pkg/openbsd

That remembered me that I need to refactor Copacabana's build-system and work on the idea of pure cross-compiling. I'm a little bit worried about more complex parts such as OpenPAM, Xorg/Wayland, KDE and Pipewire, but I think we can.

I realized that what I was doing was totally wrong, I should have first checked if OpenBSD to POSIX ports already existed.

Per my experience, the most annoying part is to remove OpenBSD-specific code such as pledge() calls, the rest is kind of easy because, unlike proprietary Solaris code (I think it has changed with illumos, though), it's portable C.

But you can do a quick and dirty trick, use the pledge() implementation of Cosmopolitan libc using -I/path/to/cosmopolitanSrc/libc/calls

I remember having a plan to port pledge() into Linux and ship it per default on Copacabana, can we do that in small time or it would take more than two weeks?

[xplshn]

I remember having a plan to port pledge() into Linux and ship it per default on Copacabana, can we do that in small time or it would take more than two weeks?

Cosmopolitan even ships a program called pledge that uses the Cosmopolitan libC pledge()'s that allows you to change what a program can do, I think it could be done in a few days, since Cosmopolitan libc is very portable, do you want to import only pledge() and adapt it to work with the rest of the codebase?

[takusuman]

do you want to import only pledge() and adapt it to work with the rest of the codebase?

I think we can go further: port it and implement as a patch/extension to musl --- or maybe into the kernel itself.

Justine has written a blog post about porting it, so I think we can write some things from scratch: https://justine.lol/pledge/.

[takusuman]

I will be doing various "pre-releases" of Copacabana before getting on 0.5, so we can experiment a lot and fix along the way.

[mamccollum]

By the way, I'm currently trying to implement wall (#46) and write (#47), which come from UNIX v7.

Just a quick note -- While util-linux is licensed under the GPL 2.0, the code for write and wall appears to be still one of the BSD 4-Clause licenses from what I think is 4.4BSD's implementation, as the license headers at the top of the files for wall.c and write.c are the BSD licenses from UCB, but modified for Linux.

The README for licensing of the files states that is indeed files under different (such as BSD) licenses.