Available FreeBSD port? #13

fmunteanu · 2021-05-14T15:32:29Z

@huo-ju since you are experienced, do you think is possible to create a fresh port? It will make things so much easier for everyone.

pmhausen · 2021-05-14T18:14:20Z

This code here is a FreeBSD port. All that is needed is submitting it in the FreeBSD Bugzilla to be included in the official collection.

https://bugs.freebsd.org/bugzilla/

fmunteanu · 2021-05-14T21:24:56Z

@pmhausen Patrick do you know who could help with this task? I'm not familiar with ports and FreeBSD, only CentOS.

This port should be split in two, one for libtensorflow library and one for photoprism.

pmhausen · 2021-05-14T22:36:33Z

I think I could help with that. Time permitting.

Why do you include some magic extracting libtensorflow from a docker image? There is a port of py-tensorflow already. Can't you just record a dependency on that?

https://www.freshports.org/science/py-tensorflow/

huo-ju · 2021-05-14T22:53:49Z

We have the same issue of py-tensorflow. Tensorflow is required bazel to build and it will download files during the build stage, this is not allowed by official ports tree. So I can't submit it to freshports.

pmhausen · 2021-05-14T23:04:32Z

OK, py-tensorflow is marked as "BROKEN". So one would have to make that work first. I fail to get the problem with the bazel dependency, there's this in the py-tensorflow Makefile:

BUILD_DEPENDS=	${RUN_DEPENDS} \
		${PYTHON_PKGNAMEPREFIX}grpcio-tools>=1.22.0:devel/py-grpcio-tools@${PY_FLAVOR} \
		bash:shells/bash \
		cython:lang/cython \
		swig:devel/swig \
		bazel:devel/bazel029

So it should just go ahead and build/install bazel. I can give that port a try tomorrow and try to find out why it is marked "BROKEN".

huo-ju · 2021-05-14T23:21:55Z

@pmhausen That's will be great. If we can make py-tensorflow works, then we can write a new build script for photoprism and submit it to the official ports tree.

fmunteanu · 2021-05-15T01:08:50Z

@huo-ju they reference your port into bug. For the interactive questions during build, I believe make has a setting to respond with default option?

huo-ju · 2021-05-15T03:27:39Z

@fmunteanu Thanks. I'm not sure that bazel will accept make's setting. But I suppose bazel should have a similar option. I will check the manual.

fmunteanu · 2021-05-16T22:14:23Z

@huo-ju Bazel flags, can you give me an example of the Bazel command to build the library?

huo-ju · 2021-05-17T15:49:50Z

@fmunteanu sure, please see this line:

photoprism-freebsd-port/Makefile

Line 56 in bc17ab3

    
           cd ${WRKSRC}/docker/tensorflow/tensorflow-${TF_VERSION} && ./configure && bazel --output_user_root="${WRKDIR}/.bazel" build --config=opt //tensorflow:libtensorflow.so ${BAZEL_COPT} && ./create_archive.sh freebsd-cpu ${TF_VERSION}

lapo-luchini · 2021-05-19T06:57:01Z

@huo-ju they reference your port into bug. For the interactive questions during build, I believe make has a setting to respond with default option?

Yeah, I did that to try and let the two porting efforts meet each other. :)

huo-ju · 2021-06-06T04:28:46Z

5a6019d

The issue about interactive questions has been fixed.

distefam · 2021-06-24T12:11:07Z

I am very much looking forward to the port being available in the main ports tree.

fmunteanu · 2021-06-29T04:56:20Z

5a6019d

The issue about interactive questions has been fixed.

Amazing, do you plan to release an official port?

huo-ju · 2021-06-29T18:08:26Z

5a6019d
The issue about interactive questions has been fixed.

Amazing, do you plan to release an official port?

The libtensorflow still not meet ports tree's requirements. Sorry I can not submit it to the official tree.

fmunteanu · 2021-07-08T23:42:54Z

I believe they have a working build of py-tensorflow, to be released?

lapo-luchini · 2021-07-09T07:27:40Z

Upgrade to 1.15.5 has just been committed to FreeBSD Ports.

dla-fritz · 2021-07-09T11:55:32Z

I think I could help with that. Time permitting.

Why do you include some magic extracting libtensorflow from a docker image? There is a port of py-tensorflow already. Can't you just record a dependency on that?

https://www.freshports.org/science/py-tensorflow/

So with a working science/py-tensorflow this would be an option, right?

huo-ju · 2021-07-09T21:24:11Z

That's a good news. I can try to remove the libtensorflow building scripts from our port and add dependency to py-tensorflow.

huo-ju · 2021-07-14T15:00:08Z

I have tested the py-tensorflow but it didn't install the libtensorflow.so, although it has been built. I thought borrowing some code from the port and adding an independent libtensorflow port could be a better solution.

psa · 2021-07-15T01:06:09Z

I also noticed this and have been poking at it.

FYI, it doesn't currently work as science/py-tensorflow does not package libtensorflow.so.1

psa · 2021-07-21T12:24:08Z

Here's the bug report with a patch for science/py-tensorflow to include the shared libs.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257311

huo-ju · 2021-07-22T14:27:37Z

@psa That's great, thanks!

dml33 · 2021-10-02T11:19:12Z

Hi.

Any news about this? Still, in (Upgrade to 210925-96168e4b), tensorflow 1.15.2 is being built from zero with bazel.

BTW, are you sure that the extra patches from the Makefile are correctly addressed to be applied? I mean, I have just run make, but the repo.bzl file remained as the original one. I had to apply the patch manually.

root@Photoprism:~ # freebsd-version
12.2-RELEASE-p10

Thanks!

huo-ju · 2021-10-02T13:56:44Z

@dml33 It should be patched on Freebsd 12:

photoprism-freebsd-port/Makefile

Line 65 in 58bbd75

.if ${OPSYS} == FreeBSD && ${OSVERSION} > 1200000 && ${OSVERSION} < 1300000

Could you show me the value of OPSYS and OSVERSION?

make -C /usr/ports -V OSVERSION
make -C /usr/ports -V OPSYS

dml33 · 2021-10-02T14:09:09Z

This is my output:

root@Photoprism:/ # make -C /usr/ports -V OSVERSION
1202000
root@Photoprism:/ # make -C /usr/ports -V OPSYS
FreeBSD

When is it applied? Bazel was already compiling for a while when I lost my terminal. Later, when I opened another one, I went to

root@Photoprism:~/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2/third_party

and I realized that repo.bzl was without the patch. Then, I had to do it manually. Here are the differences:

root@Photoprism:~/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2/third_party # diff repo.bzl.orig repo.bzl
64c64
< # Runs 'git apply' on Unix, 'patch -p1' on Windows.
---
> # Runs 'patch -p1'
66,70c66,72
<     if _is_windows(ctx):
<         patch_command = ["patch", "-p1", "-d", ctx.path("."), "-i", ctx.path(patch_file)]
<     else:
<         patch_command = ["git", "apply", "-v", ctx.path(patch_file)]
<     cmd = _wrap_bash_cmd(ctx, patch_command)
---
>     # Don't check patch on Windows, because patch is only available under bash.
>     if not _is_windows(ctx) and not ctx.which("patch"):
>         fail("patch command is not found, please install it")
>     cmd = _wrap_bash_cmd(
>         ctx,
>         ["patch", "-p1", "-d", ctx.path("."), "-i", ctx.path(patch_file)],
>     )

huo-ju · 2021-10-02T15:20:36Z

The patching target should be executed after the post-extract target. So you may need run make clean first?

Please see the building output on my testing jails, and It has been applied successfully to the file.

2021-10-02 11:13:43 (2.85 MB/s) - 'v1.15.2.tar.gz' saved [46196945]
tar -xzf v1.15.2.tar.gz
cp .tf_configure.bazelrc Makefile *.sh tensorflow-1.15.2
gmake[2]: Leaving directory '/skeleton/portbuild/mnt/photoprism-freebsd-port2/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow'
===> Patching for photoprism-g20210925
===> Applying extra patch /mnt/photoprism-freebsd-port2/files/extra-patch-docker_tensorflow_tensorflow-1.15.2_third_party_repo.bzl
===> Applying FreeBSD patches for photoprism-g20210925 from /mnt/photoprism-freebsd-port2/files
===> photoprism-g20210925 depends on package: gmake>=4.3 - found
===> photoprism-g20210925 depends on file: /usr/local/bin/python3.8 - found
===> Configuring for photoprism-g20210925
===> Building for photoprism-g20210925
cd /skeleton/portbuild/mnt/photoprism-freebsd-port2/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2 && PYTHON_BIN_PATH="/usr/local/bin/python3.8" PYTHON_LIB_PATH="/usr/local/bin/python3.8/site-packages" TF_ENABLE_XLA="False" TF_DOWNLOAD_CLANG=0 TF_NEED_MPI=0 TF_SET_ANDROID_WORKSPACE=0 CC_OPT_FLAGS="-march=native -Wno-sign-compare" TF_NEED_CUDA=0 TF_NEED_OPENCL_SYCL=0 TF_NEED_ROCM=0 ./configure
Extracting Bazel installation...

dml33 · 2021-10-02T15:46:07Z

Sorry, I lost that part in the terminal which was closed... Afterwards, in the new terminal, it started directly as follows:

root@Photoprism:~/photoprism-freebsd-port # make
===>  Building for photoprism-g20210925
cd /root/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2 && PYTHON_BIN_PATH="/usr/local/bin/python3.8" PYTHON_LIB_PATH="/usr/local/bin/python3.8/site-packages" TF_ENABLE_XLA="False" TF_DOWNLOAD_CLANG=0 TF_NEED_MPI=0 TF_SET_ANDROID_WORKSPACE=0 CC_OPT_FLAGS="-march=native -Wno-sign-compare" TF_NEED_CUDA=0 TF_NEED_OPENCL_SYCL=0 TF_NEED_ROCM=0 ./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.29.0 installed.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
        --config=gdr            # Build with GDR support.
        --config=verbs          # Build with libverbs support.
        --config=ngraph         # Build with Intel nGraph support.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
        --config=v2             # Build TensorFlow 2.x instead of 1.x.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=noaws          # Disable AWS S3 filesystem support.
        --config=nogcp          # Disable GCP support.
        --config=nohdfs         # Disable HDFS support.
        --config=noignite       # Disable Apache Ignite support.
        --config=nokafka        # Disable Apache Kafka support.
        --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished
cd /root/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2 && bazel --output_user_root="/root/photoprism-freebsd-port/work/.bazel" build --jobs 2 --verbose_failures --config=opt //tensorflow:libtensorflow.so
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=94
INFO: Reading rc options for 'build' from /usr/local/etc/bazel/bazel.bazelrc:
  'build' options: --strategy=Javac=standalone --host_javabase=@local_jdk//:jdk --action_env=PATH=/bin:/usr/bin/:/usr/local/bin
INFO: Reading rc options for 'build' from /root/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2/.bazelrc:
  'build' options: --apple_platform_type=macos --define framework_shared_object=true --define open_source_build=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --copt=-w --config=v1
INFO: Reading rc options for 'build' from /root/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/local/bin/python3.8 --action_env PYTHON_LIB_PATH=/usr/local/bin/python3.8/site-packages --python_path=/usr/local/bin/python3.8 --action_env TF_CONFIGURE_IOS=0
INFO: Found applicable config definition build:v1 in file /root/photoprism-freebsd-port/work/photoprism-96168e4b36cfd95520267e6dadc7ccee1b69ec31/docker/tensorflow/tensorflow-1.15.2/.bazelrc: --define=tf_api_version=1 --action_env=TF2_BEHAVIOR=0

Anyway, when the compilation finishes, I'll do it again (from a clean state) and I'll check those lines.

A new PhotoPrism release has just been published: 211002-bf015326. Could you update the freebsd-port? Many thanks!

huo-ju · 2021-10-03T01:39:06Z

@dml33 Sure. a665337

dml33 · 2021-10-06T21:27:09Z

I can confirm that with g20211002 and a make clean before invoking the make config and make, patched perfectly the repo.bzl file 👍

But, I had to add the --jobs 2 --verbose_failures in the Makefile in the pre-build section so my NAS would not kill the process for avoiding eating the whole computer for itself and getting more information about any failure which could arise while compiling:

cd ${WRKSRC}/docker/tensorflow/tensorflow-${TF_VERSION} && bazel --output_user_root="${WRKDIR}/.bazel" build --jobs 2 --verbose_failures --config=opt //tensorflow:libtensorflow.so ${BAZEL_COPT}

How about letting the user choose whether to include these options in the menu of make config?

Thanks again

psa · 2021-12-27T07:51:38Z

I have a port for science/libtensorflow1 in progress. Next steps to make www/photoprism into a port are to add the node and go packages as GH_TUPLE to the Makefile and rebuild the distinfo.

huo-ju · 2021-12-27T15:22:07Z

I have a port for science/libtensorflow1 in progress. Next steps to make www/photoprism into a port are to add the node and go packages as GH_TUPLE to the Makefile and rebuild the distinfo.

Thanks, we will. I asked a port committer to help you port, hoping it will be processed faster.

psa · 2021-12-27T17:22:12Z

I've been poking at this a bit more today. The Go modules are going to be fairly straightforward I believe (I've already got the tuples sorted) and the tensorflow training sets are very easy (done). The node modules are, so far, proving to be quite annoying.

Anyone know of an existing tool for packages.json -> GH_TUPLE?

kowalcj0 · 2022-01-04T22:12:09Z

Hi @psa
I've installed libtensorflow1 binaries from your port https://github.com/psa/libtensorflow1-freebsd-port
but I'm getting following error when trying to set the initial photoprism admin password:

photoprism --assets-path=/var/db/photoprism/assets --storage-path=/var/db/photoprism/storage --originals-path=/var/db/photoprism/storage/originals --import-path=/var/db/photoprism/storage/import passwd
ld-elf.so.1: /usr/local/lib/libtensorflow.so.1: Undefined symbol "rand@FBSD_1.6"

I'm trying to get it to work in a jail on TrueNas:

uname -a
FreeBSD prism20211212 12.2-RELEASE-p11 FreeBSD 12.2-RELEASE-p11 75566f060d4(HEAD) TRUENAS  amd64

Thanks

dml33 · 2022-01-04T22:25:22Z

Hi, @psa.
In my case, I could compile libtensorflow1 for FreeBSD 12.2, as needed by @kowalcj0, by doing some adaptations as described in psa/libtensorflow1-freebsd-port#2. I have the pkg ready. If you give me permissions, I could upload it to Releases, together with yours.
Thanks

gvelim · 2022-01-05T16:40:42Z

With @dml33 pointers I have put some workaround instructions here psa/libtensorflow1-freebsd-port#2 (comment)

Tensorflow build complete successfully and finished installing with make install

kowalcj0 · 2022-01-05T23:02:54Z

Hi, @psa. In my case, I could compile libtensorflow1 for FreeBSD 12.2, as needed by @kowalcj0, by doing some adaptations as described in psa/libtensorflow1-freebsd-port#2. I have the pkg ready. If you give me permissions, I could upload it to Releases, together with yours. Thanks

@dml33 I've installed libtensorflow1 using psa's binary package https://github.com/psa/libtensorflow1-freebsd-port/releases/tag/1.15.5

The error that I'm getting happens when I'm trying to run already compiled and installed photoprism.

dml33 · 2022-01-05T23:20:16Z

Hi, @psa. In my case, I could compile libtensorflow1 for FreeBSD 12.2, as needed by @kowalcj0, by doing some adaptations as described in psa/libtensorflow1-freebsd-port#2. I have the pkg ready. If you give me permissions, I could upload it to Releases, together with yours. Thanks

@dml33 I've installed libtensorflow1 using psa's binary package https://github.com/psa/libtensorflow1-freebsd-port/releases/tag/1.15.5

The error that I'm getting happens when I'm trying to run already compiled and installed photoprism.

I think that your error comes because psa's binary package https://github.com/psa/libtensorflow1-freebsd-port/releases/tag/1.15.5 is for FreeBSD 13, but your FreeBSD jail is in 12.2. I have compiled a package for FreeBSD 12.2, but I cannot upload it to psa repository until he gives me permissions. In my case, it will be with AVX instructions disabled.

kowalcj0 · 2022-01-05T23:26:56Z

Gotcha @dml33
Thank you for explaining it. Indeed, it's for FreeBSD 13 :)

lastzero · 2022-01-09T06:45:32Z

Is there anything we can help with so maintaining the port gets easier? We now have FreeBSD 13 running on an iMac 😈

psa · 2022-01-09T07:07:46Z

@dml33 I've added you as a collaborator to the libtensorflow1 port.

Thanks for the build (and the updates).

dml33 · 2022-01-09T08:32:51Z

I've updated the libtensorflow1 port release 1.15.5 with a package for FreeBSD 12.2 and noAVX support.

@psa Thanks!! 👍

crees · 2022-06-10T09:11:36Z

Anyone know of an existing tool for packages.json -> GH_TUPLE?

@psa I just made a sed script for it :)

I'm hoping to commit science/libtensorflow1 this weekend-- thanks very much for the work on the port!

Perhaps I could add a maintainer target to the port to extract the stuff and make the GH_TUPLES for you?

reyman · 2022-08-08T10:09:15Z

Hi !
Same here, I try to compile into freebsd 13.1 release into jail by cloning https://github.com/psa/libtensorflow1-freebsd-port libtensorflow1 but making package fail at linking :

ERROR: /usr/ports/science/libtensorflow1/work-default/tensorflow-1.15.5/tensorflow/BUILD:563:1: Linking of rule '//tensorflow:libtensorflow_framework.so.1.15.5' failed (Exit 1)
ld: error: duplicate symbol: adler32_z
>>> defined at adler32.c
>>>            adler32.pic.o:(adler32_z) in archive bazel-out/host/bin/external/zlib/libzlib.pic.a
>>> defined at adler32.c
>>>            adler32.pic.o:(.text.adler32_z+0x0) in archive bazel-out/host/bin/external/zlib_archive/libzlib.pic.a

@crees any help appreciated :D !

Update 1 :

Applying this patch on ./work-default/tensorflow-1.15.5/third_party/repo.bzl solve the problem, is it possible to integrate directly this into port ?
tensorflow/tensorflow#31196

MostFrabjous · 2023-06-07T19:34:37Z

This works here on FreeBSD 13:

portsnap fetch && portsnap extract

wget https://github.com/psa/libtensorflow1-freebsd-port/releases/download/1.15.5-pre-release-0/libtensorflow1-1.15.5-FreeBSD-13.0-AVX.pkg
pkg add libtensorflow1-1.15.5-FreeBSD-13.0-AVX.pkg

pkg install git
git clone https://github.com/huo-ju/photoprism-freebsd-port

cd ~/photoprism-freebsd-port
make config-recursive
make && make install

lapo-luchini mentioned this issue Aug 30, 2022

Build marked BROKEN with 404 file psa/libtensorflow1-freebsd-port#5

Closed

Available FreeBSD port? #13

Available FreeBSD port? #13

Comments

fmunteanu commented May 14, 2021 • edited Loading

pmhausen commented May 14, 2021 • edited Loading

fmunteanu commented May 14, 2021 • edited Loading

pmhausen commented May 14, 2021

huo-ju commented May 14, 2021 • edited Loading

pmhausen commented May 14, 2021 • edited Loading

huo-ju commented May 14, 2021

fmunteanu commented May 15, 2021 • edited Loading

huo-ju commented May 15, 2021

fmunteanu commented May 16, 2021 • edited Loading

huo-ju commented May 17, 2021

lapo-luchini commented May 19, 2021

huo-ju commented Jun 6, 2021

distefam commented Jun 24, 2021

fmunteanu commented Jun 29, 2021

huo-ju commented Jun 29, 2021

fmunteanu commented Jul 8, 2021 • edited Loading

lapo-luchini commented Jul 9, 2021

dla-fritz commented Jul 9, 2021

huo-ju commented Jul 9, 2021

huo-ju commented Jul 14, 2021

psa commented Jul 15, 2021

psa commented Jul 21, 2021

huo-ju commented Jul 22, 2021

dml33 commented Oct 2, 2021

huo-ju commented Oct 2, 2021

dml33 commented Oct 2, 2021

huo-ju commented Oct 2, 2021

dml33 commented Oct 2, 2021

huo-ju commented Oct 3, 2021

dml33 commented Oct 6, 2021

psa commented Dec 27, 2021

huo-ju commented Dec 27, 2021 • edited Loading

psa commented Dec 27, 2021

kowalcj0 commented Jan 4, 2022

dml33 commented Jan 4, 2022

gvelim commented Jan 5, 2022

kowalcj0 commented Jan 5, 2022

dml33 commented Jan 5, 2022

kowalcj0 commented Jan 5, 2022

lastzero commented Jan 9, 2022

psa commented Jan 9, 2022

dml33 commented Jan 9, 2022

crees commented Jun 10, 2022 • edited Loading

reyman commented Aug 8, 2022 • edited Loading

MostFrabjous commented Jun 7, 2023 • edited Loading

fmunteanu commented May 14, 2021 •

edited

Loading

pmhausen commented May 14, 2021 •

edited

Loading

fmunteanu commented May 14, 2021 •

edited

Loading

huo-ju commented May 14, 2021 •

edited

Loading

pmhausen commented May 14, 2021 •

edited

Loading

fmunteanu commented May 15, 2021 •

edited

Loading

fmunteanu commented May 16, 2021 •

edited

Loading

fmunteanu commented Jul 8, 2021 •

edited

Loading

huo-ju commented Dec 27, 2021 •

edited

Loading

crees commented Jun 10, 2022 •

edited

Loading

reyman commented Aug 8, 2022 •

edited

Loading

MostFrabjous commented Jun 7, 2023 •

edited

Loading