Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nextcloud: add recognize app #333545

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

beardhatcode
Copy link
Contributor

@beardhatcode beardhatcode commented Aug 9, 2024

Description of changes

Alters the way nextcloud28Packages etc is defined to allow adding custom packages.

In this case to add the recognize app which downloads the ML models at runtime (which is not compatible with the read only filesystem). Additionally, I also patch in the version of nodejs (in a very dirty way, for now). I tested it and it says

✔️ The machine learning models have been downloaded successfully.

and

✔️ Node.js v20.14.0 binary was installed successfully.

but also

⚠️ Could not load libtensorflow in Node.js. You can try to manually install libtensorflow or run in WASM mode.

This is mostly a request for comments on of how a thing like this can be done. So, @Ma27, @dotlambda, @onny, and @pyrox0 (the last 4 editors of the Nextcloud packages folder), what do you think?

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@beardhatcode beardhatcode force-pushed the feat/ncapps-recognise branch 2 times, most recently from 69a4a79 to 2c18298 Compare August 10, 2024 08:04
@teutat3s
Copy link
Member

Related PR: #211631

let packages = self:
let
apps = lib.importJSON (../servers/nextcloud/packages/. + (builtins.toPath "${ncVersion}.json"));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
apps = lib.importJSON (../servers/nextcloud/packages/. + (builtins.toPath "${ncVersion}.json"));
apps = lib.importJSON (../servers/nextcloud/packages/ + "${ncVersion}.json");

IIRC

fetchurl,
lib,
nodejs,
...
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
...

tar -xzpf "${builtins.elemAt srcs 0}" recognize;
tar -xzpf "${builtins.elemAt srcs 1}" recognize-${version}/models;
mv recognize-${version}/models recognize
sed -i "/'node_binary'/s:'""':'${nodejs}/bin/node':" recognize/lib/Service/SettingsService.php
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use substituteInPlace

meta = with lib; {
license = licenses.agpl3Only;
maintainers = with maintainers; [ beardhatcode ];
longDescription = ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We always must add a description

Copy link
Member

@Ma27 Ma27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means we don't need something like https://github.com/NuschtOS/nixos-modules/blob/da5286bc062adee0e0aaf2bd3b784b477c623422/modules/nextcloud.nix#L172-L183 but still https://github.com/NuschtOS/nixos-modules/blob/da5286bc062adee0e0aaf2bd3b784b477c623422/modules/nextcloud.nix#L210-L217 ?

To be clear: I'm very, very 👎 on doing any build steps at runtime.

I expect the JS code to be built correctly and to have the settings patched by this package to point to other Nix packages (I.e. node, ffmpeg etc.). I looked through those super-questionable[0] "migration" "steps"[1] and I really don't want to rely on any of that.

[0] to say the least
[1] https://github.com/nextcloud/recognize/blob/main/lib/Migration/InstallDeps.php

@@ -23,6 +23,9 @@ let packages = self:
inherit (data) url hash description homepage;
}) {};


recognize = (pkgs.callPackage ./adapted/nextcloud-app-recognize.nix { });

} // lib.mapAttrs (type: pkgs:
lib.makeExtensible (_: lib.mapAttrs (pname: data: self.mkNextcloudDerivation { inherit pname; inherit data; }) pkgs))
generatedJson;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right below the line here there's an extend. Would it be an option to add recognize to the JSON and do the necessary overrides there?

Would have the nice side-efffect that it's updated along with all other apps.

See also #328634 for related changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the package in the apps.json of nextcloud expects that it is in a writable filesystem. And then at first run it downloads some more stuff, the 2 sources need to be aligned, therefore I would keep it out of the apps.json (also making it clear that it is a special case app).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should download those files at build time if possible.
We can verify if the right version was downloaded with grep or so.

@beardhatcode
Copy link
Contributor Author

This means we don't need something like https://github.com/NuschtOS/nixos-modules/blob/da5286bc062adee0e0aaf2bd3b784b477c623422/modules/nextcloud.nix#L172-L183 but still https://github.com/NuschtOS/nixos-modules/blob/da5286bc062adee0e0aaf2bd3b784b477c623422/modules/nextcloud.nix#L210-L217 ?

I was planning to wrap the nodejs to add those packages so they do not need to be in the module. This way it "just works" if you enable this. But I'm still figuring out how to do this nicely.

@onny
Copy link
Contributor

onny commented Aug 27, 2024

I'm also going to add a third party app to the package set #328634

@SuperSandro2000
Copy link
Member

To be clear: I'm very, very 👎 on doing any build steps at runtime.

I don't use declerative nextcloud apps, so I must do it like this.

So we want to patch the app before in nextcloudApps.

@Ma27
Copy link
Member

Ma27 commented Aug 28, 2024

I don't use declerative nextcloud apps, so I must do it like this.

That's OK, my comment was strictly about what I'd like to see (or not see) in nixpkgs.

So we want to patch the app before in nextcloudApps.

Most likely yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants