Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get settings info out of headers, out of libutil #11033

Draft
wants to merge 48 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
f90e771
feat: Refactor unix domain socket store config
fzakaria Jul 15, 2024
1f72267
uds-remote-store is now clang-formatted
fzakaria Jul 15, 2024
660d868
Removed clang-format changes
fzakaria Jul 15, 2024
6585c6e
Reference public variable
fzakaria Jul 15, 2024
10c5c65
Update src/libstore/uds-remote-store.hh
fzakaria Jul 15, 2024
030d91d
Move constant to within Config object
fzakaria Jul 15, 2024
7169905
Lowercase constexpr
fzakaria Jul 15, 2024
4af5092
Update src/libstore/store-api.cc
fzakaria Jul 15, 2024
dd74b72
Revert back returning daemon for URI if path empty
fzakaria Jul 15, 2024
01fbb4b
Added unit test
fzakaria Jul 15, 2024
6eb3303
Added to meson
fzakaria Jul 15, 2024
d3275e8
Changed to protected and added another test
fzakaria Jul 16, 2024
0087d7d
Clang formatted test file
fzakaria Jul 16, 2024
1e97346
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 16, 2024
e733f6d
Move `UdsRemoteStore{,Config}` defs out of headers, and other cleanups
Ericson2314 Jul 16, 2024
e435f3e
Add two more headers to try to provide missing template instantiations
Ericson2314 Jul 16, 2024
ed2ccdc
Add globals.hh to uds-remote-store.cc
fzakaria Jul 16, 2024
e9c0636
Fix linking issue, convert many more
Ericson2314 Jul 16, 2024
8162dbd
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 16, 2024
f23d168
`make format`
Ericson2314 Jul 16, 2024
c4c148f
Declare some template specializations in the header
Ericson2314 Jul 16, 2024
13d573b
WIP get settings info out of headers, out of libutil
Ericson2314 Jul 2, 2024
4e79b4e
Move `uriSchemes` to `*StoreConfig`
Ericson2314 Jul 16, 2024
1e25c22
Revert "Declare some template specializations in the header"
Ericson2314 Jul 16, 2024
ea3a420
Merge branch 'move-schemes-set' into new-store-settings
Ericson2314 Jul 16, 2024
414d6ad
`make format`
Ericson2314 Jul 17, 2024
fcf3e32
Merge branch 'move-schemes-set' into HEAD
Ericson2314 Jul 17, 2024
3b0edea
WIP Start getting store settings off config.hh
Ericson2314 Jul 17, 2024
afee3ee
Merge branch 'new-store-settings' into settings-split
Ericson2314 Jul 17, 2024
d095fa3
WIP
Ericson2314 Jul 17, 2024
121e607
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 17, 2024
7b135d5
Comment out tests
fzakaria Jul 17, 2024
596a682
Dummy store compiles
Ericson2314 Jul 17, 2024
95b36f4
Merge branch 'issue-10766-unix-socket' of github.com:fzakaria/nix int…
Ericson2314 Jul 17, 2024
0e65638
Merge remote-tracking branch 'upstream/master' into issue-10766-unix-…
Ericson2314 Jul 17, 2024
34f876f
Format, CI hopefully happy now
Ericson2314 Jul 17, 2024
a7d9394
Merge branch 'issue-10766-unix-socket' into new-store-settings
Ericson2314 Jul 17, 2024
4909f9a
Get `LocalFSStore` building
Ericson2314 Jul 18, 2024
9052c09
Merge commit '17051ca80ab9bcee896e040cfe0ea699702ac0ff' into new-stor…
Ericson2314 Jul 18, 2024
118b050
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 18, 2024
307440b
Convert `LocalStoreConfig` to new system
Ericson2314 Jul 18, 2024
de1ec5a
A bunch of small fixes
Ericson2314 Jul 18, 2024
f7df182
Merge branch 'new-store-settings' into settings-split
Ericson2314 Jul 18, 2024
9b30c20
Merge remote-tracking branch 'upstream/master' into new-store-settings
Ericson2314 Jul 18, 2024
bb7bf73
More progress converting
Ericson2314 Jul 18, 2024
ca1e895
WIP
Ericson2314 Jul 18, 2024
901b34f
WIP
Ericson2314 Jul 18, 2024
9e6d1eb
Merge branch 'new-store-settings' into settings-split
Ericson2314 Jul 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions maintainers/flake-module.nix
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,14 @@
''^src/libflake/flake/lockfile\.cc$''
''^src/libflake/flake/lockfile\.hh$''
''^src/libflake/flake/url-name\.cc$''
''^src/libmain/args/root\.hh$''
''^src/libmain/args\.cc$''
''^src/libmain/args\.hh$''
''^src/libmain/common-args\.cc$''
''^src/libmain/common-args\.hh$''
''^src/libmain/config-impl\.hh$''
''^src/libmain/config\.cc$''
''^src/libmain/config\.hh$''
''^src/libmain/loggers\.cc$''
''^src/libmain/loggers\.hh$''
''^src/libmain/progress-bar\.cc$''
Expand Down Expand Up @@ -258,19 +264,13 @@
''^src/libutil-c/nix_api_util_internal\.h$''
''^src/libutil/archive\.cc$''
''^src/libutil/archive\.hh$''
''^src/libutil/args\.cc$''
''^src/libutil/args\.hh$''
''^src/libutil/args/root\.hh$''
''^src/libutil/callback\.hh$''
''^src/libutil/canon-path\.cc$''
''^src/libutil/canon-path\.hh$''
''^src/libutil/chunked-vector\.hh$''
''^src/libutil/closure\.hh$''
''^src/libutil/comparator\.hh$''
''^src/libutil/compute-levels\.cc$''
''^src/libutil/config-impl\.hh$''
''^src/libutil/config\.cc$''
''^src/libutil/config\.hh$''
''^src/libutil/current-process\.cc$''
''^src/libutil/current-process\.hh$''
''^src/libutil/english\.cc$''
Expand Down
2 changes: 1 addition & 1 deletion src/build-remote/build-remote.cc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#include "globals.hh"
#include "serialise.hh"
#include "build-result.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "strings.hh"
#include "derivations.hh"
#include "local-store.hh"
Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/command.cc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

#include "command.hh"
#include "markdown.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "local-fs-store.hh"
#include "derivations.hh"
#include "nixexpr.hh"
Expand Down
2 changes: 1 addition & 1 deletion src/libcmd/common-eval-args.cc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
#include "registry.hh"
#include "flake/flakeref.hh"
#include "flake/settings.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "command.hh"
#include "tarball.hh"
#include "fetch-to-store.hh"
Expand Down
4 changes: 2 additions & 2 deletions src/libcmd/repl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
#include "eval-settings.hh"
#include "attr-path.hh"
#include "signals.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "log-store.hh"
#include "common-eval-args.hh"
#include "get-drvs.hh"
Expand Down Expand Up @@ -635,7 +635,7 @@ ProcessLineResult NixRepl::processLine(std::string line)
// When missing, trigger the normal exception
// e.g. :doc builtins.foo
// behaves like
// nix-repl> builtins.foo
// nix-repl> builtins.foo
// error: attribute 'foo' missing
evalString(arg, v);
assert(false);
Expand Down
2 changes: 1 addition & 1 deletion src/libexpr/primops/fetchClosure.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#include "primops.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "realisation.hh"
#include "make-content-addressed.hh"
#include "url.hh"
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
44 changes: 44 additions & 0 deletions src/libmain/config-upstream.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
namespace nix {

#include "fs-sink.hh"
#include "logging.hh"

struct LoggerSettings : Config
{
Setting<bool> showTrace{
this,
false,
"show-trace",
R"(
Whether Nix should print out a stack trace in case of Nix
expression evaluation errors.
)"};
};

static GlobalConfig::Register r1(&restoreSinkSettings);

struct RestoreSinkSettings : Config
{
Setting<bool> preallocateContents{
this, false, "preallocate-contents", "Whether to preallocate files when writing objects with known size."};
};

static GlobalConfig::Register rLoggerSettings(&loggerSettings);

struct ArchiveSettings : Config
{
Setting<bool> useCaseHack
{
this,
#if __APPLE__
true,
#else
false,
#endif
"use-case-hack", "Whether to enable a Darwin-specific hack for dealing with file name collisions."
};
};

static GlobalConfig::Register rArchiveSettings(&archiveSettings);

}
22 changes: 0 additions & 22 deletions src/libutil/config.cc → src/libmain/config.cc
Original file line number Diff line number Diff line change
Expand Up @@ -452,26 +452,4 @@ void OptionalPathSetting::operator =(const std::optional<Path> & v)
this->assign(v);
}

bool ExperimentalFeatureSettings::isEnabled(const ExperimentalFeature & feature) const
{
auto & f = experimentalFeatures.get();
return std::find(f.begin(), f.end(), feature) != f.end();
}

void ExperimentalFeatureSettings::require(const ExperimentalFeature & feature) const
{
if (!isEnabled(feature))
throw MissingExperimentalFeature(feature);
}

bool ExperimentalFeatureSettings::isEnabled(const std::optional<ExperimentalFeature> & feature) const
{
return !feature || isEnabled(*feature);
}

void ExperimentalFeatureSettings::require(const std::optional<ExperimentalFeature> & feature) const
{
if (feature) require(*feature);
}

}
3 changes: 0 additions & 3 deletions src/libutil/config.hh → src/libmain/config.hh
Original file line number Diff line number Diff line change
Expand Up @@ -257,9 +257,6 @@ public:
, documentDefault(documentDefault)
{ }

operator const T &() const { return value; }
operator T &() { return value; }
const T & get() const { return value; }
template<typename U>
bool operator ==(const U & v2) const { return value == v2; }
template<typename U>
Expand Down
3 changes: 2 additions & 1 deletion src/libstore-c/nix_api_store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

#include "path.hh"
#include "store-api.hh"
#include "store-open.hh"
#include "build-result.hh"

#include "globals.hh"
Expand Down Expand Up @@ -42,7 +43,7 @@ Store * nix_store_open(nix_c_context * context, const char * uri, const char ***
if (!params)
return new Store{nix::openStore(uri_str)};

nix::Store::Params params_map;
nix::StoreReference::Params params_map;
for (size_t i = 0; params[i] != nullptr; i++) {
params_map[params[i][0]] = params[i][1];
}
Expand Down
72 changes: 69 additions & 3 deletions src/libstore/binary-cache-store.cc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
#include "callback.hh"
#include "signals.hh"
#include "archive.hh"
#include "config-parse-impl.hh"

#include <chrono>
#include <future>
Expand All @@ -24,9 +25,74 @@

namespace nix {

BinaryCacheStore::BinaryCacheStore(const Params & params)
: BinaryCacheStoreConfig(params)
, Store(params)
BinaryCacheStore::Config::Descriptions::Descriptions()
: Store::Config::Descriptions{Store::Config::descriptions}
, BinaryCacheStoreConfigT<config::SettingInfo>{
.compression = {
.name = "compression",
.description = "NAR compression method (`xz`, `bzip2`, `gzip`, `zstd`, or `none`).",
},
.writeNARListing = {
.name = "write-nar-listing",
.description = "Whether to write a JSON file that lists the files in each NAR.",
},
.writeDebugInfo = {
.name = "index-debug-info",
.description = R"(
Whether to index DWARF debug info files by build ID. This allows [`dwarffs`](https://github.com/edolstra/dwarffs) to
fetch debug info on demand
)",
},
.secretKeyFile{
.name = "secret-key",
.description = "Path to the secret key used to sign the binary cache.",
},
.localNarCache{
.name = "local-nar-cache",
.description = "Path to a local cache of NARs fetched from this binary cache, used by commands such as `nix store cat`.",
},
.parallelCompression{
.name = "parallel-compression",
.description = "Enable multi-threaded compression of NARs. This is currently only available for `xz` and `zstd`.",
},
.compressionLevel{
.name = "compression-level",
.description = R"(
The *preset level* to be used when compressing NARs.
The meaning and accepted values depend on the compression method selected.
`-1` specifies that the default compression level should be used.
)",
},
}
{}

const BinaryCacheStore::Config::Descriptions BinaryCacheStore::Config::descriptions{};

decltype(BinaryCacheStore::Config::defaults) BinaryCacheStore::Config::defaults = {
.compression = {"xz"},
.writeNARListing{false},
.writeDebugInfo{false},
.secretKeyFile{""},
.localNarCache{""},
.parallelCompression{false},
.compressionLevel{-1},
};

BinaryCacheStore::Config::BinaryCacheStoreConfig(const StoreReference::Params & params)
: StoreConfig{params}
, BinaryCacheStoreConfigT<config::JustValue>{
CONFIG_ROW(compression),
CONFIG_ROW(secretKeyFile),
CONFIG_ROW(localNarCache),
CONFIG_ROW(parallelCompression),
CONFIG_ROW(compressionLevel),
}
{
}

BinaryCacheStore::BinaryCacheStore(const Config & config)
: Config{config}
, Store{static_cast<const Store::Config &>(*this)}
{
if (secretKeyFile != "")
signer = std::make_unique<LocalSigner>(
Expand Down
66 changes: 34 additions & 32 deletions src/libstore/binary-cache-store.hh
Original file line number Diff line number Diff line change
Expand Up @@ -13,48 +13,46 @@ namespace nix {

struct NarInfo;

struct BinaryCacheStoreConfig : virtual StoreConfig
template<template<typename> class F>
struct BinaryCacheStoreConfigT
{
using StoreConfig::StoreConfig;

const Setting<std::string> compression{this, "xz", "compression",
"NAR compression method (`xz`, `bzip2`, `gzip`, `zstd`, or `none`)."};

const Setting<bool> writeNARListing{this, false, "write-nar-listing",
"Whether to write a JSON file that lists the files in each NAR."};

const Setting<bool> writeDebugInfo{this, false, "index-debug-info",
R"(
Whether to index DWARF debug info files by build ID. This allows [`dwarffs`](https://github.com/edolstra/dwarffs) to
fetch debug info on demand
)"};
const F<std::string> compression;
const F<bool> writeNARListing;
const F<bool> writeDebugInfo;
const F<Path> secretKeyFile;
const F<Path> localNarCache;
const F<bool> parallelCompression;
const F<int> compressionLevel;
};

const Setting<Path> secretKeyFile{this, "", "secret-key",
"Path to the secret key used to sign the binary cache."};
struct BinaryCacheStoreConfig :
virtual Store::Config,
BinaryCacheStoreConfigT<config::JustValue>
{
struct Descriptions :
virtual Store::Config::Descriptions,
BinaryCacheStoreConfigT<config::SettingInfo>
{
Descriptions();
};

const Setting<Path> localNarCache{this, "", "local-nar-cache",
"Path to a local cache of NARs fetched from this binary cache, used by commands such as `nix store cat`."};
static const Descriptions descriptions;

const Setting<bool> parallelCompression{this, false, "parallel-compression",
"Enable multi-threaded compression of NARs. This is currently only available for `xz` and `zstd`."};
static BinaryCacheStoreConfigT<config::JustValue> defaults;

const Setting<int> compressionLevel{this, -1, "compression-level",
R"(
The *preset level* to be used when compressing NARs.
The meaning and accepted values depend on the compression method selected.
`-1` specifies that the default compression level should be used.
)"};
BinaryCacheStoreConfig(const StoreReference::Params &);
};


/**
* @note subclasses must implement at least one of the two
* virtual getFile() methods.
*/
class BinaryCacheStore : public virtual BinaryCacheStoreConfig,
public virtual Store,
public virtual LogStore
struct BinaryCacheStore :
virtual BinaryCacheStoreConfig,
virtual Store,
virtual LogStore
{
using Config = BinaryCacheStoreConfig;

private:
std::unique_ptr<Signer> signer;
Expand All @@ -64,7 +62,7 @@ protected:
// The prefix under which realisation infos will be stored
const std::string realisationsPrefix = "realisations";

BinaryCacheStore(const Params & params);
BinaryCacheStore(const Config &);

public:

Expand Down Expand Up @@ -96,7 +94,11 @@ public:

public:

virtual void init() override;
/**
* Perform any necessary effectful operation to make the store up and
* running
*/
virtual void init();

private:

Expand Down
1 change: 1 addition & 0 deletions src/libstore/build/drv-output-substitution-goal.cc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
#include "worker.hh"
#include "substitution-goal.hh"
#include "callback.hh"
#include "store-open.hh"

namespace nix {

Expand Down
1 change: 1 addition & 0 deletions src/libstore/build/substitution-goal.cc
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#include "worker.hh"
#include "store-open.hh"
#include "substitution-goal.hh"
#include "nar-info.hh"
#include "finally.hh"
Expand Down
Loading
Loading