Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prop/ref value search for datasets #138

Open
andebor opened this issue Nov 17, 2022 · 0 comments
Open

Add prop/ref value search for datasets #138

andebor opened this issue Nov 17, 2022 · 0 comments
Labels
enhancement New feature or request Priority:High

Comments

@andebor
Copy link
Contributor

andebor commented Nov 17, 2022

We currently don't have a built in feature to search for specific property values in a datasets. Currently we solve this by piping the whole dataset to jq.

mim dataset entities stuff.Mydata --limit -1 --json | jq ' map(select(.props["ns80:my_property"] == "foo"))'

This can be very memory demanding for large datasets and quite unreliable. I propose that until we have a proper search to handle this in the backend, we add a command that will do more or less the same in cli. That would allow us to go through the dataset batch by batch, avoiding filling up the memory with temp entities.

It could look something like this:

mim dataset search stuff.Mydata --prop-filter ns80:my_property "foo"
mim dataset search stuff.Mydata --ref-filter ns80:my_ref "ns:60:bar"

I suggest that we also add a pterm progressbar output while searching. https://github.com/pterm/pterm/tree/master/_examples/progressbar

@andebor andebor added the enhancement New feature or request label Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Priority:High
Projects
None yet
Development

No branches or pull requests

1 participant