Unlimited history

[CyberShadow]

Hi,

The Windows open-source clipboard manager Ditto has an unique feature which I've yet to find in any Linux clipboard manager: unlimited history.

That is, there can be no limit to the number of stored entries. All entries are saved in an SQLite database, which is indexed with SQLite's excellent full text indexing. This allows incrementally searching through hundreds of thousands of clipboard entries in seconds.

I have found this to be incredibly convenient, as it means that I can always cut instead of delete anything, with the peace of mind that I can always recover it later. It also allows me to quickly paste in code snippets written any amount of time ago.

Would this be a viable feature request for this project?

[memeplex]

Is this so different than, say, set history size to 50000000?

What I would find more useful is a handy way to specify "output all entries" with -n since writing 50000000 is a bit cumbersome at the CLI.

[mrichar1]

@memeplex I've just taken a look at the code for this, and since the value of -nis just used to slice the history list, I've very slightly modified the code so that -n can now take 0 and negative numbers.

clipster -o -n 0 - return entire history.

clipster -o -n " -5" - return all but first 5 items in history. (this odd syntax is due to an argparse bug, fixed in some releases: https://code.google.com/archive/p/argparse/issues/37

Changes are in: 7ff78f1 (plus docs changes in a later commit).

[mrichar1]

@CyberShadow apologies for not replying to your original issue sooner... was sure I had!

While the idea of a DB backend is appealing, it does add an extra layer of complexity and dependencies to the code, which I'm not convinced is entirely justified.

As has been pointed out, history size can be set very large (and could probably be made to be unlimited relatively easily by ading config to disable the 'pruning' code).

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend. I suspect however that the number will be surprisingly large! (for a bit of fun, have a look at https://computers-are-fast.github.io/)

[magnetophon]

I've been wanting to update this issue for a while now, since clipster has turned slow for me, presumably because of my history size.
Unfortunately my ssd died a few days ago, and now I don'd know what my size was.
All I know is that it was less than 50000 entries, as that is what my history was capped at.

[mrichar1]

Just to follow up on this - I've done some rough and ready speed testing.

The 'baseline' speed for the clipster client on my laptop is 0.1secs - i.e if I tell it to output the history when the daemon isn't running, and it exits immediately.

If I query a 'small' history this adds about 0.025secs to the process.

If I generate a 500MB history file, made up of 50,000 10KB entries, it still only adds 0.025secs.

The only thing that is 'slow' is if I then do clipster -o -n 10000 and even then the total time is under a second.

I can't easily test a larger history as I start to run out of memory!

So I'm reluctant to accept that clipster is slowing down for you due to history size - unless of course the history has got slow large that it's starting to move into swap or something?

There are other bugs open related to performance (like #74) which I need to dig into - this might also be the cause of your slowdowns.

[CyberShadow]

As has been pointed out, history size can be set very large (and could probably be made to be unlimited relatively easily by ading config to disable the 'pruning' code).

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend. I suspect however that the number will be surprisingly large! (for a bit of fun, have a look at https://computers-are-fast.github.io/)

Hi, sorry also for the late reply - looks like I missed your message.

I don't use Windows (and therefore Ditto) much any longer, but as I've left it, my Ditto database measures at over 7 GB. Suffice to say, keeping it all in RAM would be an unreasonable misuse in resources. The 7 GB figure comes after periodic cleanups for deleting screenshots or other very large (over several MB) clips. The database is currently at 195539 clips, which puts them at a mean size of 37508 bytes per clip (incl. metadata such as search indexes), which to me seems not unreasonable.

Some of the larger clips are things like:

• log files
• terminal session transcripts
• entire source files
• program outputs
• chat logs

Again, IMO, not unreasonable.

So, I do still think there is a use case here :)

[magnetophon]

I'd be interested to know at what history size the current code starts to be noticeably slower (to a human) - if that number is suitably small then I might consider an alternative history backend.

Mine is 38M, and very slow.
Getting the output of clipster -o -n 0 -0 | fzf --read0 --no-sort --reverse --preview='echo {}' | sed -ze 's/\n$//' | clipster takes a few seconds, and pasting even longer.
I am in swap though.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[CyberShadow]

@mrichar1 Please don't use stale bots, they do more harm than good. Not just in terms of value they bring to the particular project, but they generally erode public trust and willingness to participate in open-source software development.
See the discussions here: https://www.google.com/search?q=stale+bot+site%3Aycombinator.com

[mrichar1]

The stalebot was actually added in August last year with a view to tweaking it to do minimal tidying. Unfortunately it never fired, so I forgot about it until this last week when it 'woke up'. Unfortunately I was away so missed the pending stale events to stop it before it happened.

I generally try to apply tags to 'pending' issues which haven't been fully resolved and had configured stalebot to ignore some of these - but unfortunately missed a couple, which is why so many 'useful' issues got closed. I've now reopened all of these so they can continue to receive input.