diff --git a/.gitignore b/.gitignore index a05c3f8..eeade7a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ -settings.py - +settings.yaml +logo/ # Setuptools distribution folder. /dist/ diff --git a/README.md b/README.md index be55505..c10f155 100644 --- a/README.md +++ b/README.md @@ -1,30 +1,22 @@ -# pocketsnack +# KonMari your Pocket tsundoku from the command line -When your Pocket list is overwhelming, pocket-snack lets you see just what you can read today +`pocketsnack` is a command line application offering various commands to make your [Pocket](https://getpocket.com) account more manageable. You can de-duplicate your list, purge unwanted tags, and hide your enormous 'to be read' list in a special archive so that looking at your list doesn't become paralysing. -This is the version 2 documentation. If you haven't yet upgraded you can still use the [version 1 README](v1_README.md). +This is the version 3 documentation. If you don't want to upgrade you can still read the [version 2 README](v2_README.md) or [version 1 README](v1_README.md). -## A note on version 2 +## A note on version 3 -All commands have changed since version 1 - read the _Usage_ section carefully. This was necessary in order to provide better functionality without making the code too confusing. +Version 3 introduces a new YAML format for the settings file. This approach also allows for changes to the settings file without having to re-install `pocketsnack`, which was an unintended side effect of the previous approach of simply importing the file. Additional documentation on the settings file can be found below. -One of the changes is that the `refresh` command from version 1 no longer exists. This is so that `--since`, and `--before` can be used with both `--stash` and `lucky_dip`. If you want replicate `refresh` you simply need to run `--stash` followed by `--lucky_dip`. From the command line you could do: - -```bash -pocketsnack -s && pocketsnack -d -``` - -The automation of `pocketsnack refresh` has _also_ been removed. This didn't really work very consistently, and was causing a lot of maintenance headaches. I'm looking at how to bring it back in a different way, but for now it's been removed. - -## Getting started (tl;dr) +## Getting started 1. make sure you have installed Python version 3.x (preferably 3.7 or higher) 2. download `pocketsnack` using git or the download link in [releases](releases) 3. move into the top `pocketsnack` directory (i.e. `cd pocketsnack`) -4. `pip3 install .` or if pip points to Python3, `pip install .` -5. Add Pocket consumer key to `settings/settings.py` +4. `pip install .` or if pip points to Python2, `pip3 install .` +5. Add Pocket consumer key to `settings/settings.yaml` 6. `pocketsnack --authorise` -7. You are now ready to enjoy using pocketsnack +7. You are now ready to enjoy using pocketsnack from any directory ### Installing Python 3 @@ -32,24 +24,42 @@ You will need Python 3.x installed. On MacOS the easiest thing to do is to [inst ### Settings -You will need to copy `settings/settings-example.py` to a new file - `settings/settings.py` before you start. You can do this however you like, but from the command line you could use: -`cp settings/settings-example.py settings/settings.py`, -and then edit it with a text editor like `nano` or VS Code, but any text editor will do the job - you could even use TextEdit or Notepad. +You will need to copy `settings/settings-example.yaml` to a new file - `settings/settings.yaml` before you start. You can do this however you like, but from the command line you could use: +```shell +cp settings/settings-example.py settings/settings.yaml +``` -You can adjust most settings, but the defaults in `settings-example.py` should be ok for most users. Check the comments in `settings.example.py` for an explanation of each setting. +Then edit it with a text editor like `nano`, Atom or VS Code, but any text editor will do the job - you could even use TextEdit or Notepad. + +You can adjust most settings, but the defaults in `settings-example.yaml` should be sensible for most users. + +| setting | type | description | +| :------------------- | :---: | :------------------------------------ | +| pocket_consumer_key | string | the consumer key provided by Pocket when you register your 'app' (see below)| +| items_per_cycle | integer | how many items you want to bring in to the List from your `tbr` archive when using `--lucky_dip`| +| archive_tag | string | the tag to use to identify items in your 'to be read' archive| +| ignore_tags | list | a list of tag names - items with any of these tags will be ignored by `--stash` and remain in your Pocket List| +| ignore_faves | boolean | if set to `true` favorited items will be ignored by `--stash` and remain in your Pocket List| +| replace_all_tags | boolean | if set to `true` all tags will be removed by `--stash` when adding the `archive_tag`, except anything in `retain_tags`| +| retain_tags | list | a list of tag names - these tags will not be removed by `--purge`, nor by `--stash` if `replace_all_tags` is set to `true`| +| longreads_wordcount | integer | determines how long a 'longread' is. | +| num_videos | integer | how many videos (if there are videos in your list) should be included in each `--lucky_dip`. This is a subset of `item_per_cycle`, not in addition to the total.| +| num_images | integer | how many images (if there are images in your list) should be included in each `--lucky_dip`. This is a subset of `item_per_cycle`, not in addition to the total.| +| num_longreads | integer | how many long reads (if there are long reads in your list) should be included in each `--lucky_dip`. This is a subset of `item_per_cycle`, not in addition to the total. The definition of a long read is determined by `longreads_wordcount`| +| pocket_access_token | string | access token required to interact with the Pocket API. This will be updated when you run `--authorise` and should not be edited manually.| ### Creating a Pocket consumer key for your app 1. Log in to Pocket in a web browser -2. Go to `https://getpocket.com/developer` and click 'CREATE NEW APP' +2. Go to [`https://getpocket.com/developer`](https://getpocket.com/developer) and click 'CREATE NEW APP' 3. Complete the form: you will need all permissions, and the platform should be _Desktop (other)_ -4. Your new app will show a **consumer key**, which you need to paste into the first line in `settings.py` +4. Your new app will show a **consumer key**, which you need to paste into the first line in `settings.yaml` ### Authorising your app with a Pocket access token -Pocket uses OAuth to confirm that your app has permission from your user account to do stuff in your account. This means you need to authorise the app before you can do anything else. Once you have copied your app consumer key into settings.py, run `pocketsnack --authorise` to get your token. +Pocket uses OAuth to confirm that your app has permission from your user account to do stuff in your account. This means you need to authorise the app before you can do anything else. Once you have copied your app consumer key into settings.yaml, run `pocketsnack --authorise` to get your token. -You should now have a line at the bottom of settings.py saying something like `pocket_access_token = 'aa11bb-zz9900xx'` +You should now have a line at the bottom of settings.yaml saying something like `pocket_access_token: 'aa11bb-zz9900xx'` ## Usage @@ -63,45 +73,59 @@ Outputs help for each command ### -t, --test -Outputs the first article returned by a call to the API. Normally you will never need to use this. +Outputs the full JSON from the first article returned by a call to the API. Normally you will never need to use this. ### -u, --authorise This command has an 's', not a 'z', and the short version is a 'u', not an 'a'. -You need this to authorise your app. Everything else works exclusively on the command line, but _authorise_ needs to open a browser to complete the authorisation process, so you need to run this on a machine with a web browser. It will authorise your app with your user, wait for you to confirm that you have completed the authorisation (by typing 'done') and then add the token to `settings.py`. You also need to run `--authorise` if you want to change the Pocket account you are using with `pocketsnack`. +You need this to authorise your app. Everything else works exclusively on the command line, but _authorise_ needs to open a browser to complete the authorisation process, so you need to run this on a machine with a web browser. It will authorise your app with your user, wait for you to confirm that you have completed the authorisation (by typing 'done') and then add the token to `settings.yaml`. You also need to run `--authorise` if you want to change the Pocket account you are using with `pocketsnack`. ## action commands +### --dedupe + +Removes duplicates from your List, TBR archive, full Archive, or everything, depending on the flag you use with it. This is an extension of the functionality provided by [pickpocket](https://github.com/hughrun/pickpocket). + ### -d, --lucky_dip -Returns items with the archive tag from the archive to the list, and removes the archive tag. The number of items returned is determined by `items_per_cycle` in `settings.py`. Note that if `num_videos` and `num_images` add up to more than `items_per_cycle`, _lucky_dip_ will only return the total specified in `items_per_cycle`. Videos take precedence. +Returns items with the archive tag from the archive to the list, and removes the archive tag. The number of items returned is determined by `items_per_cycle` in `settings.yaml`. Note that if `num_videos` and `num_images` add up to more than `items_per_cycle`, then `--lucky_dip` will only return the total specified in `items_per_cycle`. Videos take precedence. ### -p, --purge -You can use **purge_tags** to clear all tags in your List, Archive, or both, excluding the `archive_tag` and any `retain_tags`. This is useful if you've been using the Aus GLAM Blogs Pocket tool or anything else that retains the original tags from articles. +You can use `--purge` to clear all tags in your List, TBR achive, full Archive, or everything - excluding the `archive_tag` and any `retain_tags`. This is useful if you've been using the _Aus GLAM Blogs_ Pocket tool or anything else that retains the original tags from articles. -`--purge` requires a second argument: `--list`, `--archive`, or `--all`, depending on where you want to purge tags. - -**NOTE** that by design, `--purge` will process **all** items in your archive, not just items with the `archive_tag`. This may lead to miss-matches between the number returned by `--info --archive` and the number of items processed by `--purge --archive`. +`--purge` requires a second argument: `--list`, `--tbr`, `--archive`, or `--all`, depending on where you want to purge tags. ### -s, --stash -Adds the archive tag to everything in your list, and then archives them. Depending on the value of `ignore_faves` and `ignore_tags` in `settings.py`, and any before/since values, some items may be excluded and remain in the List. +Adds the archive tag to everything in your list, and then archives them. Depending on the value of `ignore_faves` and `ignore_tags` in `settings.yaml`, and any before/since values, some items may be excluded and remain in the List. ## optional flags ### -a, --archive -Used in combination with `--info`, this tells you how many items are in your archive and how many of them are 'long reads'. You can set the wordcount defining a long read in `settings.py`. Used with `--purge`, it purges tags on items in the archive. +Used in combination with `--info`, this tells you how many items are in your archive and how many of them are 'long reads'. You can set the wordcount defining a long read in `settings.yaml`. + +Used with `--purge`, it purges tags on items in the archive. + +Used with `--dedupe`, it restricts de-duplication to the full Pocket Archive. ### -l, --list -Same as _archive_ but for your list instead of your archive. +Same as _archive_ but for your Pocket List instead of your archive. + +### --tbr + +Used in conjuction with `--dedupe` to dedupe only items in the `tbr` archive. + +Used with `--purge`, it purges tags only on items in the `tbr` archive. ### -b, -all -For use with `--purge` - purge tags from _both_ the List and the Archive. +For use with `--purge` - purge tags from your entire Pocket account. + +Used with `--dedupe`, it de-duplicates your entire Pocket account. ### -n, --since SINCE @@ -137,13 +161,13 @@ Run lucky_dip but only choose from items last updated longer ago than one week: `pocketsnack -d -o 7` -## Uninstalling or moving to a new directory +## Uninstalling or moving v1 script to a new directory ### If you installed with pip Just run `pip uninstall pocketsnack` or `pip3 uninstall pocketsnack`. -### If you installed using the legacy install.sh script +### If you installed using the legacy version 1 install.sh script 1. Delete the executable link: `rm /usr/local/bin/pocketsnack` diff --git a/bin/pocketsnack b/bin/pocketsnack index b453388..6c6190c 100644 --- a/bin/pocketsnack +++ b/bin/pocketsnack @@ -1,8 +1,7 @@ #!/usr/bin/env python3 -# pocket-snack - a Python3 tool to help you retain your sanity when using your Pocket account - -# Copyright (C) 2019 Hugh Rundle +# pocketsnack - KonMari your Pocket tsundoku from the command line +# Copyright (C) 2018 - 2020 Hugh Rundle # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -17,9 +16,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# You can contact Hugh on Mastodon at @hugh@ausglam.space -# or Twitter at @hughrundle -# or email hugh [at] hughrundle [dot] net +# You can contact Hugh on email: hugh [at] hughrundle [dot] net # ---------------- # Import libraries @@ -32,176 +29,236 @@ import os import subprocess # local modules -from settings import settings from lib import toolkit as pt +import yaml +configyaml = open('settings/settings.yaml', 'r') +for S in yaml.safe_load_all(configyaml): + + # ---------------- + # Settings + # ---------------- + + # assign short variable names from the settings file + access_token = S['pocket_access_token'] + consumer_key = S['pocket_consumer_key'] + archive_tag = S['archive_tag'] + ignore_tags = set(S['ignore_tags']) + retain_tags = set(S['retain_tags']) + + # ---------------- + # argparser arguments + # ---------------- + + parser = ArgumentParser(description='\033[1;36mpocketsnack: KonMari your Pocket tsundoku from the command line.\033[1;m') + admin = parser.add_argument_group('admin commands') + actions = parser.add_argument_group('action commands') + mex = parser.add_mutually_exclusive_group() + timers = parser.add_mutually_exclusive_group() + + mex.add_argument( + "-a", "--archive", action="store_true", help="get information on TBR items in archive (with -i) or purge tags in archive (with -p)" + ) + mex.add_argument( + "-b", "--all", action="store_true", help="purge all tags in both list and archive (with -p)" + ) + actions.add_argument( + "-d", "--lucky_dip", action="store_true", help="move random items tagged 'tbr' from archive to list, depending on settings" + ) + actions.add_argument( + "--dedupe", action="store_true", help="de-duplicate list (-l), archive (-a), tbr items (--tbr) or all (-b)- defaults to list" + ) + actions.add_argument( + "-i", "--info", action="store_true", help="get information on items in list or TBR items in archive" + ) + mex.add_argument( + "-l", "--list", action="store_true", help="get information on items in list (with -i) or purge tags in list (with -p)" + ) + timers.add_argument( + "-n", "--since", type=int, help="only act on items where last activity is newer than a given number of days. Use with any action command" + ) + timers.add_argument( + "-o", "--before", type=int, help="only act on items where last activity is older than a given number of days. Use with any action command" + ) + actions.add_argument( + "-p", "--purge", action="store_true", help="remove all tags from list, archive, or both, depending on the second argument provided and excepting tags listed in 'retain_tags' in settings" + ) + actions.add_argument( + "-s", "--stash", action="store_true", help="add 'tbr' tag to all items in user list and archive them, with exceptions as per settings" + ) + admin.add_argument( + "-t", "--test", action="store_true", help="test whether API call returns data" + ) + mex.add_argument( + "--tbr", action="store_true", help="used in conjuction with --dedupe to dedupe only items in the tbr archive." + ) + admin.add_argument( + "-u", "--authorise", action="store_true", help="authorise app to connect to a Pocket account" + ) + + options = parser.parse_args() + + # ---------------- + # What happens with each combination? + # ---------------- + + if __name__ == '__main__': + + # Find all args that have a value other than False + # This helps with error messages for optional args + # that need to be used in combination with something else + true_vars = [] + orphans = ['list', 'archive', 'all', 'since', 'before'] + for x in vars(options): + if vars(options)[x]: + true_vars.append(x) + + if options.authorise: + # Run authorise once first to retrieve a pocket_access_token + auth = pt.authorise(consumer_key) + print(auth) + + elif options.dedupe: + if options.tbr: + state = "archive" + tag = S['archive_tag'] + else: + state = "archive" if options.archive else "all" if options.all else "unread" + tag = False + + location = tag if tag else state if state != 'unread' else 'list' + print(' \033[46;97mChecking for duplicates in ' + location + '\033[0;m') + dd = pt.dedupe(state, tag, consumer_key, access_token) + + elif options.lucky_dip: + print(' \033[46;97mRunning lucky dip...\033[0;m') + dip = pt.lucky_dip( + consumer_key, + access_token, + S['archive_tag'], + S['items_per_cycle'], + S['num_videos'], + S['num_images'], + S['num_longreads'], + S['longreads_wordcount'], + options.before, + options.since) + print(' \033[46;97m' + dip + '\033[0;m') + + elif options.info: + + def print_info(response, collection): + items = str(len(response)) + longreads = 0 + for item in response: + # is it a long read? + if 'word_count' in response[item]: + words = int(response[item]['word_count']) + longread = True if words > S['longreads_wordcount'] else False + else: + longread = False + if longread: + longreads += 1 + + if options.before: + print(collection + 'has ' + items + ' items ' + 'updated prior to ' + str(options.before) + ' days ago and ' + str(longreads) + ' are longreads.\033[0;m') + elif options.since: + print(collection + 'has ' + items + ' items ' + 'updated since ' + str(options.since) + ' days ago and ' + str(longreads) + ' are longreads.\033[0;m') + else: + print(collection + 'has ' + items + ' items and ' + str(longreads) + ' are longreads.\033[0;m') -# ---------------- -# Settings -# ---------------- - -# TODO: clean this up - -# assign short variable names from the settings file -consumer_key = settings.pocket_consumer_key -redirect_uri = settings.pocket_redirect_uri -archive_tag = settings.archive_tag - -# settings dict for other vars -refresh_settings = [ - consumer_key, - settings.pocket_access_token, - archive_tag, - settings.replace_all_tags, - settings.retain_tags, - settings.ignore_faves, - settings.ignore_tags, - settings.items_per_cycle, - settings.num_videos, - settings.num_images, - settings.num_longreads, - settings.longreads_wordcount -] - -# ---------------- -# argparser arguments -# ---------------- - -parser = ArgumentParser(description='\033[1;36mpocketsnack: a command line tool for decluttering your Pocket account\033[1;m') -admin = parser.add_argument_group('admin commands') -actions = parser.add_argument_group('action commands') -mex = parser.add_mutually_exclusive_group() -timers = parser.add_mutually_exclusive_group() - -mex.add_argument( - "-a", "--archive", action="store_true", help="get information on TBR items in archive (with -i) or purge tags in archive (with -p)" -) -mex.add_argument( - "-b", "--all", action="store_true", help="purge all tags in both list and archive (with -p)" -) -actions.add_argument( - "-d", "--lucky_dip", action="store_true", help="move random items tagged 'tbr' from archive to list, depending on settings" -) -actions.add_argument( - "-i", "--info", action="store_true", help="get information on items in list or TBR items in archive" -) -mex.add_argument( - "-l", "--list", action="store_true", help="get information on items in list (with -i) or purge tags in list (with -p)" -) -timers.add_argument( - "-n", "--since", type=int, help="only act on items where last activity is newer than a given number of days. Use with any action command" -) -timers.add_argument( - "-o", "--before", type=int, help="only act on items where last activity is older than a given number of days. Use with any action command" -) -actions.add_argument( - "-p", "--purge", action="store_true", help="remove all tags from list, archive, or both, depending on the second argument provided and excepting tags listed in 'retain_tags' in settings" -) -actions.add_argument( - "-s", "--stash", action="store_true", help="add 'tbr' tag to all items in user list and archive them, with exceptions as per settings" -) -admin.add_argument( - "-t", "--test", action="store_true", help="test whether API call returns data" -) -admin.add_argument( - "-u", "--authorise", action="store_true", help="authorise app to connect to a Pocket account" -) - -options = parser.parse_args() - -# ---------------- -# What happens with each combination? -# ---------------- + if options.archive: + response = pt.info(consumer_key, access_token, archive_tag, options.before, options.since) + if len(response) > 0: + print_info(response, ' \033[46;97mThe TBR archive ') + else: + print(' \033[46;97mNo items match that query\033[0;m') -if __name__ == '__main__': - - # Find all args that have a value other than False - # This helps with error messages for optional args - # that need to be used in combination with something else - true_vars = [] - orphans = ['list', 'archive', 'all', 'since', 'before'] - for x in vars(options): - if vars(options)[x]: - true_vars.append(x) - - if options.authorise: - # Run authorise once first to retrieve a pocket_access_token - auth = pt.authorise(consumer_key, redirect_uri) - print(auth) - - elif options.lucky_dip: - print('\033[0;36m Running lucky dip...\033[0;m') - dip = pt.lucky_dip(consumer_key, settings.pocket_access_token, settings.archive_tag, settings.items_per_cycle, settings.num_videos, settings.num_images, settings.num_longreads, settings.longreads_wordcount, options.before, options.since) - print('\033[0;36m' + dip + '\033[0;m') - - elif options.info: - - def print_info(response, collection): - items = str(len(response)) - longreads = 0 - for item in response: - # is it a long read? - if 'word_count' in response[item]: - words = int(response[item]['word_count']) - longread = True if words > settings.longreads_wordcount else False + elif options.list: + response = pt.info(consumer_key, access_token, False, options.before, options.since) + if len(response) > 0: + print_info(response, ' \033[46;97mThe user List ') else: - longread = False - if longread: - longreads += 1 - - if options.before: - print(collection + 'has ' + items + ' items ' + 'updated prior to ' + str(options.before) + ' days ago and ' + str(longreads) + ' are longreads.\033[0;m') - elif options.since: - print(collection + 'has ' + items + ' items ' + 'updated since ' + str(options.since) + ' days ago and ' + str(longreads) + ' are longreads.\033[0;m') - else: - print(collection + 'has ' + items + ' items and ' + str(longreads) + ' are longreads.\033[0;m') + print(' \033[46;97mNo items match that query\033[0;m') - if options.archive: - response = pt.info(consumer_key, settings.pocket_access_token, archive_tag, options.before, options.since) - if len(response) > 0: - print_info(response, '\033[0;36m The TBR archive ') else: - print('\033[0;36m No items match that query\033[0;m') + print('\n \033[46;97m--info\033[0;m requires a second argument (-a or -l). Check \033[46;97mpocketsnack --help\033[0;m for more information\n') + + elif options.purge: + + if options.list: + print(' \033[46;97mPurging tags in the list\033[0;m') + purge = pt.purge_tags( + 'unread', + retain_tags, + archive_tag, + consumer_key, + access_token, + options.before, + options.since + ) + print(purge) + + elif options.archive: + print(' \033[46;97mPurging tags in the archive\033[0;m') + purge = pt.purge_tags( + 'archive', + retain_tags, + archive_tag, + consumer_key, + access_token, + options.before, + options.since + ) + print(purge) + + elif options.all: + print(' \033[46;97mPurging tags in both the archive and the list\033[0;m') + purge = pt.purge_tags( + 'all', + retain_tags, + archive_tag, + consumer_key, + access_token, + options.before, + options.since + ) + print(purge) + + elif options.tbr: + print(' \033[46;97mPurging tags in tbr archive\033[0;m') + purge = pt.purge_tags( + 'tbr', + retain_tags, + archive_tag, + consumer_key, + access_token, + options.before, + options.since + ) + print(purge) - elif options.list: - response = pt.info(consumer_key, settings.pocket_access_token, False, options.before, options.since) - if len(response) > 0: - print_info(response, '\033[0;36m The user List ') else: - print('No items match that query') - - else: - print('\n \033[0;36m--info\033[0;m requires a second argument (-a or -l). Check \033[0;36mpocketsnack --help\033[0;m for more information\n') - - elif options.purge: - - if options.list: - print('\033[0;36m Purging tags in the list\033[0;m') - purge = pt.purge_tags('unread', settings.retain_tags, archive_tag, consumer_key, settings.pocket_access_token, options.before, options.since) - print(purge) - - elif options.archive: - print('\033[0;36m Purging tags in the archive\033[0;m') - purge = pt.purge_tags('archive', settings.retain_tags, archive_tag, consumer_key, settings.pocket_access_token, options.before, options.since) - print(purge) - - elif options.all: - print('\033[0;36m Purging tags in both the archive and the list\033[0;m') - purge = pt.purge_tags('all', settings.retain_tags, archive_tag, consumer_key, settings.pocket_access_token, options.before, options.since) - print(purge) + print('\n \033[46;97m--purge\033[0;m requires a second argument (-a, -l or -b). Check \033[46;97mpocketsnack --help\033[0;m for more information\n') + + elif options.stash: + stash = pt.stash( + consumer_key, + access_token, + archive_tag, + S['replace_all_tags'], + retain_tags, + S['ignore_faves'], + ignore_tags, + options.before, + options.since) + print(' \033[46;97m' + stash + '\033[0;m') + + elif options.test: + result = pt.test(consumer_key, access_token) + print(result) + + elif set(true_vars).intersection(orphans): + print('\n That command cannot be used by itself. Check \033[46;97mpocketsnack --help\033[0;m for more information\n') else: - print('\n \033[0;36m--purge\033[0;m requires a second argument (-a, -l or -b). Check \033[0;36mpocketsnack --help\033[0;m for more information\n') - - elif options.stash: - stash = pt.stash(consumer_key, settings.pocket_access_token, archive_tag, settings.replace_all_tags, settings.retain_tags, settings.ignore_faves, settings.ignore_tags, options.before, options.since) - print('\033[0;36m' + stash + '\033[0;m') - - elif options.test: - result = pt.test(consumer_key, settings.pocket_access_token) - print(result) - - elif set(true_vars).intersection(orphans): - print('\n That command cannot be used by itself. Check \033[0;36mpocketsnack --help\033[0;m for more information\n') - - else: - print('\033[0;36m pocketsnack\033[0;m requires commands and/or flags to do anything useful. Try \033[0;36mpocketsnack -h\033[0;m for more information') \ No newline at end of file + print(' \033[46;97mpocketsnack\033[0;m requires commands and/or flags to do anything useful. Try \033[46;97mpocketsnack -h\033[0;m for more information') \ No newline at end of file diff --git a/lib/toolkit.py b/lib/toolkit.py index 559e25a..4e17fcd 100755 --- a/lib/toolkit.py +++ b/lib/toolkit.py @@ -1,6 +1,6 @@ # pocket-toolkit - a collection of functions to manage your Pocket account -# Copyright (C) 2019 Hugh Rundle +# Copyright (C) 2018 - 2020 Hugh Rundle # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -125,7 +125,8 @@ def process_items(actions, consumer_key, pocket_access_token): # Authorise # ---------------- -def authorise(consumer_key, redirect_uri): # With an 's'. Deal with it. +def authorise(consumer_key): # With an 's'. Deal with it. + redirect_uri = 'https://hugh.run/success' paramsOne = {"consumer_key": consumer_key, "redirect_uri": redirect_uri} # set up step 1 request - this should return a 'code' aka 'request token' requestOne = requests.post('https://getpocket.com/v3/oauth/request', headers=headers, params=paramsOne) @@ -158,8 +159,8 @@ def authorise(consumer_key, redirect_uri): # With an 's'. Deal with it. # Assign the access token to a parameter called access_token access_token = res['access_token'] # replace the pocket_access_token line rather than just adding an extra at the end - settings_file = fileinput.FileInput("settings/settings.py", inplace=True) - repl = "pocket_access_token = " + "'" + access_token + "'" + settings_file = fileinput.FileInput("settings/settings.yaml", inplace=True) + repl = "pocket_access_token: " + access_token for line in settings_file: line = re.sub('(pocket_access_token)+.*', repl, line) print(line.rstrip()) @@ -186,7 +187,10 @@ def info(consumer_key, pocket_access_token, archive_tag, before, since): items = get_item_list(params, before, since) return items -# choose items to put back into the user List +# ----------------------------------- +# lucky dip from archive back to list +# ----------------------------------- + def lucky_dip(consumer_key, pocket_access_token, archive_tag, items_per_cycle, num_videos, num_images, num_longreads, longreads_wordcount, before, since): def run_lucky_dip(attempts): @@ -219,9 +223,9 @@ def readd(selection): # we'll use this to report what was added chosen = {} - ##################### + # filtering formats - ##################### + # ----------------- videos = [] images = [] @@ -255,9 +259,8 @@ def readd(selection): items_needed -= len(selected_images) chosen['images'] = len(selected_images) - ##################### # filtering longreads - ##################### + # ------------------- if num_longreads: long_reads = [] for item in tbr: @@ -353,7 +356,7 @@ def readd(selection): for v in chosen.values(): tot_added += v remaining = available - tot_added - completed_message = ' \033[0;36mSuccess! ' + str(tot_added) + ' items added to your reading list, including ' + completed_message = '\033[46;97mSuccess!\033[0;m ' + str(tot_added) + ' items added to your reading list, including ' if random_choice: completed_message += str(chosen['random']) + ' random articles, ' if tot_images: @@ -368,19 +371,19 @@ def readd(selection): caveat = 'last updated earlier than ' + str(before) + ' days ago ' if since: caveat = 'last updated more recently than ' + str(since) + ' days ago ' - completed_message += 'with ' + str(remaining) + ' other items ' + caveat + 'remaining to be read.\033[0;m' + completed_message += 'with ' + str(remaining) + ' other items ' + caveat + 'remaining to be read.' return completed_message # else if there's nothing tagged with the archive_tag else: - return '\033[0;36m Nothing to be read!\033[0;m' + return ' \033[46;97mNothing to be read!\033[0;m' else: if attempts < 4: attempts += 1 time.sleep(10) - print('\033[0;36m Attempting to connect...\033[0;m') + print(' \033[46;97mAttempting to connect...\033[0;m') return run_lucky_dip(attempts) else: - msg = "\033[0;36m Sorry, no connection after 4 attempts.\033[0;m" + msg = " \033[46;97mSorry, no connection after 4 attempts.\033[0;m" return msg return run_lucky_dip(0) @@ -398,6 +401,10 @@ def purge_tags(state, retain_tags, archive_tag, consumer_key, pocket_access_toke "detailType": "complete" } + if state == "tbr": + params['state'] = 'archive' + params['tag'] = archive_tag + # check we're online if connection_live() == True: # GET the list @@ -438,8 +445,12 @@ def purge_tags(state, retain_tags, archive_tag, consumer_key, pocket_access_toke Options: -retain_tags - a list of tags that should not be removed from items when adding the archive_tag. If you don't want to retain any tags, this value should be False. Defaults to False -favorite - boolean indicating whether to ignore (i.e. leave in the user list) favorite items. Defaults to True + retain_tags - a list of tags that should not be removed from items when adding the archive_tag. + If you don't want to retain any tags, this value should be False. + Defaults to False + + favorite - boolean indicating whether to ignore (i.e. leave in the user list) favorite items + Defaults to True """ @@ -448,12 +459,12 @@ def purge_tags(state, retain_tags, archive_tag, consumer_key, pocket_access_toke # ----------------- def stash(consumer_key, pocket_access_token, archive_tag, replace_all_tags, retain_tags, favorite, ignore_tags, before, since): - print('\033[0;36m Stashing items...\033[0;m') + print(' \033[46;97mStashing items...\033[0;m') # if ignore_faves is set to True, don't get favorite items params = {"consumer_key": consumer_key, "access_token": pocket_access_token, "detailType": "complete", "state": "unread"} if favorite: params['favorite'] = "0" - print('\033[0;36m Skipping favorited items...\033[0;m') + print(' Skipping favorited items...') def run_stash(attempts): if connection_live() == True: @@ -505,7 +516,7 @@ def run_stash(attempts): item_action = {"item_id": item, "action": "archive"} archive_actions.append(item_action) - print('\033[0;36m Archiving ' + str(len(archive_actions)) + ' items...\033[0;m') + print(' \033[46;97mArchiving ' + str(len(archive_actions)) + ' items...\033[0;m') # archive items process_items(archive_actions, consumer_key, pocket_access_token) @@ -517,10 +528,10 @@ def run_stash(attempts): if attempts < 4: attempts += 1 time.sleep(10) - print('\033[0;36m Attempting to connect...\033[0;m') + print(' \033[46;97mAttempting to connect...\033[0;m') return run_stash(attempts) else: - msg = "\033[0;31m Sorry, no connection after 4 attempts.\033[0;m" + msg = " \033[46;97mSorry, no connection after 4 attempts.\033[0;m" return msg return run_stash(0) @@ -537,4 +548,120 @@ def test(consumer_key, pocket_access_token): request = get(params) list_items = request.json() - return json.dumps(list_items, indent=4, default=str) \ No newline at end of file + return json.dumps(list_items, indent=4, default=str) + +# ----------------- +# de-duplicate +# ----------------- + +def dedupe(state, tag, consumer_key, pocket_access_token): + + # Retrieving items + # ---------------- + # retrieve all 'unread' items (i.e. not archived) + # we use the Retrieve API call = https://getpocket.com/developer/docs/v3/retrieve + # The endpoint for retrieving is 'get': https://getpocket.com/v3/get + # detailType should be 'simple' because we don't need any information except for the item_id and the resolved_url + parameters = { + "consumer_key" : consumer_key, + "access_token" : pocket_access_token, + "detailType" : "simple", + "state" : state + } + + if tag: + parameters['tag'] = tag # if tag exists, add it to parameters + + unread = get(parameters) + # our items will be under the JSON object's "list" key + item_list = unread.json()['list'] + + # make a new dictionary called 'summary' + # we will use this to look for duplicates + summary = {} + + # and make a list called 'items_to_delete' + items_to_delete = [] + + # loop over each key (not the whole object) in item_list + # 'item' here refers to each item's key, not the whole object/dictionary + print(' \033[46;97mchecking ' + str(len(item_list)) + ' items...\033[0;m') + for item in item_list: + # conveniently the key Pocket uses is the item_id! + item_id = item + + # we need the item_id from this request so we can use it in the next API call to delete it + # get the URL by pulling out the value from the dict using the key + # generally we want to use the 'resolved url' but sometimes that might not exist + # if so, use the 'given url' instead + if not 'resolved_url' in item_list[item]: + # item_list is a Python dictionary where each value is itself another dictionary + # or in JSON terms, it's an object where each value is another object + # below we are getting the value of the current item id (i.e the first dict), then checking if there is a value within the second dict for the key 'given_url' + item_url = item_list[item]['given_url'] + else: + item_url = item_list[item]['resolved_url'] + + # check whether the resolved_url is already in 'summary' + # if it isn't, make a new entry with resolved_url as the key and a list holding item_id as the value - basically we're reversing the logic of 'item_list'. This will allow us to check for duplicates easily in a moment. + if not item_url in summary: + summary[item_url] = [item_id] + # if it is there already, add the item_id into the existing list + else: + summary[item_url].append(item_id) + + # ------------------ + # Finding duplicates + # ------------------ + + # now we look for duplicates (this is why we use the url as the key) + for item in summary: + + # if the length of the list is more than 1, then by definition there must be a duplicate + if len(summary[item]) > 1: + print(' \033[46;97m' + item + '\033[0;m occurs ' + str(len(summary[item])) + ' times') + # keep only the most recently added item by slicing the list to make a new list of everything except the last one (which will be the *first* one that was found) + duplicates = summary[item][:-1] + # add each duplicate in the duplicates list for this url to the items_to_delete list + for item in duplicates: + items_to_delete.append(item) + + # Deleting duplicates + # ------------------- + + # now use the modify API call to delete duplicate items + # Docs - https://getpocket.com/developer/docs/v3/modify + + # With our list of duplicate item ids, we create one final list of a bunch of JSON objects + actions = [] + + # for each item to be deleted, append a dictionary to the actions list + for item_id in items_to_delete: + actions.append({"action":"delete", "item_id": item_id}) + + # Double check you really want to delete them + if len(actions) > 0: + print(' \033[107;95mAbout to delete ' + str(len(actions)) + ' duplicate items.\033[0;m') + print(' \033[107;95mDelete these items? Type "delete" to confirm.\033[0;m') + check = input('>>') + if check == 'delete': + # first turn the list and its component dictionaries into a JSON string + actions_string = json.dumps(actions) + # now URL encode it using urllib + actions_escaped = urllib.parse.quote(actions_string) + # now POST to pocket and assign the response to a parameter at the same time. + deleted = send(actions_escaped, pocket_access_token, consumer_key) + # provide feedback on what happened + # 'deleted' is a raw http response (it should return '') + # so we need to turn it into a Python string before we can do a comparison + if str(deleted) == '': + print(' \033[46;97m🚮 These duplicates have been deleted:\033[0;m') + for item in actions: + print(' \033[46;97m' + item['item_id'] + '\033[0;m') + # that's it! + else: + print(' \033[46;97mSomething went wrong 😟\033[0;m') + else: + print(' \033[46;97m✋ deletion cancelled\033[0;m') + else: + print(' 🎉 \033[46;97mNo duplicates found!\033[0;m') \ No newline at end of file diff --git a/settings/settings-example.py b/settings/settings-example.py deleted file mode 100644 index 28f631b..0000000 --- a/settings/settings-example.py +++ /dev/null @@ -1,13 +0,0 @@ -pocket_consumer_key = '' # paste the 'consumer key' from your Pocket app between the quotes here -pocket_redirect_uri = 'https://hugh.run/success' -items_per_cycle = 10 # integer, not a string -archive_tag = 'tbr' # apply this tag before archiving items. This is case-insensitive and Pocket will always downcase it -ignore_tags = {'ignore'} # don't stash any items with these tags -ignore_faves = True # don't stash items that are favorited -replace_all_tags = False # if set to True this will replace ALL tags on an item in the user's List with the archive_tag and anything in retain_tags. If set to False, the archive tag is still added but any existing tags are retained. -retain_tags = {'glam blog club', 'aus glam blogs', 'empocketer'} # if replace_all_tags is set to True, you can still retain particular tags by adding them to the exclude_tags list. Note that this is a Set, not a List -longreads_wordcount = 3000 # integer - if there are more than this many words, the article will be classified as a 'long read' -num_videos = None # if you change this to an integer this many items_per_cycle will be videos (if there are any videos in your list) -num_images = None # if you change this to an integer this many items_per_cycle will be images (if there are any images in your list) -num_longreads = 2 # if this is an integer this many items_per_cycle will be longreads (if there are any longreads in your list). If all your items are longreads you will still get the total items_per_cycle. Change to None or False if you don't want to filter longreads -pocket_access_token = None \ No newline at end of file diff --git a/settings/settings-example.yaml b/settings/settings-example.yaml new file mode 100644 index 0000000..be4e388 --- /dev/null +++ b/settings/settings-example.yaml @@ -0,0 +1,16 @@ +pocket_consumer_key: YOUR_KEY_HERE +items_per_cycle: 10 +archive_tag: tbr +ignore_tags: + - ignore +ignore_faves: true +replace_all_tags: false +retain_tags: + - glam blog club + - aus glam blogs + - empocketer +longreads_wordcount: 3000 +num_videos: null +num_images: null +num_longreads: 2 +pocket_access_token: null \ No newline at end of file diff --git a/setup.py b/setup.py index 0ee3506..0363a63 100644 --- a/setup.py +++ b/setup.py @@ -1,17 +1,17 @@ from setuptools import setup, find_packages setup(name='pocketsnack', - version='2.1.0', + version='3.0.0', url='https://hughrundle.net', license='GPL-3.0-or-later', packages=find_packages(), scripts=['bin/pocketsnack'], - install_requires=['request'], + install_requires=['requests', 'pyyaml'], zip_safe=False, author='Hugh Rundle', author_email='hugh@hughrundle.net', - description='A command line package for managing Pocket accounts.', - keywords='pocket, pocket api', + description='KonMari your Pocket tsundoku from the command line', + keywords='pocket, pocket api, cli', project_urls={ 'Source Code': 'https://github.com/hughrun/pocketsnack', 'Documentation': 'https://github.com/hughrun/pocketsnack' diff --git a/v2_README.md b/v2_README.md new file mode 100644 index 0000000..493a92f --- /dev/null +++ b/v2_README.md @@ -0,0 +1,159 @@ +# pocketsnack + +When your Pocket list is overwhelming, pocket-snack lets you see just what you can read today + +This is the version 2 documentation. + +Read the [latest documentation](README.md). + +If you haven't yet upgraded from version 1 you can still use the [version 1 README](v1_README.md). + +## A note on version 2 + +All commands have changed since version 1 - read the _Usage_ section carefully. This was necessary in order to provide better functionality without making the code too confusing. + +One of the changes is that the `refresh` command from version 1 no longer exists. This is so that `--since`, and `--before` can be used with both `--stash` and `lucky_dip`. If you want replicate `refresh` you simply need to run `--stash` followed by `--lucky_dip`. From the command line you could do: + +```bash +pocketsnack -s && pocketsnack -d +``` + +The automation of `pocketsnack refresh` has _also_ been removed. This didn't really work very consistently, and was causing a lot of maintenance headaches. I'm looking at how to bring it back in a different way, but for now it's been removed. + +## Getting started (tl;dr) + +1. make sure you have installed Python version 3.x (preferably 3.7 or higher) +2. download `pocketsnack` using git or the download link in [releases](releases) +3. move into the top `pocketsnack` directory (i.e. `cd pocketsnack`) +4. `pip3 install .` or if pip points to Python3, `pip install .` +5. Add Pocket consumer key to `settings/settings.py` +6. `pocketsnack --authorise` +7. You are now ready to enjoy using pocketsnack + +### Installing Python 3 + +You will need Python 3.x installed. On MacOS the easiest thing to do is to [install Python 3 using Homebrew](https://docs.brew.sh/Homebrew-and-Python): `brew install python`. + +### Settings + +You will need to copy `settings/settings-example.py` to a new file - `settings/settings.py` before you start. You can do this however you like, but from the command line you could use: +`cp settings/settings-example.py settings/settings.py`, +and then edit it with a text editor like `nano` or VS Code, but any text editor will do the job - you could even use TextEdit or Notepad. + +You can adjust most settings, but the defaults in `settings-example.py` should be ok for most users. Check the comments in `settings.example.py` for an explanation of each setting. + +### Creating a Pocket consumer key for your app + +1. Log in to Pocket in a web browser +2. Go to `https://getpocket.com/developer` and click 'CREATE NEW APP' +3. Complete the form: you will need all permissions, and the platform should be _Desktop (other)_ +4. Your new app will show a **consumer key**, which you need to paste into the first line in `settings.py` + +### Authorising your app with a Pocket access token + +Pocket uses OAuth to confirm that your app has permission from your user account to do stuff in your account. This means you need to authorise the app before you can do anything else. Once you have copied your app consumer key into settings.py, run `pocketsnack --authorise` to get your token. + +You should now have a line at the bottom of settings.py saying something like `pocket_access_token = 'aa11bb-zz9900xx'` + +## Usage + +To run commands, use `pocketsnack [command]`. + +### -h, --help + +Outputs help for each command + +## admin commands + +### -t, --test + +Outputs the first article returned by a call to the API. Normally you will never need to use this. + +### -u, --authorise + +This command has an 's', not a 'z', and the short version is a 'u', not an 'a'. + +You need this to authorise your app. Everything else works exclusively on the command line, but _authorise_ needs to open a browser to complete the authorisation process, so you need to run this on a machine with a web browser. It will authorise your app with your user, wait for you to confirm that you have completed the authorisation (by typing 'done') and then add the token to `settings.py`. You also need to run `--authorise` if you want to change the Pocket account you are using with `pocketsnack`. + +## action commands + +### -d, --lucky_dip + +Returns items with the archive tag from the archive to the list, and removes the archive tag. The number of items returned is determined by `items_per_cycle` in `settings.py`. Note that if `num_videos` and `num_images` add up to more than `items_per_cycle`, _lucky_dip_ will only return the total specified in `items_per_cycle`. Videos take precedence. + +### -p, --purge + +You can use **purge_tags** to clear all tags in your List, Archive, or both, excluding the `archive_tag` and any `retain_tags`. This is useful if you've been using the Aus GLAM Blogs Pocket tool or anything else that retains the original tags from articles. + +`--purge` requires a second argument: `--list`, `--archive`, or `--all`, depending on where you want to purge tags. + +**NOTE** that by design, `--purge` will process **all** items in your archive, not just items with the `archive_tag`. This may lead to miss-matches between the number returned by `--info --archive` and the number of items processed by `--purge --archive`. + +### -s, --stash + +Adds the archive tag to everything in your list, and then archives them. Depending on the value of `ignore_faves` and `ignore_tags` in `settings.py`, and any before/since values, some items may be excluded and remain in the List. + +## optional flags + +### -a, --archive + +Used in combination with `--info`, this tells you how many items are in your archive and how many of them are 'long reads'. You can set the wordcount defining a long read in `settings.py`. Used with `--purge`, it purges tags on items in the archive. + +### -l, --list + +Same as _archive_ but for your list instead of your archive. + +### -b, -all + +For use with `--purge` - purge tags from _both_ the List and the Archive. + +### -n, --since SINCE + +Restrict the current _action command_ to only items updated more recently than _SINCE_ number of days. + +### -o, --before BEFORE + +Restrict the current _action command_ to only items updated less recently than _BEFORE_ number of days. + +### What does 'updated' mean? + +The Pocket API does not store a value for the date an items was first added. The only value we can get is _since_, which is a timestamp updated every time there is an update made to an item via or equivalent to any `add` or `modify` [API action](https://getpocket.com/developer/docs/overview). This could be when it is added to the List, move to the archive, moved out of the archive back into the List, or has changes made to tags (even if that tag update results in no actual change - i.e. if `--purge` has been run against the item, regardless of whether it had any tags to begin with). + +## examples + +Stash only items updated in the last 2 days: + +`pocketsnack --stash -n 2` + +Stash only items NOT updated in the last 7 days: + +`pocketsnack --stash -o 7` + +Purge tags on all items in the List that were updated in the last day: + +`pocketsnack -pln 1` + +Run lucky_dip: + +`pocketsnack --lucky_dip` + +Run lucky_dip but only choose from items last updated longer ago than one week: + +`pocketsnack -d -o 7` + +## Uninstalling or moving to a new directory + +### If you installed with pip + +Just run `pip uninstall pocketsnack` or `pip3 uninstall pocketsnack`. + +### If you installed using the legacy install.sh script + +1. Delete the executable link: `rm /usr/local/bin/pocketsnack` + +If you don't do this when re-installing in a different directory, running `pocketsnack` will fail because it will still be pointing at the old directory. +2. Now you can safely delete the pocket-snack directory: `rm -r pocketsnack` + +## Bugs and suggestions + +Please log an issue - but check the existing issues first in case it's already been/being dealt with. \ No newline at end of file