Merge pull request #61 from robertoszek/develop

Bump to v1.0.0

.coveragerc

@@ -0,0 +1,5 @@
+[report]
+exclude_lines =
+    pragma
+    pragma: no cover
+    pragma: win32 cover

.github/FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: robertoszek
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: robertoszek
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

CHANGELOG.md

@@ -1,3 +1,19 @@
+## [1.0.0] 16-01-2022 
+## Fixed
+- `max_tweets` not accepting values higher than 100
+- Video: Not getting the best bitrate version of video attachments in some cases
+- Polls: not being retrieved for accounts with protected tweets
+- RTs: not getting original tweet media attachments 
+## Added
+- `twitter_username` value can be a list now, for having multiple Twitter accounts as sources for one target Fediverse account.
+- A Twitter [archive](https://twitter.com/settings/your_twitter_data) can be provided with `--archive`([more info in the docs](https://robertoszek.github.io/pleroma-bot/gettingstarted/usage/#using-an-archive))
+- Links to Twitter attachments (video, images) are no longer explicitly included on the post's body text by default. You can choose to keep adding them with `keep_media_links`. This option does *not* affect the upload of attachments.
+- Youtube links can be replaced with `invidious` and `invidious_base_url`
+## Enhancements
+- `bio_text` is no longer a mandatory mapping on the config
+- Hugely improved performance (around 4x) when processing tweets
+- Implemented safety measures for avoiding collision with multiple instances of the bot running at the same time
+
 ## [0.8.9] - 2021-12-05
 ## Added
 - ```original_date``` and ```original_date_format``` for adding the original tweet's creation date to the post body

FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: robertoszek
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: robertoszek
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
+custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']

LICENSE.md

@@ -1,4 +1,4 @@
-MIT License Copyright (c) 2021 Roberto Chamorro / project contributors
+MIT License Copyright (c) 2022 Roberto Chamorro / project contributors
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,6 +1,6 @@
 # Stork (pleroma-bot)
 
-[![Build Status](https://travis-ci.com/robertoszek/pleroma-bot.svg?branch=develop)](https://travis-ci.com/robertoszek/pleroma-bot)
+[![Build Status](https://travis-ci.com/robertoszek/pleroma-bot.svg?branch=develop)](https://app.travis-ci.com/github/robertoszek/pleroma-bot)
 [![Version](https://img.shields.io/pypi/v/pleroma-bot.svg)](https://pypi.org/project/pleroma-bot/)
 [![AUR version](https://img.shields.io/aur/version/python-pleroma-bot)](https://aur.archlinux.org/packages/python-pleroma-bot)
 [![codecov](https://codecov.io/gh/robertoszek/pleroma-bot/branch/master/graph/badge.svg?token=0c4Gzv4HjC)](https://codecov.io/gh/robertoszek/pleroma-bot)
@@ -9,7 +9,7 @@
 
 ![Stork](img/stork-smaller.svg)
 
-Mirror one or multiple Twitter accounts in Pleroma/Mastodon.
+Mirror your favourite Twitter accounts in the Fediverse, so you can follow their updates from the comfort of your favorite instance. Or migrate your own to the Fediverse using a Twitter [archive](https://twitter.com/settings/your_twitter_data).
 
 [![Documentation](img/docs.png)](https://robertoszek.github.io/pleroma-bot)
 
@@ -23,7 +23,7 @@ For precisely those cases I've written this Python project that automates them,
 ## Features 
 
 So basically, it does the following:
-
+* Can parse a Twitter [archive](https://twitter.com/settings/your_twitter_data), moving all your tweets to the Fediverse
 * Retrieves **tweets** and posts them on the Fediverse account if their timestamp is newer than the last post.
   * Can filter out RTs or not
   * Can filter out replies or not
@@ -53,7 +53,7 @@ Here's a list of the available packages.
 
 ## Usage
 ```console
-$ pleroma-bot [--noProfile] [--forceDate [FORCEDATE]] [-c CONFIG]
+$ pleroma-bot [-c CONFIG] [-l LOG] [--noProfile] [--forceDate [FORCEDATE]] [-a ARCHIVE]
 ```
 
 ```console
@@ -65,6 +65,9 @@ optional arguments:
                         path of config file (config.yml) to use and parse. If
                         not specified, it will try to find it in the current
                         working directory.
+  -l LOG, --log LOG     path of log file (error.log) to create. If not
+                        specified, it will try to store it at your config file
+                        path
   -n, --noProfile       skips Fediverse profile update (no background image,
                         profile image, bio text, etc.)
   --forceDate [FORCEDATE]
@@ -73,6 +76,9 @@ optional arguments:
                         supplied to only force it for that particular user in
                         the config
   -s, --skipChecks      skips first run checks
+  -a ARCHIVE, --archive ARCHIVE
+                        path of the Twitter archive file (zip) to use for
+                        posting tweets.
   --verbose, -v
   --version             show program's version number and exit
 ```
@@ -87,107 +93,30 @@ If you plan on retrieving tweets from an account which has their tweets **protec
 * Access Token Key and Secret.  You'll also find them on your project app keys and tokens section at [Twitter's Developer Portal](https://developer.twitter.com/en/portal/dashboard). 
 Alternatively, you can obtain the Access Token and Secret by running [this](https://github.com/joestump/python-oauth2/wiki/Twitter-Three-legged-OAuth-Python-3.0) locally, while being logged in with a Twitter account which follows or is the owner of the protected account
 
-Create a ```config.yml``` file in the same path where you are calling ```pleroma-bot```. There's a config example in this repo called ```config.yml.sample``` that can help you when filling yours out:
+### Configuration
+
+Create a ```config.yml``` file in the same path where you are calling ```pleroma-bot``` (or use the `--config` argument to specify a different path). 
+
+There's a config example in this repo called ```config.yml.sample``` that can help you when filling yours out.
+
+For more information you can refer to the ["Configuration" page](https://robertoszek.github.io/pleroma-bot/gettingstarted/configuration/) on the docs.
+
+Here's what a minimal config looks like:
 ```yaml
-twitter_base_url: https://api.twitter.com/1.1
-# Change this to your Fediverse instance
-pleroma_base_url: https://pleroma.robertoszek.xyz
-# (optional) Change this to your preferred nitter instance
-nitter_base_url: https://nitter.net
+# Change this to your target Fediverse instance
+pleroma_base_url: https://pleroma.instance
 # How many tweets to get in every execution
 # Twitter's API hard limit is 3,200
 max_tweets: 40
 # Twitter bearer token
-twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-# List of users and their attributes
+twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 users:
-- twitter_username: KyleBosman
-  pleroma_username: KyleBosman
-  # Mastodon/Pleroma token obtained by following the README.md
+- twitter_username: User1
+  pleroma_username: MyPleromaUser1
+  # Mastodon/Pleroma bearer token
   pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-  # (optional) keys and secrets for using OAuth 1.0a (for protected accounts)
-  consumer_key: xxxxxxxxxxxxxxxxxxxxxxxxx
-  consumer_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-  access_token_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-  access_token_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-  # If you want to add a link to the original status or not
-  signature: true
-  # (optional) If you want to download Twitter attachments and add them to the Pleroma posts.
-  # By default they are not
-  media_upload: true
-  # (optional) If twitter links should be changed to nitter ones. By default they are not
-  nitter: true
-  # (optional) If mentions should be transformed to links to the mentioned Twitter profile
-  rich_text: true
-  # (optional) visibility of the post. Must one of the following: public, unlisted, private, direct
-  # by default is "unlisted"
-  visibility: "unlisted"
-  # (optional) Force all posts for this account to be sensitive or not
-  # The NSFW banner for the instance will be shown for attachments as a warning if true
-  # If not defined, the original tweet sensitivity will be used on a tweet by tweet basis
-  sensitive: false
-  # (optional) Include the creation date of the tweet on the Fediverse post body
-  original_date: true
-  # (optional) Date format to use when adding the creation date of the tweet to the Fediverse post
-  original_date_format: "%Y/%m/%d %H:%"
-  # (optional) If RTs are to be also be posted in the Fediverse account. By default they are included
-  include_rts: false
-  # (optional) If replies are to be also posted in the Fediverse account. By default they are included
-  include_replies: false
-  # (optional) List of hashtags to use for filtering out tweets which don't include any of them
-  hashtags:
-    - sponsored
-  # (optional) How big attachments can be before being ignored and not being uploaded to the Fediverse post
-  # Examples: "30MB", "1.5GB", "0.5TB"
-  file_max_size: 500MB
-  # additional custom-named attributes
-  support_account: robertoszek
-  # you can use any attribute from 'user' inside a string with {{ attr_name }} and it will be replaced
-  # with the attribute value. e.g. {{ support_account }}
-  bio_text: "\U0001F916 BEEP BOOP \U0001F916 \nI'm a bot that mirrors {{ twitter_username }} Twitter's\
-    \ account. \nAny issues please contact @{{ support_account }} \n \n " # username will be replaced by its value
-  # Optional metadata fields and values for the Pleroma profile
-  fields:
-  - name: "\U0001F426 Birdsite"
-    value: "{{ twitter_url }}"
-  - name: "Status"
-    value: "I am completely operational, and all my circuits are functioning perfectly."
-  - name: "Source"
-    value: "https://gitea.robertoszek.xyz/robertoszek/pleroma-twitter-info-grabber"
-# Mastodon instance example
-- twitter_username: WoolieWoolz
-  pleroma_username: 24660
-  pleroma_base_url: https://botsin.space
-  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-  # Mastodon doesn't support rich text!
-  rich_text: false
-  signature: true
-  nitter: true
-  visibility: "unlisted"
-  media_upload: true
-  max_tweets: 50
-  bio_text: "\U0001F916 BEEP BOOP \U0001F916 \nI'm a bot that mirrors {{ twitter_username }} Twitter's\
-    \ account. \nAny issues please contact @{{ support_account }} \n \n " # username will be replaced by its value
-  fields:
-  - name: "\U0001F426 Birdsite"
-    value: "{{ twitter_url }}"
-  - name: "Status"
-    value: "I am completely operational, and all my circuits are functioning perfectly."
-  - name: "Source"
-    value: "https://gitea.robertoszek.xyz/robertoszek/pleroma-twitter-info-grabber"
-# Minimal config example
-- twitter_username: arstechnica
-  pleroma_username: mynewsbot
-  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-  bio_text: ""
 ```
 
-Changing the ```users``` to the desired ones. You can add as many users as needed.
-
-Also change the following to your Pleroma/Mastodon instance URL:
-```yaml
-pleroma_base_url: https://pleroma.robertoszek.xyz
-```
 ### Running
 
 If you're running the bot for the first time it will ask you for the date you wish to start retrieving tweets from (it will gather all from that date up to the present). 
@@ -203,7 +132,7 @@ For example:
 $ pleroma-bot --forceDate WoolieWoolz
 ```
 
-If the ```--noProfile``` argument is passed, *only* new tweets will be posted. The profile picture, banner, display name and bio will **not** be updated on the Fediverse account.
+If the --noProfile argument is passed, the profile picture, banner, display name and bio will **not** be updated on the Fediverse account. However, it will still gather and post the tweets following your config's parameters.
 
 NOTE: An ```error.log``` file will be created at the path from which ```pleroma-bot``` is being called.
 
@@ -243,7 +172,7 @@ Patches, pull requests, and bug reports are more than [welcome](https://github.c
 
 MIT License
 
-Copyright (c) 2021 Roberto Chamorro / project contributors
+Copyright (c) 2022 Roberto Chamorro / project contributors
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -264,6 +193,6 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
 
 **Tested and confirmed working against** :
-* ```Pleroma BE 2.0.50-2547-g5c2b6922-develop```
+* ```Pleroma BE 2.2.50-711-g100e34b4-develop```
 * ```Mastodon v3.2.1```
 * ```Twitter API v1.1 and v2```

config-minimal.yml.sample

@@ -0,0 +1,12 @@
+# Change this to your Fediverse instance
+pleroma_base_url: https://pleroma.instance
+# How many tweets to get in every execution
+# Twitter's API hard limit is 3,200
+max_tweets: 40
+# Twitter bearer token
+twitter_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+users:
+- twitter_username: User1
+  pleroma_username: MyPleromaUser1
+  # Mastodon/Pleroma bearer token
+  pleroma_token: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

config.yml.sample

@@ -50,6 +50,7 @@ users:
   support_account: robertoszek
   # you can use any attribute from 'user' inside a string with {{ attr_name }} and it will be replaced
   # with the attribute value. e.g. {{ support_account }}
+  # (optional) Text to be appended to the Twitter account bio text
   bio_text: "\U0001F916 BEEP BOOP \U0001F916 \nI'm a bot that mirrors {{ twitter_username }} Twitter's\
     \ account. \nAny issues please contact @{{ support_account }} \n \n " # username will be replaced by its value
   # Optional metadata fields and values for the Pleroma profile

docs/contribute/contributing.md

@@ -1,5 +1,8 @@
 # Contributing
 
+## Funding
+[![Donate using Liberapay](https://liberapay.com/assets/widgets/donate.svg)](https://liberapay.com/robertoszek/donate) [![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/robertoszek)
+
 ## Code
 
 Patches, pull requests, and bug reports are more than [welcome](https://github.com/robertoszek/pleroma-bot/issues/new/choose), please keep the style consistent with the original source.

docs/gettingstarted/beforerunning.md

@@ -4,9 +4,15 @@ If you haven't already, you need to [apply for a Twitter developer account](http
 
 The process involves some review of the developer account application by Twitter and it's very likely you'll be asked for some details pertaining your usecase. It usually doesn't take longer than a day or two to complete the application, the back and forth is mostly automated on their part.
 
+Additionally, Twitter introduced a new tier of access (Elevated) to their API projects and although existing projects (before Nov 2020) were promoted automatically, new users will only get Essential access instead by default, in which requests to API v1.1 are disabled. 
+
+We still use v1.1 for downloading videos and profile banners, and as of now there is no available alternative in v2, so you'll need Elevated access for the bot to function properly until further notice.
+
+You can apply for Elevated access [here](https://developer.twitter.com/en/portal/products/elevated).
+
 ## Twitter tokens
 
-Once you have a Twitter developer account, you need to access your [dashboard](https://developer.twitter.com/en/portal/dashboard) and create a new project (so your app has V2 access) and also create a new app associated to that new project.
+Once you have a Twitter developer account, you need to access your [dashboard](https://developer.twitter.com/en/portal/dashboard) and create a new project (so your app has v2 access) and also create a new app associated to that new project.
 
 Now, enter your new application "Keys and tokens" section, copy and safely store all of your tokens.