Fix: #892 DeviantArt not ripping full sized images #1000
Conversation
|
1000th pull request ... |
| @@ -4,7 +4,7 @@ | |||
| <groupId>com.rarchives.ripme</groupId> | |||
| <artifactId>ripme</artifactId> | |||
| <packaging>jar</packaging> | |||
| <version>1.7.67</version> | |||
| <version>1.7.68-Q0-1.0.0</version> | |||
There was a problem hiding this comment.
Please change this back to 1.7.67
| @@ -267,6 +267,8 @@ public String getUsername(Document doc) { | |||
| try { | |||
| String imageURL = doc.select("span").first().attr("data-super-full-img"); | |||
| if (!imageURL.isEmpty() && imageURL.startsWith("http")) { | |||
| String page = doc.select("span").first().attr("href"); | |||
| imageURL = smallToFull(imageURL,page); | |||
There was a problem hiding this comment.
Ripping the album https://www.deviantart.com/justgenitals/gallery/64358240/SPECIAL-MinamiShots I'm not seeing any difference between the images downloaded using smallToFull and the ones not, can you give me a test link so I can confirm that the ripper now downloads full sized images?
|
I investigated this, and it turns out this whole ripper was badly messed up. The first image on any given page (or possibly more) were being skipped due to the way JSON was parsed, cookies were not being maintained correctly at all, and the method remaining intended to check on original links was not being used, though it was mostly functional as is. I should have a pull request soon with a major update, though it A: slows down the ripper a fair bit as it both has to load every image page and then possibly again another request for the download link and B: may still have issues with the original download links 403 erroring if we hit them fast enough. It may be worth building in a whole setting for checking on originals or accepting the smaller ones. |
I'm very surprised how that can be given the record so far. |
Me as much as you, I just recently used it to download a huge collection and re-parse my existing directories and looking at it it consistently didn't download the first 1-2 files of each page (not even just artist, but every single page) when I started from a ?catpath=/. I should have suspected something when in my re-rips it was randomly grabbing a lot of older files I should have had already, but it took dealing with e621 to see this pull and look into it. |
Category
This change is exactly one of the following (please change
[ ]to[x]) to indicate which:Description
I've fixed and enabled the full size image download for Deviantart
Testing
Required verification:
mvn test(there are no new failures or errors). - Additional Regression testing is needed.Optional but recommended: