[DeviantArt] Can the URLs for all the gallery pages for the specified deviant be gotten, along with their page numbers for each?

[smaybius]

It's said that the DeviantArt API allows for it, but I'm wondering if gallery-dl has that option. It's to be able to send all pages to do with the deviant to the wayback machine without needing to manually scrub through each gallery page to save each page number one or five at a time.
I only know how to catalog all submissions instead of gallery pages using gallery-dl:

gallery-dl -o original=false --filter "print(url)" https://www.deviantart.com/someone/gallery/all > someone.txt

[mikf]

You mean generating links like https://www.deviantart.com/USER/gallery?page=PAGE ?

The API has no concept of pages, only post offsets or cursor values. I guess one could convert the highest offset to the last page number and count down, but fetching the highest page number can be done much simpler with one HTTP request to the artist's first gallery page and some regex.

[smaybius]

Yes, along with all the pages under "/gallery/\d+/".

[mikf]

So folder pages? The OAuth API identifies each folder by UUID and not the index number used by the website itself, so gallery-dl's output would be pretty useless for these.

You can use gallery-dl -o flat=0 -g https://www.deviantart.com/USER/gallery to list pseudo folder URLs, but they don't actually work outside of gallery-dl.

[smaybius]

I found a different way with ChatGPT's help, but I don't know if it's practical as it uses live pages instead of API calls.

import requests
from bs4 import BeautifulSoup
import re
import sys

base_url = f"https://www.deviantart.com/{sys.argv[1]}/gallery"
pattern = re.compile(f"https://www.deviantart.com/{sys.argv[1]}/" + r"gallery/\d+/.*")
page_pattern = re.compile(r'\?page=(\d+)')

def fetch_links(url):
    response = requests.get(url)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        links = [a['href'] for a in soup.find_all('a', href=True) if pattern.match(a['href'])]
        return links
    raise NameError(f"No results for https://www.deviantart.com/{sys.argv[1]}/, returned code {response.status_code}")

def get_highest_page_number(link):
    max_page = 0
    response = requests.get(link)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        for a in soup.find_all('a', href=True):
            match = page_pattern.search(a['href'])
            if match:
                page_number = int(match.group(1))
                if page_number > max_page:
                    max_page = page_number
    return max_page

if __name__ == "__main__":
    links = fetch_links(base_url)
    if links:
        for link in links:
            highest_page_number = get_highest_page_number(link)
            print(f"{link}")
            if highest_page_number >= 2:
                for page in range(2, highest_page_number):
                    print(f"{link}?page={page}")
        highest_page_number = get_highest_page_number(f"https://www.deviantart.com/{sys.argv[1]}/gallery/all")
        print(f"https://www.deviantart.com/{sys.argv[1]}/gallery/all")
        if highest_page_number >= 2:
            for page in range(2, highest_page_number):
                print(f"https://www.deviantart.com/{sys.argv[1]}/gallery/all?page={page}")
    else:
        print("No matching links found.")

[mikf]

This will also only grab /gallery/all pages and not any /gallery/\d+/ ones, but depending on your goal this is good enough as /gallery/all contains all of an artist's posts.

I don't know if it's practical as it uses live pages instead of API calls.

As I said before, the API doesn't use the concept of "pages" to iterate over its results. Using "live pages" is better for the wayback machine anyway I'd say.

[smaybius]

I found that it only gets the first folders that were loaded. I had a frustrating effort back and forth with GPT to make something that works and goes past the first ones loaded, and here's what's hopefully the final result which I tested: https://gist.github.com/smaybius/7dd26eebee5447511329e3768124786a

Whenever I see anything wrong with the result, I talk it out with ChatGPT until all the folders of the artist that it had problems with are gotten, mainly using fresh new chats because it often ruins the script beyond repair if I put everything in the same chat.