Pagination url from Facebook

[liulilli]

Hi everyone!
This might be a stupid question, but I wanted to scrape posts from the CNN International page starting from the 1st of March backwards. I've read about start_url but, as written here #310, I would need to get a pagination url. However I don't know how to obtain a pagination url. Could someone help me or redirect to the appropriate Facebook documentation? (Either here or in private)
Thank you so much in advance and have a nice day

[neon-ninja]

Here's how you can retrieve posts quickly, logging timestamps and pagination URLs:

def handle_pagination_url(url):
    print(url)

for post in get_posts("cnninternational", pages=None, options={'allow_extra_requests': False, 'posts_per_page': 200}, request_url_callback=handle_pagination_url):
    print(post["time"])

Sample output:

2022-03-03 10:30:09
https://m.facebook.com/page_content_list_view/more/?page_id=18793419640&start_cursor={"timeline_cursor":"AQHRETPViw_PpYB8dvuQv_nxWo4tDuVr9kDnxwrV6GsELF02wsIOTobJTeozaNOOK1iPD6iEAxbPJRs5sIT9QYh3YPwRpaOnv6oLSBa7u6P4zkX9EQYxXLZwZcLg6B4NuP4Z","timeline_section_cursor":null,"has_next_page":true}&num_to_fetch=200&surface_type=posts_tab
2022-03-03 10:01:09

start_url = 'https://m.facebook.com/page_content_list_view/more/?page_id=18793419640&start_cursor={"timeline_cursor":"AQHRETPViw_PpYB8dvuQv_nxWo4tDuVr9kDnxwrV6GsELF02wsIOTobJTeozaNOOK1iPD6iEAxbPJRs5sIT9QYh3YPwRpaOnv6oLSBa7u6P4zkX9EQYxXLZwZcLg6B4NuP4Z","timeline_section_cursor":null,"has_next_page":true}&num_to_fetch=200&surface_type=posts_tab'

should work for you I think

[liulilli]

Thank you so much for your help! Really appreciated

[liulilli]

I need to ask again because I'm a bit of a Python newbie and I'm not sure where I'm making a mistake. I'm trying to scrape around 60/70 posts starting from the 1st of March backwards, but it seems that the code I'm writing sees as a single post all the posts that are present in the first pagination. In addition, after the first "post" retrieved it progresses in collecting a second "post" which is just a copy of the first. As a consequence, it's very slow (I haven't been able to append the posts even after hours of letting the code run). Could you please correct me? I'm writing the code below

from facebook_scraper import get_posts

channel_str = "cnninternational"

import time
start_url = 'https://m.facebook.com/page_content_list_view/more/?page_id=18793419640&start_cursor={"timeline_cursor":"AQHRETPViw_PpYB8dvuQv_nxWo4tDuVr9kDnxwrV6GsELF02wsIOTobJTeozaNOOK1iPD6iEAxbPJRs5sIT9QYh3YPwRpaOnv6oLSBa7u6P4zkX9EQYxXLZwZcLg6B4NuP4Z","timeline_section_cursor":null,"has_next_page":true}&num_to_fetch=200&surface_type=posts_tab'

def handle_pagination_url(url):
global start_url
start_url = url

post_list = []
for post in get_posts(channel_str, cookies = 'facebook.com_cookies.txt', extra_info = True, page_limit = 1, start_url = start_url, request_url_callback = handle_pagination_url,
options={"comments": True, "progress":False}):
print("ID: ", post["post_id"])
post_list.append(post)

print("TOTAL POSTS RECOVERED: ", len(post_list))

Thank you very much in advance!

[neon-ninja]

Fetching all comments for 200 posts would take a very long time. I would recommend decoupling these processes, and fetching the posts first, then coming back for the comments

[liulilli]

Thank you for your kind answer! Much appreciated :)