Where does scholarly increment to the next page of search results (in a pubs search)?

[stanleyrhodes]

Where does this package control going from page to page of results? I've been chasing the code up and down and I can't figure it out. When calling search_pubs() the start_index defaults to 0. But I've been through the whole code, especially _construct_url(), search_publications(), and _SearchScholarIterator(), and I can't find anything that looks like it either increments start_index to construct a new URL, or somehow updates the URL itself while looping through all the results.

The reason I want to know, besides understanding the package overall, is that I would like to script a wait-time between page requests (and later, fill requests) so GS doesn't become upset about the traffic. That's in addition to using ProxyManager, which I have set up and working. I've been laboring to understand the inner workings of this package for a while, but intensely in the last month, because I need to use search_pubs() for a quantitative literature review of thousands of papers. GS and Scholarly are the only game in town that can do it. I've been using Python for years, but I'm not very familiar with this depth of OOP with so many nested methods. I must have read the code that goes to the next page, but not have recognized it as such.

[arunkannawadi]

Hi Stanley, look at this implementation of __next__ here: https://github.com/scholarly-python-package/scholarly/blob/main/scholarly/publication_parser.py#L80-L93 The iterator is not user-facing, but manages fetching results from multiple pages. You might want to add a delay in the elif block there. Overall, I think that'd be a good feature to have, so feel free to submit a pull request with the delay that can be specified by the user.

[arunkannawadi]

You introduced a delay before the if block that is unnecessary. If self._pos < len(self._rows) holds true, it means that we are creating publication objects from a page that is already loaded, there's no delay needed there since we're not requesting anything from GS. If that fails, then the iterator looks for the next button and see if it's active. If it's not disabled, then it fetches the next page. That's where we might want a delay.

Since you've created a function that mocks up human behavior, we can have a boolean variable passed through search_pubs all the way down to _SearchScholarIterator that can either choose to have this delay or not.

[arunkannawadi]

Feel free to open a PR with your preliminary design and it's easier to suggest modifications or comment on your contribution. Here's the section on how to contribute: https://github.com/scholarly-python-package/scholarly/blob/main/.github/CONTRIBUTING.md#how-to-contribute

[stanleyrhodes]

Yes, thanks for pointing out the sleep placements. I did simplify the code a bit too much in my example for convenience sake, but I would put it before the next button check. I can see the benefit of having a boolean, but also the value of users being able to easily customize their delay timing. Working more on the project that necessitated including this delay may give me a better idea of what I think would better serve the intended users. I don't know without some experimentation whether, say, the kind of proxy being used changes the delay time preference. As you said, we can discuss with a PR and I'll have more experience by then.

Thanks also for linking me to the contribution how-to. To be honest, I don't have the time right now to fulfill all the conditions in that list. Particularly figuring out how to modify or create new test cases, as I haven't done that before outside a very basic tutorial I took a few years ago. Once the draft paper--the one I'm using scholarly for--is in internal review I'll circle back and give it all a shot using your checklist. That may be closer to the end of the year. But I'll do it before journal submission, when I'm making sure the results are replicable and citing the software version.

I appreciate you taking the time to give me some guidance! This--scholarly--is a really important project.

[arunkannawadi]

No worries. I might take the snippet you posted and make you a co-author on those commits in the mean time. Great to see this library being used scholarly work.

[stanleyrhodes]

Sure, of course I'm fine with that if you think that's worthwhile. I still hope to run through making test cases in the future. I made the delay code updates and pushed the changes to the scholarly repo fork in my Github repos. The changes are untested and I'm not sure they're working properly. I need to see if I can get the logging working and have a more verbose, play-by-play account of what's happening when I run search_pubs(). I can say, unfortunately, I had no luck with search_pubs() before making those changes, and I've had no luck afterward. GS doesn't seem to like Brightdata's data center proxies either; if anything, it's worse now than it was a couple months ago.

I think I should start a discussion in a new thread about what we (users) can realistically expect from search_pubs() right now, how people have used it to get results, dos and don'ts with proxies, and so on. It may be that I'm doing a lot of things wrong, or it may be that GS is just too good at blocking anything and everything search_pubs().