Specific keywords queries blocked

[danielanapoli]

(Not sure if this is best to post in discussion or issues... Please let me know if I should move it!)

Hi,

I'm having a hard time getting specific keyword queries to work. I set up a Luminati proxy to avoid being blocked, but particular keyword combinations (that I know can elicit results on Google's web UI) seem to be resolving to StopIteration... Is there anyway around this?

from scholarly import scholarly, ProxyGenerator

pg = ProxyGenerator()
pg.Luminati(usr='xyz', passwd='xyz', proxy_port='xyz')
scholarly.use_proxy(pg)

# example query 1 - succeeds
search_query = scholarly.search_keywords(['usability', 'security', 'privacy'])
scholarly.pprint(next(search_query))

# example query 2 - succeeds 
search_query = scholarly.search_keyword('nudge')
scholarly.pprint(next(search_query))

# example query 3 - fails
search_query = scholarly.search_keywords(['usability', 'security', 'privacy', 'nudge'])
scholarly.pprint(next(search_query))

[ipeirotis]

There is only one author that includes the term "nudge" in their profile: https://scholar.google.com/citations?hl=en&user=lMkvfl8AAAAJ

There are no authors that include the terms nudge and usability. (See https://scholar.google.com/citations?hl=en&view_op=search_authors&mauthors=nudge+usability&btnG=)

The next does not really fail, it just tells you that you reached the end of the results.

[danielanapoli]

Ah, the problem is I confused search_pubs with search_keywords. I see that my query is targeting author information because it returns an Author object not a Publication object. Thanks.

Quick follow up -- does the str passed as a query param to pubs() only target publication titles?

[ipeirotis]

It just does what Google Scholar does when you type a query. I think you need to put the prefix allintitle: if you want the terms to appear in the title.

[danielanapoli]

Thanks for the help. Feel free to delete (since this probably isn't super helpful)!

[Aniruth-ai]

Hey @ipeirotis and @danielanapoli, I started using scholarly API recently and not so sure about how to setup the Luminati Proxy. I am getting blocked by google scholar while using search_pubs for more than one time in less than a minute. Does installing the proxy going to solve this issue? If yes, then could you be able to help me on this?

[danielanapoli]

Using a proxy should help with getting blocked.

This video was helpful to me for getting started with Luminati: https://www.youtube.com/watch?v=LF1u3QFNqIQ&ab_channel=LuminatiNetworks

And here is the documentation for integrating Luminati with the scholarly library: https://scholarly.readthedocs.io/en/latest/quickstart.html#using-proxies

Hope that helps!

[Aniruth-ai]

Thank you @danielanapoli for a quick reply and for sharing the information👍 !

[stanleyrhodes]

@danielanapoli With Luminati you used residential proxies? And I'm scratching my head a bit about how the pricing scheme works.

I think there needs to be a Scholarly proxy guide for the less savvy. I'll try to document what I'm doing as I go through it.

[ipeirotis]

Empirically, I did not notice a big difference between the data center and the residential proxies in terms of blocking. If anything, the data center proxies were faster.

[Tranhoa150284]

Thank sir