How can i get 12 million requests to complete my graduation thesis?

[hxby112345678]

Due to my graduation thesis, I need all projects related to China from 2008 to 2024, as well as corresponding information in other categories. I have crawled about 70000 Chinese users who meet the requirements, and now I need to crawl projects under their names. After calculation, I need a total of about 2000000 * 6=12000000, which is about 12 million requests to obtain all the data. However, my account only has 5000 visits per hour, and it only takes about 120000 requests per day. That is to say, it will take me about 100 days to obtain the data I want using my account. Is there any way supported by GitHub to obtain this data for research? It would be even better if we could obtain these data within a month, thank you!

[hxby112345678]

if you have any ideas that can help me,i would be extremely grateful!

[AmalDarKhalil]

GitHub has strict rate limits for API requests, especially for unverified or individual accounts. However, there are several strategies you can employ to potentially gather the required data more efficiently or work around the limits, especially for research purposes:

1. Use Personal Access Tokens with Rate Limit Increases

• GitHub API Rate Limit: The rate limit for GitHub's API is typically 5,000 requests per hour for authenticated requests with a personal access token (PAT). This limit is per account and can be further restricted based on the endpoint.
• Apply for Academic Research Access: You could contact GitHub and explain your academic research. Sometimes, GitHub may provide higher rate limits or special access for research purposes.
  • You can try reaching out through GitHub’s support or their specific GitHub Research Archive for researchers.

2. Distribute Requests Across Multiple Accounts

• Use Multiple GitHub Accounts: By distributing the requests across multiple accounts, each authenticated with a unique personal access token, you could scale up the request capacity. Each account would get 5,000 requests per hour, meaning if you have 10 accounts, you can make 50,000 requests per hour.
  • This would allow you to make 1.2 million requests per day, cutting your time frame down to just 10 days.
  • To stay compliant with GitHub's policies, ensure you’re not violating their terms of service, and ideally, get approval for this approach.

3. Optimize API Requests

• Use GraphQL API: GitHub’s GraphQL API allows for more efficient querying compared to REST API because you can batch multiple requests into one query and retrieve multiple fields at once. This could help you reduce the total number of requests needed.
  • For example, instead of making one request per user, you could potentially retrieve multiple projects per user in one query.
• Crawl Only Updated Data: If you're looking to reduce the number of requests further, consider fetching only projects that have been updated since your last crawl using the updated_at parameter. This reduces redundancy.

4. Parallelization and Request Scheduling

• Distributed Crawling: You could distribute your scraping task over multiple servers, each authenticated with a different GitHub account. If you have access to cloud infrastructure (AWS, GCP, Azure), you can distribute the task across multiple instances to speed up the process.
• Scheduled Crawling: By running the crawling process in intervals (every hour or 15 minutes), you can ensure you're always utilizing the available rate limit, spreading the requests evenly over the day.

5. Consider Bulk Data Sources

• GitHub Archive: GitHub provides a public archive of events that happen across its platform, which might contain the data you need. It’s hosted on Google BigQuery, and you can access it through Google's GitHub Archive. You could potentially query and download bulk data from there rather than crawling the API yourself.
• GHTorrent: This is another project that mirrors all GitHub activity in a more queryable format. It may already have the data you’re looking for, which could save time. You can find it on GHTorrent.

6. Check GitHub’s Data Grant Programs

• GitHub’s Data for Research Program: GitHub has specific data grants for researchers who need access to larger volumes of data than the public API allows. This program might allow you to access the data you need much more quickly.
  • You can check it out here: GitHub Data for Research.

7. Rate Limiting with Exponential Backoff

• Ensure that your scraping scripts respect GitHub's rate limits by incorporating a mechanism for backoff and retries when you hit the limit, to avoid being blocked.

Summary of Steps

1. Apply for GitHub Academic Research access or explore GitHub Archive for bulk data.
2. Leverage multiple GitHub accounts with individual access tokens.
3. Use the GitHub GraphQL API to optimize requests.
4. Distribute and schedule requests using cloud infrastructure.
5. Consider external resources like GitHub Archive and GHTorrent.