Limit to how many calls you can retrieve?

[jam1245]

hello there - I'm loving your package and finding it super helpful. thanks for all the work you've put into this. I'm wondering if there is a limit to how much data I can retrieve. I'm trying to get daily reports for dates and lat long points over the past 50 years. My data has around 70,000 rows. I tried testing this with a smaller number of rows 2000 and got a connection issue that looks like it dropped me after 620 records. I don't want to disrupt the service and want to be respectful here so looking for guidance. Thank you in advance!

[clampr]

Hi @jam1245,

I guess you're using the Python library, right?

There should be no limit. We're using Cloudflare to scale our service - so you should be good to go.

Can you share your code + the error?

[jam1245]

Hi @clampr - thanks so much for the quick reply. I'm sharing my code and the error I'm getting. It mentions the connection was forcibly closed by the remote host: URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>

Please let me know if you have any suggestions! Thanks in advance and appreciate all the work that goes into this service!

----- Python Code -------

from meteostat import Point, Daily
from datetime import datetime 
import pandas as pd    

### filter on U.S. to drop 10k rows
test = df.loc[df['cntry_name'] == 'United States']

### reduce from 70k rows to 2000
test = test.head(2000)

test['datetime'] = test['datetime'].apply(lambda x: datetime.strptime(x, '%Y-%m-%dT%H:%M:%SZ').strftime('%Y-%m-%d'))

### Create an empty DataFrame to store the results
results = pd.DataFrame()

### Loop over the rows in the DataFrame
for index, row in test.iterrows():
    ##### Create a Point for each row
    point = Point(row['latitude'], row['longitude'])
    
    #### Convert the 'datetime' column to a datetime object with year-month-day
    start = datetime.strptime(row['datetime'], '%Y-%m-%d')
    end = start  # I want the same start and end date
    
    #### Get the daily data
    data = Daily(point, start, end).fetch()
    
    #### Add the result to the 'results' DataFrame
    results = pd.concat([results, data])

--------- Error Message ------------
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

ConnectionResetError Traceback (most recent call last)
File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\urllib\request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
1347 try:
-> 1348 h.request(req.get_method(), req.selector, req.data, headers,
1349 encode_chunked=req.has_header('Transfer-encoding'))
1350 except OSError as err: # timeout error

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\http\client.py:1286, in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
1285 """Send a complete request to the server."""
-> 1286 self._send_request(method, url, body, headers, encode_chunked)

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\http\client.py:1332, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
1331 body = _encode(body, 'body')
-> 1332 self.endheaders(body, encode_chunked=encode_chunked)

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\http\client.py:1281, in HTTPConnection.endheaders(self, message_body, encode_chunked)
1280 raise CannotSendHeader()
-> 1281 self._send_output(message_body, encode_chunked=encode_chunked)

File C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\http\client.py:1041, in HTTPConnection._send_output(self, message_body, encode_chunked)
1040 del self._buffer[:]
-> 1041 self.send(msg)
1043 if message_body is not None:
1044
...
-> 1351 raise URLError(err)
1352 r = h.getresponse()
1353 except:

URLError: <urlopen error [WinError 10054] An existing connection was forcibly closed by the remote host>