Error when using private_extractions: too many 500 error responses

Hello,

I’m trying to use the WorldCereal private_extractions service to access my private collections. I have successfully authenticated using the Copernicus Data Space user code and initiated the session with RdmInteraction(), but when I run get_collections(include_public=False, include_private=True) I get the following error:

RetryError: HTTPSConnectionPool(host='ewoc-rdm-api.iiasa.ac.at', port=443): Max retries exceeded with url: /collections/search?Bbox=-180&Bbox=-90&Bbox=180&Bbox=90 (Caused by ResponseError('too many 500 error responses'))

It seems the RDM API is returning multiple internal server errors (500), and the request fails after too many retries.

Has anyone experienced the same issue, or is the service currently down?

Thanks in advance for your support!

Best regards,
Filippo

The print of the errors is here:

2025-07-18 14:37:06.090 | INFO | worldcereal.rdm_api.rdm_interaction:get_collections:152 - To access private collections, you need to authenticate.
Visit Sign in to Copernicus Data Space Ecosystem :clipboard: to authenticate.
:white_check_mark: Authorized successfully

ResponseError Traceback (most recent call last)
ResponseError: too many 500 error responses

The above exception was the direct cause of the following exception:

MaxRetryError Traceback (most recent call last)
File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/requests/adapters.py:667, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
666 try:
→ 667 resp = conn.urlopen(
668 method=request.method,
669 url=url,
670 body=request.body,
671 headers=request.headers,
672 redirect=False,
673 assert_same_host=False,
674 preload_content=False,
675 decode_content=False,
676 retries=self.max_retries,
677 timeout=timeout,
678 chunked=chunked,
679 )
681 except (ProtocolError, OSError) as err:

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/urllib3/connectionpool.py:942, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
941 log.debug(“Retry: %s”, url)
→ 942 return self.urlopen(
943 method,
944 url,
945 body,
946 headers,
947 retries=retries,
948 redirect=redirect,
949 assert_same_host=assert_same_host,
950 timeout=timeout,
951 pool_timeout=pool_timeout,
952 release_conn=release_conn,
953 chunked=chunked,
954 body_pos=body_pos,
955 preload_content=preload_content,
956 decode_content=decode_content,
957 **response_kw,
958 )
960 return response

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/urllib3/connectionpool.py:942, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
941 log.debug(“Retry: %s”, url)
→ 942 return self.urlopen(
943 method,
944 url,
945 body,
946 headers,
947 retries=retries,
948 redirect=redirect,
949 assert_same_host=assert_same_host,
950 timeout=timeout,
951 pool_timeout=pool_timeout,
952 release_conn=release_conn,
953 chunked=chunked,
954 body_pos=body_pos,
955 preload_content=preload_content,
956 decode_content=decode_content,
957 **response_kw,
958 )
960 return response

[... skipping similar frames: HTTPConnectionPool.urlopen at line 942 (2 times)]

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/urllib3/connectionpool.py:942, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
941 log.debug(“Retry: %s”, url)
→ 942 return self.urlopen(
943 method,
944 url,
945 body,
946 headers,
947 retries=retries,
948 redirect=redirect,
949 assert_same_host=assert_same_host,
950 timeout=timeout,
951 pool_timeout=pool_timeout,
952 release_conn=release_conn,
953 chunked=chunked,
954 body_pos=body_pos,
955 preload_content=preload_content,
956 decode_content=decode_content,
957 **response_kw,
958 )
960 return response

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/urllib3/connectionpool.py:932, in HTTPConnectionPool.urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, preload_content, decode_content, **response_kw)
931 try:
→ 932 retries = retries.increment(method, url, response=response, _pool=self)
933 except MaxRetryError:

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/urllib3/util/retry.py:519, in Retry.increment(self, method, url, response, error, _pool, _stacktrace)
518 reason = error or ResponseError(cause)
→ 519 raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
521 log.debug(“Incremented Retry for (url=‘%s’): %r”, url, new_retry)

MaxRetryError: HTTPSConnectionPool(host=‘ewoc-rdm-api.iiasa.ac.at’, port=443): Max retries exceeded with url: /collections/search?Bbox=-180&Bbox=-90&Bbox=180&Bbox=90 (Caused by ResponseError(‘too many 500 error responses’))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
Cell In[1], line 6
3 rdm = RdmInteraction()
5 # Get a list of your private collections
----> 6 collections = rdm.get_collections(include_public=False, include_private=True)
8 if len(collections) == 0:
9 print(‘No private collections found, cannot proceed.’)

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/worldcereal/rdm_api/rdm_interaction.py:195, in RdmInteraction.get_collections(self, spatial_extent, temporal_extent, include_public, include_private, ewoc_codes)
192 url = f"{self.RDM_ENDPOINT}/collections/search?{bbox_str}{val_time}{ewoc_codes_str}"
194 # Process the request
→ 195 response = self.session.get(url=url, headers=self._get_headers(), timeout=10)
197 if response.status_code != 200:
198 raise Exception(f"Error fetching collections: {response.text}")

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/requests/sessions.py:602, in Session.get(self, url, **kwargs)
594 r""“Sends a GET request. Returns :class:Response object.
595
596 :param url: URL for the new :class:Request object.
597 :param **kwargs: Optional arguments that request takes.
598 :rtype: requests.Response
599 “””
601 kwargs.setdefault(“allow_redirects”, True)
→ 602 return self.request(“GET”, url, **kwargs)

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/requests/sessions.py:589, in Session.request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
584 send_kwargs = {
585 “timeout”: timeout,
586 “allow_redirects”: allow_redirects,
587 }
588 send_kwargs.update(settings)
→ 589 resp = self.send(prep, **send_kwargs)
591 return resp

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
700 start = preferred_clock()
702 # Send the request
→ 703 r = adapter.send(request, **kwargs)
705 # Total elapsed time of the request (approximately)
706 elapsed = preferred_clock() - start

File /opt/conda/envs/worldcereal/lib/python3.11/site-packages/requests/adapters.py:691, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
688 raise ConnectTimeout(e, request=request)
690 if isinstance(e.reason, ResponseError):
→ 691 raise RetryError(e, request=request)
693 if isinstance(e.reason, _ProxyError):
694 raise ProxyError(e, request=request)

RetryError: HTTPSConnectionPool(host=‘ewoc-rdm-api.iiasa.ac.at’, port=443): Max retries exceeded with url: /collections/search?Bbox=-180&Bbox=-90&Bbox=180&Bbox=90 (Caused by ResponseError(‘too many 500 error responses’))

Hi there! Thanks for posting this. Should indeed not be happening. We’re taking this up with the developer of the API and will get back to you asap.

We identified an issue due to a recent fix in the database. This issue should now be fixed. Could you try once again? Let us know if you’re still facing issues.

Thanks, now the cell is working. There were a few other lines missing in Step 3: EO data extractions — I had to add !pip install geopandas and import geopandas as gpd. Also, the line that saves the samples_df to file was commented out (# samples_df.to_file(samples_df_path, driver='GPKG')), so I uncommented it. After that, everything worked fine. Thank you!

Great to hear it’s working now.
And yes, you are right that the line responsible for saving the dataframe was wrongly commented out.
We have fixed this issue and are working on a significant update of custom crop type notebook as well, to be released later this week.