Skip to content

Add robust and performant HTTP data fetcher

Gustavo Padovan requested to merge robust-fetcher into main

We use requests.Session to maintain the connection alive between requests and add an HTTPAdapter with a robust retry logic to save us when bad things happen.

There is also a url validator.

This project will be running thousands of requests. We need to remove our fragility and add more performance.

If this code is good, then I'll go ahead and convert all the request code we have.

Merge request reports