Design for dealing with the large amount of outbound rest API calls


We are building an application where we need to get the data from the external system required for case processing. To get the data, we need to call two rest API.

1: API -1 gives 1 million or more of records, but we are interested only in one field of these records.

2: API-2 is a call for these 1- Million records separately, i.e. (the same API needs to call for 1 million times ). Again, we are interested in the 1 or 2 fields.

This data can not be cached in pega for a long time. Data get updated in the external system at least close to real-time.

Performance and close to real-time calls are the main factors to consider while designing the solution.



Keep up to date on this post and subscribe to comments

November 11, 2019 - 10:24am

am I right that the API-2 is a filtered API-1? Like

API-1 is /api/v1/companies

API-2 is /api/v1/company/{id}

November 11, 2019 - 10:41am
Response to vaspoz

Yes, it's similar to the example you mentioned 

November 12, 2019 - 2:19am
Response to PravinM

You mentioned, that you're interested in only 1-2 records from the whole set.

So why then use API-1? Just make 1-2 separate calls to API-2 to get the exact data what you need

November 12, 2019 - 9:14am
Response to vaspoz

I am interested in all the records but only 1or 2 properties.

November 12, 2019 - 11:55am
Response to PravinM

well, it's out of your hands. It's the service owner responsibility to define what exactly will be in the response.

From your side you could add Response Data transform to remove all unneeded properties. But keeping in mind that the connector will retrieve ~1 million records, it will be very heavy =/ Test it beforehand.

Better approach would be to change the webservice to use some kind of pagination from their side