Question

Timed out borrowing service requestor from requestor pool for service package: monitor

We have multiple nodes running and the requests to the application are made using IAC. In one node ,In SMa the requestor count is growing high.Stale requestors are still present in System Management Application (SMA). In other node the below exception is comming in the logs and the system is non responsive.

Logs are getting filled with the below message and the system is becomming non-responsive.

2017-04-05 16:59:31,226 [ http-nio-80-exec-59] [ ] [ ] [ ] ( internal.mgmt.RequestorPool) ERROR username|ipaddress - Timed out borrowing service requestor from requestor pool for service package: monitor

Could any one please assist.

Thanks,

Srikanth J

***Updated by moderator: Lochan to remove proprietary information***

**Moderation Team has archived post**

This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.

Comments

Keep up to date on this post and subscribe to comments

Mod
April 5, 2017 - 4:14am
Lochana | Community Moderator | Pegasystems Inc.

April 5, 2017 - 6:23am
Response to Lochan_DV

Hi Lochan,

Thanks for reply. 

i am facing  2 issues here :

1) Time out borrowing service requestor error (  the above link suggests to maximize the  max requestors count to 100 . does it mean i need to add DATA-ADMIN-SERVICEPACKAGE MONITOR service package max requestors to 100 ? does this configuration need to be done manually in all environments ?

2) The second issue is the requestors are not getting killed and the requestors count is high when viewd in sma.

Could you please assist .

Thanks,

Srikanth J

 

Pega
April 5, 2017 - 6:34am

Hi Srikanth,

For the first issue check the number of requests for the service package from the SMA and if it is more than 100, you will have to update the requestor pool settings manually on all the nodes (please keep a note that by increasing the requestor pool setting will have an impact on the memory footprint)

For the second issue, could you please share the Pegarules logs and application server log files to verify the pattern and to make sure that passivation daemon is getting triggered or not.

 

Regards,

Basavaraj Kadakol

May 15, 2017 - 10:41pm
Response to BASAVARAJ

Appologies for late reply. do to some other project deadlines i did not got chance to look in to this issue. Below is the scenaios i am facing the issue

I am facing this issue when the sever is restarting or some time when more requestors are ideal and stil the requestor is not killed.

on The monitor Service package i am seeing the default values as 

Maximum idle requestors :10

Maximum active requestors :10

Maximum wait :10 

Do you mean can i increase the Max active requestors to 100 or below 100 ? Also i was curious to know why was i am getting the  "Timed out borrowing service requestor from requestor pool"  error when i am restarting the server. Does all the requestors will not be killed when we restart the server ?

PFA Log screen shot.

Thanks,

Srikanth J

 

 

 

May 17, 2017 - 2:46pm
Response to Srikanth J

Hi Srikanth,

The maximum 10 requestors indicated the max number of requests that it can handle at any time.  If you have an 11th request, it will wait until one of the first 10 requestors finishes or until it hits the 10 second maximum wait time.  When you see the "Timed out borrowing service requestor from requestor pool", that means the request trying to get processed has been waiting the maximum wait time and has timed out.  In regards to the 100 or below question, you need to determine how big your pool size should be based on the number of anticipated requests at any one time.   Once you determine that, you will want to set you max requestors appropriately (for example:  50 or 100) keeping in mind the memory increased that was pointed out above.  You will also need to factor in how long the requests take so that your timeout/maximum active requestors are set appropriately to accommodate the expected load.  You can review more information at the following links:  https://pdn.pega.com/sites/pdn.pega.com/files/help_v722/procomhelpmain.htm#data-/data-admin-/data-admin-servicepackage/pooling.htm and https://pdn.pega.com/sites/pdn.pega.com/files/help_v722/procomhelpmain.htm#concepts/concepts2/servicerequestorpooling.htm

In regards to the requestors not being killed when you restart the server, you may still have requests coming into your service even while you are restarting the server.  If that is the case, you may be experiencing the same scenario where you fill up the 10 max active requestors and then the 11th and subsequent requests start timing out as a requestor doesn't become available within the 10 second timeout.  Again, you need to understand the load of the requests coming in to set the pool appropriately.

Pega
May 17, 2017 - 4:43pm

And this is a little abstract / high-level: but it might provide you with some ideas on how large to make your Requestor Pools (you will need an idea of what your expected throughput is for this) : https://en.wikipedia.org/wiki/Queueing_theory

 

June 1, 2017 - 5:11am

Hi All,
if the service package has configuration: Maximum active requestors :10

Does it mean that one "node" can have 10 requestors active at one time or the all nodes can handle 10 requestors?

June 6, 2017 - 11:21am
Response to JakubB86

Each node can have 10 active requestors at a time.  If you have 3 nodes and you are configured to 10 max active requestors in your pool, spread across the 3 nodes you can handle 30 active requestors (10 for each node).