# Overloaded server - various issues

**URL:** <https://forum.genieacs.com/t/overloaded-server-various-issues/5135>\
**Category:** Uncategorized\
**Created:** [February 26, 2024, 8:54am UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135 "2024-02-26T08:54:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmeiring](https://avatars.discourse-cdn.com/v4/letter/j/c2a13f/32.png) [@jmeiring](https://forum.genieacs.com/u/jmeiring)\
**Post date:** [February 26, 2024, 8:54am UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135/1 "2024-02-26T08:54:15Z")

</div>

Hi,

I have an genieacs servcer which is very overloaded.  
Approximately 16000 CPEs

I am getting various errors, which looks like race contidions (I have not examined the code)  
If I increase the number of CPU’s the server has, the issue becomes worse.

Getting various errors cwmp errors like:  
exceptionMessage=“Cache snapshot does not exist” exceptionStack=  
"Error: Cache snapshot does not exist\n at Bt.get (/opt/genieacs-source/genieacs/lib/local-cache.ts:29:26)

exceptionName=“MongoServerError” exceptionMessage=“Updating the path ‘InternetGatewayDevice.LANDevice.1.WLANConfiguration.1.AssociatedDevice’ would create a conflict at ‘InternetGatewayDevice.LANDevice.1.WLANConfiguration.1.AssociatedDevice’” exceptionStack="MongoServerError: Updating the path ‘InternetGatewayDevice.LANDevice.1.WLANConfiguration.1.AssociatedDevice’ would create a conflict at ‘InternetGatewayDevice.LANDevice.1.WLANConfiguration.1.AssociatedDevice’\n at (/opt/genieacs-source/genieacs/node\_modules/mongodb/src/operations/update.ts:146:44)

exceptionName=“MongoExpiredSessionError” exceptionMessage=“Cannot use a session that has ended” exceptionStack="/usr/lib/node\_modules/genieacs/node\_modules/mongodb/src/sessions.ts:978

exceptionName=“MongoNotConnectedError” exceptionMessage=“Client must be connected before running operations” exceptionStack="/usr/lib/node\_modules/genieacs/node\_modules/mongodb/src/operations/execute\_operation.ts:89

exceptionName=“Error” exceptionMessage=“Cache snapshot does not exist” exceptionStack=  
"Error: Cache snapshot does not exist\n at Bt.get (/opt/genieacs-source/genieacs/lib/local-cache.ts:29:26)

Please help!

Kind Regards,

Johan

---

<div class="post-metadata">

**Author:** ![akcoder](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.genieacs.com/akcoder/32/11_2.png) [@akcoder](https://forum.genieacs.com/u/akcoder)\
**Post date:** [February 26, 2024, 5:39pm UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135/2 "2024-02-26T17:39:28Z")

</div>

You probably need to provide more details on your setup. What is your inform interval. Does the MongoDB and GenieACS run on the same server? What are you server specs?

---

<div class="post-metadata">

**Author:** ![jmeiring](https://avatars.discourse-cdn.com/v4/letter/j/c2a13f/32.png) [@jmeiring](https://forum.genieacs.com/u/jmeiring)\
**Post date:** [February 27, 2024, 1:35pm UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135/3 "2024-02-27T13:35:25Z")

</div>

Morning Akcoder,

Setup is (was - see below) a single server running Mongo/CWMP/UI/FS/NBI.  
12G RAM / 2 x 6 core CPU.  
VM running on Proxmox.

I have in the meantime split it over 4 different servers.  
Server 1 (12G RAM / 2 x 6 core CPU) CWMP/UI/FS/NBI  
Server 2 (8G RAM / 2 x 4 core CPU) CWMP  
Server 3 (8G RAM / 2 x 4 core CPU) CWMP  
Server 4 (8G RAM / 2 x 4 core CPU) Mongo

Nginx on Server 1 is load balancing to CWMP on server 1/2/3  
Nginx load balancing is ip based, so the same CPE reaches the same CWMP instance every time

I’ve also incresed the timeouts on all 3 CWMP servers  
GENIEACS\_DEVICE\_ONLINE\_THRESHOLD=22000  
GENIEACS\_CONNECTION\_REQUEST\_TIMEOUT=20000

Still see a lot of this in “systemctl status genieacs-cwmp”:

exceptionName=“Error” exceptionMessage=“Lock expired” exceptionStack="Error: Lock expired\n at Wt (/opt/genieacs-source/genieacs/lib/lock.ts:44:37)

exceptionName=“Error” exceptionMessage=“Cache snapshot does not exist” exceptionStack="Error: Cache snapshot does not exist\n at Bt.get (/opt/genieacs-source/genieacs/lib/local-cache.ts:29:26)

Also a lot of this in /var/log/genieacs/genieacs-cwmp.log:

2024-02-27T13:30:27.871Z [INFO] 41.85.21.63 5895D8-ONT-CXNKD8A85BF8: ACS request; acsRequestId=“18deac1dbef0107” acsRequestName=“GetParameterNames”  
2024-02-27T13:30:27.873Z [ERROR] 41.85.21.63 5895D8-ONT-CXNKD8A85BF8: Connection dropped

Happens on all three servers

Kind Regards,

Johan Meiring

---

<div class="post-metadata">

**Author:** ![jmeiring](https://avatars.discourse-cdn.com/v4/letter/j/c2a13f/32.png) [@jmeiring](https://forum.genieacs.com/u/jmeiring)\
**Post date:** [February 27, 2024, 1:36pm UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135/4 "2024-02-27T13:36:36Z")

</div>

O yes. Inform interval was originally set to 10 minutes.  
Now updated to 60 minutes.

ONTs are slowly backing off as they successfully connect.

---

<div class="post-metadata">

**Author:** ![akcoder](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.genieacs.com/akcoder/32/11_2.png) [@akcoder](https://forum.genieacs.com/u/akcoder)\
**Post date:** [February 28, 2024, 5:33pm UTC](https://forum.genieacs.com/t/overloaded-server-various-issues/5135/5 "2024-02-28T17:33:45Z")

</div>

You should probably set your inform interval to 6 hours. Unless you have a need for an hourly inform interval, you are just burning up your infrastructure time.