None of this turned up in my testing before upgrading to v1.2.9 from v1.1.3.
Since upgrading our production server to v1.2.9 yesterday, I’m seeing a bunch of faults with the message “The TR-069 session was unsuccessfully terminated.” The faults always happen upon boot. I’ve not been able to find a rhyme or reason for the faults. Sometimes the CPE will boot up with no issue, other times 4-5 faults will show up when rebooting.
Any help you can provide would be most appreciated!
-dan
TLDR: EXT TIMEOUT and/or Conn limit variable? Not sure but will test it.
how you doing.
just to put some context first, I like to say that In our case we have near 4k of CPE on our ACS. We don’t use periodic informs because I use a series of scripts responsible to call each CPE based on certain tags – which roughly translated means: our ACS don’t get bombarded with CPE event’s except after a power blackouts and even that it seems able to handle them pretty well. One of those scripts are also responsible to collect faults information from API, mail it to us, and then clear the faults, so the next time an event occurs over those CPE the affected provision won’t be skipped.
What I want to point out is that I always got some emails with the fault’s description you mention (since version 1.2.5), but only a few of them per week (an average of 5 or 10 per week? dunno, will check tomorrow). Looking at the logs I got the impression it was always related with physical issues: on cwmp.log we get a VALUE CHANGE event, which occurs every time there’s a PPPoE reconnection because the CPE detected a loss of signal.
Since IMHO the number faults we get is kinda low I decided to ignore it for the moment, but I will look into it after the upgrade (we are still using 1.2.8). Probably the EXT TIMEOUT variable might have to do something with it, I have it set to 15 seconds, will try to lower it to 2 seconds and see what happen. Also we have the number of concurrent connections set to 1000.-
As usual I might not be saying something very useful but perhaps it can help. I will add something later after the upgrade and post it here if I think it’s relevant.
How/where did you do that? Are you using some sort of reverse proxy in front of the CWMP process?
Most everything we do is tag based as well. All the issues I’m seeing are on boot.
FWIW, I don’t think you will have issues going from 1.2.8 to 1.2.9. We were upgrading from v1.1.3 to v1.2.9.
Apologies for the confusion. This happened during a period of frequents blackouts. I was desperate to solve the problem of one thousands CPE going to the ACS at the same time. I have placed a parameter inside genieacs.env:
MAX_CONCURRENT_REQUESTS=1000
but I think it is useless since it is only mentioned in config.ts and it loads a different file
// Configuration file
const configFilename = configDir
? `${configDir}/config.json`
: `${ROOT_DIR}/config/config.json`;
Probably found it some mention of it on the net.
agpologies again for not paying attention to that part. Have you considered creating a new server and simply restore the dump of the mongodb database? That’s what we did while upgrading 1.2.5 to 1.2.8 (the real reason behind it: installation wasn’t clean and it was not possible to upgrade using npm).
I just looked at the code for cwmp.ts and config.ts, MAX_CONCURRENT_REQUESTS is being used, line 1440 and 1631. If the number of concurrent sessions is > than max, then a 503 is sent to the CPE telling it to retry after 60 seconds.
This is actually exactly what we did. Built a new v1.2.9 server, ran these commands on the existing server, then imported the data into the mongo db on the new server.
# Dump just the modems that have informed since the beginning of 2022
mongodump --db=genieacs --collection=devices --query='{"_lastInform":{"$gte": {"$date": "2022-01-01T00:00:00.000Z"} } }' --out=dump
# Now export everything else
mongodump --db=genieacs --excludeCollection=devices,tasks,faults,presets,provisions,virtualParameters --out=dump
Then ran my own script to populate the vparams/presets/provisions.
Hi @akcoder and @rudymartin
I am new to GenieACS, According to my research, the default value of MAX_CONCURRENT_REQUESTS is only 20. How does it relate to the size of deployment or CPU capacity of GenieACS machine?
i.e: what is the best formula for MAX_CONCURRENT_REQUESTS value based on CPU threads and size of deployment (the number of connected CPE)?
Thank you so much~
Do not resurrect a year old thread to ask a question. Necromancy is frowned upon.
Hi @akcoder
I’ve created a new topic
https://forum.genieacs.com/t/best-estimation-for-max-concurrent-requests/4484
Thank you~