500 Internal Server Error when loading tasks

Hello,

I have a device that has some issues in GenieACS.

Even if I delete and re-add it, when I try to edit/reload some property, it loads:
https://xxxxx:3000/api/devices/XXXXXXXXXXXX/tasks

But it fails with HTTP error 500, and body content:
Internal Server Error

How can I debug this?

I’ve tried to set cwmp.debug to
id = ‘XXXX’
And I see there is some communication, without no apparent reason to have a 500 error, but it’s there.
I should debug GenieACS itself, and not the communication with the device.

I had a small progress…

Using
# journalctl -u genieacs-ui.service -f

I can see:

  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: (node:34477) UnhandledPromiseRejectionWarning: RangeError [ERR_SOCKET_BAD_PORT]: Port should be > 0 and < 65536. Received NaN.
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at validatePort (internal/validators.js:182:11)
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at Socket.send (dgram.js:610:12)
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at /usr/lib/node_modules/genieacs/bin/genieacs-ui:2:72647
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at new Promise ()
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at /usr/lib/node_modules/genieacs/bin/genieacs-ui:2:72624
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: at processTicksAndRejections (internal/process/task_queues.js:97:5)
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: (node:34477) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see Command-line options | Node.js v15.14.0 Documentation). (rejection id: 2)
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: (node:34477) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: (node:34477) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 2)
  • Apr 09 19:28:54 tr069server genieacs-ui[34477]: 2021-04-09T17:28:54.725Z [ERROR] Uncaught exception; pid=34477 exceptionName=“RangeError” exceptionMessage=“Port should be > 0 and < 65536. Received NaN.” exceptionStack=“RangeError [ERR_SOCKET_BAD_PORT]: Port should be > 0 and < 65536. Received NaN.\n at validatePort (internal/validators.js:182:11)\n at Socket.send (dgram.js:610:12)\n at /usr/lib/node_modules/genieacs/bin/genieacs-ui:2:72647\n at new Promise ()\n at /usr/lib/node_modules/genieacs/bin/genieacs-ui:2:72624\n at processTicksAndRejections (internal/process/task_queues.js:97:5)”
  • Apr 09 19:28:54 tr069server genieacs-ui[728]: 2021-04-09T17:28:54.867Z [ERROR] Worker died; pid=34477 exitCode=0
  • Apr 09 19:28:55 tr069server genieacs-ui[728]: 2021-04-09T17:28:55.302Z [INFO] Worker listening; pid=34518 address=“0.0.0.0” port=3000

What port is it talking about?

@zaidka do you have any idea?

Hi, only this device gives errors? If so, my guess would be the connection request URL contains a invalid port. Look at the value for InternetGatewayDevice.ManagementServer.ConnectionRequestURL property.

Hi @lavira, yes, only on this device.
InternetGatewayDevice.ManagementServer.ConnectionRequestURL
is:
http://xxx.xxx.xxx.xxx:7547/tr069

So it seems to be correct

look also at the ConnectionRequestUsername and Password … if either one contains ‘:’ or ‘@’, it could be a sign of a bug in the way genieacs handles these.

Thanks @lavira,
It seems this is not the case:

InternetGatewayDevice.ManagementServer.ConnectionRequestUsername:
xxxx-xxxxx-xxxxxx

InternetGatewayDevice.ManagementServer.ConnectionRequestPassword:
xxxxx

Oh, I misunderstood the initial post … the device ID is all just alphanumeric + 2 dash? or are there url encoded chars in it ? (has at least one % in the device ID)

It’s
E4C32A-IGD-XXXXXX
No encoded chars in it

then sorry, I don’t have any other clue about what could cause this error.

1 Like

@lavira :frowning: Thank you for your help!
Maybe @zaidka can help me

Can’t think of an possible cause other than the connection request URL being invalid. Double check the URL for any spaces or special characters.

@zaidka thank you for your answer. Is there a way to debug exactly what GenieACS is doing? So I can check the URL is using?

The debug log shows everything working properly.

Exactly @zaidka, but unfortunately GenieACS shows up this error and the GUI crashes.
Could you please send me an email? I would like to help to solve this issue, but can’t understand how to debug it

What is the value of the parameter UDPConnectionRequestAddress? The error is getting raised from the code that handles UDP connection requests so I suspect that param might have something to do with it.

@zaidka is:
InternetGatewayDevice.ManagementServer.UDPConnectionRequestAddress http://xxx.xxx.xxx.xxx:3478

@zaidka did you had a chance to look at this? Thanks :pray:t2:

According to the spec this is invalid. UDPConnectionRequestAddress is supposed to be either host:port or host (i.e. without the “http://” prefix). Having the http:// in front makes no sense because HTTP is TCP not UDP. This is probably the cause of the issue. GenieACS would not expect an http:// prefix for this parameter since it is invalid syntax and is nonsensical.

2 Likes

Thank you very much for your answer @mjducharme.
Now I have two ways to solve it:

  1. Make GenieACS to filter this value programmatically, maybe removing the http:// part, or deleting this parameter entirely. Is it possible? How?

  2. Patch GenieACS to don’t crash when it receives an unexpected value and make this patch available to anyone, but I don’t have an idea on GitHub of what part of the code I have to check :sweat_smile:

I should solve this ASAP, any idea?

Something you can try as a workaround, I don’t know if it will work, try creating a provision with this code:

const now = Date.now();
clear(“InternetGatewayDevice.ManagementServer.UDPConnectionRequestAddress”, now);

To run for any event. I think that should delete the UDPConnectionRequestAddress which would force it to use TCP instead. However the thing I’m not sure about is it might just pull the UDPConnectionRequestAddress again immediately after this deletes it, so that’s why I’m not 100% sure if it will work.