GenieACS 1.2 - CPE keeps in a reboot loop sended by ACS

Hello.

I faced a error with a ONT from Huawei: I do a reboot (only once time…), and then the ACS keeps in “stale”. Then the ONT connects, but keeps in a loop of “reboot”. I dont have any provision that sends the reboot. The only way to make exit from that loop, is erase the device from the list, or reboot the ACS.
Seems that the ACS if not recieve any message after reboot, keeps in that loop.

Queued: 0 Pending: 0 Fault: 0 Stale: 1 CommitClear 00259E-HG8247Q-485754439323389A Reboot.

This are the logs (sorry if this is not the best way to get help, pasting all this stuff like this):

  1. Logs from cwmp:

2020-05-12T02:08:54.966Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Inform; cpeRequestId=“71” informEvent=“2 PERIODIC” informRetryCount=0
2020-05-12T02:08:55.006Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“17206a45e930000” acsRequestName=“Reboot”
2020-05-12T02:08:55.104Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Updating Virtual parameters…
2020-05-12T02:08:55.114Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Fetch gpon mac-address
2020-05-12T02:08:55.115Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Running unified Maccadress script…
2020-05-12T02:08:55.125Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“17206a45e930100” acsRequestName=“GetParameterValues”

2020-05-12T02:10:02.277Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Inform; cpeRequestId=“37” informEvent=“1 BOOT,M Reboot,4 VALUE CHANGE” informRetryCount=0
2020-05-12T02:10:02.311Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“17206a565780000” acsRequestName=“Reboot”
2020-05-12T02:10:02.423Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Updating Virtual parameters…
2020-05-12T02:10:02.424Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Setting setting management routes 192.168.0.0/16 and 10.0.0.0/8
2020-05-12T02:10:02.425Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Polling information NOW from BOOT trigger…
2020-05-12T02:10:02.425Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: DONE polling information from BOOT trigger…
2020-05-12T02:10:02.438Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Fetch gpon mac-address
2020-05-12T02:10:02.439Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Running unified Maccadress script…
2020-05-12T02:10:02.452Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“17206a565780200” acsRequestName=“GetParameterValues”

2)Logs from debug:

event: incoming HTTP request
timestamp: 2020-05-12T02:16:49.407Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
soapaction: “”
connection: Keep-Alive
content-type: text/xml; charset=UTF-8
user-agent: HW_WAP_CWMP_V02
content-length: “2550”
body: >
<SOAP-ENV:Envelope xmlns:SOAP-ENV=“http://schemas.xmlsoap.org/soap/envelope/
xmlns:SOAP-ENC=“http://schemas.xmlsoap.org/soap/encoding/
xmlns:xsd=“http://www.w3.org/2001/XMLSchema
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance
xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>

SOAP-ENV:Header

<cwmp:ID SOAP-ENV:mustUnderstand=“1”>37</cwmp:ID>

</SOAP-ENV:Header>

SOAP-ENV:Body

cwmp:Inform

Huawei Technologies Co., Ltd

00259E

HG8247Q

485754439323389A

1 BOOT

M Reboot

4 VALUE CHANGE

1

2020-05-11T23:16:48+00:00

0

InternetGatewayDevice.ManagementServer.ParameterKey

InternetGatewayDevice.ManagementServer.ConnectionRequestURL

http://190.—.---.—:7547/f90b5a8ee811c7d786150f9759d3bf7e

InternetGatewayDevice.DeviceSummary

InternetGatewayDevice:1.4[](Baseline:1, EthernetLAN:1, WiFiLAN:2, Time:1, IPPing:1, DeviceAssociation:1), VoiceService:1.0[1](Endpoint:1, SIPEndpoint:1)

InternetGatewayDevice.DeviceInfo.SpecVersion

1.0

InternetGatewayDevice.DeviceInfo.HardwareVersion

929.A

InternetGatewayDevice.DeviceInfo.SoftwareVersion

V3R017C10S118

InternetGatewayDevice.DeviceInfo.ProvisioningCode

InternetGatewayDevice.WANDevice.1.WANConnectionDevice.4.WANIPConnection.1.ExternalIPAddress

0.0.0.0

</cwmp:Inform>

</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

event: outgoing HTTP response
timestamp: 2020-05-12T02:16:49.409Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
statusCode: 200
headers:
content-length: 522
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
set-cookie: session=61055f955a5d5bcb
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>37</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:InformResponse1</cwmp:InformResponse></soap-env:Body></soap-env:Envelope>

event: incoming HTTP request
timestamp: 2020-05-12T02:16:49.433Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
connection: Keep-Alive
cookie: session=61055f955a5d5bcb
user-agent: HW_WAP_CWMP_V02
content-length: “0”
body: “”

event: outgoing HTTP response
timestamp: 2020-05-12T02:16:49.442Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
statusCode: 200
headers:
content-length: 514
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>17206ab9bef0000</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:Reboot</cwmp:Reboot></soap-env:Body></soap-env:Envelope>

event: incoming HTTP request
timestamp: 2020-05-12T02:16:49.512Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
soapaction: “”
connection: Keep-Alive
content-type: text/xml; charset=UTF-8
cookie: session=61055f955a5d5bcb
user-agent: HW_WAP_CWMP_V02
content-length: “453”
body: >
<SOAP-ENV:Envelope xmlns:SOAP-ENV=“http://schemas.xmlsoap.org/soap/envelope/
xmlns:SOAP-ENC=“http://schemas.xmlsoap.org/soap/encoding/
xmlns:xsd=“http://www.w3.org/2001/XMLSchema
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance
xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>

SOAP-ENV:Header

<cwmp:ID SOAP-ENV:mustUnderstand=“1”>17206ab9bef0000</cwmp:ID>

</SOAP-ENV:Header>

SOAP-ENV:Body

cwmp:RebootResponse/

</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

event: outgoing HTTP response
timestamp: 2020-05-12T02:16:49.534Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:16:49.407Z
statusCode: 200
headers:
content-length: 652
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>17206ab9bef0200</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:GetParameterValuesInternetGatewayDevice.ManagementServer.UpgradesManaged</cwmp:GetParameterValues></soap-env:Body></soap-env:Envelope>

event: incoming HTTP request
timestamp: 2020-05-12T02:31:56.131Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
soapaction: “”
connection: Keep-Alive
content-type: text/xml; charset=UTF-8
user-agent: HW_WAP_CWMP_V02
content-length: “2550”
body: >
<SOAP-ENV:Envelope xmlns:SOAP-ENV=“http://schemas.xmlsoap.org/soap/envelope/
xmlns:SOAP-ENC=“http://schemas.xmlsoap.org/soap/encoding/
xmlns:xsd=“http://www.w3.org/2001/XMLSchema
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance
xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>

SOAP-ENV:Header

<cwmp:ID SOAP-ENV:mustUnderstand=“1”>37</cwmp:ID>

</SOAP-ENV:Header>

SOAP-ENV:Body

cwmp:Inform

Huawei Technologies Co., Ltd

00259E

HG8247Q

485754439323389A

1 BOOT

M Reboot

4 VALUE CHANGE

1

2020-05-11T23:31:55+00:00

0

InternetGatewayDevice.ManagementServer.ParameterKey

InternetGatewayDevice.ManagementServer.ConnectionRequestURL

http://190.—.---.—:7547/f90b5a8ee811c7d786150f9759d3bf7e

InternetGatewayDevice.DeviceSummary

InternetGatewayDevice:1.4[](Baseline:1, EthernetLAN:1, WiFiLAN:2, Time:1, IPPing:1, DeviceAssociation:1), VoiceService:1.0[1](Endpoint:1, SIPEndpoint:1)

InternetGatewayDevice.DeviceInfo.SpecVersion

1.0

InternetGatewayDevice.DeviceInfo.HardwareVersion

929.A

InternetGatewayDevice.DeviceInfo.SoftwareVersion

V3R017C10S118

InternetGatewayDevice.DeviceInfo.ProvisioningCode

InternetGatewayDevice.WANDevice.1.WANConnectionDevice.4.WANIPConnection.1.ExternalIPAddress

0.0.0.0

</cwmp:Inform>

</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

event: outgoing HTTP response
timestamp: 2020-05-12T02:31:56.133Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
statusCode: 200
headers:
content-length: 522
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
set-cookie: session=ecc03c97ac545dc2
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>37</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:InformResponse1</cwmp:InformResponse></soap-env:Body></soap-env:Envelope>

event: incoming HTTP request
timestamp: 2020-05-12T02:31:56.153Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
connection: Keep-Alive
cookie: session=ecc03c97ac545dc2
user-agent: HW_WAP_CWMP_V02
content-length: “0”
body: “”

event: outgoing HTTP response
timestamp: 2020-05-12T02:31:56.160Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
statusCode: 200
headers:
content-length: 514
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>17206b971d30000</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:Reboot</cwmp:Reboot></soap-env:Body></soap-env:Envelope>

event: incoming HTTP request
timestamp: 2020-05-12T02:31:56.235Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
localPort: 7547
method: POST
url: /
headers:
host: 186.—.---.—:37547
soapaction: “”
connection: Keep-Alive
content-type: text/xml; charset=UTF-8
cookie: session=ecc03c97ac545dc2
user-agent: HW_WAP_CWMP_V02
content-length: “453”
body: >
<SOAP-ENV:Envelope xmlns:SOAP-ENV=“http://schemas.xmlsoap.org/soap/envelope/
xmlns:SOAP-ENC=“http://schemas.xmlsoap.org/soap/encoding/
xmlns:xsd=“http://www.w3.org/2001/XMLSchema
xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance
xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>

SOAP-ENV:Header

<cwmp:ID SOAP-ENV:mustUnderstand=“1”>17206b971d30000</cwmp:ID>

</SOAP-ENV:Header>

SOAP-ENV:Body

cwmp:RebootResponse/

</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

event: outgoing HTTP response
timestamp: 2020-05-12T02:31:56.254Z
remoteAddress: 190.—.---.—
deviceId: 00259E-HG8247Q-485754439323389A
connection: 2020-05-12T02:31:56.131Z
statusCode: 200
headers:
content-length: 652
server: GenieACS/1.2.0-beta.0+20200412005912
soapserver: GenieACS/1.2.0-beta.0+20200412005912
content-type: text/xml; charset=“utf-8”
body: >-

<?xml version="1.0" encoding="UTF-8"?>

<soap-env:Envelope xmlns:soap-enc=“http://schemas.xmlsoap.org/soap/encoding/” xmlns:soap-env=“http://schemas.xmlsoap.org/soap/envelope/” xmlns:xsd=“http://www.w3.org/2001/XMLSchema” xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance” xmlns:cwmp=“urn:dslforum-org:cwmp-1-0”>soap-env:Header<cwmp:ID soap-env:mustUnderstand=“1”>17206b971d30200</cwmp:ID></soap-env:Header>soap-env:Bodycwmp:GetParameterValuesInternetGatewayDevice.ManagementServer.UpgradesManaged</cwmp:GetParameterValues></soap-env:Body></soap-env:Envelope>
^C
root@GenieACS2:~#

Something in your scripts/presets is asking the device to reboot (second line):

2020-05-12T02:08:55.006Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“17206a45e930000” acsRequestName=“Reboot”

no, i only do a “reboot” via button, but i get “staled” and the ONT reboots, but the ACS seems to never know and reboots when it comes again online. The only way is to delete the device, then rediscovers and keeps online.

i do check a reboot via OLT and then online, and works ok:

2020-05-12T04:11:58.382Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Inform; cpeRequestId=“37” informEvent=“1 BOOT” informRetryCount=0
2020-05-12T04:11:58.414Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Updating Virtual parameters…
2020-05-12T04:11:58.415Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Setting setting management routes 192.168.0.0/16 and 10.0.0.0/8
2020-05-12T04:11:58.416Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Polling information NOW from BOOT trigger…
2020-05-12T04:11:58.418Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: DONE polling information from BOOT trigger…
2020-05-12T04:11:58.421Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Fetch gpon mac-address
2020-05-12T04:11:58.422Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Running unified Maccadress script…
2020-05-12T04:11:58.425Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“172071508200100” acsRequestName=“GetParameterValues”
2020-05-12T04:11:58.729Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: DONE unified Maccadress script…
2020-05-12T04:11:58.730Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Reading Wifi 2.4Ghz current password.
2020-05-12T04:11:58.730Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Reading Wifi 5Ghz current password.
2020-05-12T04:11:58.735Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“172071508200101” acsRequestName=“GetParameterValues”
2020-05-12T04:11:59.072Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: ACS request; acsRequestId=“172071508200102” acsRequestName=“GetParameterNames”
2020-05-12T04:11:59.111Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Configuring HOTSPOT feature for; model=“HG8247Q”
2020-05-12T04:11:59.111Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Model not supported for HOTSPOT…; model=“HG8247Q”
2020-05-12T04:11:59.111Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Done configuring HOTSPOT features…
2020-05-12T04:11:59.112Z [INFO] 190.—.---.— 00259E-HG8247Q-485754439323389A: Script: Configure MSO user access for Vendor Huawei Technologies Co., Ltd model HG8247Qfor inital BOOTSRAP configuration

What does the task queue of the CPE look like when its stuck in the reboot loop? Is there a stuck “Reboot” task? If so, then whats more than likely happening is the CPE isn’t acknowledging the reboot task to the ACS so the ACS doesn’t remove it from the task queue.

yes, i think that is what happen (i dont had any task to reboot).
only happens when i send the reboot and ont wont ack that.
Where can be the timers that are related to retrying the reboot all time? I think also that it should not do a reboot task after BOOT event is happen.

May be some of this parameters can be relationated, but as no documentation i see that here is so much “try and test”.

cwmp.connectionRequestTimeout
cwmp.deviceOnlineThreshold
cwmp.maxCommitIterations
cwmp.sessionTimeout

I’m having the same problem if I issue a reboot command but the device is Offline. In this case, the reboot task will remain in the queue and will start a reboot loop once the device comes online.
I did a bit of more research and I think the problem appears when you also have a provisioning script that runs at 1 BOOT event and that (i guess) prevents genieacs from clearing the reboot task as it also waits for the provision task to complete which never happens because of session timeout. For now, I found a workaround by deleting the reboot tasks.

it is very strange since it only happen with huawei ont, does had problem with other ont, like sagemcom.

Sounds like a defect in the firmware of the CPE…

Yes, i will do some captures comparing sagem vs huawei.
Do you suggest any point, to focus? May be i can find something to get a new firmware from them.

Hello akcoder, seems that is not a issue on huawei. Since i see that on captures, that huawei responds ok with “Rebootresponse”.

Seems to had some bug on genieacs as, lavira says, some process keeps pending and enter in a boot loop, and do again the reboot after reboot when cpe get online.

Paste the full response from the CPE. It could be that the CPE is returning the wrong cwmp id in the soap data. I’m not sure how the very deep internals of GenieACS work to know if thats necessary.

There are some differences, but dont know really wich is the wrong: (at left, is the goodone reboot cpe, at right the hawei response)

Thanks a lot.