VM Crashing with 120k CPEs

elanderson · October 28, 2025, 1:51pm

Hi everyone,

I’m running GenieACS on a Debian VM and it’s been crashing frequently with kernel soft lockup errors, making the system unresponsive. This has happened multiple times, and it seems tied to high load from the CPEs. We currently have around 120,000 active CPEs connected, mostly Huawei and similar devices using TR-069.

Here’s a bit more detail on the issue:

The VM locks up, and I see messages like this in the console/logs (from syslogd):

Message from syslogd@genieacs at Oct 28 10:45:11 ...
 kernel:[ 4463.061294] watchdog: BUG: soft lockup - CPU#36 stuck for 33s! [conn2617:10368]

Message from syslogd@genieacs at Oct 28 10:45:11 ...
 kernel:[ 4463.061424] watchdog: BUG: soft lockup - CPU#38 stuck for 37s! [conn2046:9116]

Message from syslogd@genieacs at Oct 28 10:45:11 ...
 kernel:[ 4463.061775] watchdog: BUG: soft lockup - CPU#24 stuck for 33s! [node:1568]

Message from syslogd@genieacs at Oct 28 10:45:11 ...
 kernel:[ 4463.062186] watchdog: BUG: soft lockup - CPU#31 stuck for 29s! [node:3468]
(And many more similar messages for various CPUs and processes like node, migration, bdsecd, etc.)

The system is running Linux kernel 5.10.0-23-amd64 on Debian.
VM specs:
- CPU: 4x 12-Core Intel Xeon E5-4650 0 (-MCP SMP) 2700 MHz
- Kernel: 5.10.0-23-amd64 x86_64
- Mem: 22828.9/32135.4 MiB (71.0%)
- Storage: 100 GiB (47.7% used)
GenieACS version: v1.2.9+20220822165235

It’s hitting max memory and CPU usage during peaks and even killing services, like mongod. Once it happens, the ACS services try to use it and crash one by one.

I’ve tried increasing the watchdog threshold temporarily (echo 60 > /proc/sys/kernel/watchdog_thresh), but it’s just a workaround. Also monitored with htop, and Node.js processes are spiking CPU.

Questions:

What could be causing these soft lockups? Is it likely due to overload from the CPE count, or something like inefficient configs/scripts in GenieACS?
Is 120k CPEs too much for a single instance? From what I’ve read, some setups handle 10k-30k, but scaling to 100k+ might need multiple instances or optimizations like adjusting MAX_CONCURRENT_REQUESTS.
Any tips on tuning for better performance, like worker processes, MongoDB indexing, or hardware upgrades?

Thanks in advance for any insights or similar experiences!

akcoder · October 28, 2025, 4:17pm

120k CPEs while a large deployment, isn’t outside of what GenieACS can handle.

How many provision scripts run each time a CPE informs? What is your inform interval?

elanderson · October 28, 2025, 5:03pm

We don’t usually run more than one script at a time for each CPE and we don’t run any script on inform event.

This is what our inform script is like:

// Default "inform" Script \\

const now = Date.now();
const daily = Date.now(86400000);

const username = String(declare("DeviceID.ID", { value: 1 }).value[0]);
const password = String(Math.trunc(Math.random() * Number.MAX_SAFE_INTEGER).toString(36));

const informInterval = 43200;
const informIntervalLower = 7200;
const informIntervalDebug = 10;
const informTime = daily % 86400000;

const manufacturer = declare("DeviceID.Manufacturer", { value: 1 }).value[0];
const productClass = declare("DeviceID.ProductClass", { value: 1 }).value[0];

declare("InternetGatewayDevice.ManagementServer.Username", { value: daily }, { value: "xx" });
declare("InternetGatewayDevice.ManagementServer.Password", { value: daily }, { value: xxx" });

const productConfigs = {
    "EG8145X6": {
      ConnectionRequestUsername: username,
      ConnectionRequestPassword: password,
      PeriodicInformEnable: true,
      PeriodicInformInterval: informInterval
    },
    "EG8145X6-10": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: true,
        PeriodicInformInterval: informInterval
    },
    "EN8145B7Ns": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: true,
        PeriodicInformInterval: informIntervalLower
    },
    "Huawei": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: true,
        PeriodicInformInterval: informInterval
    },
    "AC10": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: 0
    },
    "TX2ProV1.0": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: true,
        PeriodicInformInterval: informInterval,
        PeriodicInformTime: informTime
    },
    "default": {
        ConnectionRequestUsername: username,
        ConnectionRequestPassword: password,
        PeriodicInformEnable: true,
        PeriodicInformInterval: informInterval,
        PeriodicInformTime: informTime
    }
}

const config = productConfigs[productClass] || productConfigs["default"];

for (const [param, value] of Object.entries(config)) {
    declare(`InternetGatewayDevice.ManagementServer.${param}`, { value: daily }, { value: value });
    declare(`Device.ManagementServer.${param}`, { value: daily }, { value: value });
}

Except by the 3 default provisions we have only 2 scripts. One for common routers and the other one to ONTs (such as Huawei EG8145X6, EG8145X6-10 and so) as shown below:

akcoder · October 28, 2025, 6:48pm

Why are you using daily as your refresh interval for InternetGatewayDevice.ManagementServer params? Do you expect the CPE to change its connection request un/pw/inform values?

Send me a screenshot of your presets page please, redact as appropriate.

What is the output of sudo lshw -C memory and grep -i swap /etc/fstab? Have you checked your host for memory faults? Have you tried moving the VM to a new host to see if the errors go away (or change)?

akcoder · October 28, 2025, 6:54pm

Everything I’m reading online says kernel:[ 4463.061294] watchdog: BUG: soft lockup - CPU#36 stuck for 33s! is a hardware/kernel issue. And reviewing the logs you provided, I would agree given that the error message shows for many different processes.

elanderson · October 28, 2025, 7:14pm

We had some CPEs that kept changing for no reason a while back, so that’s what we did to fix it back then. I’ll check with the team to change it today.

Is the last image not appropriate? There’s not sensitive data to redact if that’s what you said. If not, let me know, please.

lshw -C memory:

root@genieacs:~# lshw -C memory
  *-firmware                
       description: BIOS
       vendor: SeaBIOS
       physical id: 0
       version: rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org
       date: 04/01/2014
       size: 96KiB
  *-memory
       description: System Memory
       physical id: 1000
       size: 32GiB
       capabilities: ecc
       configuration: errordetection=multi-bit-ecc
     *-bank:0
          description: DIMM RAM
          vendor: QEMU
          physical id: 0
          slot: DIMM 0
          size: 16GiB
     *-bank:1
          description: DIMM RAM
          vendor: QEMU
          physical id: 1
          slot: DIMM 1
          size: 16GiB

grep -i swap /etc/fstab:

root@genieacs:~#  grep -i swap /etc/fstab
# swap was on /dev/sda5 during installation
#UUID=f022794a-4744-4659-aec3-b3ba73c2246d none            swap    sw              0       0
/swapfile none swap sw 0 0

We did check the memory and it was brand new. The server running this VM used to run our entire ERP. That’s why we think it’s at least weird that all the resources attributed to this one VM for GenieACS is not being enough.

elanderson · October 28, 2025, 7:28pm

Presets

Provisions

Below are the Provision Scripts:

ONTs

// Auditoria das ONTs\\

const now = Date.now();
const provisionado = declare("Tags.Provisionado", { value: 1 })?.value?.[0];
const bootstrap = args[0];
log(`provisionado: ${provisionado} || bootstrap: ${bootstrap}`);

if (provisionado && !bootstrap) {
  bindLANToWAN();
  log("CPE está (supostamente) configurada, retornando...");
  return;
}

const serial = declare('DeviceID.SerialNumber', { value: 1 }).value[0];
const productClass = declare('DeviceID.ProductClass', { value: 1 }).value[0];
const pppoe = ext("cpe-config", "GetPPPoE", serial);

declareBasicDevices();

log("[1]");
refreshWlan();
log("[2]");
checkWAN();
log("[3]");
setupAdditionalConfigs();
updateTags();

log('>> Reiniciando CPE para finalizar configuração...');
declare("Reboot", null, { value: now });

function updateTags() {
  declare("Tags.Provisionado", null, { value: true });
}

function refreshWlan() {
  log('>> Refreshing WLAN...');
  declareWLANConfiguration(1);
  declareWLANConfiguration(5);
}

function checkWAN() {
  declare("InternetGatewayDevice.WANDevice.*.WANConnectionDevice.*", { value: now });
  const wanNames = [1, 2, 3].map(getWANName);

  if (!wanNames[0] && !wanNames[1] && !wanNames[2]) {
    setupBaseWanPppConnection(true, 1);
  } else {
    for (let i = 0; i < wanNames.length; i++) {
      if (wanNames[i] && wanNames[i].includes("internet")) {
        setupBaseWanPppConnection(false, i + 1);
        break;
      }
    }
  }
}

function getWANName(connectionDeviceNumber) {
  const name = declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${connectionDeviceNumber}.WANPPPConnection.1.Name`, { value: now })?.value?.[0];
  return name ? name.toLowerCase() : null;
}

function setupBaseWanPppConnection(createWan, WANConnectionDevicePATH) {
  if (createWan) {
    if (productClass.includes('EN8255X6s-8X')) {
      declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.*`, null, { path: 1 });
    } else {
      declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.*`, null, { path: 1 });
    }
  } 
  declareWanPppConnection(WANConnectionDevicePATH, pppoe.username, pppoe.password);
  huaweiOntX6AdditionalConfigs(WANConnectionDevicePATH);
}

function declareWanPppConnection(WANConnectionDevicePATH, username, password) {
  log('>> Atualizando valores WANPPP Base...');
  
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.Enable`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.MaxMTUSize`, { value: now }, { value: "1492" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.MaxMRUSize`, { value: now }, { value: "1492" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.CurrentMRUSize`, { value: now }, { value: "1492" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.DNSEnabled`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.NATEnabled`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.DNSServers`, { value: now });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.Username`, { value: now }, { value: username });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.Password`, { value: now }, { value: password });
}

function huaweiOntX6AdditionalConfigs(WANConnectionDevicePATH) {
  declare(`InternetGatewayDevice.WANDevice.1.WANCommonInterfaceConfig.EnabledForInternet`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.WANDevice.1.WANCommonInterfaceConfig.WANAccessType`, { value: now }, { value: "Ethernet" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_VLAN`, { value: now }, { value: 10 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_NatType`, { value: now }, { value: 0 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6Enable`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Address.[]`, { value: now }, { path: 0 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Prefix.[]`, { value: now }, { path: 0 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Address.[]`, { value: now }, { path: 1 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Address.1.Origin`, { value: now }, { value: "AutoConfigured" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Prefix.[]`, { value: now }, { path: 1 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_IPv6.IPv6Prefix.1.Origin`, { value: now }, { value: "PrefixDelegation" });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_LANBIND.*`, { value: now });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_LANBIND.SSID1Enable`, { value: now }, { value: 1 });
  declare(`InternetGatewayDevice.WANDevice.1.WANConnectionDevice.${WANConnectionDevicePATH}.WANPPPConnection.1.X_HW_LANBIND.SSID5Enable`, { value: now }, { value: 1 });
}

function setupAdditionalConfigs() {
  log('>> Finalizando configurações adicionais...');

  declare("InternetGatewayDevice.X_HW_MainUPnP.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_SlvUPnP.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_Security.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.*.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.AccessControlListEnable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.AccessControlListNumberOfEntries", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*", null, { path: 1 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.Mode", { value: now }, { value: 0 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.Priority", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.HTTPLanEnable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.HTTPSLanEnable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.SamBaLanEnable", { value: now }, { value: true });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.WanAccess", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.WanAccess.*", null, { path: 1 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.WanAccess.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.WanAccess.*.*", { value: now });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.WanAccess.1.Enable", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.ServicePort", { value: now }, { value: "HTTP,ICMP" });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.SrcIp", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.SrcPortName", { value: now }, { value: "ALL" });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.SrcPortType", { value: now }, { value: 2 });
  declare("InternetGatewayDevice.X_HW_AppRemoteManage.CurrentMgtURL", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.X_HW_AppRemoteManage.CurrentPort", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.X_HW_AppRemoteManage.PhoneAppURL", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.LANDevice.*", { value: now });
  //declare("InternetGatewayDevice.LANDevice.1.LANEthernetInterfaceConfig.*", null, { path: 4 });
  declare("InternetGatewayDevice.LANDevice.1.LANEthernetInterfaceConfig.*.X_HW_L3Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.Time.*", { value: now });
  declare("InternetGatewayDevice.Time.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.Time.LocalTimeZone", { value: now }, { value: "-03:00" });
  declare("InternetGatewayDevice.Time.LocalTimeZoneName", { value: now }, { value: "Brasilia" });
  declare("InternetGatewayDevice.Time.NTPServer1", { value: now }, { value: "clock.fmt.he.net" });
  declare("InternetGatewayDevice.Time.NTPServer2", { value: now }, { value: "time.windows.com" });
  declare("InternetGatewayDevice.Time.X_HW_SynInterval", { value: now }, { value: "3600" });
  declare("InternetGatewayDevice.X_HW_Security.Firewall.Enable", { value: now }, { value: true });

  const ont = ext("cpe-config", "GetONTPassword", pppoe.username);
  declare("InternetGatewayDevice.UserInterface.X_HW_WebUserInfo.*", { value: now });
  declare("InternetGatewayDevice.UserInterface.X_HW_WebUserInfo.*.*", { value: now });
  declare("InternetGatewayDevice.UserInterface.X_HW_WebUserInfo.*.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.UserInterface.X_HW_WebUserInfo.2.Password", { value: 1 }, { value: ont.password });
}

function declareBasicDevices() {
  declare("InternetGatewayDevice", { value: now });
  declare("InternetGatewayDevice.*", { value: now });
  declare("InternetGatewayDevice.WANDevice.*", { value: now });
  declare("InternetGatewayDevice.WANDevice.*.WANConnectionDevice.*", { value: now });
}

function bindLANToWAN() {
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.1.X_HW_LANBIND.Lan1Enable", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.1.X_HW_LANBIND.Lan2Enable", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.1.X_HW_LANBIND.Lan3Enable", { value: now }, { value: 1 });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.1.X_HW_LANBIND.Lan4Enable", { value: now }, { value: 1 });
  //declare("InternetGatewayDevice.Layer3Forwarding.X_HW_policy_route.2.PhyPortName", { value: now }, { value: "LAN1,LAN2,LAN3,LAN4,SSID1,SSID5" });
  declare("InternetGatewayDevice.X_HW_Security.AclServices.AccessControl.List.*.ServicePort", { value: now }, { value: "HTTP,ICMP" });
}

function declareWLANConfiguration(configNumber) {
  declare(`InternetGatewayDevice.LANDevice.1.WLANConfiguration.${configNumber}.*`, { value: now });
  declare(`InternetGatewayDevice.LANDevice.1.WLANConfiguration.${configNumber}.Enable`, { value: now }, { value: true });
  declare(`InternetGatewayDevice.LANDevice.1.WLANConfiguration.${configNumber}.X_HW_AttachConf.X_HW_AirtimeFairness`, { value: now }, { value: true });
}

Routers

// Configs Checker Routers only \\

const now = Date.now();

let OK = declare("Tags.Provisionado", { value: 1 });
if (OK.value !== undefined) {
  log("CPE está (supostamente) configurada, retornando...");
  return;
}


log("[1]");
refreshWlan();
log("[2]");
setupBaseWanPppConnection();
log("[3]");
setupAdditionalConfigs();

//Refresh the mac and external ip
declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.*.MACAddress", { value: now });
declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.*.WANPPPConnection.*.IpAddress", { value: now });

updateTags();

log('>> Reiniciando CPE para finalizar configuração...');
declare("Reboot", null, { value: now });

function updateTags() {
  log('Auditoria realizada! Retornando Script...');
  declare("Tags.Provisionado", null, { value: true });
}

function refreshWlan() {
  log('>> Refreshing WLAN...');
  declare("InternetGatewayDevice.LANDevice.*.WLANConfiguration.*", { value: now });
  declare("InternetGatewayDevice.LANDevice.*.WLANConfiguration.*.*", { value: now });
  declare("InternetGatewayDevice.LANDevice.1.WLANConfiguration.*.SSID", { value: now });
}

function setupBaseWanPppConnection() {
  //Garantir que temos a instância WanPPPConnection
  log('>> Criando instância WANPPPConnection...');
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.*", null, { path: 1 });

  //Common WanPPPConnection
  log('>> Atualizando valores WANPPP Base...');  
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.MaxMTUSize", { value: now }, { value: "1492" })
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.MaxMRUSize", { value: now }, { value: "1492" });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.CurrentMRUSize", { value: now }, { value: "1492" });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.DNSEnabled", { value: now }, { value: true });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.NATEnabled", { value: now }, { value: true });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.DNSServers", { value: now });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.Username", { value: now });
  declare("InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANPPPConnection.1.Password", { value: now });
  declare("InternetGatewayDevice.WANDevice.1.WANCommonInterfaceConfig.EnabledForInternet", { value: now }, { value: true });
  
  //Huawei Ax2/Ax2S
  declare("InternetGatewayDevice.WANDevice.1.WANCommonInterfaceConfig.WANAccessType", { value: now }, { value: "Ethernet" });

  //Tenda Tx2ProV1.0
  declare("InternetGatewayDevice.WANDevice.1.WANCommonInterfaceConfig.EnabledUpnp", {value: now}, {value: true});
}

function setupAdditionalConfigs() {
  log('>> Finalizando configurações adicionais...');

  //Tenda Tx2ProV1.0  
  declare("InternetGatewayDevice.UserInterface.RemoteAccess.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.UserInterface.RemoteAccess.IP", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.UserInterface.RemoteAccess.Port", { value: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.UserInterface.RemoteAccess.Protocol", { value: now }, { value: "HTTP" })

  //Huawei Ax2/Ax2S
  declare("InternetGatewayDevice.Services.X_HUAWEI_WANRemoteAccess.Enable", { value: now }, { value: true });
  declare("InternetGatewayDevice.Services.X_HUAWEI_WANRemoteAccess.IPAddress1", { vale: now }, { value: "XXXX" });
  declare("InternetGatewayDevice.Services.X_HUAWEI_WANRemoteAccess.Port", { value: now }, { value: "XXXX" });
}

akcoder · October 28, 2025, 7:41pm

Your provisions Firewall_ONT, default and inform run every time a CPE communicates with the ACS. What is the contents of those scripts?

Did your ERP VM have the same kernel version? You might try updating your kernel too. Debian 11 is currently at v5.10.163.

elanderson · October 28, 2025, 7:53pm

default and inform are both default presets. We kept it the same way they came upon installation back then.

Firewall_ONT is a provision i just created our NOC request to guarantee that the Firewall will be enabled for ONTs. Btw, it only run on ONTs, like some of the other presets above. It is rather simple, check it out:

const now = Date.now()

const firewallStatus = declare("InternetGatewayDevice.X_HW_Security.Firewall.Enable", {value: now}).value[0]

if (firewallStatus) {
  return;
}

declare("InternetGatewayDevice.X_HW_Security.Firewall.Enable", { value: now }, { value: true });

root@genieacs:~# uname -r
5.10.0-23-amd64

akcoder · October 28, 2025, 8:00pm

Upgrade your kernel :). Current version for Debian 11 is v5.10.163.

GenieACS is declarative, so there is no need to put the guard statement in your provisions.


const now = Date.now()

declare("InternetGatewayDevice.X_HW_Security.Firewall.Enable", { value: now }, { value: true });

This is all you need for your Firewall_ONT script. GenieACS will refresh the parameter because the age (the value param in the second arg) of the cached data is older than now. Then if the desired value (third arg) does not match, a SPV will be done by GenieACS.

elanderson · October 28, 2025, 8:05pm

Im doing it in like… 20 minutes or so, then, i’ll talk back about it.

Right, i’ll change it too.

Is there any other sugestion? Could any scripts above be causing that mess? What you think?

akcoder · October 28, 2025, 8:08pm

From what I’ve seen, no. Your scripts look fine. I think its a kernel bug.

elanderson · October 29, 2025, 12:28pm

After updating the kernel and disabling some services, we thought it got solved. But it didn’t. Mongod by himself is actually using all the resources, but we can’t think of a possible reason for this…

elanderson · October 31, 2025, 8:45pm

Just updtating in case someone gets into the same problem. We fixed it.

We configured the VM with many virtual sockets and cores (48 vCPUs). This forces Proxmox (our old hypervisor) to synchronize dozens of virtual threads with the physical hardware, generating internal clock drift and kernel lockups. Each vCPU tries to “compete” for real CPU time, and the guest operating system (Debian) interprets this as a CPU freeze.

Correction: Reduce the virtual CPU topology.

Stable configuration for GenieACS:
Sockets: 1 or 2
Cores per socket: 4 to 8
NUMA: disabled

This simplification eliminates synchronization overhead, makes the KVM scheduler more efficient, and stabilizes the VM clock.

Result: no soft lockups, no watchdog freezes, and much more predictable CPU usage.

Topic		Replies	Views
Genieacs-cwmp: CPE not picking up Tasks	2	3425	April 29, 2019
GenieACS 1.2 - CPE keeps in a reboot loop sended by ACS	20	4037	September 25, 2020
CONNECTION REQUEST PROBLEM - EasyCWMP and GenieACS	11	10438	September 5, 2020
Max CPE supported by ACS	13	3189	January 9, 2024
Connection dropped on multiple cpe connections at the same time	5	1931	September 11, 2021

VM Crashing with 120k CPEs

Presets

Provisions

ONTs

Routers

Related topics