Relaxed mode for parsing invalid MIBs

[Jellyfrog]

Librenms got a decent collection of MIBs from various vendors, some more broken than others.

It might be of interest to add a "relaxed mode" for common problems, such as the ones with invalid dates: unexpected "201506080000A" (expected <extutctime> ...)

$ git clone https://github.com/librenms/librenms.git /tmp/librenms
$ find /tmp/librenms/mibs/ -type f | xargs -P0 -n1 ./parse 1>/dev/null
2022/06/03 16:22:32 Parse file: /tmp/librenms/mibs/SNMPv2-TC-v1:121:1: unexpected token "SNMPv2-CONF-v1"
2022/06/03 16:22:33 Parse file: /tmp/librenms/mibs/allied/AT-DOS-MIB:72:11: unexpected "/" (expected "(" ...)
2022/06/03 16:22:35 Parse file: /tmp/librenms/mibs/eltek/VALERE-DC-POWER-MIB:2460:30: unexpected "bdtAlarmPresent" (expected "}")
2022/06/03 16:22:37 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-PBR-MIB:13:22: unexpected "201506080000A" (expected <extutctime> ...)
2022/06/03 16:22:39 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-ROUTEMAP-MIB:12:22: unexpected "201506080000A" (expected <extutctime> ...)
2022/06/03 16:22:39 Parse file: /tmp/librenms/mibs/imco/IMCO-BIG-MIB:12202:2: unexpected "aSupply1UnitAny" (expected "}")
2022/06/03 16:22:40 Parse file: /tmp/librenms/mibs/cisco/CISCOSB-SYSMNG-MIB:16:22: unexpected "201010310000a" (expected <extutctime> ...)
2022/06/03 16:22:43 Parse file: /tmp/librenms/mibs/endrun/TEMPUSLXUNISON-MIB:285:7: unexpected "1" (expected "}")
2022/06/03 16:22:43 Parse file: /tmp/librenms/mibs/aerohive/AH_TRAP_MIB:37:21: unexpected "8021" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/riedo/NETTRACK-E3METER-CTR-SNMP-MIB:466:3: unexpected "1" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/terra/TERRA-PRODUCTS-MIB:18:9: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/ekinops/EKINOPS-MGNT2-NMS-MIB:357:4: unexpected "undefinedError" (expected "}")
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/terra/TERRA-DEFINITIONS-MIB:20:9: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:22:44 Parse file: /tmp/librenms/mibs/transition/TN-OTDR-MIB:690:13: unexpected "1" (expected <ident>)
2022/06/03 16:22:45 Parse file: /tmp/librenms/mibs/transition/TN-LLDP-EXT-MIB:23:21: unexpected "2013030251930Z" (expected <extutctime> ...)
2022/06/03 16:22:47 Parse file: /tmp/librenms/mibs/juniper/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:22:49 Parse file: /tmp/librenms/mibs/riello/RIELLOMDU-MIB:216:109: unexpected "Imminent" (expected <assign> ...)
2022/06/03 16:22:49 Parse file: /tmp/librenms/mibs/poweralert/TRIPPLITE-12X:45:5: unexpected "Corrected spelling errors" (expected "DESCRIPTION" ...)
2022/06/03 16:22:51 Parse file: /tmp/librenms/mibs/eds/EDS-MIB:1352:23: unexpected "," (expected (<octetstring> | <objectidentifier> | <ident>) ...)
2022/06/03 16:22:57 Parse file: /tmp/librenms/mibs/hp/HP-SN-MIBS:74:1: unexpected token "HP-SN-AGENT-MIB"
2022/06/03 16:23:07 Parse file: /tmp/librenms/mibs/linksys/LINKSYS-SYSMNG-MIB:16:22: unexpected "201010310000a" (expected <extutctime> ...)
2022/06/03 16:23:08 Parse file: /tmp/librenms/mibs/linksys/LINKSYS-MODEL-MIB:3554:1: no match found for
2022/06/03 16:23:10 Parse file: /tmp/librenms/mibs/fs/GBNPlatformOAM-MIB:954:16: unexpected "10" (expected "}")
2022/06/03 16:23:16 Parse file: /tmp/librenms/mibs/dragonwave/HORIZON-ODU-MIB:204:7: unexpected "hc40-57-qpsk" (expected "}")
2022/06/03 16:23:18 Parse file: /tmp/librenms/mibs/delta/DeltaUPS-MIB:205:12: unexpected "3" (expected "}")
2022/06/03 16:23:18 Parse file: /tmp/librenms/mibs/fs/FS-MIB:18630:78: unexpected "cpemac" (expected "}")
2022/06/03 16:23:21 Parse file: /tmp/librenms/mibs/zyxel/ZYXEL-NWA-SERIES_v1-4-2:104:4: unexpected "success" (expected "}")
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/tplink/TPLINK-SYSINFO-MIB:115:16: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/proware/proware-SNMP-MIB:94:17: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/zyxel/ZYXEL-GS4012F-MIB:7529:22: unexpected "mandatory" (expected ("current" | "deprecated" | "obsolete") ...)
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/edgecos/ECS4100-52T-MIB:10879:4: unexpected "radiusServerType" (expected "}")
2022/06/03 16:23:23 Parse file: /tmp/librenms/mibs/ibm/IBM-NetFinity-Text-Alert-MIB:132:1: no match found for
2022/06/03 16:23:25 Parse file: /tmp/librenms/mibs/ibm/IBMIROCAUTH-MIB:426:19: no match found for .
2022/06/03 16:23:28 Parse file: /tmp/librenms/mibs/zyxel/IES5206-MIB:10374:17: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:33 Parse file: /tmp/librenms/mibs/arris/d5/ARRIS-D5-SLOT-EXT-MIB:50:34: unexpected "(" (expected "END")
2022/06/03 16:23:37 Parse file: /tmp/librenms/mibs/perle/PERLE-IOLAN-SDS-MIB:657:6: unexpected "Description" (expected <assign> ...)
2022/06/03 16:23:38 Parse file: /tmp/librenms/mibs/junos/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:23:41 Parse file: /tmp/librenms/mibs/dlink/DLINKSW-NETWORK-ACCESS-MIB:487:44: unexpected "xffffffff" (expected ")")
2022/06/03 16:23:42 Parse file: /tmp/librenms/mibs/teldat/TELDAT-MON-INTERF-WLAN-MIB:42:10: unexpected "mandatory" (expected ("current" | "deprecated" | "obsolete") ...)
2022/06/03 16:23:43 Parse file: /tmp/librenms/mibs/dell/DELLEMC-OS10-CHASSIS-MIB:137:13: unexpected "os10ChassisProductBase" (expected "}")
2022/06/03 16:23:44 Parse file: /tmp/librenms/mibs/bdcom/NMS-CARD-SYS-MIB:70:23: unexpected "`" (expected (<octetstring> | <objectidentifier> | <ident>) ...)
2022/06/03 16:23:44 Parse file: /tmp/librenms/mibs/bdcom/NMS-CHASSIS:822:25: unexpected "TLB-modification-exception" (expected "}")
2022/06/03 16:23:45 Parse file: /tmp/librenms/mibs/cambium/800/CAMBIUM-PTP800-MIB:34:70: no match found for .
2022/06/03 16:23:45 Parse file: /tmp/librenms/mibs/calix/E7-TC:348:9: unexpected "ontAutoQuarantine" (expected "}")
2022/06/03 16:23:46 Parse file: /tmp/librenms/mibs/calix/E7-Calix-MIB:28:5: unexpected "E7OperStatus" (expected "FROM" ...)
2022/06/03 16:23:47 Parse file: /tmp/librenms/mibs/fortinet/FORTINET-FORTIGATE-MIB:9462:33: unexpected "1" (expected "}")
2022/06/03 16:23:48 Parse file: /tmp/librenms/mibs/brocade/FOUNDRY-SN-ROUTER-TRAP-MIB:439:1: no match found for
2022/06/03 16:23:48 Parse file: /tmp/librenms/mibs/calix/E5-120-MIB:7822:18: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:49 Parse file: /tmp/librenms/mibs/calix/E5-121-MIB:7932:18: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:50 Parse file: /tmp/librenms/mibs/screenos/NETSCREEN-TRAP-MIB:189:9: unexpected "low-memory" (expected "}")
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-Accounting-CONF:250:13: unexpected "DESCRIPTION" (expected <assign> ...)
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-ATM-1483-Profile-CONF:101:13: unexpected "juniAtm1483ProfileGroup4" (expected "STATUS" ...)
2022/06/03 16:23:51 Parse file: /tmp/librenms/mibs/junose/broken/Juniper-Internet-CONF:1593:18: unexpected "INCLUDES" (expected "{" ...)
2022/06/03 16:23:52 Parse file: /tmp/librenms/mibs/junose/JUNIPER-WX-MIB:241:1: unexpected token "JUNIPER-WX-GLOBAL-TC"
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SYSTEM-ACCESS-MIB:528:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SYSLOG-MIB:339:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-RSVP-MIB:308:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-CPU-MAC-FILTER:195:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-SERVICE-POLICY-GROUP-MIB:354:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-CFM-MIB:2551:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-OSPF-MIB:332:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-QOS-MIB:166:7: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-SOFTWARE-MANAGEMENT-GROUP-MIB:580:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-MIB:667:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-INTERFACE-MIB:1092:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ARP-MIB:248:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PON-MAC-GROUP-MIB:3155:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-ONTMANAGER-GROUP-MIB:178:5: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PM-MIB:487:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-MIB:1286:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ERPS-MIB:795:21: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-MPLS-PW-MIB:655:6: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-AUTO-RESET-MIB:505:14: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ENTITY-ALARM-MIB:287:22: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-REDUNDANCY-MIB:697:22: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PON-PROFILE-GROUP-MIB:3812:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-SLD-MIB:314:9: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:53 Parse file: /tmp/librenms/mibs/ubiquoss/UBIQUOSS-EPON-PM-GROUP-MIB:3038:16: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-DOT1BRIDGE-MIB:200:24: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-NTP-MIB:196:21: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/ubiquoss/UBQS-ACCESS-LIST-MIB:259:4: unexpected "(" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:54 Parse file: /tmp/librenms/mibs/extreme/Brocade-TC:89:1: unexpected token "BROCADE-PRODUCTS-MIB"
2022/06/03 16:23:55 Parse file: /tmp/librenms/mibs/dasan/SLE-ISIS-MIB:29:4: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/dasan/DASAN-SWITCH-MIB:3511:5: unexpected "{" (expected ("ACCESS" | "MAX-ACCESS") ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/dasan/DASAN-NOTIFICATION:1216:13: unexpected "}" (expected <ident> ...)
2022/06/03 16:23:56 Parse file: /tmp/librenms/mibs/extreme/EXTREME-BASE-MIB:356:1: unexpected token "EXTREME-SYSTEM-MIB"
2022/06/03 16:23:57 Parse file: /tmp/librenms/mibs/dasan/SLE-VRRP-MIB:29:4: unexpected "REVISION" (expected <assign> ...)
2022/06/03 16:23:59 Parse file: /tmp/librenms/mibs/dasan/SLE-EPON-MIB:31:17: unexpected "" (expected <extutctime> ...)
2022/06/03 16:24:00 Parse file: /tmp/librenms/mibs/huawei/HUAWEI-SLOG-EUDM-MIB:113:19: unexpected "0" (expected "}")
2022/06/03 16:24:00 Parse file: /tmp/librenms/mibs/huawei/HUAWEI-MGMD-STD-MIB:747:5: unexpected "?" (expected "}")
2022/06/03 16:24:05 Parse file: /tmp/librenms/mibs/himoinsa/DISMUNTELv00-MIB:1294:1: unexpected token "cj"
2022/06/03 16:24:08 Parse file: /tmp/librenms/mibs/4rf/4RF-COMMON-TC:22:5: unexpected "Integer32" (expected "FROM" ...)
2022/06/03 16:24:08 Parse file: /tmp/librenms/mibs/4rf/4RF-APRISAXE-TC:111:45: unexpected "'E0'H" (expected <int>)

[sleepinggenius2]

This isn't really practical with the current parser. I also don't know how many people might be using the library for linting, so I would want to be very careful about introducing any such workarounds without also including flags to be able to control them.

I have cloned the LibreNMS MIBs myself and have used them for some testing. CISCOSB-SYSMNG-MIB, for example, looks like it was correct until a bulk update ~10 months ago that messed up the revision dates. Unfortunately, it doesn't look like there have been checks added to their CI process to do linting or any other validation on the included MIB files, so they do get committed with errors, sometimes even regressions it seems.

I'm always happy to address bugs in the library for spec-compliant MIB files, but I would suggest in this scenario to open an issue against the LibreNMS repo to work on getting things corrected there.

[Jellyfrog]

This isn't really practical with the current parser. I also don't know how many people might be using the library for linting, so I would want to be very careful about introducing any such workarounds without also including flags to be able to control them.

I'll rename the issue to reflect a bit better the goal of the issue.

I have cloned the LibreNMS MIBs myself and have used them for some testing. CISCOSB-SYSMNG-MIB, for example, looks like it was correct until a bulk update ~10 months ago that messed up the revision dates. Unfortunately, it doesn't look like there have been checks added to their CI process to do linting or any other validation on the included MIB files, so they do get committed with errors, sometimes even regressions it seems.

All (guessing 99%) comes directly, unmodified, from the vendor, this is more a reflection of the [sad] state of mibs out in the wild.
If we take the mentioned MIB, latest download I can find (https://software.cisco.com/download/home/284077831/type/283415684/release/1.4.10.06) contains;

$ grep LAST-UPDATED CISCOSBsysmng.mib
        LAST-UPDATED "201010310000a"

I'm always happy to address bugs in the library for spec-compliant MIB files, but I would suggest in this scenario to open an issue against the LibreNMS repo to work on getting things corrected there.

I'm on of the maintainers there, and I don't think we (LibreNMS) can/should modify vendor mibs

[sleepinggenius2]

I'll rename the issue to reflect a bit better the goal of the issue.

The new title I agree with. I've started to work on a major rewrite of the library's core that should start to help with some common errors in MIBs (like textual convention inheritance), but unfortunately I can't address parsing-related issues until I can replace the existing parser. As stated previously, I want to make sure that there is still a way to use the library as a linter, so I don't want to introduce any changes without still providing an option for spec compliance. The immediate goal it to reach parity with the original libsmi C library in what it is able to load (despite some errors), but my ultimate goal is to be able to load any MIB that Net-SNMP can. Unfortunately, my time is limited, so I can't commit to a timeline on this. A lot of work has already been completed, but not much on the parser yet.

All (guessing 99%) comes directly, unmodified, from the vendor, this is more a reflection of the [sad] state of mibs out in the wild. If we take the mentioned MIB, latest download I can find (https://software.cisco.com/download/home/284077831/type/283415684/release/1.4.10.06) contains;

$ grep LAST-UPDATED CISCOSBsysmng.mib
        LAST-UPDATED "201010310000a"

Yeah, the ones out in the wild are often pretty bad. Some vendors are certainly worse than others. I'm mostly happy when I can even get MIBs for a product, but too often even the latest available seem to be outdated when actually trying to walk a device.

I'm one of the maintainers there, and I don't think we (LibreNMS) can/should modify vendor mibs

I don't necessarily agree with that statement. If there's a change that can be made to correct a syntax issue and it allows for the MIB file to be more fully parsed or otherwise create more useful information, then I think it would be in everyone's best interest to do so. In my career, I've definitely done that for some vendor MIBs, even going so far as correcting types, enum values, adding missing objects, and so on, so that I can actually consume the MIB and do something useful with it. You definitely have to stay vigilant though, because I have had vendors fix some of those kind of bugs in newer firmware releases. That's why it's always important to try to use not just the correct MIBs, but the correct revisions for a particular device and firmware version. MIBs are supposed to be backwards compatible, but that's another thing that vendors all too often mess up as well.

Depending on your relationship with the vendor, you can try opening a ticket with them to "properly" correct the issue, but I have not had great success with most vendors going that route (even ones with active support contracts). Especially for older/unsupported products or companies that no longer exist, there's really little recourse to get the problem fixed at the source.