workaround for hanging rig-pump communication (with WW pump) #445

PieterGit · 2017-04-19T20:29:46Z

This will reboot the rig if hanging rig-pump communication is detected.
This pull request also improves logging, has more robust killing of reset.py if it hangs.
The root cause of this issue is discussed here oskarpearson/mmeowlink#60

and improve logging, more robust killing of reset.py if it hangs

scottleibrand · 2017-04-19T20:34:33Z

bin/oref0-radio-reboot.sh

+# Note that this is a workaround, until we found the root cause of why the rig pump communication fails
+# Note that there is one known false positive for the second issue. It will reboot if the port/device and/or radio_type have been misconfigured
+
+radio_errors=`tail --lines=20 /var/log/openaps/pump-loop.log | egrep "spidev5.1 already in use|mmeowlink.exceptions.CommsException: Could not get subg_rfspy state or version|retry 0|TimeoutExpired. Killing process"`


I've seen Could not get subg_rfspy state or version lots of times and it never requires a reboot for me: a simple mmtune (with its radio reset) will do.

You're right. This might be the trigger for the hanging reset.py for WW pump users (in tonights log), but it will probably not hang on the rig. So I won't trigger a reboot on mmeowlink.exceptions.CommsException: Could not get subg_rfspy state or version

scottleibrand · 2017-04-19T20:34:51Z

bin/oref0_init_pump_comms.py

-        logging.debug("spi_serial not installed. Assuming not using spidev")
-    except Exception as ex: 
-        logging.error("Exception in oref0-init-pump-comms spi_serial: %s" % ex)
+        logging.exception("spi_serial not installed. Assuming not using spidev")


Why should this be an exception and not a debug warning?

oops, refactored too much. rolled back

scottleibrand

Thanks. Looks like it's ready for testing.

PieterGit · 2017-04-20T08:55:11Z

This code runs on my rig now. Installs fine and I did some tests that the scripts work. Feel free to test and improve. I will report later if there are no downsides, so that it can be merged to dev.

libxmike · 2017-04-20T22:20:32Z

@PieterGit here the pump log you requested:

Settings refreshError, retrying
Starting pump-loop at Thu Apr 20 23:52:21 CEST 2017:
2017-04-20 23:52:22,722 DEBUG Import spi_serial
2017-04-20 23:52:22,725 DEBUG spi_serial not installed. Assuming not using spidev
2017-04-20 23:52:22,726 DEBUG Import oref0_subg_ww_radio_parameters
2017-04-20 23:52:22,765 DEBUG Parsing ./pump.ini
2017-04-20 23:52:22,769 DEBUG Serial device (port) for pump is: /dev/ttyACM0
2017-04-20 23:52:22,770 DEBUG env RFSPY_RTSCTS=0
2017-04-20 23:52:22,770 DEBUG excuting ['oref0-subg-ww-radio-parameters', '/dev/ttyACM0']
+ echo

+ echo The CC111x is located at /dev/ttyACM0
The CC111x is located at /dev/ttyACM0
+ cd /home/pi/src/subg_rfspy/tools
+ case "$2" in
+ ./change_setting.py /dev/ttyACM0 0x06 0x00
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0C 0x59
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0D 0x66
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0E 0x33
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0F 0x62
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x10 0x1A
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x11 0x13
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x09 0x24
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0A 0x2E
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0B 0x38
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ exit 0
2017-04-20 23:52:28,856 DEBUG script exited with 0
2017-04-20 23:52:28,856 DEBUG sleeping for 0.5 seconds 
mmtune: pump://JSON/mmtune/monitor/mmtune.json
reporting monitor/mmtune.json
"868.630", 5, -63  Listening: ........No pump comms detected from other rigs

is that the log part you mean?
let me know if you want me to test from scratch (oref0-setup) or something else

libxmike · 2017-04-20T22:37:04Z

here with error:

Starting pump-loop at Fri Apr 21 00:30:05 CEST 2017:
Traceback (most recent call last):
  File "/usr/local/bin/mmeowlink-any-pump-comms.py", line 15, in <module>
    app.run(None)
  File "/usr/local/lib/python2.7/dist-packages/decocare/helpers/cli.py", line 113, in run
    self.prelude(args)
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/cli/any_pump_comms_app.py", line 28, in prelude
    super(AnyPumpCommsApp, self).prelude(args)
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/cli/base_mmeowlink_app.py", line 26, in prelude
    self.link = link = LinkBuilder().build(args.radio_type, port)
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/link_builder.py", line 16, in build
    return SubgRfspyLink(port)
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/vendors/subg_rfspy_link.py", line 55, in __init__
    self.open()
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/vendors/serial_interface.py", line 32, in open
    self.check_setup()
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/vendors/subg_rfspy_link.py", line 72, in check_setup
    self.serial_rf_spy.sync()
  File "/usr/local/lib/python2.7/dist-packages/mmeowlink/vendors/serial_rf_spy.py", line 121, in sync
    raise CommsException("Could not get subg_rfspy state or version. Have you got the right port/device and radio_type?")
mmeowlink.exceptions.CommsException: Could not get subg_rfspy state or version. Have you got the right port/device and radio_type?
2017-04-21 00:30:08,010 DEBUG Import spi_serial
2017-04-21 00:30:08,013 DEBUG spi_serial not installed. Assuming not using spidev
2017-04-21 00:30:08,013 DEBUG Import oref0_subg_ww_radio_parameters
2017-04-21 00:30:08,054 DEBUG Parsing ./pump.ini
2017-04-21 00:30:08,058 DEBUG Serial device (port) for pump is: /dev/ttyACM0
2017-04-21 00:30:08,059 DEBUG env RFSPY_RTSCTS=0
2017-04-21 00:30:08,060 DEBUG excuting ['oref0-subg-ww-radio-parameters', '/dev/ttyACM0']
+ echo

+ echo The CC111x is located at /dev/ttyACM0
The CC111x is located at /dev/ttyACM0
+ cd /home/pi/src/subg_rfspy/tools
+ case "$2" in
+ ./change_setting.py /dev/ttyACM0 0x06 0x00
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0C 0x59
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0D 0x66
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0E 0x33
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0F 0x62
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x10 0x1A
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x11 0x13
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x09 0x24
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0A 0x2E
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ ./change_setting.py /dev/ttyACM0 0x0B 0x38
RileyLink OK
Version: subg_rfspy 0.9
SUCCESS
+ sleep 0.5
+ exit 0
2017-04-21 00:30:14,209 DEBUG script exited with 0
2017-04-21 00:30:14,209 DEBUG sleeping for 0.5 seconds 
mmtune: pump://JSON/mmtune/monitor/mmtune.json
reporting monitor/mmtune.json
"868.558", 5, -77  Listening: ...........

scottleibrand · 2017-05-01T04:59:09Z

Fixed the merge conflict. Is this ready to merge to dev and be tested by more folks?

PieterGit · 2017-05-01T19:42:48Z

yes, it's ready to go to dev. i have been running for more than a week on this branch

workaround for hanging rig-pump communication (with WW pump)

0f35206

and improve logging, more robust killing of reset.py if it hangs

scottleibrand reviewed Apr 19, 2017

View reviewed changes

fix scotts review remarks

8fe625f

scottleibrand reviewed Apr 19, 2017

View reviewed changes

fix python closing "

a39a0f5

PieterGit mentioned this pull request Apr 19, 2017

Problem with hanging reset.py and stacktrace with mmeowlink-any-pump-comms.py oskarpearson/mmeowlink#60

Open

PieterGit added 2 commits April 20, 2017 00:52

improve log statement for US pump users

9c5645b

also create cgm-loop for G4-upload

a866d34

PieterGit mentioned this pull request Apr 20, 2017

Improve reliability of ww pump / oref0-subg-ww-radio-parameters.sh (especially with TI USB) #357

Merged

Merge branch 'dev' into wwresetpyworkaround

36675b4

danamlewis merged commit 0fc8151 into openaps:dev May 1, 2017

PieterGit deleted the wwresetpyworkaround branch May 1, 2017 22:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workaround for hanging rig-pump communication (with WW pump) #445

workaround for hanging rig-pump communication (with WW pump) #445

PieterGit commented Apr 19, 2017 •

edited

Loading

scottleibrand Apr 19, 2017

PieterGit Apr 19, 2017

scottleibrand Apr 19, 2017

PieterGit Apr 19, 2017

scottleibrand left a comment

PieterGit commented Apr 20, 2017

libxmike commented Apr 20, 2017 •

edited

Loading

libxmike commented Apr 20, 2017

scottleibrand commented May 1, 2017

PieterGit commented May 1, 2017

workaround for hanging rig-pump communication (with WW pump) #445

workaround for hanging rig-pump communication (with WW pump) #445

Conversation

PieterGit commented Apr 19, 2017 • edited Loading

scottleibrand Apr 19, 2017

Choose a reason for hiding this comment

PieterGit Apr 19, 2017

Choose a reason for hiding this comment

scottleibrand Apr 19, 2017

Choose a reason for hiding this comment

PieterGit Apr 19, 2017

Choose a reason for hiding this comment

scottleibrand left a comment

Choose a reason for hiding this comment

PieterGit commented Apr 20, 2017

libxmike commented Apr 20, 2017 • edited Loading

libxmike commented Apr 20, 2017

scottleibrand commented May 1, 2017

PieterGit commented May 1, 2017

PieterGit commented Apr 19, 2017 •

edited

Loading

libxmike commented Apr 20, 2017 •

edited

Loading