Crash: Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",) #4652

timdawborn · 2018-04-10T07:50:25Z

Possibly a bug in kombu or py-amqp, not celery, but I couldn't see this logged in either of those.

Checklist

I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).

software -> celery:3.1.25 (Cipater) kombu:3.0.37 py:2.7.12
            billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

CELERY_ACKS_LATE: True
CELERY_TASK_SERIALIZER: u'json'
CELERY_ENABLE_UTC: True
CELERY_ACCEPT_CONTENT:
    (u'json',)
BROKER_HEARTBEAT_CHECKRATE: 2
BROKER_USE_SSL: {
    u'ca_certs': u'<redacted>',
    u'cert_reqs': 2,
    u'certfile': u'<redacted>',
    u'keyfile': '********'}
CELERY_IMPORTS:
    (<redacted>)
BROKER_HEARTBEAT: 30
BROKER_TRANSPORT_OPTIONS: {
    u'confirm_publish': True}
CELERY_CREATE_MISSING_QUEUES: True
BROKER_URL: 'amqp://<redacted>:5671//<redacted>'
CELERY_IGNORE_RESULT: True
CELERY_ROUTES: {<redacted>}
CELERY_TIMEZONE: u'UTC'
CELERY_SEND_TASK_ERROR_EMAILS: True
ADMINS: [<redacted>]
CELERY_DISABLE_RATE_LIMITS: True

I have verified that the issue exists against the master branch of Celery.

Hard to do as this would require a celery 3 -> celery 4 upgrade, which is non-trivial and I don't have time to do right now.

Steps to reproduce

See celery log below. Possibly related to the socket disconnect just before, but otherwise, I don't know.

Expected behavior

To not crash with an AttributeError.

Actual behavior

[2018-04-10 07:13:32,531: INFO/MainProcess] Connected to amqp://<redacted>:5671//<redacted>
[2018-04-10 07:13:32,580: INFO/MainProcess] mingle: searching for neighbors
[2018-04-10 07:13:33,653: INFO/MainProcess] mingle: all alone
[2018-04-10 07:13:33,757: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 280, in start
    blueprint.start(self)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 884, in start
    c.loop(*c.loop_args())
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
    next(loop)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/kombu/async/hub.py", line 279, in create_loop
    item()
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/amqp/utils.py", line 42, in __call__
    self.set_error_state(exc)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/amqp/utils.py", line 39, in __call__
    **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/kombu/transport/base.py", line 144, in _read
    raise RecoverableConnectionError('Socket was disconnected')
RecoverableConnectionError: Socket was disconnected
[2018-04-10 07:13:33,844: INFO/MainProcess] Connected to amqp://<redacted>:5671//<redacted>
[2018-04-10 07:13:33,888: INFO/MainProcess] mingle: searching for neighbors
[2018-04-10 07:13:34,958: INFO/MainProcess] mingle: all alone
[2018-04-10 07:13:35,070: ERROR/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",)
Traceback (most recent call last):
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 280, in start
    blueprint.start(self)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 884, in start
    c.loop(*c.loop_args())
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/celery/worker/loops.py", line 76, in asynloop
    next(loop)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/kombu/async/hub.py", line 279, in create_loop
    item()
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/amqp/utils.py", line 42, in __call__
    self.set_error_state(exc)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/amqp/utils.py", line 39, in __call__
    **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/kombu/connection.py", line 288, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "<redacted>/virtualenv/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events
    return connection.drain_events(**kwargs)
AttributeError: 'NoneType' object has no attribute 'drain_events'

The text was updated successfully, but these errors were encountered:

auvipy · 2018-04-10T08:24:23Z

this version is not supported anymore. try 4.2rc2

WisdomPill · 2018-07-28T08:16:44Z

I have the same error using python 3.6 and celery 4.2.0
All of this happens when I'm using different workers on different queues and using autoscale from 2 to 10 in the worker that gets me the error. If you need more info just ask

[2018-07-28 10:11:38,418: INFO/MainProcess] Connected to redis://localhost:6379//
[2018-07-28 10:11:38,430: INFO/MainProcess] mingle: searching for neighbors
[2018-07-28 10:11:39,440: INFO/MainProcess] mingle: all alone
[2018-07-28 10:11:39,462: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'fileno'",)
Traceback (most recent call last):
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 322, in start
    blueprint.start(self)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 598, in start
    c.loop(*c.loop_args())
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/loops.py", line 50, in asynloop
    obj.controller.register_with_event_loop(hub)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/worker.py", line 219, in register_with_event_loop
    description='hub.register',
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/bootsteps.py", line 151, in send_all
    fun(parent, *args)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/worker/components.py", line 179, in register_with_event_loop
    w.pool.register_with_event_loop(hub)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/concurrency/prefork.py", line 134, in register_with_event_loop
    return reg(loop)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 482, in register_with_event_loop
    [self._track_child_process(w, hub) for w in self._pool]
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 482, in <listcomp>
    [self._track_child_process(w, hub) for w in self._pool]
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/celery/concurrency/asynpool.py", line 465, in _track_child_process
    hub.add_reader(fd, self._event_process_exit, hub, proc)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 207, in add_reader
    return self.add(fds, callback, READ | ERR, args)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 156, in add
    fd = fileno(fd)
  File "/Users/anas/.envs/timing_sensor/lib/python3.6/site-packages/kombu/utils/compat.py", line 97, in fileno
    return f.fileno()
AttributeError: 'NoneType' object has no attribute 'fileno'

jamespeacock · 2018-10-05T13:56:06Z

I also see this error after the worker recovers from a ConnectionError with redis
redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.

On recovery, this is what I see
[2018-10-05 09:51:00,888: WARNING/MainProcess] Restoring 4 unacknowledged message(s)
[2018-10-05 09:51:00,908: INFO/MainProcess] Connected to redis://redis:6379//
[2018-10-05 09:51:00,917: INFO/MainProcess] mingle: searching for neighbors
[2018-10-05 09:51:01,926: INFO/MainProcess] mingle: sync with 4 nodes
[2018-10-05 09:51:01,926: INFO/MainProcess] mingle: sync complete
[2018-10-05 09:51:01,957: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'NoneType' object has no attribute 'fileno'",)
...Followed by identical stacktrace as @WisdomPill

auvipy · 2018-10-12T16:30:41Z

what version and code you are u using?

auvipy · 2019-05-06T08:09:06Z

will reopen after feedback on celery 4.3 or master

ckcollab · 2019-05-11T20:00:42Z

Getting this on Python 3.7.3 docker using Celery 4.3.0:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/worker/worker.py", line 238, in run_wrapper
    run(task_id, task_args)
  File "/worker/worker.py", line 810, in run
    'metadata': debug_metadata
  File "/worker/worker.py", line 211, in _send_update
    queue="submission-updates",
  File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 798, in __exit__
    self.release()
  File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 372, in release
    self._close()
  File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 338, in _close
    self._do_close_self()
  File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 328, in _do_close_self
    self.maybe_close_channel(self._default_channel)
  File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 320, in maybe_close_channel
    channel.close()
  File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 226, in close
    wait=spec.Channel.CloseOk,
  File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 60, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
  File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 80, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/amqp/connection.py", line 500, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python3.7/site-packages/amqp/connection.py", line 506, in blocking_read
    return self.on_inbound_frame(frame)
  File "/usr/local/lib/python3.7/site-packages/amqp/method_framing.py", line 55, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.7/site-packages/amqp/connection.py", line 510, in on_inbound_method
    method_sig, payload, content,
  File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 126, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 280, in _on_close
    self._do_revive()
  File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 164, in _do_revive
    self.open()
  File "/usr/local/lib/python3.7/site-packages/amqp/channel.py", line 437, in open
    spec.Channel.Open, 's', ('',), wait=spec.Channel.OpenOk,
  File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 60, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
  File "/usr/local/lib/python3.7/site-packages/amqp/abstract_channel.py", line 80, in wait
    self.connection.drain_events(timeout=timeout)
AttributeError: 'NoneType' object has no attribute 'drain_events'

auvipy · 2019-05-11T21:21:01Z

actually a pyamq issue celery/py-amqp#218

auvipy added the Version: 3.1 label Apr 10, 2018

auvipy closed this as completed Apr 10, 2018

auvipy reopened this Jul 30, 2018

auvipy added the Status: Needs Verification ✘ label Oct 12, 2018

auvipy closed this as completed May 6, 2019

auvipy added this to the 4.5 milestone May 11, 2019

auvipy reopened this May 11, 2019

auvipy added pyamqp and removed Status: Needs Verification ✘ Version: 3.1 labels May 11, 2019

auvipy removed this from the 4.5 milestone May 11, 2019

auvipy closed this as completed May 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash: Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",) #4652

Crash: Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",) #4652

timdawborn commented Apr 10, 2018

auvipy commented Apr 10, 2018 •

edited

Loading

WisdomPill commented Jul 28, 2018

jamespeacock commented Oct 5, 2018

auvipy commented Oct 12, 2018

auvipy commented May 6, 2019

ckcollab commented May 11, 2019

auvipy commented May 11, 2019

Crash: Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",) #4652

Crash: Unrecoverable error: AttributeError("'NoneType' object has no attribute 'drain_events'",) #4652

Comments

timdawborn commented Apr 10, 2018

Checklist

Steps to reproduce

Expected behavior

Actual behavior

auvipy commented Apr 10, 2018 • edited Loading

WisdomPill commented Jul 28, 2018

jamespeacock commented Oct 5, 2018

auvipy commented Oct 12, 2018

auvipy commented May 6, 2019

ckcollab commented May 11, 2019

auvipy commented May 11, 2019

auvipy commented Apr 10, 2018 •

edited

Loading