It seems that a recent version of the apify package broke proxies for Scrapy

[honzajavorek]

Just upgrading the apify package on my Scrapy + Apify project has caused the following error. I don't have time at this particular moment to investigate the issue, but I came here to at least report the issue and provide the stack trace. I can provide more details later if needed:

2025-01-22T08:26:23.473Z [scrapy.core.scraper] ERROR Error downloading <GET https://www.example.com/redacted/redacted> ({"spider": "<Spider 'courses-up' at 0x76f1c90990a0>"})
2025-01-22T08:26:23.475Z       Traceback (most recent call last):
2025-01-22T08:26:23.477Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks
2025-01-22T08:26:23.479Z           result = context.run(
2025-01-22T08:26:23.481Z                    ^^^^^^^^^^^^
2025-01-22T08:26:23.483Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
2025-01-22T08:26:23.484Z           return g.throw(self.value.with_traceback(self.tb))
2025-01-22T08:26:23.486Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.488Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/scrapy/core/downloader/middleware.py", line 56, in process_request
2025-01-22T08:26:23.491Z           response = yield deferred_from_coro(
2025-01-22T08:26:23.493Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.494Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/twisted/internet/defer.py", line 1253, in adapt
2025-01-22T08:26:23.496Z           extracted: _SelfResultT | Failure = result.result()
2025-01-22T08:26:23.498Z                                               ^^^^^^^^^^^^^^^
2025-01-22T08:26:23.500Z         File "/usr/local/lib/python3.12/asyncio/futures.py", line 202, in result
2025-01-22T08:26:23.501Z           raise self._exception.with_traceback(self._exception_tb)
2025-01-22T08:26:23.503Z         File "/usr/local/lib/python3.12/asyncio/tasks.py", line 314, in __step_run_and_handle_result
2025-01-22T08:26:23.505Z           result = coro.send(None)
2025-01-22T08:26:23.507Z                    ^^^^^^^^^^^^^^^
2025-01-22T08:26:23.509Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/scrapy/middlewares/apify_proxy.py", line 82, in process_request
2025-01-22T08:26:23.511Z           url = await self._get_new_proxy_url()
2025-01-22T08:26:23.513Z                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.514Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/scrapy/middlewares/apify_proxy.py", line 133, in _get_new_proxy_url
2025-01-22T08:26:23.516Z           else await Actor.create_proxy_configuration(actor_proxy_input=self._proxy_settings)
2025-01-22T08:26:23.518Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.520Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/actor.py", line 1315, in create_proxy_configuration
2025-01-22T08:26:23.521Z           return await cls._get_default_instance().create_proxy_configuration(
2025-01-22T08:26:23.523Z                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.525Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/actor.py", line 1355, in _create_proxy_configuration_internal
2025-01-22T08:26:23.527Z           await proxy_configuration.initialize()
2025-01-22T08:26:23.529Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/proxy_configuration.py", line 206, in initialize
2025-01-22T08:26:23.530Z           await self._check_access()
2025-01-22T08:26:23.533Z         File "/usr/src/app/.venv/lib/python3.12/site-packages/apify/proxy_configuration.py", line 328, in _check_access
2025-01-22T08:26:23.542Z           async with httpx.AsyncClient(proxies=await self.new_url()) as client:
2025-01-22T08:26:23.544Z                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T08:26:23.546Z       TypeError: AsyncClient.__init__() got an unexpected keyword argument 'proxies'

[honzajavorek]

I'm sorry, this isn't relevant. I checked again and I realized I had some older versions of the packages. The following combo doesn't produce this error:

apify = {version = "2.2.1", extras = ["scrapy"]}
apify-client = "1.8.1"
apify-shared = "*"