[BUG] Test are breaking for restricted access to the MIND blob

[miguelgfierro]

Description

tmp_path = PosixPath('/tmp/pytest-of-root/pytest-0/test_download_mind_large0')

    def test_download_mind_large(tmp_path):
>       train_path, valid_path = download_mind(size="large", dest_path=tmp_path)

tests/data_validation/recommenders/datasets/test_mind.py:128: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
recommenders/datasets/mind.py:66: in download_mind
    train_path = maybe_download(url=url_train, work_directory=path)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:56: in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:266: in call
    raise attempt.get()
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:301: in get
    six.reraise(self.value[0], self.value[1], self.value[2])
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/six.py:719: in reraise
    raise value
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:251: in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
recommenders/datasets/download_utils.py:52: in maybe_download
    r.raise_for_status()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [409]>

    def raise_for_status(self):
        """Raises :class:`HTTPError`, if one occurred."""
    
        http_error_msg = ""
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode("utf-8")
            except UnicodeDecodeError:
                reason = self.reason.decode("iso-8859-1")
        else:
            reason = self.reason
    
        if 400 <= self.status_code < 500:
            http_error_msg = (
                f"{self.status_code} Client Error: {reason} for url: {self.url}"
            )
    
        elif 500 <= self.status_code < 600:
            http_error_msg = (
                f"{self.status_code} Server Error: {reason} for url: {self.url}"
            )
    
        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip

/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/requests/models.py:1024: HTTPError
------------------------------ Captured log call -------------------------------
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
___________________________ test_extract_mind_large ____________________________

tmp = '/tmp/pytest-of-root/pytest-0/tmptwsi8df7'

    def test_extract_mind_large(tmp):
>       train_zip, valid_zip = download_mind(size="large", dest_path=tmp)

tests/data_validation/recommenders/datasets/test_mind.py:136: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
recommenders/datasets/mind.py:66: in download_mind
    train_path = maybe_download(url=url_train, work_directory=path)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:56: in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:266: in call
    raise attempt.get()
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:301: in get
    six.reraise(self.value[0], self.value[1], self.value[2])
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/six.py:719: in reraise
    raise value
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:251: in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
recommenders/datasets/download_utils.py:52: in maybe_download
    r.raise_for_status()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Response [409]>

    def raise_for_status(self):
        """Raises :class:`HTTPError`, if one occurred."""
    
        http_error_msg = ""
        if isinstance(self.reason, bytes):
            # We attempt to decode utf-8 first because some servers
            # choose to localize their reason strings. If the string
            # isn't utf-8, we fall back to iso-8859-1 for all other
            # encodings. (See PR #3538)
            try:
                reason = self.reason.decode("utf-8")
            except UnicodeDecodeError:
                reason = self.reason.decode("iso-8859-1")
        else:
            reason = self.reason
    
        if 400 <= self.status_code < 500:
            http_error_msg = (
                f"{self.status_code} Client Error: {reason} for url: {self.url}"
            )
    
        elif 500 <= self.status_code < 600:
            http_error_msg = (
                f"{self.status_code} Server Error: {reason} for url: {self.url}"
            )
    
        if http_error_msg:
>           raise HTTPError(http_error_msg, response=self)
E           requests.exceptions.HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip

/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/requests/models.py:1024: HTTPError
------------------------------ Captured log call -------------------------------
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
ERROR    recommenders.datasets.download_utils:download_utils.py:51 Problem downloading https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip
_____________________________ test_mind_utils_runs _____________________________

notebooks = {'als_deep_dive': '/mnt/azureml/cr/j/0ff71de6470e4f4eb33b284cd911e6c9/exe/wd/examples/02_model_collaborative_filtering...rk_movielens': '/mnt/azureml/cr/j/0ff71de6470e4f4eb33b284cd911e6c9/exe/wd/examples/06_benchmarks/movielens.ipynb', ...}
output_notebook = 'output.ipynb', kernel_name = 'python3'
tmp = '/tmp/pytest-of-root/pytest-0/tmpnb2e_a7h'

    def test_mind_utils_runs(notebooks, output_notebook, kernel_name, tmp):
        notebook_path = notebooks["mind_utils"]
>       execute_notebook(
            notebook_path,
            output_notebook,
            kernel_name=kernel_name,
            parameters=dict(mind_type="small", word_embedding_dim=300),
        )

tests/data_validation/examples/test_mind.py:9: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
recommenders/utils/notebook_utils.py:102: in execute_notebook
    executed_notebook, _ = execute_preprocessor.preprocess(
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py:103: in preprocess
    self.preprocess_cell(cell, resources, index)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/nbconvert/preprocessors/execute.py:124: in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/jupyter_core/utils/__init__.py:165: in wrapped
    return loop.run_until_complete(inner)
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/asyncio/base_events.py:649: in run_until_complete
    return future.result()
/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/nbclient/client.py:1062: in async_execute_cell
    await self._check_raise_for_error(cell, cell_index, exec_reply)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nbconvert.preprocessors.execute.ExecutePreprocessor object at 0x145c1b978d00>
cell = {'cell_type': 'code', 'execution_count': 3, 'metadata': {'execution': {'iopub.status.busy': '2024-08-01T02:06:38.72166...lid'), clean_zip_file=False)\noutput_path = os.path.join(data_path, 'utils')\nos.makedirs(output_path, exist_ok=True)"}
cell_index = 4
exec_reply = {'buffers': [], 'content': {'ename': 'HTTPError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'ff776c0c-0b2d-4c00-...e, 'engine': 'ff776c0c-0b2d-4c00-8284-f504cc9cd4c9', 'started': '2024-08-01T02:06:38.722124Z', 'status': 'error'}, ...}

    async def _check_raise_for_error(
        self, cell: NotebookNode, cell_index: int, exec_reply: dict[str, t.Any] | None
    ) -> None:
        if exec_reply is None:
            return None
    
        exec_reply_content = exec_reply["content"]
        if exec_reply_content["status"] != "error":
            return None
    
        cell_allows_errors = (not self.force_raise_errors) and (
            self.allow_errors
            or exec_reply_content.get("ename") in self.allow_error_names
            or "raises-exception" in cell.metadata.get("tags", [])
        )
        await run_hook(
            self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
        )
        if not cell_allows_errors:
>           raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)
E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
E           ------------------
E           tmpdir = TemporaryDirectory()
E           data_path = tmpdir.name
E           train_zip, valid_zip = download_mind(size=mind_type, dest_path=data_path)
E           unzip_file(train_zip, os.path.join(data_path, 'train'), clean_zip_file=False)
E           unzip_file(valid_zip, os.path.join(data_path, 'valid'), clean_zip_file=False)
E           output_path = os.path.join(data_path, 'utils')
E           os.makedirs(output_path, exist_ok=True)
E           ------------------
E           
E           ----- stderr -----
E           Problem downloading https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip
E           ----- stderr -----
E           Problem downloading https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip
E           ----- stderr -----
E           Problem downloading https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip
E           ----- stderr -----
E           Problem downloading https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip
E           ----- stderr -----
E           Problem downloading https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip
E           ------------------
E           
E           �[0;31m---------------------------------------------------------------------------�[0m
E           �[0;31mHTTPError�[0m                                 Traceback (most recent call last)
E           Cell �[0;32mIn[3], line 3�[0m
E           �[1;32m      1�[0m tmpdir �[38;5;241m=�[39m TemporaryDirectory()
E           �[1;32m      2�[0m data_path �[38;5;241m=�[39m tmpdir�[38;5;241m.�[39mname
E           �[0;32m----> 3�[0m train_zip, valid_zip �[38;5;241m=�[39m �[43mdownload_mind�[49m�[43m(�[49m�[43msize�[49m�[38;5;241;43m=�[39;49m�[43mmind_type�[49m�[43m,�[49m�[43m �[49m�[43mdest_path�[49m�[38;5;241;43m=�[39;49m�[43mdata_path�[49m�[43m)�[49m
E           �[1;32m      4�[0m unzip_file(train_zip, os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(data_path, �[38;5;124m'�[39m�[38;5;124mtrain�[39m�[38;5;124m'�[39m), clean_zip_file�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m)
E           �[1;32m      5�[0m unzip_file(valid_zip, os�[38;5;241m.�[39mpath�[38;5;241m.�[39mjoin(data_path, �[38;5;124m'�[39m�[38;5;124mvalid�[39m�[38;5;124m'�[39m), clean_zip_file�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m)
E           
E           File �[0;32m/mnt/azureml/cr/j/0ff71de6470e4f4eb33b284cd911e6c9/exe/wd/recommenders/datasets/mind.py:66�[0m, in �[0;36mdownload_mind�[0;34m(size, dest_path)�[0m
E           �[1;32m     64�[0m url_train, url_valid �[38;5;241m=�[39m URL_MIND[size]
E           �[1;32m     65�[0m �[38;5;28;01mwith�[39;00m download_path(dest_path) �[38;5;28;01mas�[39;00m path:
E           �[0;32m---> 66�[0m     train_path �[38;5;241m=�[39m �[43mmaybe_download�[49m�[43m(�[49m�[43murl�[49m�[38;5;241;43m=�[39;49m�[43murl_train�[49m�[43m,�[49m�[43m �[49m�[43mwork_directory�[49m�[38;5;241;43m=�[39;49m�[43mpath�[49m�[43m)�[49m
E           �[1;32m     67�[0m     valid_path �[38;5;241m=�[39m maybe_download(url�[38;5;241m=�[39murl_valid, work_directory�[38;5;241m=�[39mpath)
E           �[1;32m     68�[0m �[38;5;28;01mreturn�[39;00m train_path, valid_path
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:56�[0m, in �[0;36mretry.<locals>.wrap.<locals>.wrapped_f�[0;34m(*args, **kw)�[0m
E           �[1;32m     54�[0m �[38;5;129m@six�[39m�[38;5;241m.�[39mwraps(f)
E           �[1;32m     55�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mwrapped_f�[39m(�[38;5;241m*�[39margs, �[38;5;241m*�[39m�[38;5;241m*�[39mkw):
E           �[0;32m---> 56�[0m     �[38;5;28;01mreturn�[39;00m �[43mRetrying�[49m�[43m(�[49m�[38;5;241;43m*�[39;49m�[43mdargs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m*�[39;49m�[38;5;241;43m*�[39;49m�[43mdkw�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mcall�[49m�[43m(�[49m�[43mf�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m*�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m*�[39;49m�[38;5;241;43m*�[39;49m�[43mkw�[49m�[43m)�[49m
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:266�[0m, in �[0;36mRetrying.call�[0;34m(self, fn, *args, **kwargs)�[0m
E           �[1;32m    263�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstop(attempt_number, delay_since_first_attempt_ms):
E           �[1;32m    264�[0m     �[38;5;28;01mif�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wrap_exception �[38;5;129;01mand�[39;00m attempt�[38;5;241m.�[39mhas_exception:
E           �[1;32m    265�[0m         �[38;5;66;03m# get() on an attempt with an exception should cause it to be raised, but raise just in case�[39;00m
E           �[0;32m--> 266�[0m         �[38;5;28;01mraise�[39;00m �[43mattempt�[49m�[38;5;241;43m.�[39;49m�[43mget�[49m�[43m(�[49m�[43m)�[49m
E           �[1;32m    267�[0m     �[38;5;28;01melse�[39;00m:
E           �[1;32m    268�[0m         �[38;5;28;01mraise�[39;00m RetryError(attempt)
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:301�[0m, in �[0;36mAttempt.get�[0;34m(self, wrap_exception)�[0m
E           �[1;32m    299�[0m         �[38;5;28;01mraise�[39;00m RetryError(�[38;5;28mself�[39m)
E           �[1;32m    300�[0m     �[38;5;28;01melse�[39;00m:
E           �[0;32m--> 301�[0m         �[43msix�[49m�[38;5;241;43m.�[39;49m�[43mreraise�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mvalue�[49m�[43m[�[49m�[38;5;241;43m0�[39;49m�[43m]�[49m�[43m,�[49m�[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mvalue�[49m�[43m[�[49m�[38;5;241;43m1�[39;49m�[43m]�[49m�[43m,�[49m�[43m �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mvalue�[49m�[43m[�[49m�[38;5;241;43m2�[39;49m�[43m]�[49m�[43m)�[49m
E           �[1;32m    302�[0m �[38;5;28;01melse�[39;00m:
E           �[1;32m    303�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mvalue
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/six.py:719�[0m, in �[0;36mreraise�[0;34m(tp, value, tb)�[0m
E           �[1;32m    717�[0m     �[38;5;28;01mif�[39;00m value�[38;5;241m.�[39m__traceback__ �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m tb:
E           �[1;32m    718�[0m         �[38;5;28;01mraise�[39;00m value�[38;5;241m.�[39mwith_traceback(tb)
E           �[0;32m--> 719�[0m     �[38;5;28;01mraise�[39;00m value
E           �[1;32m    720�[0m �[38;5;28;01mfinally�[39;00m:
E           �[1;32m    721�[0m     value �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/retrying.py:251�[0m, in �[0;36mRetrying.call�[0;34m(self, fn, *args, **kwargs)�[0m
E           �[1;32m    248�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39m_before_attempts(attempt_number)
E           �[1;32m    250�[0m �[38;5;28;01mtry�[39;00m:
E           �[0;32m--> 251�[0m     attempt �[38;5;241m=�[39m Attempt(�[43mfn�[49m�[43m(�[49m�[38;5;241;43m*�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m*�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m�[43m)�[49m, attempt_number, �[38;5;28;01mFalse�[39;00m)
E           �[1;32m    252�[0m �[38;5;28;01mexcept�[39;00m:
E           �[1;32m    253�[0m     tb �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
E           
E           File �[0;32m/mnt/azureml/cr/j/0ff71de6470e4f4eb33b284cd911e6c9/exe/wd/recommenders/datasets/download_utils.py:52�[0m, in �[0;36mmaybe_download�[0;34m(url, filename, work_directory, expected_bytes)�[0m
E           �[1;32m     50�[0m     �[38;5;28;01melse�[39;00m:
E           �[1;32m     51�[0m         log�[38;5;241m.�[39merror(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mProblem downloading �[39m�[38;5;132;01m{�[39;00murl�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m)
E           �[0;32m---> 52�[0m         �[43mr�[49m�[38;5;241;43m.�[39;49m�[43mraise_for_status�[49m�[43m(�[49m�[43m)�[49m
E           �[1;32m     53�[0m �[38;5;28;01melse�[39;00m:
E           �[1;32m     54�[0m     log�[38;5;241m.�[39minfo(�[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mFile �[39m�[38;5;132;01m{�[39;00mfilepath�[38;5;132;01m}�[39;00m�[38;5;124m already downloaded�[39m�[38;5;124m"�[39m)
E           
E           File �[0;32m/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/requests/models.py:1024�[0m, in �[0;36mResponse.raise_for_status�[0;34m(self)�[0m
E           �[1;32m   1019�[0m     http_error_msg �[38;5;241m=�[39m (
E           �[1;32m   1020�[0m         �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39mstatus_code�[38;5;132;01m}�[39;00m�[38;5;124m Server Error: �[39m�[38;5;132;01m{�[39;00mreason�[38;5;132;01m}�[39;00m�[38;5;124m for url: �[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39murl�[38;5;132;01m}�[39;00m�[38;5;124m"�[39m
E           �[1;32m   1021�[0m     )
E           �[1;32m   1023�[0m �[38;5;28;01mif�[39;00m http_error_msg:
E           �[0;32m-> 1024�[0m     �[38;5;28;01mraise�[39;00m HTTPError(http_error_msg, response�[38;5;241m=�[39m�[38;5;28mself�[39m)
E           
E           �[0;31mHTTPError�[0m: 409 Client Error: Public access is not permitted on this storage account. for url: https://mind201910small.blob.core.windows.net/release/MINDsmall_train.zip

/azureml-envs/azureml_ee05c39251ad373fce7f4861d6ed56fe/lib/python3.10/site-packages/nbclient/client.py:918: CellExecutionError

In which platform does it happen?

How do we replicate the issue?

https://github.com/recommenders-team/recommenders/actions/runs/13447074780/job/37577703132

Expected behavior (i.e. solution)

Willingness to contribute

• Yes, I can contribute for this issue independently.
• Yes, I can contribute for this issue with guidance from Recommenders community.
• No, I cannot contribute at this time.

Other Comments

[miguelgfierro]

In local it works:

$ pytest tests/data_validation/recommenders/datasets/test_mind.py 
======================================================================== test session starts ========================================================================
platform linux -- Python 3.11.9, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/miguel/MS/recommenders
configfile: pyproject.toml
plugins: anyio-4.4.0, cov-5.0.0, typeguard-4.3.0, hypothesis-6.104.2, mock-3.14.0
collected 15 items                                                                                                                                                  

tests/data_validation/recommenders/datasets/test_mind.py ...............                                                                                      [100%]

================================================================== 15 passed in 204.00s (0:03:23) ===================================================================

This is weird @SimonYansenZhao @anargyri do you think it could be because there is some kind of restrictions on Azure to download that dataset?

Can you check if it works for you: pytest tests/data_validation/recommenders/datasets/test_mind.py

[SimonYansenZhao]

I tried on GitHub Codespaces, it worked.

[miguelgfierro]

I get the same, why is it not working on azure?

[SimonYansenZhao]

From the error logs, I find the error is from downloading the data from an unknown URL. It looks like the test machine is hijacked.

[SimonYansenZhao]

After searching in the repo, I find the URL in the error log (https://mind201910small.blob.core.windows.net/release/MINDlarge_train.zip) was replaced by https://recodatasets.z20.web.core.windows.net/newsrec/MINDlarge_train.zip. See Add new URL of MIND small and MIND large and [BUG] Unable to access MIND dataset due to public access restriction on the storage account #2133.

[SimonYansenZhao]

It looks like the invalid URLs are from this version committed on Sep 16, 2023.

[SimonYansenZhao]

However, after going through the logs of the failed run, I find the version used in the run is d58014367f66cea4596a728ba3994c2ed3d12fe3 which looks correct.

[miguelgfierro]

Merging to main to see if the issue is because the code is not in main: #2222

[miguelgfierro]

These are the jobs running, some of them time out

This run is yellow_plate_6xjzxw87t7 https://github.com/recommenders-team/recommenders/actions/runs/14447665419/job/40525811066
In azureml it says it completed

These are the logs:

============================= test session starts ==============================
platform linux -- Python 3.9.22, pytest-8.3.5, pluggy-1.5.0
rootdir: /mnt/azureml/cr/j/e3a2825ed3154bbdacd2d1881d67a48e/exe/wd
configfile: pyproject.toml
plugins: cov-6.1.1, typeguard-4.4.2, anyio-4.9.0, mock-3.14.0, hypothesis-6.131.0
collected 4 items

tests/unit/examples/test_notebooks_gpu.py ...                            [100%]

=============================== warnings summary ===============================
../../../../../../../root/conda/envs/Recommenders/lib/python3.9/site-packages/jupyter_client/connect.py:22
  /root/conda/envs/Recommenders/lib/python3.9/site-packages/jupyter_client/connect.py:22: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
  given by the platformdirs library.  To remove this warning and
  see the appropriate new directories, set the environment variable
  `JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
  The use of platformdirs will be the default in `jupyter_core` v6
    from jupyter_core.paths import jupyter_data_dir, jupyter_runtime_dir, secure_write

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== slowest durations ===============================
45.04s call     tests/unit/examples/test_notebooks_gpu.py::test_xdeepfm
37.26s call     tests/unit/examples/test_notebooks_gpu.py::test_wide_deep
1.53s call     tests/unit/examples/test_notebooks_gpu.py::test_gpu_vm
0.01s teardown tests/unit/examples/test_notebooks_gpu.py::test_wide_deep

(5 durations < 0.005s hidden.  Use -vv to show these durations.)
=================== 3 passed, 1 warning in 85.21s (0:01:25) ====================

Something weird is going on. It's like GitHub is not receiving some of the logs.

Where is the 2h limit, and why it is timing out if the computation of the test are 9m 13.30s? @SimonYansenZhao

[SimonYansenZhao]

Not sure, let's re-run to see if they can be solved.

[miguelgfierro]

related to #2226