Skip to content

Add D205 pydocstyle check #11627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add D205 pydocstyle check #11627

wants to merge 2 commits into from

Conversation

ktrueda
Copy link

@ktrueda ktrueda commented Oct 18, 2020

Enabled D205 Pydocstyle check

  • Fixed all pre-commit errors

Part of issue: #10742

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:dev-tools area:webserver Webserver related Issues labels Oct 18, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Oct 18, 2020

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://s.apache.org/airflow-slack

@kaxil
Copy link
Member

kaxil commented Oct 19, 2020

Static checks are failing::


airflow/providers/amazon/aws/sensors/step_function_execution.py:28:111: E501 line too long (118 > 110 characters)
airflow/jobs/backfill_job.py:290:111: E501 line too long (142 > 110 characters)
airflow/jobs/backfill_job.py:711:111: E501 line too long (122 > 110 characters)
airflow/jobs/backfill_job.py:768:111: E501 line too long (129 > 110 characters)
airflow/jobs/backfill_job.py:847:111: E501 line too long (147 > 110 characters)
airflow/providers/amazon/aws/example_dags/example_google_api_to_s3_transfer_advanced.py:18:111: E501 line too long (167 > 110 characters)
airflow/operators/generic_transfer.py:27:111: E501 line too long (124 > 110 characters)
airflow/providers/google/cloud/hooks/kubernetes_engine.py:89:111: E501 line too long (124 > 110 characters)
airflow/providers/exasol/hooks/exasol.py:178:111: E501 line too long (120 > 110 characters)
airflow/settings.py:127:111: E501 line too long (161 > 110 characters)
airflow/providers/google/suite/transfers/gcs_to_gdrive.py:35:111: E501 line too long (115 > 110 characters)
airflow/providers/amazon/aws/hooks/batch_client.py:239:111: E501 line too long (116 > 110 characters)
airflow/providers/amazon/aws/hooks/batch_client.py:282:111: E501 line too long (129 > 110 characters)
airflow/providers/http/sensors/http.py:29:111: E501 line too long (123 > 110 characters)
airflow/providers/amazon/aws/hooks/s3.py:47:111: E501 line too long (132 > 110 characters)
airflow/jobs/scheduler_job.py:552:111: E501 line too long (131 > 110 characters)
airflow/jobs/scheduler_job.py:939:111: E501 line too long (122 > 110 characters)
airflow/jobs/scheduler_job.py:1193:111: E501 line too long (116 > 110 characters)
airflow/jobs/scheduler_job.py:1564:111: E501 line too long (146 > 110 characters)
airflow/jobs/scheduler_job.py:1776:111: E501 line too long (130 > 110 characters)
airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:31:111: E501 line too long (131 > 110 characters)
airflow/providers/amazon/aws/operators/glue.py:30:111: E501 line too long (112 > 110 characters)
airflow/configuration.py:56:111: E501 line too long (139 > 110 characters)
airflow/configuration.py:230:111: E501 line too long (122 > 110 characters)
airflow/providers/amazon/aws/example_dags/example_imap_attachment_to_s3.py:19:111: E501 line too long (144 > 110 characters)
airflow/providers/google/cloud/hooks/functions.py:61:111: E501 line too long (117 > 110 characters)
airflow/utils/log/colored_log.py:43:111: E501 line too long (140 > 110 characters)
airflow/example_dags/example_branch_python_dop_operator_3.py:20:111: E501 line too long (141 > 110 characters)
airflow/providers/google/cloud/operators/gcs.py:283:111: E501 line too long (134 > 110 characters)

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left some review, but you have made many of the lines too long, and (needlessly?) changed the line wrapping -- please don't do that unless there is a reason.

For example, this sort of change is not needed:

@@ -1856,8 +1861,7 @@ def get_default_view(self):
    @provide_session
    def deactivate_unknown_dags(active_dag_ids, session=None):
        """
-        Given a list of known DAGs, deactivate any other DAGs that are
-        marked as active in the ORM
+        Given a list of known DAGs, deactivate any other DAGs that are marked as active in the ORM
        :param active_dag_ids: list of DAG IDs that are active
        :type active_dag_ids: list[unicode]

Authentication is implemented using flask_login and different environments can
implement their own login mechanisms by providing an `airflow_login` module
in their PYTHONPATH. airflow_login should be based off the
Authentication is implemented using flask_login and different environments can implement their own login mechanisms by providing an `airflow_login` module in their PYTHONPATH.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is too long. Please wrap at 110 characters.

Comment on lines +49 to +51
GunicornMonitor

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
GunicornMonitor

The existing first paragraph was a good summary.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The existing first paragraph was good. But it's long as a first line summary .
Obeying D205 and E501 summary line needs to obey two rules.

  • only 1 line (not 2+ line)
  • length < 110

Comment on lines 56 to 57
Expands (potentially nested) env vars by repeatedly applying
`expandvars` and `expanduser` until interpolation stops having
any effect.
Expands (potentially nested) env vars by repeatedly applying `expandvars` and `expanduser` until interpolation stops having any effect.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous version was fine as it was.

@@ -17,8 +17,7 @@
# under the License.

"""
Example DAG demonstrating the usage of BranchPythonOperator with depends_on_past=True, where tasks may be run
or skipped on alternating runs.
Example DAG demonstrating the usage of BranchPythonOperator with depends_on_past=True, where tasks may be run or skipped on alternating runs.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this -- previous one was fine.

@ktrueda ktrueda marked this pull request as draft October 19, 2020 15:22
@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@mik-laj mik-laj removed the area:API Airflow's REST/HTTP API label Oct 19, 2020
@ashb ashb added area:docs and removed area:CLI area:webserver Webserver related Issues area:dev-tools labels Oct 20, 2020
@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@mik-laj
Copy link
Member

mik-laj commented Oct 20, 2020

Static checks are sad.

airflow/kubernetes/kube_client.py:69 in private function `_enable_tcp_keepalive`:
        D205: 1 blank line required between summary line and description (found 0)
airflow/providers/discord/operators/discord_webhook.py:28 in public class `DiscordWebhookOperator`:
        D205: 1 blank line required between summary line and description (found 0)
airflow/providers/google/cloud/operators/bigquery_dts.py:199 in public class `BigQueryDataTransferServiceStartTransferRunsOperator`:
        D205: 1 blank line required between summary line and description (found 0)
airflow/providers/microsoft/azure/operators/azure_cosmos.py:25 in public class `AzureCosmosInsertDocumentOperator`:
        D205: 1 blank line required between summary line and description (found 0)

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@ktrueda
Copy link
Author

ktrueda commented Oct 22, 2020

CI Build / Build docs (pull_request) Failing after 10m — Build docs said

==================== Error   1 ====================
 WARNING: Definition list ends without a blank line; unexpected unindent.

File path: /opt/airflow/docs/_api/airflow/providers/slack/operators/slack/index.rst (71)

  67 |            dag=dag,
  68 |            token="XXX",
  69 |            text="hello there!",
  70 |            channel="#random",
  71 |        )
  72 | 
  73 |    :param channel: channel in which to post message on slack name (#general) or
  74 |        ID (C12318391). (templated)
  75 |    :type channel: str
  76 |    :param username: Username that airflow will be posting to Slack as. (templated)
==================== Error   2 ====================
 WARNING: Definition list ends without a blank line; unexpected unindent.

File path: /opt/airflow/docs/_api/airflow/providers/slack/operators/slack/index.rst (123)

 119 |            initial_comment="Hello World!",
 120 |            filename="hello_world.csv",
 121 |            filetype="csv",
 122 |            content="hello,world,csv,file",
 123 |        )
 124 | 
 125 |    :param channel: channel in which to sent file on slack name (templated)
 126 |    :type channel: str
 127 |    :param initial_comment: message to send to slack. (templated)
 128 |    :type initial_comment: str
==================================================

CI Build / Spell check docs (pull_request) said

/opt/airflow/docs/_api/airflow/providers/slack/operators/slack/index.rst:71:Definition list ends without a blank line; unexpected unindent.
==================== Error   1 ====================
Sphinx spellcheck returned non-zero exit status: 2.

==================================================

I didn't modify these files. How can I pass these CI.

And CI Build / MySQL8, Py3.6 (pull_request) failed. I have no idea about this test.

@potiuk
Copy link
Member

potiuk commented Oct 22, 2020

I believe it was a merge of a change that failed the static checks - it is already fixed by 95be3ee and you need to rebase to latest master to fix it (that's what I did with my PRs)

@ktrueda
Copy link
Author

ktrueda commented Oct 23, 2020

@potiuk
Thank you for your advice!!!
I rebased it and pushed it.

🙏

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@ktrueda
Copy link
Author

ktrueda commented Oct 24, 2020

@potiuk
I tried rebase, but it doesn't work.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@ktrueda
Copy link
Author

ktrueda commented Oct 24, 2020

@kaxil
Thank you for your suggestion very much!!

🙏

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.

@ktrueda
Copy link
Author

ktrueda commented Oct 28, 2020

@kaxil
#11627 (comment)
If you have any solution, please let me know it.

@apache apache deleted a comment from github-actions bot Oct 29, 2020
@mik-laj
Copy link
Member

mik-laj commented Oct 29, 2020

CI is sad. I working on fix.

black.............................................................................................................................Failed
- hook id: black
- files were modified by this hook

reformatted /home/runner/work/airflow/airflow/airflow/providers/slack/operators/slack.py
All done! ✨ 🍰 ✨
1 file reformatted, 1558 files left unchanged.
All changes made by hooks:
diff --git a/airflow/providers/slack/operators/slack.py b/airflow/providers/slack/operators/slack.py
index 18bc86f..29b53cf 100644
--- a/airflow/providers/slack/operators/slack.py
+++ b/airflow/providers/slack/operators/slack.py
@@ -90,7 +90,7 @@ class SlackAPIOperator(BaseOperator):
         slack.call(self.method, json=self.api_params)
 
 
-class SlackAPIPostOperator(SlackAPIOperator):   # noqa: D412
+class SlackAPIPostOperator(SlackAPIOperator):  # noqa: D412
     """
     Posts messages to a slack channel.
 
@@ -160,7 +160,7 @@ class SlackAPIPostOperator(SlackAPIOperator):   # noqa: D412
         }
 
 
-class SlackAPIFileOperator(SlackAPIOperator):   # noqa: D412
+class SlackAPIFileOperator(SlackAPIOperator):  # noqa: D412
     """
     Send a file to a slack channel
 
Error: Process completed with exit code 1.

@mik-laj
Copy link
Member

mik-laj commented Oct 29, 2020

Spell check is sad now. :-/

_api/airflow/models/dag/index.rst:1109: (parametrize)  Accepts kwargs for operator kwarg. Can be used to parametrize DAGs.

@@ -388,6 +390,8 @@ def __exit__(self, _type, _value, _tb):
@staticmethod
def _upgrade_outdated_dag_access_control(access_control=None):
"""
_upgrade_outdated_dag_access_control
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks unreleated

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not be adding a dummy Summary heading for docstring @ktrueda

@@ -525,6 +531,8 @@ def next_dagrun_info(

def next_dagrun_after_date(self, date_last_automated_dagrun: Optional[pendulum.DateTime]):
"""
next_dagrun_after_date
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks unrelated.

@@ -810,6 +817,8 @@ def normalized_schedule_interval(self) -> Optional[ScheduleInterval]:
@provide_session
def handle_callback(self, dagrun, success=True, reason=None, session=None):
"""
handle_callback
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks unrelated.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@kaxil
Copy link
Member

kaxil commented Oct 29, 2020

parametrize

Fixed spelling and pushed

@apache apache deleted a comment from github-actions bot Oct 29, 2020
@ktrueda
Copy link
Author

ktrueda commented Oct 30, 2020

@kaxil @mik-laj
Thank you for fixing very much.
I'll fix dummy summary heading.

@mik-laj
Copy link
Member

mik-laj commented Oct 30, 2020

@ktrueda CI is sad. Can you do a rebase?

@ktrueda
Copy link
Author

ktrueda commented Oct 31, 2020

@ktrueda CI is sad. Can you do a rebase?

Thank you.
I've rebased.

@github-actions
Copy link

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@github-actions
Copy link

The Workflow run is cancelling this PR. It in earlier duplicate of 1029499 run.

@@ -18,6 +18,8 @@
#

"""
__init__.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a Dummy header

@@ -218,6 +221,8 @@ def verify_dagruns(dag_runs, commit, state, session, current_task):

def verify_dag_run_integrity(dag, dates):
"""
verify_dag_run_integrity
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a Dummy header

@@ -105,6 +107,8 @@ def __init__(

def _generate_plugin_state(self) -> Dict[str, float]:
"""
_generate_plugin_state
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a Dummy header

@@ -52,6 +52,8 @@

def expand_env_var(env_var):
"""
expand_env_var
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a Dummy header

@@ -225,6 +227,8 @@ def _validate(self):

def _validate_config_dependencies(self):
"""
_validate_config_dependencies
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are lot of dummy headers instead of actual headers that needs to be fixed

@ktrueda
Copy link
Author

ktrueda commented Nov 2, 2020

@kaxil
Thank you for your comment.
I'm currently working on it. I need some times since there are a lot of dummies.

@kaxil
Copy link
Member

kaxil commented Nov 4, 2020

Can you please rebase your PR on latest Master since we have applied Black and PyUpgrade on Master.

It will help if your squash your commits into single commit first so that there are less conflicts.

@ktrueda
Copy link
Author

ktrueda commented Nov 4, 2020

@kaxil
I rebased and wrote summary line instead of dummy.
I did my best but some text is redundant and some is same with function/class name. If you have better idea, please suggest.
I'm sorry for my lack of skill and understanding the project.

@github-actions
Copy link

github-actions bot commented Nov 4, 2020

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

@ktrueda
Copy link
Author

ktrueda commented Nov 16, 2020

@kaxil
If my code is bad, please suggest.
Or I'll close this PR.

@kaxil
Copy link
Member

kaxil commented Nov 16, 2020

@kaxil
If my code is bad, please suggest.
Or I'll close this PR.

Thanks @ktrueda , I am on leave today, will check back once I am back

@github-actions
Copy link

github-actions bot commented Mar 1, 2021

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Mar 1, 2021
@github-actions github-actions bot closed this Mar 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants