🎉Source Amazon S3: increase unit test coverage at least 90% #11967

lazebnyi · 2022-04-13T11:58:20Z

What

#11676 - Source Amazon S3: increase unit test coverage at least 90%

How

Increased unit test coverage to 90%

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

lazebnyi · 2022-04-13T11:58:56Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2160999760
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2160999760
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  28      1    96%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           184     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   29     10    66%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              42     22    48%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               665    134    80%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              42      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  28      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      1    98%
source_s3/source_files_abstract/source.py                            37      2    95%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/formats/csv_parser.py                74     18    76%
source_s3/utils.py                                                   29      8    72%
source_s3/source_files_abstract/stream.py                           184     74    60%
source_s3/stream.py                                                  43     18    58%
-------------------------------------------------------------------------------------
TOTAL                                                               665    124    81%

CLAassistant · 2022-05-05T13:12:14Z

All committers have signed the CLA.

davydov-d · 2022-05-19T17:41:10Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2353640456
❌ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2353640456
🐛 https://gradle.com/s/zedlzy6ppr4dw
Python short test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs0]
FAILED test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs1]
FAILED test_core.py::TestBasicRead::test_airbyte_trace_message_on_failure[inputs2]
=================== 3 failed, 47 passed in 110.83s (0:01:50) ===================

davydov-d · 2022-05-20T07:02:04Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2356690202
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2356690202
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  29      1    97%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           185     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              43     22    49%
source_s3/source_files_abstract/formats/avro_parser.py               38     25    34%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               715    159    78%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              43      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  29      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      1    98%
source_s3/stream.py                                                  43      1    98%
source_s3/source_files_abstract/source.py                            37      2    95%
source_s3/source_files_abstract/formats/avro_parser.py               38      3    92%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/utils.py                                                   31      8    74%
source_s3/source_files_abstract/stream.py                           185     74    60%
-------------------------------------------------------------------------------------
TOTAL                                                               715    110    85%

davydov-d · 2022-05-20T13:26:33Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2358549679
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2358549679
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  29      1    97%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           185     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              43     22    49%
source_s3/source_files_abstract/formats/avro_parser.py               38     25    34%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               715    159    78%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              43      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  29      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      1    98%
source_s3/stream.py                                                  43      1    98%
source_s3/source_files_abstract/source.py                            37      2    95%
source_s3/source_files_abstract/formats/avro_parser.py               38      3    92%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/stream.py                           185     43    77%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/utils.py                                                   31      8    74%
-------------------------------------------------------------------------------------
TOTAL                                                               715     79    89%

davydov-d · 2022-05-20T14:18:50Z

/test connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2358830992
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2358830992
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source.py                                                  29      1    97%
source_s3/source_files_abstract/storagefile.py                       23      1    96%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      2    94%
source_s3/source_files_abstract/stream.py                           185     11    94%
source_s3/stream.py                                                  43      3    93%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/source_files_abstract/file_info.py                         26      8    69%
source_s3/utils.py                                                   31     10    68%
source_s3/source_files_abstract/source.py                            37     14    62%
source_s3/source_files_abstract/spec.py                              43     22    49%
source_s3/source_files_abstract/formats/avro_parser.py               38     25    34%
source_s3/source_files_abstract/formats/parquet_parser.py            61     44    28%
-------------------------------------------------------------------------------------
TOTAL                                                               715    159    78%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
source_s3/source_files_abstract/storagefile.py                       23      0   100%
source_s3/source_files_abstract/spec.py                              43      0   100%
source_s3/source_files_abstract/formats/parquet_spec.py               9      0   100%
source_s3/source_files_abstract/formats/csv_spec.py                  16      0   100%
source_s3/source_files_abstract/formats/avro_spec.py                  5      0   100%
source_s3/source_files_abstract/formats/abstract_file_parser.py      35      0   100%
source_s3/source.py                                                  29      0   100%
source_s3/s3file.py                                                  37      0   100%
source_s3/s3_utils.py                                                19      0   100%
source_s3/__init__.py                                                 2      0   100%
source_s3/source_files_abstract/formats/parquet_parser.py            61      1    98%
source_s3/stream.py                                                  43      1    98%
source_s3/source_files_abstract/source.py                            37      2    95%
source_s3/source_files_abstract/formats/avro_parser.py               38      3    92%
source_s3/source_files_abstract/file_info.py                         26      3    88%
source_s3/source_files_abstract/stream.py                           185     35    81%
source_s3/source_files_abstract/formats/csv_parser.py                76     18    76%
source_s3/utils.py                                                   31      8    74%
-------------------------------------------------------------------------------------
TOTAL                                                               715     71    90%

davydov-d · 2022-05-23T07:15:32Z

/publish connector=connectors/source-s3

🕑 connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2369613218
🚀 Successfully published connectors/source-s3
🚀 Auto-bumped version for connectors/source-s3
✅ connectors/source-s3 https://github.com/airbytehq/airbyte/actions/runs/2369613218

Increased unittest coverage

d027b67

github-actions bot added the area/connectors Connector related issues label Apr 13, 2022

lazebnyi linked an issue Apr 13, 2022 that may be closed by this pull request

Source Amazon S3: increase unit test coverage at least 90% #11676

Closed

lazebnyi requested review from grubberr and annalvova05 April 13, 2022 12:03

grubberr approved these changes Apr 14, 2022

View reviewed changes

annalvova05 approved these changes Apr 27, 2022

View reviewed changes

#11676 test coverage 85%

d15ada8

Merge branch 'master' into lazebnyi/11676-s3--unit-test-coverage-to-90

29360ba

#11676 unit tests 90%

ed8124b

#11676 two more unit tests

c68e22a

davydov-d requested a review from bazarnov May 20, 2022 14:45

bazarnov approved these changes May 20, 2022

View reviewed changes

#11676 bump version

523ea04

github-actions bot added the area/documentation Improvements or additions to documentation label May 23, 2022

auto-bump connector version

c8efa1d

octavia-squidington-iii temporarily deployed to more-secrets May 23, 2022 07:47 Inactive

octavia-squidington-iii temporarily deployed to more-secrets May 23, 2022 08:16 Inactive

lazebnyi marked this pull request as ready for review May 23, 2022 10:36

lazebnyi merged commit 9132674 into master May 23, 2022

lazebnyi deleted the lazebnyi/11676-s3--unit-test-coverage-to-90 branch May 23, 2022 10:37

octavia-squidington-iii mentioned this pull request May 25, 2022

Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha #13188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉Source Amazon S3: increase unit test coverage at least 90% #11967

🎉Source Amazon S3: increase unit test coverage at least 90% #11967

lazebnyi commented Apr 13, 2022

lazebnyi commented Apr 13, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

davydov-d commented May 19, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 23, 2022 •

edited by github-actions bot

Loading

🎉Source Amazon S3: increase unit test coverage at least 90% #11967

🎉Source Amazon S3: increase unit test coverage at least 90% #11967

Conversation

lazebnyi commented Apr 13, 2022

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

lazebnyi commented Apr 13, 2022 • edited by github-actions bot Loading

CLAassistant commented May 5, 2022 • edited Loading

davydov-d commented May 19, 2022 • edited by github-actions bot Loading

davydov-d commented May 20, 2022 • edited by github-actions bot Loading

davydov-d commented May 20, 2022 • edited by github-actions bot Loading

davydov-d commented May 20, 2022 • edited by github-actions bot Loading

davydov-d commented May 23, 2022 • edited by github-actions bot Loading

lazebnyi commented Apr 13, 2022 •

edited by github-actions bot

Loading

CLAassistant commented May 5, 2022 •

edited

Loading

davydov-d commented May 19, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 20, 2022 •

edited by github-actions bot

Loading

davydov-d commented May 23, 2022 •

edited by github-actions bot

Loading