Encoding is ISO-8859-1 #12552

girarda · 2022-05-03T23:18:42Z

What

We currently fail to decode some bytes received by salesforce. As mentioned by Marcos, downgrading down to 1.0.2 fixed the issue, which indicates it was introduced in 644ace4
Fixes https://github.com/airbytehq/oncall/issues/221
Fixes Source Salesforce: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 8379: unexpected end of data #12372

How

Change the expected encoding from UTF-8 to ISO-8859-1 based on these 2 two threads which seem to indicate the encoding should be ISO-8859

Add a unit test to confirm we can decode the byte we failed to decode in production

girarda · 2022-05-03T23:26:19Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2266902416
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2266902416
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              285    106    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  896    259    71%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/__init__.py            2      0   100%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/api.py               150     19    87%
source_salesforce/streams.py           295     68    77%
source_salesforce/rate_limiting.py      22      6    73%
source_salesforce/source.py             77     33    57%
source_salesforce/utils.py               8      7    12%
--------------------------------------------------------
TOTAL                                  562    134    76%
Name                                 Stmts   Miss  Cover
--------------------------------------------------------
source_salesforce/utils.py               8      0   100%
source_salesforce/__init__.py            2      0   100%
source_salesforce/source.py             77      6    92%
source_salesforce/api.py               150     14    91%
source_salesforce/exceptions.py          8      1    88%
source_salesforce/rate_limiting.py      22      3    86%
source_salesforce/streams.py           295     43    85%
--------------------------------------------------------
TOTAL                                  562     67    88%

girarda · 2022-05-03T23:28:34Z

airbyte-integrations/connectors/source-salesforce/unit_tests/api_test.py

+
+
+def test():
+    assert b"0\xe5".decode(SalesforceStream.encoding) == "0å"


confirmed this fails if SalesforceStream.encoding is 'utf-8'

brianjlai

➕ for the unit test for encoding

girarda · 2022-05-03T23:50:55Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2266983181
🚀 Successfully published connectors/source-salesforce
🚀 Auto-bumped version for connectors/source-salesforce
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2266983181

codecov · 2022-05-04T00:18:39Z

Codecov Report

❗ No coverage uploaded for pull request base (master@075bec3). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #12552   +/-   ##
=========================================
  Coverage          ?   88.07%           
=========================================
  Files             ?        7           
  Lines             ?      562           
  Branches          ?        0           
=========================================
  Hits              ?      495           
  Misses            ?       67           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 075bec3...233ee6e. Read the comment docs.

marcosmarxm · 2022-05-04T00:32:18Z

airbyte-integrations/connectors/source-salesforce/source_salesforce/streams.py

@@ -34,6 +34,7 @@
 class SalesforceStream(HttpStream, ABC):
    page_size = 2000
    transformer = TypeTransformer(TransformConfig.DefaultSchemaNormalization)
+    encoding = "ISO-8859-1"


There is any link to Salesforce docs explaining or saying about this encoding?

https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/implementation_considerations.htm

marcosmarxm

thanks @girarda

wissevrowl · 2022-05-04T07:36:35Z

@girarda Different sources including your links indicate most Salesforce instances UTF-8 expcept for some instances

If your org logs in to ssl.salesforce.com, your encoding is ISO-8859-1. All other instances use UTF-8.

While your fix prevents the problem happening by switching to a non-variable length encoding, characters are now incorrectly decoded.

For example I added '间单的说 🪐' in an opportunity's description. The Airbyte source now converts this to 'Ã©Â�Â´Ã¥Â�Â�Ã§Â�Â�Ã¨Â¯Â´ Ã°Â�ÂªÂ�'

* Encoding is ISO-8859-1 * rename test * bump * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

Encoding is ISO-8859-1

8274e43

github-actions bot added the area/connectors Connector related issues label May 3, 2022

girarda commented May 3, 2022

View reviewed changes

girarda added 2 commits May 3, 2022 16:29

rename test

f18dea2

bump

a6c2133

github-actions bot added the area/documentation Improvements or additions to documentation label May 3, 2022

girarda marked this pull request as ready for review May 3, 2022 23:40

girarda requested review from brianjlai and marcosmarxm May 3, 2022 23:41

brianjlai approved these changes May 3, 2022

View reviewed changes

auto-bump connector version

233ee6e

octavia-squidington-iii temporarily deployed to more-secrets May 4, 2022 00:12 Inactive

marcosmarxm reviewed May 4, 2022

View reviewed changes

marcosmarxm approved these changes May 4, 2022

View reviewed changes

girarda merged commit 90162b6 into master May 4, 2022

girarda deleted the alex/salesforceEncoding branch May 4, 2022 02:23

octavia-squidington-iii mentioned this pull request May 4, 2022

Bump Airbyte version from 0.36.7-alpha to 0.36.8-alpha #12553

Merged

girarda mentioned this pull request May 4, 2022

source salesforce: use utf8 by default and iso as fallback #12576

Merged

38 tasks

suhomud pushed a commit that referenced this pull request May 23, 2022

Encoding is ISO-8859-1 (#12552)

d34d621

* Encoding is ISO-8859-1 * rename test * bump * auto-bump connector version Co-authored-by: Octavia Squidington III <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encoding is ISO-8859-1 #12552

Encoding is ISO-8859-1 #12552

girarda commented May 3, 2022 •

edited

Loading

girarda commented May 3, 2022 •

edited by github-actions bot

Loading

girarda May 3, 2022

brianjlai left a comment

girarda commented May 3, 2022 •

edited by github-actions bot

Loading

codecov bot commented May 4, 2022 •

edited

Loading

marcosmarxm May 4, 2022

girarda May 4, 2022

marcosmarxm left a comment

wissevrowl commented May 4, 2022



		def test():
		assert b"0\xe5".decode(SalesforceStream.encoding) == "0å"

Encoding is ISO-8859-1 #12552

Encoding is ISO-8859-1 #12552

Conversation

girarda commented May 3, 2022 • edited Loading

What

How

girarda commented May 3, 2022 • edited by github-actions bot Loading

girarda May 3, 2022

Choose a reason for hiding this comment

brianjlai left a comment

Choose a reason for hiding this comment

girarda commented May 3, 2022 • edited by github-actions bot Loading

codecov bot commented May 4, 2022 • edited Loading

Codecov Report

marcosmarxm May 4, 2022

Choose a reason for hiding this comment

girarda May 4, 2022

Choose a reason for hiding this comment

marcosmarxm left a comment

Choose a reason for hiding this comment

wissevrowl commented May 4, 2022

girarda commented May 3, 2022 •

edited

Loading

girarda commented May 3, 2022 •

edited by github-actions bot

Loading

girarda commented May 3, 2022 •

edited by github-actions bot

Loading

codecov bot commented May 4, 2022 •

edited

Loading