The requested stream BalanceTransactions was not found in the source (Shopify)

[hbd]

Environment

• Airbyte version: 0.35.62-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker (Compose)
• Source Connector and version: Shopify v0.1.36
• Destination Connector and version: S3 v0.2.3
• Severity: Medium
• Step where error happened: Sync job

Current Behavior

After creating a new Shopify source using credentials from a Shopify Development store, syncing to an S3 destination (Parquet), the connection reports requested streams are not available. However, these streams are available for other merchants with the same configuration.

Expected Behavior

The streams should be available.

Logs

LOG

2022-04-01 02:50:14 source > "The requested stream balance_transactions was not found in the source. Available streams: dict_keys(['orders', 'abandoned_checkouts', 'metafields', 'order_refunds', 'order_risks', 'tender_transactions', 'transactions', 'fulfillments', 'shop'])"

Steps to Reproduce

1. Create source using Shopify development store credentials
2. Sync to S3 (probably works with any other destination)
3. View logs for the sync

Are you willing to submit a PR?

Yes

[marcosmarxm]

Looks Shopify connector filters some stream based on the permission of your API key.

airbyte/airbyte-integrations/connectors/source-shopify/source_shopify/source.py

Lines 509 to 542 in 669e6ed

	permitted_streams = [
	stream
	for user_scope in user_scopes
	if user_scope["handle"] in SCOPES_MAPPING
	for stream in SCOPES_MAPPING.get(user_scope["handle"])
	] + always_permitted_streams
	# before adding stream to stream_instances list, please add it to SCOPES_MAPPING
	stream_instances = [
	Customers(config),
	Orders(config),
	DraftOrders(config),
	Products(config),
	AbandonedCheckouts(config),
	Metafields(config),
	CustomCollections(config),
	Collects(config),
	OrderRefunds(config),
	OrderRisks(config),
	TenderTransactions(config),
	Transactions(config),
	BalanceTransactions(config),
	Pages(config),
	PriceRules(config),
	DiscountCodes(config),
	Locations(config),
	InventoryItems(config),
	InventoryLevels(config),
	FulfillmentOrders(config),
	Fulfillments(config),
	Shop(config),
	]
	return [stream_instance for stream_instance in stream_instances if self.format_name(stream_instance.name) in permitted_streams]

Probably the connector needs to improve this workflow to not allow this to happen.

[hbd]

Agreed, best to prevent creation of the connection altogether if permissions are not available for the selected stream.

Alternatively, we could accept connection creation, then handle the streams for which permissions have not been met, and log that the stream is unavailable due to permission issues. This would allow at least the available streams to succeed.