Skip to content

Compactor silently fails to create No Compaction Mark when it has Read Only permissions to LTS #4709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
shybbko opened this issue Apr 7, 2022 · 1 comment
Closed

Compactor silently fails to create No Compaction Mark when it has Read Only permissions to LTS #4709

shybbko opened this issue Apr 7, 2022 · 1 comment
Labels
stale

Comments

@shybbko
Copy link

shybbko commented Apr 7, 2022

Describe the bug
I was testing 4d751f2 and noticed that if Compactor is run with Read Only permission for blocks Long Term Storage (in my case it was GCS bucket) the only time it gets reported in the logs is upon startup, when blocks_cleaner routine is called (see logs shared below).
Later, when Compactor wants to write something to LTS (in my case it was a no compaction mark) it fails, but does not report it in the logs neither stops the process. In the result it loops endlessly without a visible cause. I assume the similar problem might occur ie. when writing a deletion mark.

I think a change should be introduced to either:

  • make Compactor fail upon startup if it cannot write to LTS or
  • report to logs every time it fails to write to LTS (ie. upon trying to create no compaction / deletion mark) or
  • make it fail after it cannot write to LTS (later than on startup)

To Reproduce
Steps to reproduce the behavior:

  1. Start Cortex Compactor (version=1.12.0-rc.0, branch=master, revision=bb6b026)
  2. Make sure it has Read Only permission for blocks LTS
  3. Make sure there are out of order blocks in LTS and the skip_blocks_with_out_of_order_chunks_enabled is enabled
  4. Let it run
  5. It will loop over a single compaction without telling you there's a problem with writing to LTS (creating a no compaction mark)

Expected behavior
Upon failing to create a no compaction mark in LTS there is an error logged in logs and / or the process fails.

Environment:

  • Infrastructure: laptop
  • Deployment tool: docker

Storage Engine

  • Blocks

Additional Context
logs when run with RO permissions:

level=info ts=2022-04-07T12:34:07.430256Z caller=main.go:193 msg="Starting Cortex" version="(version=1.12.0-rc.0, branch=master, revision=bb6b026)"
(...)
level=info ts=2022-04-07T12:34:07.4334961Z caller=blocks_cleaner.go:139 component=cleaner msg="started blocks cleanup and maintenance"
level=info ts=2022-04-07T12:34:08.8837554Z caller=blocks_cleaner.go:294 component=cleaner org_id=fake msg="started blocks cleanup and maintenance"
level=warn ts=2022-04-07T12:34:13.5073365Z caller=updater.go:108 component=cleaner org_id=fake msg="skipped partial block when updating bucket index" block=01FVVEXMX7PQDSTZXZ4174HVS4
level=warn ts=2022-04-07T12:34:14.467466Z caller=blocks_cleaner.go:297 component=cleaner org_id=fake msg="failed blocks cleanup and maintenance" err="upload bucket index: googleapi: Error 403: REDACTED does not have storage.objects.create access to the Google Cloud Storage bucket., forbidden"
level=error ts=2022-04-07T12:34:14.4676933Z caller=blocks_cleaner.go:150 component=cleaner msg="failed to run blocks cleanup and maintenance" err="failed to delete blocks for user: fake: upload bucket index: googleapi: Error 403: REDACTED does not have storage.objects.create access to the Google Cloud Storage bucket., forbidden"
level=info ts=2022-04-07T12:34:14.4677394Z caller=compactor.go:549 component=compactor msg="discovering users from bucket"
level=info ts=2022-04-07T12:34:14.4678873Z caller=cortex.go:436 msg="Cortex started"
level=info ts=2022-04-07T12:34:14.6510999Z caller=compactor.go:556 component=compactor msg="discovered users from bucket" users=1
level=info ts=2022-04-07T12:34:14.8166841Z caller=compactor.go:598 component=compactor msg="starting compaction of user blocks" user=fake
level=info ts=2022-04-07T12:34:14.8193797Z caller=compact.go:1278 component=compactor org_id=fake msg="start sync of metas"
level=debug ts=2022-04-07T12:34:14.8195597Z caller=fetcher.go:323 component=compactor org_id=fake component=block.BaseFetcher msg="fetching meta data" concurrency=100
level=info ts=2022-04-07T12:36:13.4194719Z caller=fetcher.go:481 component=compactor org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1m58.5999712s duration_ms=118599 cached=16053 returned=16053 partial=1
level=info ts=2022-04-07T12:36:13.4195435Z caller=compact.go:1283 component=compactor org_id=fake msg="start of GC"
level=info ts=2022-04-07T12:36:19.8486281Z caller=compact.go:1306 component=compactor org_id=fake msg="start of compactions"
level=info ts=2022-04-07T12:36:19.9307411Z caller=compact.go:1002 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="compaction available and planned; downloading blocks" plan="[01FT707C81H893BQGEX4RSRTNN (min time: 1643047200000, max time: 1643054400000) 01FT70XSBJRBXDC0HYV98KY5NM (min time: 1643050042951, max time: 1643054400000) 01FT70YXNXYX9X6FKKYF64TR1D (min time: 1643050042951, max time: 1643054400000) 01FT72DJY9WFAAMR2HSDNC0GBA (min time: 1643051558029, max time: 1643054400000) 01FT72C11J77Q8Z8MZPXFA63CR (min time: 1643051558340, max time: 1643054400000) 01FT72DH72HWXVDNQHFCH5BDQM (min time: 1643051574777, max time: 1643054400000) 01FT72CFJFNBSPZX2SBXT6EBCT (min time: 1643051576146, max time: 1643054400000) 01FT72K0NTAPDBXRN65HKN5K1H (min time: 1643051748495, max time: 1643054400000) 01FT72JP8AMJMBHWC966B22YGE (min time: 1643051748495, max time: 1643054400000)]"
level=debug ts=2022-04-07T12:36:47.6351705Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT707C81H893BQGEX4RSRTNN/meta.json
level=debug ts=2022-04-07T12:38:42.0024544Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70XSBJRBXDC0HYV98KY5NM/meta.json
level=debug ts=2022-04-07T12:38:58.857239Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70YXNXYX9X6FKKYF64TR1D/meta.json
level=debug ts=2022-04-07T12:39:11.8828168Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT72DJY9WFAAMR2HSDNC0GBA/meta.json
level=debug ts=2022-04-07T12:39:14.1395463Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=debug ts=2022-04-07T12:39:14.1748006Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=error ts=2022-04-07T12:39:14.5997113Z caller=compactor.go:603 component=compactor msg="failed to compact user blocks" user=fake err="compaction: group 0@5679675083797525161: blocks with out-of-order chunks are dropped from compaction:  /tmp/cortex/compactor/compact/0@5679675083797525161/01FT72DJY9WFAAMR2HSDNC0GBA: 2/132124 series have an average of 1.000 out-of-order chunks: 2.000 of these are exact duplicates (in terms of data and time range)"
level=info ts=2022-04-07T12:40:13.240904Z caller=compactor.go:549 component=compactor msg="discovering users from bucket"
level=info ts=2022-04-07T12:40:13.4264793Z caller=compactor.go:556 component=compactor msg="discovered users from bucket" users=1
level=info ts=2022-04-07T12:40:13.5987182Z caller=compactor.go:598 component=compactor msg="starting compaction of user blocks" user=fake
level=info ts=2022-04-07T12:40:13.6005819Z caller=compact.go:1278 component=compactor org_id=fake msg="start sync of metas"
level=debug ts=2022-04-07T12:40:13.6012212Z caller=fetcher.go:323 component=compactor org_id=fake component=block.BaseFetcher msg="fetching meta data" concurrency=100
level=info ts=2022-04-07T12:41:44.1456847Z caller=fetcher.go:481 component=compactor org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1m30.5448389s duration_ms=90544 cached=16053 returned=16053 partial=1
level=info ts=2022-04-07T12:41:44.145819Z caller=compact.go:1283 component=compactor org_id=fake msg="start of GC"
level=info ts=2022-04-07T12:41:51.4004888Z caller=compact.go:1306 component=compactor org_id=fake msg="start of compactions"
level=info ts=2022-04-07T12:41:51.4842811Z caller=compact.go:1002 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="compaction available and planned; downloading blocks" plan="[01FT707C81H893BQGEX4RSRTNN (min time: 1643047200000, max time: 1643054400000) 01FT70YXNXYX9X6FKKYF64TR1D (min time: 1643050042951, max time: 1643054400000) 01FT70XSBJRBXDC0HYV98KY5NM (min time: 1643050042951, max time: 1643054400000) 01FT72DJY9WFAAMR2HSDNC0GBA (min time: 1643051558029, max time: 1643054400000) 01FT72C11J77Q8Z8MZPXFA63CR (min time: 1643051558340, max time: 1643054400000) 01FT72DH72HWXVDNQHFCH5BDQM (min time: 1643051574777, max time: 1643054400000) 01FT72CFJFNBSPZX2SBXT6EBCT (min time: 1643051576146, max time: 1643054400000) 01FT72K0NTAPDBXRN65HKN5K1H (min time: 1643051748495, max time: 1643054400000) 01FT72JP8AMJMBHWC966B22YGE (min time: 1643051748495, max time: 1643054400000)]"
level=debug ts=2022-04-07T12:42:18.282668Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT707C81H893BQGEX4RSRTNN/meta.json
level=debug ts=2022-04-07T12:44:14.7720932Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70YXNXYX9X6FKKYF64TR1D/meta.json
level=debug ts=2022-04-07T12:44:32.1317761Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70XSBJRBXDC0HYV98KY5NM/meta.json
level=debug ts=2022-04-07T12:44:45.37019Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT72DJY9WFAAMR2HSDNC0GBA/meta.json
level=debug ts=2022-04-07T12:44:47.598426Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=debug ts=2022-04-07T12:44:47.6371986Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=error ts=2022-04-07T12:44:48.0461042Z caller=compactor.go:603 component=compactor msg="failed to compact user blocks" user=fake err="compaction: group 0@5679675083797525161: blocks with out-of-order chunks are dropped from compaction:  /tmp/cortex/compactor/compact/0@5679675083797525161/01FT72DJY9WFAAMR2HSDNC0GBA: 2/132124 series have an average of 1.000 out-of-order chunks: 2.000 of these are exact duplicates (in terms of data and time range)"
level=info ts=2022-04-07T12:44:48.0476701Z caller=compactor.go:549 component=compactor msg="discovering users from bucket"
level=info ts=2022-04-07T12:44:48.2353732Z caller=compactor.go:556 component=compactor msg="discovered users from bucket" users=1
level=info ts=2022-04-07T12:44:48.4046448Z caller=compactor.go:598 component=compactor msg="starting compaction of user blocks" user=fake
(...)

logs when run with RW permissions:

level=info ts=2022-04-07T13:29:29.6471055Z caller=main.go:193 msg="Starting Cortex" version="(version=1.12.0-rc.0, branch=master, revision=bb6b026)"
(...)
level=info ts=2022-04-07T13:29:29.6579872Z caller=blocks_cleaner.go:139 component=cleaner msg="started blocks cleanup and maintenance"
level=info ts=2022-04-07T13:29:30.2773576Z caller=blocks_cleaner.go:294 component=cleaner org_id=fake msg="started blocks cleanup and maintenance"
level=info ts=2022-04-07T13:29:36.5298447Z caller=blocks_cleaner.go:299 component=cleaner org_id=fake msg="completed blocks cleanup and maintenance" duration=6.2524888s
level=info ts=2022-04-07T13:29:36.5302695Z caller=blocks_cleaner.go:143 component=cleaner msg="successfully completed blocks cleanup and maintenance"
level=info ts=2022-04-07T13:29:36.530465Z caller=compactor.go:549 component=compactor msg="discovering users from bucket"
level=info ts=2022-04-07T13:29:36.5308679Z caller=cortex.go:436 msg="Cortex started"
level=info ts=2022-04-07T13:29:36.695387Z caller=compactor.go:556 component=compactor msg="discovered users from bucket" users=1
level=info ts=2022-04-07T13:29:36.8617045Z caller=compactor.go:598 component=compactor msg="starting compaction of user blocks" user=fake
level=info ts=2022-04-07T13:29:36.8698958Z caller=compact.go:1278 component=compactor org_id=fake msg="start sync of metas"
level=debug ts=2022-04-07T13:29:36.8705178Z caller=fetcher.go:323 component=compactor org_id=fake component=block.BaseFetcher msg="fetching meta data" concurrency=100
level=info ts=2022-04-07T13:29:37.8814069Z caller=fetcher.go:481 component=compactor org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=1.0111858s duration_ms=1011 cached=9 returned=9 partial=0
level=info ts=2022-04-07T13:29:37.8815283Z caller=compact.go:1283 component=compactor org_id=fake msg="start of GC"
level=info ts=2022-04-07T13:29:37.8820476Z caller=compact.go:1306 component=compactor org_id=fake msg="start of compactions"
level=info ts=2022-04-07T13:29:37.883354Z caller=compact.go:1002 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="compaction available and planned; downloading blocks" plan="[01FT707C81H893BQGEX4RSRTNN (min time: 1643047200000, max time: 1643054400000) 01FT70YXNXYX9X6FKKYF64TR1D (min time: 1643050042951, max time: 1643054400000) 01FT70XSBJRBXDC0HYV98KY5NM (min time: 1643050042951, max time: 1643054400000) 01FT72DJY9WFAAMR2HSDNC0GBA (min time: 1643051558029, max time: 1643054400000) 01FT72C11J77Q8Z8MZPXFA63CR (min time: 1643051558340, max time: 1643054400000) 01FT72DH72HWXVDNQHFCH5BDQM (min time: 1643051574777, max time: 1643054400000) 01FT72CFJFNBSPZX2SBXT6EBCT (min time: 1643051576146, max time: 1643054400000) 01FT72JP8AMJMBHWC966B22YGE (min time: 1643051748495, max time: 1643054400000) 01FT72K0NTAPDBXRN65HKN5K1H (min time: 1643051748495, max time: 1643054400000)]"
level=debug ts=2022-04-07T13:30:02.6127351Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT707C81H893BQGEX4RSRTNN/meta.json
level=debug ts=2022-04-07T13:31:48.2011698Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70YXNXYX9X6FKKYF64TR1D/meta.json
level=debug ts=2022-04-07T13:32:03.912955Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70XSBJRBXDC0HYV98KY5NM/meta.json
level=debug ts=2022-04-07T13:32:17.2592199Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT72DJY9WFAAMR2HSDNC0GBA/meta.json
level=debug ts=2022-04-07T13:32:19.1268051Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=debug ts=2022-04-07T13:32:19.1690613Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=info ts=2022-04-07T13:32:19.5506106Z caller=block.go:404 component=compactor org_id=fake msg="block has been marked for no compaction" block=01FT72DJY9WFAAMR2HSDNC0GBA
level=info ts=2022-04-07T13:32:19.5516949Z caller=compact.go:1278 component=compactor org_id=fake msg="start sync of metas"
level=debug ts=2022-04-07T13:32:19.5520271Z caller=fetcher.go:323 component=compactor org_id=fake component=block.BaseFetcher msg="fetching meta data" concurrency=100
level=info ts=2022-04-07T13:32:20.5032778Z caller=fetcher.go:481 component=compactor org_id=fake component=block.BaseFetcher msg="successfully synchronized block metadata" duration=951.3879ms duration_ms=951 cached=9 returned=9 partial=0
level=info ts=2022-04-07T13:32:20.5034901Z caller=compact.go:1283 component=compactor org_id=fake msg="start of GC"
level=info ts=2022-04-07T13:32:20.5041959Z caller=compact.go:1306 component=compactor org_id=fake msg="start of compactions"
level=info ts=2022-04-07T13:32:20.5049993Z caller=compact.go:1002 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="compaction available and planned; downloading blocks" plan="[01FT707C81H893BQGEX4RSRTNN (min time: 1643047200000, max time: 1643054400000) 01FT70YXNXYX9X6FKKYF64TR1D (min time: 1643050042951, max time: 1643054400000) 01FT70XSBJRBXDC0HYV98KY5NM (min time: 1643050042951, max time: 1643054400000) 01FT72C11J77Q8Z8MZPXFA63CR (min time: 1643051558340, max time: 1643054400000) 01FT72DH72HWXVDNQHFCH5BDQM (min time: 1643051574777, max time: 1643054400000) 01FT72CFJFNBSPZX2SBXT6EBCT (min time: 1643051576146, max time: 1643054400000) 01FT72K0NTAPDBXRN65HKN5K1H (min time: 1643051748495, max time: 1643054400000) 01FT72JP8AMJMBHWC966B22YGE (min time: 1643051748495, max time: 1643054400000)]"
level=debug ts=2022-04-07T13:32:44.9014553Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT707C81H893BQGEX4RSRTNN/meta.json
level=debug ts=2022-04-07T13:34:30.0025815Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70YXNXYX9X6FKKYF64TR1D/meta.json
level=debug ts=2022-04-07T13:34:45.9169045Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT70XSBJRBXDC0HYV98KY5NM/meta.json
level=debug ts=2022-04-07T13:34:58.6321683Z caller=objstore.go:263 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="not downloading again because a provided path matches this one" file=01FT72C11J77Q8Z8MZPXFA63CR/meta.json
level=debug ts=2022-04-07T13:35:00.258157Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=debug ts=2022-04-07T13:35:00.2714224Z caller=index.go:331 component=compactor org_id=fake group="0@{__org_id__=\"fake\"}" groupKey=0@5679675083797525161 msg="found out of order series" labels="REDACTED"
level=info ts=2022-04-07T13:35:00.7732846Z caller=block.go:404 component=compactor org_id=fake msg="block has been marked for no compaction" block=01FT72C11J77Q8Z8MZPXFA63CR
level=info ts=2022-04-07T13:35:00.7736232Z caller=compact.go:1278 component=compactor org_id=fake msg="start sync of metas"
(...)

how docker is run:

docker run -d --name=cortex01 --network=cortex \
    -v $(pwd)/docker_run_config.yaml:/etc/config.yaml \
    -p 9001:9009 \
    cortexproject/cortex:master-bb6b026 \
    -config.file=/etc/config.yaml \
    -target=compactor \
    -log.level=debug \

docker_run_config.yaml:

auth_enabled: false

server:
  http_listen_port: 9009

  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: gzip

ingester:
  lifecycler:
    join_after: 0
    min_ready_duration: 0s
    final_sleep: 0s
    num_tokens: 512

    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

storage:
  engine: blocks

blocks_storage:
  tsdb:
    dir: /tmp/cortex/tsdb

  bucket_store:
    sync_dir: /tmp/cortex/tsdb-sync

  backend: gcs
  gcs:
    bucket_name: "redacted"
    service_account: |
        {
        redacted
        }

compactor:
  data_dir: /tmp/cortex/compactor
  sharding_ring:
    kvstore:
      store: inmemory
    wait_active_instance_timeout: 3m
  compaction_interval: 1m
  cleanup_interval: 12h
  block_deletion_marks_migration_enabled: false
  compaction_retries: 1
  block_sync_concurrency: 40
  meta_sync_concurrency: 100
  skip_blocks_with_out_of_order_chunks_enabled: true

frontend_worker:
  match_max_concurrent: true

ruler:
  enable_api: true
  enable_sharding: false

ruler_storage:
  backend: local
  local:
    directory: /tmp/cortex/rules

Related also to #4692 and #4677
@stale
Copy link

stale bot commented Jul 11, 2022

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jul 11, 2022
@stale stale bot closed this as completed Aug 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shybbko