Skip to content

Unreliable consumer builds after integrating producer on CI #126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
ahmednafei opened this issue May 2, 2022 · 33 comments
Closed

Unreliable consumer builds after integrating producer on CI #126

ahmednafei opened this issue May 2, 2022 · 33 comments

Comments

@ahmednafei
Copy link

Context
We have managed integrating XCRemoteCache with our project, using manual integration steps that had been added as part of project generation rules using BUCK.

Desired output had been achieved of saving around 70% of build time.

End to end integration tested for producer and consumer on different machines where cache was being pushed to remote cache server.

During testing we were setting our feature branch (for XCRemoteCache Integration) as main XCRemoteCache branch and running the following steps:

  1. Generate project with producer integration using BUCK
  2. Build project with Xcode and make sure cache is generated on caching server
  3. Run rm -rf ~/Library/Developer/Xcode/DerivedData and rm -rf ~/Library/Caches/XCRemoteCache; on machine that with be used as consumer.
  4. Generate project with consumer integration using BUCK
  5. Build project with Xcode

Output: On consumer builds 100% cache hit achieved and around 70% in build time is being saved.

Problem
After merging our feature branch to master, setting up master as main XC-RC branch, we integrated xc cache generation (building the project in producer mode) as part of our CI pipeline that runs immediately and automatically after merging any feature MR (merge request) to master.

By this setup for xc-rc cache generation, we started randomly seeing project build fails on local machines that builds the project with XC-RC integration in consumer mode.

On Xcode, we see errors like the following:
Showing All Messages /Users/anafei/git_tree/iosapp/Modules/Presentation/UI/SearchResultsList/Sources/ViewControllers/SearchResultsViewController.swift:21:12: Cannot load underlying module for 'ApplicationServices'

And on machine logs we see errors like:
(DMLScalarTypes) Local fingerprint Fingerprint(raw: "ddd713e9d85f977383c244f69787e55e", contextSpecific: "0fb9aac67d84f8e2ed5bc27bdc678489") does not match with remote one 9f8e9e9907f734f02b826d413d73905b.
And / Or:
Tons of 404 fails to fetch the cache meta files

The project build failure issue is some time being solved by just rebuilding the project again, and sometimes needed to pull latest master commit and clean the DerivedData, then rebuild again.

It worth to mention that, as our project is being extensively developed by multiple developers, so it happens that two merge requests or more are being merged to master with very small time difference and based on that CI pipeline would be running multiple times in parallel (one for each MR merged) and on different CI runner. So it happens that more than one producer job would be running in parallel on different CI runners, and generating cache for different commits at the same time.

Also as our CI pipeline for every MR takes considerable amount of time and cache generation is just one step of it, and also it runs after the MR is being merged to master so the following scenarios are expected too:

  1. When developer pulls latest master commit and try to build the project with consumer mode, the cache generation could not be started yet for that commit
  2. When developer pulls latest master commit and try to build the project with consumer mode, the xc remote cache generation for that commit could be running at the same time

Also worth to mention and as highlighted above, during integration test we were always building the app in producer mode first, wait until producer build finishes and cache generation completes, then start testing to build the project in consumer mode and this was always succeeding with 100% cache hits. Same happens in our real integration, if we waited until all CI jobs completes, pulled the latest master commit on local machine and tried to build the project with consumer mode. This also always succeeds with 100% cache hit.

Environment

  • XCRemoteCache: 0.3.11
  • HTTP cache server: Remote Cache Server
  • Xcode: 13.2.1 (13C100)
@cezarsignori
Copy link
Contributor

I would like to share an interesting example:

Last Friday EoD I pulled from master, regenerated the project and clean built the app successfully as RC consumer with 100% cache hit. Then, first thing in the morning today (Monday) I just hit rebuild and the build failed (no pull, no xcprepare).

The build error was about a missing header file (ImageNameConstants.h) in GeneratedResources module.

I found the missing header was referenced in the GeneratedResources.xcodeproj and the GeneratedResources.h umbrella header did import it. A symlink existed under buck-out but did not resolve, as the file in fact, does not exist. It doesn't exist because 3 commits prior to the one I pull and built on Friday migrated that code to Swift (now, there is a ImageNameConstants.swift instead).

So I regenerated the GeneratedResources project manually (just like it is automatically done when I regenerate the app workspace as I did on Friday). This process also triggered xcprepare calls.

Then re-building the app did not fail but I got a few cache misses due to a 404 error when fetching some file from GeneratedResources.

(GeneratedResources) Prebuild step failed with error: unsuccessfulResponse(status: 404)

I looked for the URL in the RC log. It shows the meta file URL and so I manually requested it. It resolved just fine. Then I saw the headers in the content were correct (no ImageNameConstants.h) and manually requested the zip file (fileKey) and it also resolved fine (only header present was -Swift.h).

Then I regenerated the entire workspace (calling xcprepare) and the build succeeded with the same cache misses and same 404 error for the same module.

I checked the server logs for 404 error on meta or zip file using the URL and all 3 requests I found were 200 responses.

I then deleted ~/Library/Developer/Xcode/DerivedData and re-built. This time, no 404 and 100% cache hit rate.

It is strange that the local state of GeneratedResource.xcodeproj in my machine did not break build on Friday but did today without any local changes.

And it is also strange that the very module who showed the problem had 404 errors on RC consumer side (without matching 404 errors on server log...maybe it failed a file other than meta or zip?) until I deleted DerivedData.

@cezarsignori
Copy link
Contributor

Here is a another example/use case:

I pulled from master, re-generated project, ran xcprepare.
I did not clean DerivedData nor Caches and tried to build the app.
In this case, as I had zero local changes and the producer build succeeded on CI, I expected a successful build with 100% cache hit.

But the build failed with 11 cache misses (and a lot of cache hits).

There were 4 build errors (Command CompileSwiftSources failed with a nonzero exit code). Only one with clear error message:

clang: error: no such file or directory: '/Users/csignori/Projects/iphone/Modules/Presentation/UseCases/DealsUIComponents/Sources/ViewControllers/ViewsAndModels/CampaignsInfo/BCDealsCampaignBottomSheetController.m'
clang: error: no input files

Looking at RC logs I found:

(DealsUIComponents) Prebuild step failed with error: missingFile(file:///Users/csignori/Projects/iphone/Modules/Presentation/UseCases/DealsUIComponents/Sources/ViewControllers/ViewsAndModels/CampaignsInfo/BCDealsCampaignBottomSheetController.h)

The BCDealsCampaignBottomSheetController.h is really not there (nor in the respective buck-out directory). But it shows in the meta (which I downloaded manually using the URL in the logs) file under the same location (and also under respective buck-out directory).

Issue 1: If on the current commit the source file is not there, why does it show up in the meta file?

The other 3 modules all failed due to a local fingerprint mismatch error. DealsUISwiftComponents mismatched directly and the others depended on a common module which mismatched.

(DealsUISwiftComponents)     Local fingerprint Fingerprint(raw: "08b368e2c0c66260f2b719aa17b28607", contextSpecific: "1bbc2585eb75943df45cc57a7b190d01") does not match with remote one be989aa9ef59ce4453a21fd0f2af098c.
2022-05-02 13:24:21.360 I  xcprebuild[87326:6479b2] (DealsUISwiftComponents) Remote cache cannot be used
...
2022-05-02 13:23:38.780 I  xcprebuild[85272:645596] (BasicUI)     Local fingerprint Fingerprint(raw: "ba37be22cf7dcbbf21bfe17573782fd6", contextSpecific: "15891428ea45643221f8c9094bd966d5") does not match with remote one ea26e6cba611d6408ec2b3296b0ead6f.
2022-05-02 13:23:38.780 I  xcprebuild[85272:645596] (BasicUI) Remote cache cannot be used

And all the 11 cache misses also either directly suffered the local fingerprint mismatch issue or a dependency did.

Issue 2: I need to delete the Caches directory to avoid this error.

I then deleted the Caches directy and re-built the app. I got a different set of build failures matching @Ahmed's:

/Users/csignori/Projects/iphone/Modules/Presentation/UI/SearchResultsList/Sources/ViewControllers/SearchResultsListViewController.swift:21:12: error: cannot load underlying module for 'ApplicationServices'
    import ApplicationServices

But ApplicationServices was a cache hit and built sucessfully! And the cache misses from before remained with Prebuild step failed with error: missingFile erros for a dependency that also missed cache but no reason listed in the log.

I then deleted both the Caches and DerivedData directories and re-built the app. Again the build failed with the same error: cannot load underlying module for 'ApplicationServices' error.

I then deleted both the Caches and DerivedData and buck-out directories. The Xcode workspace lost the destination options. So I re-generated the workspace without triggering xcprepare. This resulted in the same set of errors.

Issue 3: I believe this indicates the issue is not only with dirty local copy.

It is worth noting that during my testing other commits where made by other developers and pushed to master, triggering other producer runs.

So I deleted everything again, re-generated the workspace and called xcprepare.

This time, the build succeeded with 3 cache misses. The misses were because of:

2022-05-02 14:31:53.328 I  xcprebuild[25786:67b289] (PrivacyUI)     Local fingerprint Fingerprint(raw: "4d11fa244eac74dabfade1c1292ce4ee", contextSpecific: "931c853ebe4f19618e8b8273abb0ed2a") does not match with remote one 1206103db295239bf9a6d5b18f877cae.
2022-05-02 14:31:53.328 I  xcprebuild[25786:67b289] (PrivacyUI) Remote cache cannot be used

That is an error I also do not expect to see in this scenario, as everything was fetched from the cache server and there were no local changes.

Regarding the cache server, I don't see any errors there nor xcodebuild failures in the producer side (which would happen upon any server side issues). Unfortunately I don't have access to RC producer logs.

@polac24
Copy link
Collaborator

polac24 commented May 2, 2022 •
edited
Loading

Hello!
Glad to hear that you managed to integrate the project with Buck! Great success!

@cezarsignori , regarding So I regenerated the GeneratedResources project manually (...) This process also triggered xcprepare calls.. That suggests that you call xcprepare (without mark) several times, one time per .xcodeproj. That could be a reason of the described problem, if each .xcodeproj has unique remote_commit_file. Can you make sure all projects point to the same file on a disk (you could verify if more than one arc.rc is present (recursively) in the top project directory)?
I have a premonition that some projects use artifacts generated for different sha. Even at glance I don't see any obvious problems, it might lead to some undiscovered edge cases.

To analyze the original could you share more details about your setup:

  • on the producer side, how many times per commit do you call xcprepare mark ...? It seems you use .xcworkspace and maye the docs is not clear that that commit has to be called only one per combination: sha_commit/platform/architecture/Xcode/configuration. For instance if you support 1 Xcode version, simulator&hardware, have x86_64 and arm64 consumers, only iOS app (no WatchApp) and always Debug: then there should be max 3 invocations per commit : commit/iphonesimulator/arm64/1300/Debug, commit/iphonesimulator/x86_64/1300/Debug, commit/iphone/arm64/1300/Debug
  • Do you call xcparepare mark ... as a last step of the producer, when all artifacts are already uploaded to the server?
  • In the git, do you use merge commit, rebase or squash strategy? If you just rebase without squashing, then default cache_commit_history = 10 may be too low
  • That shouldn't be relevant, but maybe could help with troubleshooting: how does your backend server support duplicated PUT requests? Does it overwrite the file or leave the file intact? Assuming you first send PUT with contentA and then PUT with contentB, which will be returned in GET?
  • Do you have different projects integrated as git submodules?

If you CI generation takes significant amount of time, I can suggest having a secondary branch (e.g. main-with-cache) that is a bit delayed main but pointing the most recent commit sha with all artifacts uploaded. Then, you could recommend developers to use master-with-cache for the best dev experience.

@Ahmed
Copy link

Ahmed commented May 2, 2022 via email

@cezarsignori
Copy link
Contributor

Glad to hear that you managed to integrate the project with Buck! Great success!

Thanks! We are happy too! But we need these issues to go away in order to roll it out to others and validate impact.

That suggests that you call xcprepare (without mark) several times, one time per .xcodeproj. That could be a reason of the described problem...

I call (automatically after project generation) xcprepare on the consumer side once per repository.
That means xcprepare on the consumer side is called as many times as that happens (typically once right after pull).
Do you mean that calling it multiple times causes functional problems? Why?

Can you make sure all projects point to the same file on a disk (you could verify if more than one arc.rc is present (recursively) in the top project directory)?

Yes, there is a single arc.rc file per repo. When there is more than one xcodeproj file in the repo, their .rcinfo file refers to that arc.rc file (e.g. remote_commit_file: ../../../arc.rc)

On the producer side, how many times per commit do you call xcprepare mark ...?

mark is called once per repository (main repo and submodules) after a successsful build.

Therefore, it is possible that mark is called more than once per commit, but only for git submodules (none of the errors experienced so far are related to git submodules).

Currently, CI builds only 1 time as Producer for each commit in the main repo (Debug, arm64, Xcode 13.2.1).

Do you call xcparepare mark ... as a last step of the producer, when all artifacts are already uploaded to the server?

Yes. First I call xcodebuild then I call a custom script that calls mark once per repo.

In the git, do you use merge commit, rebase or squash strategy? If you just rebase without squashing, then default cache_commit_history = 10 may be too low.

Direct master commits are forbidden and Merge Requests are merged via merge commit with option to squash.

That shouldn't be relevant, but maybe could help with troubleshooting: how does your backend server support duplicated PUT requests?

I am using AmazonS3 Java Library and Bucket Versioning is disabled in AWS. So existing objects are overwritten.

Per official AWS documentation:

"If versioning is not enabled, this operation will overwrite an existing object with the same key; Amazon S3 will store the last write request. Amazon S3 does not provide object locking. If Amazon S3 receives multiple write requests for the same object nearly simultaneously, all of the objects might be stored. However, a single object will be stored with the final write request."

But does RC create duplicated filenames across different commits?

Do you have different projects integrated as git submodules?

Yes. 12.

If you CI generation takes significant amount of time, I can suggest having a secondary branch...

What do you mean? Are these issues expected and part of day to day when using RC?

@cezarsignori
Copy link
Contributor

I pulled commit ff954ae5 from master after CI producer job finished.
The build failed due to -Swift.h headers.
After deleting both Caches and DerivedData directories, the build succeeded with 100% cache hit rate.
After that, rebuilding the app is consistently successful, even calling xcprepare multiple times (while no other CI job run).

I started a new build of the same commit right when CI was running the second producer build. The build failed with same ApplicationServices error and after checking the RC logs I met with several Prebuild step failed with error: unsuccessfulResponse(status: 404) for different modules. I reproduced that three times in a row (before the 3rd attempt I called xcprepare).

3 commits were made (that I never pulled locally) in between my successful and failed consumer builds of the same commit. Those 3 commits were merged as a single MR.

I picked a random module that failed with a 404 and noticed the meta URL changed:

A) The successful build's meta URL: /rc/meta/ff954ae571d09151f570663db466e41b37061287-DSAssets-Debug-iphonesimulator-13C100-2d9d6c5e8a0c6beb6c7e3676196cf173.json

B) The failed build's meta URL: /rc/meta/ff954ae571d09151f570663db466e41b37061287-DSAssets-Debug-iphonesimulator-13C100-f3e580d5f4035f670832934055078cfa.json

And URL B does not resolve.

It seems like:

  1. As I did not pull any changes, even though another CI build happened on remote, the meta URL should remain A but does not.
  2. These RC errors break the consumer build. I think that should not happen. Cache should be disabled instead for that module.

When is the meta URL computed? Why is it changed? Maybe the problem is with how the fingerprint is computed?

@polac24
Copy link
Collaborator

polac24 commented May 2, 2022

Your flow seems legit and calling xcprepare (both with and without mark) multiple times is fine if you have git submodules (once per a submodule).

The case with different URL A and URL B is really weird, generating the meta URL is quite simple:

XCRemoteCache/Sources/XCRemoteCache/Network/URLBuilderImpl.swift

Line 78 in 0c2afc1

let filename = "\(commit)-\(targetName)-\(configuration)-\(platform)-\(xcode)-\(envFingerprint)"

where the envFingerprintis just a hash (MD5 at the moment) of several ENVs: ENVs listed here:

XCRemoteCache/Sources/XCRemoteCache/Fingerprint/EnvironmentFingerprint.swift

Lines 23 to 34 in 3b614c6

private static let defaultEnvFingerprintKeys = [
"GCC_PREPROCESSOR_DEFINITIONS",
"CLANG_COVERAGE_MAPPING",
"TARGET_NAME",
"CONFIGURATION",
"PLATFORM_NAME",
"XCODE_PRODUCT_BUILD_VERSION",
"CURRENT_PROJECT_VERSION",
"DYLIB_COMPATIBILITY_VERSION",
"DYLIB_CURRENT_VERSION",
"PRODUCT_MODULE_NAME",
"ARCHS",

with extra ENVs defined in custom_fingerprint_envs. version is hardcoded now to 5 (it is bumped if XCRemoteCache introduces a breaking change):

XCRemoteCache/Sources/XCRemoteCache/Config/XCRemoteCacheConfig.swift

Line 30 in 3b614c6

let schemaVersion = "5"

Can you compare these ENVs between successful and failing builds for the same commit?

@polac24
Copy link
Collaborator

polac24 commented May 3, 2022

I am still thinking about other reasons:

  1. Do you clean DerivedData (or call xcodebuild clean build... on the CI prior to the producer build? That is required.
  2. But does RC create duplicated filenames across different commits?. Yes, current implementation relentlessly makes PUT requests, even that file exists on the server (otherwise a prob HEAD would be required). It would be quite simple to add a mode that optimizes PUT requests and never overrides existing files.
  3. What do you mean? Are these issues expected and part of day to day when using RC?. Not at all. That is just a recommendation to have a higher hit rate if developers don't absolutely need the most up-to-date master. However, XCRemoteCache should still be able to selectively build locally only these targets that are not available yet.
  4. Can you compare meta files for the ApplicationServices target generated for the commit you built on Friday and the commit built on Monday morning? Do you see ImageNameConstants.h listed as a dependency only for the Friday one and not for the Monday one?

@cezarsignori
Copy link
Contributor

cezarsignori commented May 3, 2022 •
edited
Loading

I invested the whole morning in trying to reproduce the 404s until I did by re-generating the xcode projects and calling xcprepare without pulling and after other producer job rans.

In both successful and failed builds, Xcode logs show the same values for those environment variables. For example:

GCC_PREPROCESSOR_DEFINITIONS: 
CLANG_COVERAGE_MAPPING: 
TARGET_NAME: nanopb
CONFIGURATION: Debug
PLATFORM_NAME: iphonesimulator
XCODE_PRODUCT_BUILD_VERSION: 13C100
CURRENT_PROJECT_VERSION: 
DYLIB_COMPATIBILITY_VERSION: 
DYLIB_CURRENT_VERSION: 
PRODUCT_MODULE_NAME: nanopb
ARCHS: arm64

With the exception of one module that defines CURRENT_PROJECT_VERSION: 100 all others look like that.

In the failed build RC log I noticed both the 404 errors and local fingerprint fingerprint mismatch errors.

I picked one of the modules (nanopb) with 404 error and verified there were 2 meta requests in the build log for xcprebuild:

2022-05-03 11:17:22.806 I  xcprebuild[30279:817eb6] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-6ea319252a122f193b59ab30ab187422.json
....
2022-05-03 11:19:40.252 I  xcprebuild[37233:81edb3] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-262669077fa61748b755747279b40903.json

The nanopb target shows only one prebuild and one postbuild steps and the build log shows only one execution per step.

I though I maybe accumulated the RC log for two different builds, so I deleted everything and built again.

To my surprise I got a successful build with 100% cache hit rate AND again two meta requests for the nanopb target (and others too). One of them succeeded and the other did not:

2022-05-03 11:34:44.086 I  xcprebuild[61908:838bdc] (nanopb) Network request failed with unsuccessful code 404
2022-05-03 11:34:44.086 E  xcprebuild[61908:838bae] (nanopb) Prebuild step failed with error: unsuccessfulResponse(status: 404)

But on Xcode I see one execution per step.

Here is all the nanopb RC logs for the same successful app build (158.6s long):

2022-05-03 11:33:28.713 E  xcprebuild[53583:832ec2] (nanopb) Couldn't verify if should disable RC for 8b3c37c27dea9796aefde9a8a4f4c54805def119.
2022-05-03 11:33:28.720 I  xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products
2022-05-03 11:33:28.720 I  xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb
2022-05-03 11:33:28.720 I  xcprebuild[53583:832ec2] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone
2022-05-03 11:33:28.721 I  xcprebuild[53583:832ec2] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-6ea319252a122f193b59ab30ab187422.json
2022-05-03 11:33:31.335 I  xcprebuild[53583:832ec2] (nanopb) Downloading artifact to file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Intermediates.noindex/nanopb.build/Debug-iphonesimulator/nanopb.build/xccache/81d6178e5cfd5d84381c7d67ee340208.zip
2022-05-03 11:33:31.502 I  xcprebuild[53583:832ec2] (nanopb) Artifact unzipped to file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Intermediates.noindex/nanopb.build/Debug-iphonesimulator/nanopb.build/xccache/81d6178e5cfd5d84381c7d67ee340208
2022-05-03 11:33:37.289 I  xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Build/Products
2022-05-03 11:33:37.289 I  xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb
2022-05-03 11:33:37.289 I  xcpostbuild[54106:833b51] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone
2022-05-03 11:34:41.885 E  xcprebuild[61908:838bae] (nanopb) Couldn't verify if should disable RC for 8b3c37c27dea9796aefde9a8a4f4c54805def119.
2022-05-03 11:34:41.894 I  xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products
2022-05-03 11:34:41.894 I  xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb
2022-05-03 11:34:41.894 I  xcprebuild[61908:838bae] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone
2022-05-03 11:34:41.894 I  xcprebuild[61908:838bae] (nanopb) Making request https://{url}/rc/meta/8b3c37c27dea9796aefde9a8a4f4c54805def119-nanopb-Debug-iphonesimulator-13C100-262669077fa61748b755747279b40903.json
2022-05-03 11:34:44.086 I  xcprebuild[61908:838bdc] (nanopb) Network request failed with unsuccessful code 404
2022-05-03 11:34:44.086 E  xcprebuild[61908:838bae] (nanopb) Prebuild step failed with error: unsuccessfulResponse(status: 404)
2022-05-03 11:34:47.461 I  xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products/. Remapping: /Users/csignori/Library/Developer/Xcode/DerivedData/{app}-bkgyochpbymbjodvowqymxzemazn/Index/Build/Products
2022-05-03 11:34:47.461 I  xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/Pods/nanopb/. Remapping: /Users/csignori/Projects/iphone/Pods/nanopb
2022-05-03 11:34:47.461 I  xcpostbuild[62385:8392eb] (nanopb) Found url to remapp: file:///Users/csignori/Projects/iphone/. Remapping: /Users/csignori/Projects/iphone

@polac24
Copy link
Collaborator

polac24 commented May 3, 2022

The second request comes from the indexing build, more details here: #109 (comment)

Xcode 13 changed the indexing flow and Xcode in the background builds everything twice. If you are curious what happens there, unlock the Indexing build log summary:
Screenshot 2022-05-03 at 12 17 03

To confirm, you could disable indexing globally: defaults write com.apple.dt.Xcode IDEIndexEnable -bool YES and restart Xcode. There should be only one xcprebuild and one xcpostbuild per target.

@cezarsignori
Copy link
Contributor

Do you clean DerivedData (or call xcodebuild clean build... on the CI prior to the producer build? That is required.

Initially I didn't. But since yesterday I do. I suspected the uploaded content could be dirty. I still see the errors.

This is how I run the producer:

#!/bin/bash -e # This is interrupts the script in case of errors (so mark does not run if xcodebuild fails)

git clean -ffdx # This is because DerivedData stays in $WORKSPACE. See below.

rm -rf ~/Library/Caches/XCRemoteCache

./buckw clean

# This step generates xcodeproj with RC steps, and automaticaly creates .rcinfo and user.rcinfo files.
./buckw project GuestApp -c schema.no-swift-time-checks=true -c project.rc-mode=producer 

xcodebuild -workspace "$WORKSPACE/$XC_WORKSPACE_FILE" \
           -scheme "$WORKSPACE_SCHEME" \
           -configuration "Debug" \
           -destination "platform=iOS Simulator,name=iPhone 13 Pro,OS=15.2" \
           -derivedDataPath "$WORKSPACE/deriveddata" \
           -disableAutomaticPackageResolution | tee ./build_artifacts/xcodebuild.log | bundle exec xcpretty

# This will call RC mark once per repo
./Scripts/Utils/xcremotecache.py --mark 1

But does RC create duplicated filenames across different commits?. Yes.

I don't think there is a problem with overriding files, unless they have different content and same name.
In those cases, we probably want a different name (not to skip a PUT call).
So to confirm, do you mean RC can generate the same file name for files with different content (e.g. because different commit)?
I think that could be an issue.

Can you compare meta files for the ApplicationServices target generated for the commit you built on Friday and the commit built on Monday morning? Do you see ImageNameConstants.h listed as a dependency only for the Friday one and not for the Monday one?

Unfortunately not. I don't have the files and both builds ran on the same commit. I could rebuild that commit and see what I get.
But it seems like we should park that one for now and focus on the multiple request and fingerprint difference issue. Perhaps on the files with different content but same name too. Those are much more common and impactful.

@cezarsignori
Copy link
Contributor

Confirmed 404 error is due to Indexing.
Turns out indexing builds with ARCHS=x86_64.

export ARCHS\=x86_64
export ARCHS_STANDARD\=arm64\ x86_64
export ARCHS_STANDARD_32_64_BIT\=arm64\ i386\ x86_64
export ARCHS_STANDARD_32_BIT\=i386
export ARCHS_STANDARD_64_BIT\=arm64\ x86_64
export ARCHS_STANDARD_INCLUDING_64_BIT\=arm64\ x86_64
export ARCHS_UNIVERSAL_IPHONE_OS\=arm64\ i386\ x86_64
export NATIVE_ARCH\=arm64
export NATIVE_ARCH_32_BIT\=arm
export NATIVE_ARCH_64_BIT\=arm64
export NATIVE_ARCH_ACTUAL\=arm64
export PLATFORM_PREFERRED_ARCH\=x86_64
export VALID_ARCHS\=arm64\ arm64e\ i386\ x86_64
export XCRC_PLATFORM_PREFERRED_ARCH\=x86_64
export arch\=undefined_arch
export CURRENT_ARCH\=undefined_arch

Whereas a regular build is:

export ARCHS\=arm64
export ARCHS_STANDARD\=arm64\ x86_64
export ARCHS_STANDARD_32_64_BIT\=arm64\ i386\ x86_64
export ARCHS_STANDARD_32_BIT\=i386
export ARCHS_STANDARD_64_BIT\=arm64\ x86_64
export ARCHS_STANDARD_INCLUDING_64_BIT\=arm64\ x86_64
export ARCHS_UNIVERSAL_IPHONE_OS\=arm64\ i386\ x86_64
export NATIVE_ARCH\=arm64
export NATIVE_ARCH_32_BIT\=arm
export NATIVE_ARCH_64_BIT\=arm64
export NATIVE_ARCH_ACTUAL\=arm64
export PLATFORM_PREFERRED_ARCH\=x86_64
export VALID_ARCHS\=arm64\ arm64e\ i386\ x86_64
export XCRC_PLATFORM_PREFERRED_ARCH\=arm64
export arch\=undefined_arch
export CURRENT_ARCH\=undefined_arch

XCRC_PLATFORM_PREFERRED_ARCH is set as "XCRC_PLATFORM_PREFERRED_ARCH": "$(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH):dir:standardizepath:file:default=arm64)",.

But for Indexing builds $(LINK_FILE_LIST_$(CURRENT_VARIANT)_$(PLATFORM_PREFERRED_ARCH) resolves as LINK_FILE_LIST_normal_x86_64 while for regular builds it resolves to LINK_FILE_LIST_normal_arm64.

Strange thing is that PLATFORM_PREFERRED_ARCH is x86_64 in both cases at the times I can print it.

I'm investigating why and how to change that.

Could indexing builds be somehow breaking regular builds? If that's not the case, it might be all the issues I saw on logs so far are just noise (not the actual problem that breaks the build). I am going to also run a few tests with Indexing disabled.

@cezarsignori
Copy link
Contributor

After disabling indexing I was able to build sucessfully the commit I was at consistently (even after regenerating the project and calling xcprepare) as expected.

Then I pulled from master, regenerated the project, called xcprepare, opened Xcode and hit build (without cleaning up previous build files).

Like before, the build failed with a module (SearchResultsList) not being able to load underlying module for "ApplicationServices". This error is shown for several files in SearchResultsList always pointing to import ApplicationServices line. No additional context is available.

In the Xcode build log I see ApplicationServices built successfully with [RC] Cached build for ApplicationServices target message.

RC logs show (SearchResultsList) Prebuild step failed with error: missingFile for a dependency which failed with Local fingerprint Fingerprint(raw: "83a05da6423b11af6257225b1c429a87", contextSpecific: "7c6acec80ba61a984ba01d9784cca64f") does not match with remote one ce81524e4068217e579b50efdf586364..

The dependency with local fingerprint mismatch error had the expected values for defaultEnvFingerprintKeys on the consumer side.

In this case, I did not expect 100% cache hit rate as not all commits I fetched had been built by CI in producer mode, but I also don't expect a build failure as outcome.

Since ApplicationServices built fine and the issue with SearchResultsList is with a different dependency that cached missed, I don't know why Xcode shows the ApplicationServices could not be found error.

Also I am not sure if the local fingerprint mismatch error is the error I should expect to see in RC logs. The dependency in question was changed in one of those commits CI didn't pick up yet.. but from your (@polac24's) message should that error only show up when producer had different defaultEnvFingerprintKeys than consumer? (provided same RC binaries are used).

PS: Haven't found how to fix Indexing yet.

@polac24
Copy link
Collaborator

polac24 commented May 3, 2022

So to confirm, do you mean RC can generate the same file name for files with different content (e.g. because different commit)?

No, artifact filename is unique for the content so all target builds with the same filename are equivalent. Of course, assuming we don't include some input files or ENVs in the fingerprint hash. My initial concern was regarding the very short period of time when PUT overwrites a file. I read Amazon's docs again and they only mention that some content will be always served.

should that error only show up when producer had different (...)

No, there is something off with ApplicationServices, because XCRemoteCache uses remotely available artifacts with best-effort. If nothing is available yet, a locally built product is fully compatible with the rest of the XCRemoteCache flow. Sidenote: One of our requirements was to be resilient in case of disconnected VPN. If a connection comes back, XCRemoteCache should continue using remote artifacts, even some of its dependencies were compiled locally.

I think we should focus on ApplicationServices error. Seems that its product (.a, .swiftmodule etc.) are not correctly constructed from the artifact.

  1. Is it an ObjC, Swift or mixed target?
  2. If .modulemap is used, how is it created? Is it auto-generated by Xcode or commited-in to the repo?
  3. (Extra) I would inspect the content of the artifact .zip. The simples way is to trigger an error and inspect the unzipped content, e.g. in ..../DerivedData/Project/Build/Intermediates.noindex/Project.build/Debug-iphonesimulator/ApplicationServices.build/xccache/active. Can you see there .swiftmoduleand include dirs? What files are available there? I would also review meta's dependencies field in some_hash.json. Does the list makes sense - as the name suggests, the content of these files are used to generate a fingerprint that identifies an artifact.

@cezarsignori
Copy link
Contributor

Is it an ObjC, Swift or mixed target?

Mixed.

If .modulemap is used, how is it created? Is it auto-generated by Xcode or commited-in to the repo?

The .modulemap is auto-generated when buck generates the Xcode project.

It looks like this:

module ApplicationServices {
    umbrella header "ApplicationServices.h"

    export *
    module * { export * }
}

module ApplicationServices.Swift {
    header "ApplicationServices-Swift.h"
    requires objc
}

And the objc.modulemap file looks like:

module ApplicationServices {
    umbrella header "ApplicationServices.h"

    export *
    module * { export * }
}

module ApplicationServices.__Swift {
    exclude header "ApplicationServices-Swift.h"
}

(Extra) I would inspect the content of the artifact .zip.

I fetched the meta and file zip from a previous log where the error happened. In the zip file I find:

Screenshot 2022-05-03 at 17 56 06

And the meta.json field makes sense:

{"inputs":[],"generationCommit":"1e3df9044505d06a3a859e45428831af75423c88","pluginsKeys":{},"targetName":"ApplicationServices","platform":"iphonesimulator","configuration":"Debug","rawFingerprint":"01aa667906554640e3790c60358b33d1","fileKey":"4c6091b5e2e6151cc1dc85e89f1e7038","dependencies":["$(SRCROOT)\/ApplicationServices\/App Updates\/BCAppGetUpdateService.h","$(SRCROOT)\/ApplicationServices\/App Updates\/BCAppGetUpdateService.m","$(SRCROOT)\/ApplicationServices\/App Updates\/BCGetAppUpdateHandlers.h","$(SRCROOT)\/ApplicationServices\/App Updates\/BCGetAppUpdateHandlers.m","$(SRCROOT)\/ApplicationServices\/BCCreditCardConfigurator.h","$(SRCROOT)\/ApplicationServices\/BCCreditCardConfigurator.m","$(SRCROOT)\/ApplicationServices\/BCDataModelsInitializing.h","$(SRCROOT)\/ApplicationServices\/BCDataModelsInitializing.m","$(SRCROOT)\/ApplicationServices\/BCNetworkSecurityCheckService.h","$(SRCROOT)\/ApplicationServices\/BCNetworkSecurityCheckService.m","$(SRCROOT)\/ApplicationServices\/BCPropertyDataProviderImplementation.h","$(SRCROOT)\/ApplicationServices\/BCPropertyDataProviderImplementation.m","$(SRCROOT)\/ApplicationServices\/BCWechatController.h","$(SRCROOT)\/ApplicationServices\/BCWechatController.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReport.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReport.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReportAttachment.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReportAttachment.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReporter.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCBugReporter.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenRecorder.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenRecorder.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenshotTool.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/BCScreenshotTool.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCBugSubmitter.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCBugSubmitter.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCJSONSubmittedBug.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/Networking\/BCJSONSubmittedBug.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BCBugReporterViewController.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BCBugReporterViewController.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationArrowView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationArrowView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBlurView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBlurView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBoxView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationBoxView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKAnnotationView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKCheckerboardView.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKCheckerboardView.m","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKScreenshotViewController.h","$(SRCROOT)\/ApplicationServices\/Core\/BugReporter\/UI\/BugshotAnnotation\/JBSKScreenshotViewController.m","$(SRCROOT)\/ApplicationServices\/Core\/DebugViewControllerFactory.h","$(SRCROOT)\/ApplicationServices\/Core\/DebugViewControllerFactory.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainApplicationContext.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainApplicationContext.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainExperimentLoader.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/BCMainExperimentLoader.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ETExperimentToolConnector.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ETExperimentToolConnector.m","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ExperimentationObserver.h","$(SRCROOT)\/ApplicationServices\/Core\/Experimentation\/ExperimentationObserver.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizationInitializer.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizationInitializer.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizeNumberFormattingHelper.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/BCLocalizeNumberFormattingHelper.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BCI18NFactory.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BCI18NFactory.m","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BKCTranslationABTesting.h","$(SRCROOT)\/ApplicationServices\/Core\/Localization\/I18N\/BKCTranslationABTesting.m","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices.h","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices.m","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProvider+CommonServices_Private.h","$(SRCROOT)\/ApplicationServices\/Core\/Service Provider\/BCServiceProviderInjectable.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCApplicationDevelopmentServices.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCApplicationDevelopmentServices.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCRemoveUnusedDataService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCRemoveUnusedDataService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSessionTimeMeasurer.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSessionTimeMeasurer.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSettings.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/BCSettings.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/CrashReporter\/BCCrashlyticsLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/CrashReporter\/BCCrashlyticsLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogConfiguration.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogConfiguration.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLoggingService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCDebugLoggingService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCLoggingService.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCLoggingService.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCMutableDebugLogConfiguration.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCMutableDebugLogConfiguration.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCTTYDebugLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCTTYDebugLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCUITestFileLogger.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Log\/BCUITestFileLogger.m","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Screenshotter.h","$(SRCROOT)\/ApplicationServices\/Core\/Support\/Screenshotter.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProvider.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProviderRegistry.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Authentication Providers\/BCAuthenticationProviderRegistry.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProviderRegistry.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCPaymentProviderRegistry.m","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCWechatPaymentProvider.h","$(SRCROOT)\/ApplicationServices\/Core\/User\/Payments Provider\/BCWechatPaymentProvider.m","$(SRCROOT)\/ApplicationServices\/CountryCode+Flags.h","$(SRCROOT)\/ApplicationServices\/CountryCode+Flags.m","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFabricService.h","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFabricService.m","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFirebaseSessionExperiments.h","$(SRCROOT)\/ApplicationServices\/Fabric\/BCFirebaseSessionExperiments.m","$(SRCROOT)\/ApplicationServices\/Network\/Requests\/UpdateDeviceInfoRequestConfig+Defaults.swift","$(SRCROOT)\/ApplicationServices\/Network\/Webcalls\/BCMobileBackendServiceContext.h","$(SRCROOT)\/ApplicationServices\/Network\/Webcalls\/BCMobileBackendServiceContext.m","$(SRCROOT)\/ApplicationServices\/PayInSDK\/PayInSDKBuilder.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInABTestingImp.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInLocalizer.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInfoLoader.swift","$(SRCROOT)\/ApplicationServices\/PayInSDK\/Private\/PayInfoPersistenceProviderImpl.swift","$(SRCROOT)\/ApplicationServices\/PaymentComponent\/PaymentComponentBuilder.h","$(SRCROOT)\/ApplicationServices\/PaymentComponent\/PaymentComponentBuilder.m","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneConfirmationService.h","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneConfirmationService.m","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneService.h","$(SRCROOT)\/ApplicationServices\/Phone\/BCPhoneService.m","$(SRCROOT)\/ApplicationServices\/Phone\/BKPJSONStatusWithError.h","$(SRCROOT)\/ApplicationServices\/Phone\/BKPJSONStatusWithError.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCServiceProvider+Routing.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCServiceProvider+Routing.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerProxy.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerProxy.m","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerSource.h","$(SRCROOT)\/ApplicationServices\/Routing\/BCUIOpenerSource.m","$(SRCROOT)\/ApplicationServices\/Routing\/ServiceProvider+Routing.swift","$(SRCROOT)\/ApplicationServices\/Routing\/UIBaseOpener.h","$(SRCROOT)\/ApplicationServices\/Routing\/UIBaseOpener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/BCAssertionService.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/BCAssertionService.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCAlertAssertionListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCAlertAssertionListener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCBCGoalAssertListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCBCGoalAssertListener.m","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCFabricAssertionListener.h","$(SRCROOT)\/ApplicationServices\/Support\/Assertion\/Listeners\/BCFabricAssertionListener.m","$(SRCROOT)\/DeallocDebugging\/UIViewController+DeallocDebugging.h","$(SRCROOT)\/DeallocDebugging\/UIViewController+DeallocDebugging.m","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/6tEQmabZZE-pub\/StartupExperiments\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/9aIHa5GMKK-pub\/Core\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/9aIHa5GMKK-pub\/Core\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/A8iAWHPFeI-pub\/Localization\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/Gze0c3Q_xF-pub\/Analytics\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/Gze0c3Q_xF-pub\/Analytics\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cF8V00mifi-pub\/SwiftUICore\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cg7NxCVKml-pub\/Networking\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/cg7NxCVKml-pub\/Networking\/objc.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/gBnVtj5naS-pub\/Persistence\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/CoreModules\/buck-out\/gen\/_p\/tR9MA1CY0w-pub\/DependencyInjection\/module.modulemap","$(PROJECT_ROOT)\/.com\/Modules\/Platform\/Bdux\/buck-out\/gen\/_p\/CyhVfx8_xO-pub\/Bdux\/module.modulemap","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerCrossPromotionHelper.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerDeepLink.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerDeepLinkResult.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerLib.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerLinkGenerator.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Headers\/AppsFlyerShareInviteHelper.h","$(PROJECT_ROOT)\/Pods\/AppsFlyerFramework\/AppsFlyerLib.xcframework\/ios-arm64_i386_x86_64-simulator\/AppsFlyerLib.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/FirebaseAnalytics\/Frameworks\/FirebaseAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/FirebaseAnalytics.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAI.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIDictionaryBuilder.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceFields.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceProduct.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommerceProductAction.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIEcommercePromotion.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAIFields.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAILogger.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAITrackedViewController.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GAITracker.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Headers\/GoogleAnalytics-umbrella.h","$(PROJECT_ROOT)\/Pods\/GoogleAnalytics\/Frameworks\/GoogleAnalytics.xcframework\/ios-arm64_i386_x86_64-simulator\/GoogleAnalytics.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/OTPublishersHeadlessSDK-Swift.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/OTPublishersHeadlessSDK.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Headers\/TCF2Encoder.h","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Modules\/OTPublishersHeadlessSDK.swiftmodule\/arm64-apple-ios-simulator.swiftinterface","$(PROJECT_ROOT)\/Pods\/OTPublishersHeadlessSDK\/OTPublishersHeadlessSDK.xcframework\/ios-arm64_x86_64-simulator\/OTPublishersHeadlessSDK.framework\/Modules\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/0dRqK4SfsV-pub\/Promises\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/15TFc8Szm6-pub\/FLEX\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/2fbdIKI0gS-pub\/FBSDKCoreKit\/objc.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/CQw1YVPsxj-pub\/Identity\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/OZ9vhJNhFI-pub\/Valet\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/SGT46c7cUG-pub\/GoogleUtilities\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/lmGQiP_3Pb-pub\/UI\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/lmGQiP_3Pb-pub\/UI\/objc.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/vB7q01z9MO-pub\/TSMessages\/module.modulemap","$(PROJECT_ROOT)\/Pods\/buck-out\/gen\/_p\/y7WPAZHCFl-pub\/AFNetworking\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/EMMODyN0oP-pub\/PerformanceSuite\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/FjZPYUHybM-pub\/PrivacyUI\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/XrDcAievMX-pub\/CocoaHTTPServer2\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/cZAo5JftXV-pub\/Privacy\/module.modulemap","$(PROJECT_ROOT)\/Submodules\/buck-out\/gen\/_p\/nek6SbMbef-pub\/ABTesting\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/0OKxluvWCI-pub\/LocationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/1dvFki_alS-pub\/PerformanceTrackingServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/1dvFki_alS-pub\/PerformanceTrackingServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/2dzer-8x1e-pub\/Pricing\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/2vVGOTBOsy-pub\/MarketplaceAppsUI_Deprecated\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/3cAxWrNmBX-pub\/BKX\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/5x3KPSaE1B-pub\/Messaging\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/7uZ55b6nBZ-pub\/PoliciesDataModel\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/7uZ55b6nBZ-pub\/PoliciesDataModel\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/8o-TgZBMsX-pub\/BasicUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/C_i4L1yqQR-pub\/PostDataModel\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/Eyfl32kr0X-pub\/AppStartsServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/FI4P-wg1FM-pub\/LegacyUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/FNCOEm0EEp-pub\/SearchDestinationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAlertAssertionListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAppGetUpdateService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCApplicationDevelopmentServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCApplicationLaunchMethod.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAuthenticationProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAuthenticationProviderRegistry.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBCGoalAssertListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCAssertionService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCI18NFactory.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReport.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReportAttachment.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReporter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugReporterViewController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCBugSubmitter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCCrashlyticsLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCCreditCardConfigurator.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDataModelsInitializing.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLogConfiguration.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCDebugLoggingService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFabricAssertionListener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFabricService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCFirebaseSessionExperiments.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCGetAppUpdateHandlers.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCJSONSubmittedBug.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLocalizationInitializer.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLocalizeNumberFormattingHelper.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCLoggingService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMainApplicationContext.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMainExperimentLoader.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMobileBackendServiceContext.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCMutableDebugLogConfiguration.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCNetworkSecurityCheckService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPaymentProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPaymentProviderRegistry.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPhoneConfirmationService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPhoneService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCPropertyDataProviderImplementation.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCRemoveUnusedDataService.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCScreenRecorder.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCScreenshotTool.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+CommonServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+CommonServices_Private.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProvider+Routing.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCServiceProviderInjectable.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCSessionTimeMeasurer.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCSettings.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCTTYDebugLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUIOpenerProxy.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUIOpenerSource.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCUITestFileLogger.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCWechatController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BCWechatPaymentProvider.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BKCTranslationABTesting.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/BKPJSONStatusWithError.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ApplicationServices.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/CountryCode+Flags.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/DebugViewControllerFactory.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ETExperimentToolConnector.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/ExperimentationObserver.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationArrowView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationBlurView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationBoxView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKAnnotationView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKCheckerboardView.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/JBSKScreenshotViewController.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/PaymentComponentBuilder.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/Screenshotter.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/UIBaseOpener.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/UIViewController+DeallocDebugging.h","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/JbG58Qi8mg-pub\/ApplicationServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/KKK0l284C6-pub\/PropertyResponses\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/PxhODn6zhP-pub\/C360TrackerService\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/Y9ZCi6HS2l-pub\/FilteringServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/_BVlEto_GS-pub\/BEDrivenComponents\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/eXDT_yFxGS-pub\/DataModels\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/eXDT_yFxGS-pub\/DataModels\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/fba5cLfzYh-pub\/EMKUI\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/i1btfxAVkF-pub\/NotificationCenterService\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/j2Aczs_jbP-pub\/WishlistUIExperiments\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/kYucPv7jLf-pub\/BCNotifications\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/nWampcnrgg-pub\/PaymentDependencies\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/opjNULo7Dc-pub\/GeneratedResources\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/s-r4uCF9vZ-pub\/Utils\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/s-r4uCF9vZ-pub\/Utils\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/vzjHohEuGE-pub\/SegmentsServices\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/vzjHohEuGE-pub\/SegmentsServices\/objc.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/wdutdtfu5K-pub\/DirectionsSheet\/module.modulemap","$(PROJECT_ROOT)\/buck-out\/gen\/_p\/yhW2CPbkdY-pub\/TransportDiscovery\/module.modulemap","$(BUILD_DIR)\/Debug-iphonesimulator\/Apollo.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloAPI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloCombine.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/ApolloUtils.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/BUIAssets.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/BUIFoundations.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Blitzen.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Analytics.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Core.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Networking.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Privacy.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/StartupExperiments.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/UI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DMLClient.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DSAssets.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DataModels.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/DependencyInjection.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/FilteringServices.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/GeneratedResources.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/NetworkServices.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PayInSDK.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PoliciesDataModel.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/PostDataModel.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Pricing.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/Promises.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5","$(BUILD_DIR)\/Debug-iphonesimulator\/SwiftBUI.swiftmodule\/arm64-apple-ios-simulator.swiftmodule.md5"],"xcode":"13C100"}

@polac24
Copy link
Collaborator

polac24 commented May 3, 2022

The modulemap format you posted suggests that ApplicationServices is a framework built using Xcode. Otherwise the zip artifact wouldn't be created.
In general, when a framework is created in Xcode, Xcode creates some intermediate module maps and passes their paths to both compilers using virtualFileSystem (-ivfsoverlay). These modulemaps, unextended-module.modulemap and module.modulemap, have very similar format to your .modulemap and objc.modulemap. I didn't get why buck generates them. Does Xcode consume these files? If so, which build setting is used for that, MODULEMAP_FILE? Is the target compiled partially in buck and partially in Xcode?

Suggested the next step

When a build is failing with a missing module, can you copy the entire directory ApplicationServices.framework in the DerviedData's Product dir and compare its content after introducing a dummy change in the ApplicationServices target (e.g. a new line)? Obviously a binary file will differ but module.modulemap, PPP-Swift.h (and maybe other .h files in Headers dir) should be identical. *.swiftmodule files structure should be the same, except .swiftsourceinfo.

Sample .framework content:
Screenshot 2022-05-03 at 23 03 24

@cezarsignori
Copy link
Contributor

ApplicationServices is not built as a .framework but as a .a library. In DerivedData's Product dir, ApplicationServices is found as ApplicationServices.swiftmodule and ApplicationServices.a.

Anyways, the ApplicationServices error now seems to be resolved. Haven't seen it anymore. The SearchResultsList module was not declaring ApplicationServices as a dependency and was imported under a canImport condition. Removing that and declaring the dependency seems to have fixed the build failure.

With that out of the way and Indexing disabled, there is less noise.

Now, I can confirm that checking out to a specific commit CI already produced does not consistently result in 100% cache hit rate. This is what I am experiencing:

  1. I checked out to e717bd30. Verified CI job finished successfully and that it was the last RC CI job run. Deleted everything and regenerated the project (automatically triggering xcprepare). The build succeeded with 100% cache hit rate.
  2. Then I repeated step 1 but checked out to b8768420 instead. This was the commit with the SearchResultsList fix. The build succeeded with many cache misses. The reason were fingerprint mismatch errors. I see another RC CI job was running at the time for 8263d68c.
  3. Then I repeated step 1 but checked out to 8263d68c instead. This time I got one cache miss for PrivacyUI (and reverse dependencies). PrivacyUI failed due to a fingerprint mismatch error.
  4. Then I pulled from master and checked out 4dde5913. Again, I verified the CI job finished successfully for that commit and that it was the last RC CI job run. This time I did not delete anything locally. Just regenerated the project (automatically triggering xcprepare). Again I got many cache misses due to modules with high number of reverse dependencies failing with a fingerprint mismatch error.

My expectation is that whenever I check out to a commit in master that was successfully produced by CI, as long as the commit is within the last 10 limit and I have no local changes, with or without a local DerivedData and Caches directory, I should always get 100% cache hit rate when building. Is that correct?

Unfortunately I can't call log show --predicate 'sender BEGINSWITH "xc"' --style compact --info --debug due to lack of permissions on CI, so I don't have Producer logs for every commit. But I can tell all these commits built successfully (indicating no RC networking errors as those break producer builds).

When running the same CI commands in my local machine and capturing both Xcode and RC logs I did not find any defaultEnvFingerprintKeys with different values. I will do some more testing mimic-ing the CI behaviour across commits to see if I can reproduce.

Any other ideas?

@polac24
Copy link
Collaborator

polac24 commented May 4, 2022

My expectation is that whenever I check out to a commit in master that was successfully produced by CI, as long as the commit is within the last 10 limit and I have no local changes, with or without a local DerivedData and Caches directory, I should always get 100% cache hit rate when building. Is that correct?

That is correct. Moreover, there is no 10 commits limit. It is enough that CI has ever generated artifacts for that commit. Of course, assuming you don't have any eviction policy on the HTTP server.

I recall such inconsistency when code generation is not deterministic. If you have some .swift, .m, .h (or anything else) sources generated on-fly, their char-level difference can introduce random cache miss every now and then.
To troubleshoot it faster, there is an extra fields rawFingerprint in meta .json that is generated only based on all sources content. Contrary to fileKey, it doesn't interleave any ENVs like ARCHS, XCODE_VERSION etc. so locally you can quicker say if a miss is caused by a source file.

@ahmednafei
Copy link
Author

ahmednafei commented May 4, 2022 •
edited
Loading

Hey Bartosz,

We have found out one of issues that was causing fingerprint mismatch. Our main application repo was pointing to a commit that is not on master of one of our sub-repos Privacy.

merging changes of Privacy sub-repo to master and updating the main app repo to point to latest master commit of Privacy sub-repo had fixed the mismatch issue and cache misses with PrivacyUI module.

P.S. We are still investigating and validating other incidents of weird cache misses and fingerprint mismatch.

Although now, we are noting an issue related to build time for app in cases of cache misses happens.

For example, pulling latest commit on master of main application repo, and trying to build it, before the producer_ci_job completes for that commit, this is expected to cause cache misses for modules touched in that latest commit and all reverse dependencies. I can see this is normal, but the issue is with build time itself. it sometimes exceeds by far (1.5x to 2x) the regular app build time without XC Remote Cache support. This mainly happens in case building that latest commit in consumer mode without cleaning the DerivedData dir.

Steps to produce:

  1. pull a recent commit on master that has producer job completed for.
  2. build the app for that commit achieves 100% cache hits
  3. pull latest commit on master that that producer job did not ran for it
  4. re-build the app for that commit without cleaning DerivedData nor XCRemoteCache dirs
    here this app build could take 1.5x to 2x to the regular app build time
  5. cleaning DerivedData and rebuild again
    here build time reduced to to around 0,66x the regular app build

Do we need to clean DerivedData on before building on local machines as consumers?

I also just want to highlight that we are still seeing weird 404 errors on the local consumer machines logs, and did not verify that all issues related to cache misses are solved yet. We are still working on that, but trying to handle issues one by one.

@polac24
Copy link
Collaborator

polac24 commented May 5, 2022

Do we need to clean DerivedData on before building on local machines as consumers?

No, XCRemoteCache is optimized for incremental builds, including scenario after checking out to a different sha. 1.5x to 2x the regular app build time is off.

@polac24
Copy link
Collaborator

polac24 commented May 5, 2022

There is an extra optimization that could help achieving the incremental build with XCRemoteCache:

defaults write com.apple.dt.XCBuild IgnoreFileSystemDeviceInodeChanges -bool YES

That command modifies Xcode global config and shouldn't have any side effects - at least I haven't noticed any.

@ahmednafei
Copy link
Author

hey Bartosz,

Thanks for your updates on this. As status of today we got one step back, as we found out that finger print mismatch for PrivacyUI is not solved as we thought yesterday. We can still see it randomly happens with different modules but it is very common to happen with PrivacyUI.

I had changed my cache server to a local docker image instead of amazon S3, so that I can easily traverse and investigate the cache files produced from multiple producer jobs.

From investigation I can see that the same meta file with exact same name of PrivacyUI module is being overwritten with different content, for every producer build executed.

Step to produce:

  1. I had ran producer build for main app commit with hash 41e57bbf9c49f1cfb44ac930f374b57a23e263f6 on master repo. This commit points to the following commit hash 6f524499a878ad213c02f81eb36f5969996579ec on Privacy sub-repo that has PrivacyUI module.

  2. I can see the following meta file generated 6f524499a878ad213c02f81eb36f5969996579ec-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.json for PrivacyUI module with content `{
    "inputs":[

    ],
    "generationCommit":"6f524499a878ad213c02f81eb36f5969996579ec",
    "pluginsKeys":{

    },
    "targetName":"PrivacyUI",
    "platform":"iphonesimulator",
    "configuration":"Debug",
    "rawFingerprint":"7229c23320e5a9cc2a66e9adefa8a5d5",
    "fileKey":"5ef3c61d6aa040a484ded6fcc0750c17",
    "dependencies":[......`

  3. run the following commands to clean git clean -ffdx git submodule foreach git clean -ffdx rm -rf ~/Library/Caches/XCRemoteCache rm -rf ~/Library/Developer/Xcode/DerivedData

  4. re-generate project with buck ./buck project App -c project.rc-mode=producer

  5. re-build the app again with producer mode for exactly the same commits, without touching the code

  6. I can see the meta file 6f524499a878ad213c02f81eb36f5969996579ec-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.json for PrivacyUI module had been overwritten with different rawFingerprint and fileKey as follow `{
    "inputs":[

    ],
    "generationCommit":"6f524499a878ad213c02f81eb36f5969996579ec",
    "pluginsKeys":{

    },
    "targetName":"PrivacyUI",
    "platform":"iphonesimulator",
    "configuration":"Debug",
    "rawFingerprint":"e2d5a803cc5e82355bb1d6ab0b55e270",
    "fileKey":"eced69e44829c0b6eb24449e58fa000b",
    "dependencies":[......`

  7. repeating steps again every time generates new rawFingerprint & fileKey values (sometimes old values repeated).

So, if rawFingerprint is a hash for source files content, then this confirms a change happens between builds, although there is not code generated except the project files generated by buck, and modulemap files that are being generated by buck too. Working now to investigate what could be the potential reason for that source change or rawFingerprint mismatch

@ahmednafei
Copy link
Author

Hi Bartosz,

I am noting that list of dependencies in the meta file of PrivacyUI module, changes between different producer builds for the same commit, with even re-generating the project with buck. So I generated the project once with buck and ran multiple producer builds with just cleaning DerivedData dir in between

Attaching sample of these meta files generated with the same name and different content for the same module [PrivacyUI]
31d760b57ddd53f99a4373139aa4d298b073fae7-PrivacyUI-Debug-iphonesimulator-13C100-867e80b75d883611ed817cb2686c05de.zip

so what changed between these builds is only clean DerivedData and no code generation happened

@polac24
Copy link
Collaborator

polac24 commented May 6, 2022

Are ‘BUIAssets’, ‘PayInSDK’ prebuilt or generated on-fly when xcodebuild happens by buck (or any other tool)?
In other words, does ‘ /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap’ exist prior to the Xcode build (after project generation)?

@ahmednafei
Copy link
Author

ahmednafei commented May 6, 2022 •
edited
Loading

/Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap is generated as part of xcode project generation by buck. So at build time there is no code generated at all, neither forBUIAssets nor for PayInSDK. Both modules are NOT prebuilt and are compiled during xcode build. Also both of them achieve cache hit on consumer side.

On other side, is it expected that XCRemoteCache to include /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap as part of dependencies for PrivacyUI? and given these module map files are already existing before build time, any clue why sometimes could XCRemoteCache include them and sometimes not?

I fixed the project generation state (generated project once, and only cleaning derived data in between), and I can still see different rawFingerprint each time I run the producer.

@polac24
Copy link
Collaborator

polac24 commented May 6, 2022

A list of dependencies is takes directly from clang/swift compiler .d files and assuming all these .modulemap files are present on a disk when PrivacyUI target is compiled, their locations are passed into header search paths (or similar), compilers should always include them.

I would analyze isolated PrivacyUI compilation and inspect why modulemaps are placed in any of DerivedData/Proj/Build/Intermediates.noindex/Proj.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal/arm64/*.d:

  1. Does PrivacyUI import BUIAssets? (Directly/indirectly)?
  2. How the compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap should be analyzed (e.g. -Xcc -fmodule-map-file=BUIAssets/obj.modulemap?
  3. If -fmodule-map-file=...BUIAssets/objc.modulemap is used, does the generated project always contain that argument?

On the other hand, if /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap always exists on a disk (and its content is byte-level identical), including/not including it should not cause any cache miss on a consumer side, right? I agree it is not optimal as consumers can download several artifact zips of the same library (thus finding a reason should be prioritized) but that would only affect performance, not cache hit rate.

@cezarsignori
Copy link
Contributor

Hi @polac24!

I will first address your questions and then add some more detail and questions from our side.

Addressing your questions

  1. Does PrivacyUI import BUIAssets? (Directly/indirectly)?

Yes, BUIAssets is directly referenced as a dependency of PrivacyUI in the BUCK file.

  1. How the compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap should be analyzed (e.g. -Xcc -fmodule-map-file=BUIAssets/obj.modulemap)?

-fmodule-map-file is not used. The compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub should be analyzed via HEADER_SEARCH_PATHS, SWIFT_INCLUDE_PATHS and isystem flags on OTHER_CFLAGS, OTHER_CPLUSPLUSFLAGS and OTHER_SWIFT_FLAGS. In the test I ran, BUIAssets wasn't directly referenced but BUIAssets was the only directory under that path and the meta file did not contain any references to it but $(BUILD_DIR)/Debug-iphonesimulator/BUIAssets.swiftmodule/arm64-apple-ios-simulator.swiftmodule.md5.

On the other hand, if /Pods/buck-out/gen/_p/MRmmgKh-hU-pub/BUIAssets/objc.modulemap always exists on a disk (and its content is byte-level identical), including/not including it should not cause any cache miss on a consumer side, right?

I think that depends on whether it is consistently listed as a dependency. In the above mentioned test, the objc.modulemap was there but it was not referenced in the meta file so I believe not included in the rawFingerprint.

Details and questions from our side

Using the meta files produced by different producer runs for the same commit (6f524499a878ad213c02f81eb36f5969996579ec) on the PrivacyUI git submodule we identified two issues:

Issue 1. The dependency list changes;
Issue 2. The raw fingerprint changes even if the dependencies are the same;

_Category 1_. `rawFingerprint` is the *same* for the *same* `dependencies`. This OK from a fingerprint perspective.
  • Build 1 has rawFingerprint: 5f71dac5e70d9e06efa9c1284d741b87 and UI/objc.modulemap listed as dependency)
  • Build 4 has rawFingerprint: 5f71dac5e70d9e06efa9c1284d741b87 and UI/objc.modulemap listed as dependency)
  • Build 6 has rawFingerprint: 7229c23320e5a9cc2a66e9adefa8a5d5 and BUIAssets headers listed as dependency)
  • Build 7 has rawFingerprint: 7229c23320e5a9cc2a66e9adefa8a5d5 and BUIAssets headers listed as dependency)

Category 2. rawFingerprint is different for different dependencies. This is OK from a fingerprint perspective.

  • Build 5 has rawFingerprint: 7c437cf1bb1dfd0485d7b353206769e3 and BUIAssets headers + UI/objc.modulemap listed as dependency)
  • Build 8 has rawFingerprint: e6d874913d46d9b047463e32f6ff479e and BUIAssets headers + BUIAssets/objc.modulemap listed as dependency)

Category 3. rawFingerprint is different for the same dependencies. This is not OK a fingerprint perspective.

  • Build 2 has rawFingerprint: e2d5a803cc5e82355bb1d6ab0b55e270 and BUIAssets/objc.modulemap listed as dependency)
  • Build 3 has rawFingerprint: 01aa092fc7a167e98968852c70e44622 and BUIAssets/objc.modulemap listed as dependency)

We are investing both our setup (focused on the related modules) as well as RC to understand what is going on.

I reviewed RC's code on identifying dependencies (Issue 1) and generating the raw fingerprint (Issue 2).

Regarding Issue 1 / identifying dependencies:

a) isRelevantDependency excludes modulemap files in the $TARGET_BUILD_DIR directory (~/Library/Developer/Xcode/DerivedData/{app}/Build/Products/Debug-iphonesimulator). Should instead modulemap files be excluded regardless of location?

In our case, PrivacyUI lists modulemap files (e.g., BUIAssets/objc.modulemap) under /buck-out/gen. That could fix the dependency differences regarding modulemap files, though it is still unclear why they randomly show up in the /buck-out/gen directory (likely a problem unrelated to RC).

b) Otherwise isRelevantDependency excludes any .xcode, .intermediate, .ownProduct, .derivedFile.

But in your setup:
b.1) RC 0.3.11 excludes all $DERIVED_FILE_DIR files from dependency list. For PrivacyUI the value of $DERIVED_FILE_DIR is ~/Projects/iphone/Submodules/buck-out/xcode/derived-sources/PrivacyUI-16364f6141f26cc02c2e3f099242c1c0ece88448. And there are found only -Swift.h files (in this case only PrivacyUI-Swift.h) file.
b.2) Dependencies are read from $OBJECT_FILE_DIR_normal/arm64. In our case $OBJECT_FILE_DIR_normal=~/Library/Developer/Xcode/DerivedData/{app}/Build/Intermediates.noindex/Privacy.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal and we find files: .d, .dia, .o, .swiftdeps, ~partial.swiftdoc, ~partial.swiftmodule and ~partial.swiftsourceinfo

Are those values matching RC expectations? If not, can you clarify what files RC expects to be where?

c) Why is fingerprintOverrideManager.getFingerprintFile run over the dependency output and when?

Regarding Issue 2 / generating rawFingerprint:

a) The piece that is unclear to me to is Postbuild.generateFingerprintOverrides. Why and when is that run? Could it be causing Category 3 issues?

@polac24
Copy link
Collaborator

polac24 commented May 9, 2022

Hi @cezarsignori!

The compiler knows that /Pods/buck-out/gen/_p/MRmmgKh-hU-pub should be analyzed via HEADER_SEARCH_PATHS, SWIFT_INCLUDE_PATHS and isystem flags on OTHER_CFLAGS, OTHER_CPLUSPLUSFLAGS and OTHER_SWIFT_FLAGS.

I tried to reproduce that in a simple project but no matter which build setting I define, only module.modulemap files are read by the swift compiler and obj.modulemap is just skipped. The only way to force objc.modulemap processing was passing -Xcc -fmodule-map-file=BUIAssets/obj.modulemap. In the example project attached below, importing BUIAssetsInObj always fails because it is not defined in module.modulemap:
ModuleDemo.zip
Am I missing something? Do you know why does you project process obj.modulemap?

isRelevantDependency excludes modulemap files in the $TARGET_BUILD_DIR directory (~/Library/Developer/Xcode/DerivedData/{app}/Build/Products/Debug-iphonesimulator). Should instead modulemap files be excluded regardless of location?

To be honest, isRelevantDependency function is not perfect (thus TODO was added) and it unnecessary skips all .modulemaps in TARGET_BUILD_DIR. As a comment says, the method should precisely inspect if a given modulemap was actually used and then include it in a list of dependencies. It is still an open question how to decide if a modulemap was relevant or not from that place so to play safe, XCRemoteCache just excludes all modulemaps. That could lead to an edge case and overcache a target if someone changes .modulemap only. Technically, a change in a .modulemap that defines module A should invalidate all its dependants (reverse dependencies). I think it is not a problem for you as you don't have any modulemaps in TARGET_BUILD_DIR.
To conclude, I don't think we should exclude all modulemaps, quite opposite - we should consider .modulemap in a fingerprint calculation even for some modules in TARGET_BUILD_DIR.

RC 0.3.11 excludes all $DERIVED_FILE_DIR files from dependency list. For PrivacyUI the value of $DERIVED_FILE_DIR is ~/Projects/iphone/Submodules/buck-out/xcode/derived-sources/PrivacyUI-16364f6141f26cc02c2e3f099242c1c0ece88448. And there are found only -Swift.h files (in this case only PrivacyUI-Swift.h) file.

I think it is correct - PrivacyUI-Swift.h shouldn't be defined as a dependency because this file is generated as a target compilation product. In other words, prior to PrivacyUI compilation, when xcprebuild is called, PrivacyUI-Swift.h doesn't exists so if we were including it in a dependencies list, a target wouldn't be cached.

Dependencies are read from $OBJECT_FILE_DIR_normal/arm64. In our case $OBJECT_FILE_DIR_normal=~/Library/Developer/Xcode/DerivedData/{app}/Build/Intermediates.noindex/Privacy.build/Debug-iphonesimulator/PrivacyUI.build/Objects-normal and we find files: .d, .dia, .o, .swiftdeps, ~partial.swiftdoc, ~partial.swiftmodule and ~partial.swiftsourceinfo. Are those values matching RC expectations? If not, can you clarify what files RC expects to be where?

That is fine. Dependencies are read only from .d files, all other file extensions can sit next to it:

XCRemoteCache/Sources/XCRemoteCache/Dependencies/TargetDepdenciesReader.swift

Line 48 in 3b614c6

let isDependencyFile = file.pathExtension == "d"

Why is fingerprintOverrideManager.getFingerprintFile run over the dependency output and when?

This is required when a byte-level fingerprinting of some fileextension is not portable. In the blogpost, section "Build artifacts portability" you can find a problem statement. In essence, we cannot generate a fingerprint from a binary file .swiftmodule because it contains absolute paths. To overcome that, XCRemoteCache always generates an extra file next to xxx.swiftmodule (default xxx.swiftmodule.md5) that contains a normalized fingerprint of that target. By normalized I mean SRCROOT-agnostic. In the fingerprint generation, we should always consider yyy.swiftmodule.md5 content, not yyy.swiftmodule. Sidenote: that xxx.md5 file is called "an override".

The piece that is unclear to me to is Postbuild.generateFingerprintOverrides. Why and when is that run? Could it be causing Category 3 issues?

generateFingerprintOverrides is a step that generates an override (see above). It has to be called on a producer side (all Swift targets) and consumer side (only for Swift targets that have been compiled locally).
The aim of calling that on a consumer side is to make locally built .swiftmodule compatible with XCRemoteCache.

Imagine a project with 2 swift targets A and B, where B depends on A. Let's assume we didn't introduce any changes locally. First we build A, but for some reason we had to compile it locally (e.g. VPN issue). When it comes a turn for target B compilation, we call xcprebuild and generate fingerprint from all B's dependencies - including A.swiftmodule.md5. If we weren't generating an override file for A.swiftmodule, B would have been compiled locally (cache miss because a dependency doesn't exist) even remotely available artifact could be used.

You are asking if that could be a reason of the issue 3? Yes. If your list of dependencies is non-deterministic, some other dependency of PrivacyUI (e.g. Privacy) could get a different "normalized" fingerprint - Privacy.swiftmodule.md5 and even the list of PrivacyUI dependencies is the same - their content is different and a final fingerprint of the PrivacyUI target is different.

So to make the fully stable fingerprinting, the list of dependencies for all targets has to be stable too.

I think that depends on whether it is consistently listed as a dependency. In the above mentioned test, the objc.modulemap was there but it was not referenced in the meta file so I believe not included in the rawFingerprint.

Right.

Again, having inconsistent list of dependencies in a meta is annoying and could lead to excessive artifact download on a consumer side, that shouldn't affect consumer's cache hit rate.

@cezarsignori
Copy link
Contributor

Am I missing something? Do you know why does you project process obj.modulemap?

Comparing to ModuleDemo, BUIAssets (dependency of PrivacyUI) is a mixed module that depends on a bundle (BUIAssets-Bundle), on a pure Swift module (BUIFoundations), and on another mixed module (DSAssets) that also depends on a bundle (DSAssets-Bundle).

About the compiler flags, I do not know why it is included. I can confirm none of our .bzl or BUCK files contain the -fmodule-map-file flag. In fact, the only fmodule* flag we use is -fmodules. And so far I have not seen it in the build logs. I will keep an eye for that.

I don't think we should exclude all modulemaps, quite opposite - we should consider .modulemap in a fingerprint calculation even for some modules in TARGET_BUILD_DIR.

My reasoning here is that if modulemap files are excluded because their are in TARGET_BUILD_DIR, they should be also excluded if they are in buck-out. I mean, RC expects modulemap files to be in the DerivedData directory, but in the buck setup, they are found elsewhere. Until we find how to identify whether a modulemap was actually used, it stands to reason RC should consistently ignore them, as that was the original intent of the current code. Right?

Of course, making that change to RC would not make it so we no longer need to have a consistent list of dependencies for our modules, but at the same time, my understanding is that we can't say RC works as designed given in our setup buck places modulemap files outside of the TARGET_BUILD_DIR. At the very least, that change should reduce the frequency of inconsistencies in the list of dependencies on our meta files (no more objc.modulemap listed).

Makes sense?

You are asking if that could be a reason of the issue 3? Yes. If your list of dependencies is non-deterministic, some other dependency of PrivacyUI (e.g. Privacy) could get a different "normalized" fingerprint - Privacy.swiftmodule.md5 and even the list of PrivacyUI dependencies is the same - their content is different and a final fingerprint of the PrivacyUI target is different.

The Category 3 error is happening with PrivacyUI but all dependencies have successful cache hits, meaning their fingerprint was a match and therefore no content difference. I believe that is a different behaviour than what you described. Correct?

Again, having inconsistent list of dependencies in a meta is annoying and could lead to excessive artifact download on a consumer side, that shouldn't affect consumer's cache hit rate.

I do not follow. Since the raw fingerprint is computed from the list of dependencies, inconsistencies in that list do affect hit rate (due to fingerprint mismatch between local and remote). That is what we are experiencing. Am I missing something?

@polac24 polac24 mentioned this issue May 10, 2022
Merged
@polac24
Copy link
Collaborator

polac24 commented May 10, 2022

I will reply for other questions, but for now have a draft PR that could unblock you (until you find out why you have unstable dependency list).

Until we find how to identify whether a modulemap was actually used, it stands to reason RC should consistently ignore them, as that was the original intent of the current code. Right?

Once #130 is merged, you could specify something like irrelevant_dependencies_paths: "Pods/buck-out/gen/.*\\.modulemap$" to exclude buck's modulemaps only for your project. That potentially could lead to target overcaching if one introduces a change to a target's modulemap without modifying any other sources of the target - chances of happening that are low though. Please let me know if such property would unblock you and then I will create a PR.

@cezarsignori
Copy link
Contributor

@polac24 , by declaring irrelevant_dependencies_paths: ["\\.modulemap$"] across all .rcinfo files I was able to run 12 (local) producer builds where all modules consistently generated the same rawFingerprint for the same commit (Thanks!). I am rolling out the change (still in a controlled environment) so CI starts to automatically build commits as they come in to increase the amount of samples.

@polac24
Copy link
Collaborator

polac24 commented May 11, 2022

My reasoning here is that if modulemap files are excluded because their are in TARGET_BUILD_DIR, they should be also excluded if they are in buck-out. I mean, RC expects modulemap files to be in the DerivedData directory, but in the buck setup, they are found elsewhere.

Excluding modulemaps in TARGET_BUILD_DIR has another reason. Contrary to your scenario, we have a project where modulemaps are copied (dittoed) there progressively, as a postbuild phase step for each Xcode target. So these *.modulemap files do not exist prior to the Xcode build. Because an order of building Xcode targets is random, it may happen that some modulemaps that were added to a dependencies list to a meta file on a producer, do not exists on a consumer side (because that target has not been finished yet there). The simplest workaround was to skip all modulemaps in TARGET_BUILD_DIR, but the valid behaviour should only exclude modulemaps that are not actually used (there is TODO in the code for that).

that we can't say RC works as designed given in our setup buck places modulemap files outside of the TARGET_BUILD_DIR.

Hard to say what is a reason of that and I wouldn't blame XCRemoteCache yet. We cannot reproduce in a sample project processing obj.modulemap so until we know what is happening, we are blind.

At the very least, that change should reduce the frequency of inconsistencies in the list of dependencies on our meta files (no more objc.modulemap listed).

That is correct. Introducing irrelevant_dependencies_paths can improve incremental build times after rebasing/merging.

Since the raw fingerprint is computed from the list of dependencies, inconsistencies in that list do affect hit rate (due to fingerprint mismatch between local and remote). That is what we are experiencing. Am I missing something?

Consumer always computes a fingerprint from the same set of files as a producer so there will be no fingerprint mismatch. The very first step in a xcprebuild is downloading meta json for a given commit sha, which contains a list of dependencies. There always is a parity of local & remote files used to computer a fingerprint.

@ahmednafei
Copy link
Author

So I think this issue could be safely closed after the fix of adding irrelevant_dependencies_paths config to exclude modulemap files that are generated by buck outside the DerivedData Dir.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ahmednafei @Ahmed @cezarsignori @polac24