Skip to content

build: slightly improve the build for libdispatch #785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 3, 2023

Conversation

compnerd
Copy link
Member

Avoid polluting the build directory a small amount given that we can use -fmodule-map-file= for the C/C++ build of libdispatch. Unfortunately, for the Swift build, we need to have the file copied over due to the umbrella header resolution.

Hopefully this reduces some of the race conditions that we have seen in the build.

Thanks to @dgregor for reminding me of the flag!

Copy link
Contributor

@etcwilde etcwilde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could use another vfs overlay type thing, or just copy all of the headers into the build directory next to the module map instead of pushing things around inside of the source directory?

This is an improvement over the current state of things though.

@etcwilde etcwilde requested a review from rokhinip March 31, 2023 16:28
@etcwilde
Copy link
Contributor

CC @rokhinip, this is the source of our little race condition in CI that crops up from time to time.

@compnerd
Copy link
Member Author

That VFS idea 🤔

@compnerd compnerd force-pushed the references-available-upon-request branch 2 times, most recently from 387b87f to e0517b6 Compare March 31, 2023 21:11
@compnerd
Copy link
Member Author

@swift-ci please test

@etcwilde
Copy link
Contributor

Good. I like it. No more copying things around in the source directories during the build.

Avoid polluting the build directory a small amount given that we can use
`-fmodule-map-file=` for the C/C++ build of libdispatch.  Unfortunately,
for the Swift build, we need to have the file copied over due to the
umbrella header resolution.

Hopefully this reduces some of the race conditions that we have seen in
the build.

Thanks to @dgregor for reminding me of the flag!
@DougGregor
Copy link
Member

@swift-ci please test

@etcwilde
Copy link
Contributor

etcwilde commented Apr 1, 2023

Linux failure:

/home/build-user/swift-corelibs-foundation/Sources/Foundation/DispatchData+DataProtocol.swift:14:8: error: cannot load underlying module for 'Dispatch'
import Dispatch
       ^

Windows failure:

CMake Error at dispatch/cmake_install.cmake:59 (file):
  file INSTALL cannot find
  "C:/Users/swift-ci/jenkins/workspace/swift-corelibs-libdispatch-PR-windows/swift-corelibs-libdispatch/dispatch/module.modulemap":
  File exists.
Call Stack (most recent call first):
  cmake_install.cmake:42 (include)

@etcwilde
Copy link
Contributor

etcwilde commented Apr 1, 2023

Hmm, Linux VFS overlay not quite pointing in the right place maybe?

@DougGregor
Copy link
Member

Hmm. The Foundation build has include paths pointing into the Dispatch source directory:

-I /home/build-user/build/buildbot_linux/libdispatch-linux-x86_64/src/swift/swift -I /home/build-user/build/buildbot_linux/libdispatch-linux-x86_64 -I /home/build-user/swift-corelibs-libdispatch -I /home/build-user/swift-corelibs-libdispatch/src -I /home/build-user/build/buildbot_linux/libdispatch-linux-x86_64/src -I /home/build-user/swift-corelibs-libdispatch/src/BlocksRuntime

So we either need to also add the VFS flags to the Foundation build, or we need to first install Dispatch and then switch the Foundation build to use the installed Dispatch.

@etcwilde
Copy link
Contributor

etcwilde commented Apr 1, 2023

switch the Foundation build to use the installed Dispatch.

That would be a day for celebration.

Use a VFS overlay to avoid polluting the source tree.
@compnerd compnerd force-pushed the references-available-upon-request branch from e0517b6 to ed909bb Compare April 1, 2023 18:03
@compnerd
Copy link
Member Author

compnerd commented Apr 1, 2023

@DougGregor correct, I had missed that I had given the wrong visibility to the flags. This is now resolved and -vfsoverlay should propagate properly when building against the build tree. I think that the now current state should build both against an uninstalled as well as installed dispatch. If you use -D dispatch_DIR=... you will prefer the build tree and if you do not, it will prefer the "system version".

@compnerd
Copy link
Member Author

compnerd commented Apr 1, 2023

@swift-ci please test

@etcwilde
Copy link
Contributor

etcwilde commented Apr 1, 2023

For tracking purposes:

  • rdar://92756762
  • rdar://94289755
  • rdar://105782611

@DougGregor
Copy link
Member

@DougGregor correct, I had missed that I had given the wrong visibility to the flags. This is now resolved and -vfsoverlay should propagate properly when building against the build tree. I think that the now current state should build both against an uninstalled as well as installed dispatch. If you use -D dispatch_DIR=... you will prefer the build tree and if you do not, it will prefer the "system version".

Sounds like exactly what we want. 🤞

@DougGregor
Copy link
Member

The Linux build got much farther! Now the XCTest tests are failing with

# command stderr:
<unknown>:0: error: cannot load underlying module for 'Dispatch'

@DougGregor
Copy link
Member

@swift-ci please test Windows

@DougGregor
Copy link
Member

Windows got to the same point; now it's the XCTest failures, but Foundation built and tested properly (!)

@compnerd
Copy link
Member Author

compnerd commented Apr 2, 2023

Ah, nice! The XCTest failures are not surprising. Fortunately, I had recently enhanced that path for Windows - we just need to make sure that we are passing along the dispatch flags to the tests, not just the build.

@DougGregor
Copy link
Member

So exciting!

@compnerd
Copy link
Member Author

compnerd commented Apr 2, 2023

Please test with the following PRs:
swiftlang/swift-corelibs-xctest#437

@swift-ci please test

@compnerd compnerd merged commit a5167e2 into swiftlang:main Apr 3, 2023
@compnerd compnerd deleted the references-available-upon-request branch April 3, 2023 02:08
MaxDesiatov added a commit that referenced this pull request Jun 1, 2023
…on-request"

This reverts commit a5167e2, reversing
changes made to fa1b4ae.
finagolfin added a commit to finagolfin/swift that referenced this pull request Aug 26, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them.
gottesmm pushed a commit to gottesmm/swift that referenced this pull request Sep 5, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them.
finagolfin added a commit to finagolfin/swift that referenced this pull request Sep 6, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them and temporarily
disable one failing async test on linux till it can be investigated.
finagolfin added a commit to finagolfin/swift that referenced this pull request Sep 8, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them and fix one
async test on linux.
finagolfin added a commit to finagolfin/swift that referenced this pull request Sep 8, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them and fix one
async test on linux.
gottesmm pushed a commit to gottesmm/swift that referenced this pull request Sep 8, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them and fix one
async test on linux.
finagolfin added a commit to finagolfin/swift that referenced this pull request Sep 19, 2023
…g/swift-corelibs-libdispatch#785

This allows the tests that use libdispatch to find its modulemap, plus add the
libdispatch compilation flags to one test that was missing them and fix one
async test on linux.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants