MacOS runner does not pick up job

[moonbox3]

Description

For the past few days (Starting around March 25, 2025), we're not seeing MacOS runners pick up jobs. For example, with the following job, it's been queued for 1 hour 16 min and counting.

https://github.com/microsoft/semantic-kernel/actions/runs/14072710659/job/39410048335?pr=11184

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Public repo link: https://github.com/microsoft/semantic-kernel/actions/runs/14072710659/job/39410048335?pr=11184

Current runner version: '2.322.0'
Operating System
macOS
14.7.4
23H420
Runner Image
Image: macos-14-arm64
Version: 20250317.1117
Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20250317.1117/images/macos/macos-14-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20250317.1117

Is it regression?

Not sure

Expected behavior

I expect the job to be picked up as quickly as possible. Normally they finish running within a few min.

Actual behavior

Job queued for over an hour.

Repro steps

View the link to our public repo job.

[moonbox3]

A little misleading now - once the job is finally picked up, any indication about the time it was queued seems to be lost.

[sangeeths03]

Hi @moonbox3 ,I will check on the issue and keep you updated on the progress.

[imabhichow]

We have similar issue for macOS-13. #11918

[sangeeths03]

Hi @moonbox3 ,

I’ve tested the pipeline on my end by cloning the repo to my public account and running the Python unit test pipeline multiple times, and I haven’t encountered the issue you're facing. The jobs are being picked up quickly on macOS 14 ARM64, version 20250324.1158, which is the same version as yours.

Here’s the link to the tests I ran: Test Pipelines.

It seems the jobs are running successfully on my side. Perhaps the issue is specific to your environment or configuration, but everything seems fine with the macOS 14 ARM64 runners. Let me know if you need further assistance or if there’s anything more I can help with.

[moonbox3]

Hi @moonbox3 ,

I’ve tested the pipeline on my end by cloning the repo to my public account and running the Python unit test pipeline multiple times, and I haven’t encountered the issue you're facing. The jobs are being picked up quickly on macOS 14 ARM64, version 20250324.1158, which is the same version as yours.

Here’s the link to the tests I ran: Test Pipelines.

It seems the jobs are running successfully on my side. Perhaps the issue is specific to your environment or configuration, but everything seems fine with the macOS 14 ARM64 runners. Let me know if you need further assistance or if there’s anything more I can help with.

Thanks for spending time on this, @sangeeths03. Much appreciated. I haven't seen the high-latency since I posted this issue -- it previously took hours for MacOS jobs to be picked up, but now they seem okay.

[sangeeths03]

Thanks for the update! I’m glad to hear the issue seems to have been resolved. Feel free to reach out if you encounter any further problems in the future

[moonbox3]

@sangeeths03 we're seeing high latency again. As an example, the job hasn't been picked up after 45 min. This is unusual.

https://github.com/microsoft/semantic-kernel/actions/runs/14252786458/job/39949297039?pr=11358

[moonbox3]

Queued for an hour and six minutes now.

[moonbox3]

A separate PR with slow MacOS runners as well:

https://github.com/microsoft/semantic-kernel/actions/runs/14253222268/job/39950641095?pr=11321

[moonbox3]

Another run taking almost 40 minutes.

https://github.com/microsoft/semantic-kernel/actions/runs/14254144829/job/39953473532?pr=11358

[moonbox3]

@sangeeths03 please re-open this issue, thanks.

[sangeeths03]

Thanks for the detailed breakdown @moonbox3 . I'll definitely take a closer look at these

[sangeeths03]

Hi @moonbox3,

I’ve tested and monitored the pipeline for the past three days, and everything seems to be working fine on my end. Here’s a screenshot of my test runs, where the jobs are picked up without any issues:

Could you please confirm if the issue is still occurring on your side? If so, we can investigate further.