Snapshot taken on CI has slight color differences compared to local

[edulpn]

Hi everybody, I'm having an issue here with color discrepancies in an image view. I'm using KIF framework for UI Tests and SnapshotTesting to take screenshots between interactions (asserts made on the app key window). Everything works fine, except that I get color differences from snapshots taken on my local machine vs snapshots taken on CI machine (both running the same stack, Mojave + Xcode 11.3).

Some examples below:

This one is taken by CI machine

This one is taken by my local computer

This is the comparison of both images made by imagemagick

And these are the color discrepancies of a same pixel, measured by macOS Digital Color Meter

I know there are lots of variables and also using KIF, but do you guys have any idea of what might be causing this difference?

Thanks!

[iandundas]

Just to chime in that we're seeing the same issue. When run our snapshot tests from fastlane, assets (pngs etc) seem to be slightly differently rendered compared when we launch the same tests on the same machine, same simulator, from Xcode. Very strange.

[gobetti]

I'm not seeing any difference between fastlane and Xcode on my machine so far, but I do also experience tiny nuances between my machine's and CI's snapshots, specifically on iOS 13. Having a non-zero corner radius in a native view is enough to have a difference, don't really need to involve assets.

edit: including screenshots now. What scares me is that this tiny difference is consistently so, on both.

Mine	CI

if there's no workaround for that other than lowering the precision, I wonder if maybe it'd make sense to have a tolerance specifically for cases like that where the R and/or G and/or B channel(s) have a difference of only +-1...

edit 2: on iOS 12, both 2x and 3x screens tests pass, on iOS 13 both fail.

[aegzorz]

We're seeing similar results between different machines, as soon as we add "real life" images we get a small imperceptible differences causing the test to fail.

[gobetti]

@aegzorz could you please check if your differences are only on iOS 13? And how much is the difference? (i.e. what's the RGB for the different pixel in the two different runs)

[aegzorz]

We're only testing on iOS 13 currently. Seems using shadows on views throws it off sometimes as well. Haven't done any RGB measurements yet.

[gobetti]

sadly, lowering the precision or considering as "equal" pixels with an invisible difference aren't performant solutions. Tests that don't pass the memcmp comparison are orders of magnitude slower so having to go through the bytes comparison loop is probably very undesirable.

[stephencelis]

We have this in our README:

⚠️ Warning: Snapshots must be compared using a simulator with the same OS, device gamut, and scale as the simulator that originally took the reference to avoid discrepancies between images.

Is it possible to ensure that CI and developer machines test in a consistent environment?

[iandundas]

Could be something there, but we were seeing it on the exact same simulator (model, OS etc) on the same machine, with the difference being only between running the tests from Xcode vs running them (locally) from fastlane

[stephencelis]

Huh. Perhaps someone with fastlane experience can explain? Maybe it's worth opening an issue with them to ask?

[iandundas]

Good idea, will do 👍🏻

[iandundas]

I was just about to, when I realised above that some people are saying it is not fastlane which is causing this issue for them (but simply running the tests on a different machine). So a bit hesitant now to open that fastlane issue 🤔.

[gscalzo]

@iandundas it could be related to #288