Performance(?) regressions after update to Zygote.jl v0.7.5

[leburgel]

It seems the gradient and example tests are timing out for an unknown reason, as first noticed in #165. Just from looking at the test Project.toml, it seems this might be due to tests now running on Zygote.jl v0.7.5 versus v0.7.4 for the most recent successful test run.

After #160:

     Testing PEPSKit
      Status `/tmp/jl_kXLtOm/Project.toml`
  [7d9f7c33] Accessors v0.1.42
  [d360d2e6] ChainRulesCore v1.25.1
  [cdddcdb0] ChainRulesTestUtils v1.13.0
  [34da2185] Compat v4.16.0
  [26cc04aa] FiniteDifferences v0.12.32
  [0b1a1467] KrylovKit v0.9.5
⌅ [e6f89c97] LoggingExtras v1.0.3
  [bb1c41ca] MPSKit v0.12.6
  [ca635005] MPSKitModels v0.4.0
⌅ [67456a42] OhMyThreads v0.7.0
  [77e91f04] OptimKit v0.4.0
  [52969e89] PEPSKit v0.4.2 `~/work/PEPSKit.jl/PEPSKit.jl`
  [1fd47b50] QuadGK v2.11.2
  [1bc83da4] SafeTestsets v0.1.0
  [07d1fe3e] TensorKit v0.14.5
  [6aa20fa7] TensorOperations v5.1.4
  [409d34a3] VectorInterface v0.5.0
  [e88e6eb3] Zygote v0.7.4
  [37e2e46d] LinearAlgebra
  [de0858da] Printf
  [9a3f8284] Random
  [10745b16] Statistics v1.10.0
  [8dfed614] Test

Giving things like:

Test Summary:                | Pass  Total     Time
Transverse Field Ising model |    5      5  8m31.5s
511.808261 seconds (958.20 M allocations: 93.805 GiB, 2.60% gc time, 87.28% compilation time: <1% of which was recompilation)

After #165:

     Testing PEPSKit
      Status `/tmp/jl_RW27Ks/Project.toml`
  [7d9f7c33] Accessors v0.1.42
  [d360d2e6] ChainRulesCore v1.25.1
  [cdddcdb0] ChainRulesTestUtils v1.13.0
  [34da2185] Compat v4.16.0
  [26cc04aa] FiniteDifferences v0.12.32
  [0b1a1467] KrylovKit v0.9.5
⌅ [e6f89c97] LoggingExtras v1.0.3
  [bb1c41ca] MPSKit v0.12.6
  [ca635005] MPSKitModels v0.4.0
⌅ [67456a42] OhMyThreads v0.7.0
  [77e91f04] OptimKit v0.4.0
  [52969e89] PEPSKit v0.5.0 `~/work/PEPSKit.jl/PEPSKit.jl`
  [1fd47b50] QuadGK v2.11.2
  [1bc83da4] SafeTestsets v0.1.0
  [07d1fe3e] TensorKit v0.14.5
  [6aa20fa7] TensorOperations v5.1.4
  [409d34a3] VectorInterface v0.5.0
  [e88e6eb3] Zygote v0.7.5
  [37e2e46d] LinearAlgebra
  [de0858da] Printf
  [9a3f8284] Random
  [10745b16] Statistics v1.10.0
  [8dfed614] Test

Giving things like:

Test Summary:                | Pass  Total      Time
Transverse Field Ising model |    5      5  17m11.9s
1032.239334 seconds (3.68 G allocations: 213.161 GiB, 2.51% gc time, 95.91% compilation time: <1% of which was recompilation)

This is quite the difference, so we should probably look into the precise cause.

[lkdvos]

I was browsing through the Zygote code and I can't find anything that really changed meaningfully: that version mostly seems to drop support for julia 1.6

It might be that the precompilation is now no longer cached because it never manages to run through a full example, and it could revert to normal if we make it go through a full run once.
Alternatively, we can also try restricting the zygote version to check if that actually solves the problem

[pbrehmer]

Somehow, also locally for me, precompilation of AD code now takes extremely long. (Specifically differentiating leading_boundary.) This was even without updating Zygote to v0.7.5. So I'm not so sure that restricting the Zygote version will change anything. I am somewhat confused by this since locally I didn't make any obvious changes to the installed packages.