Using macro in test only

[Alex293]

Is your feature request related to a problem? Please describe.
How to use the macro only in test code ? I don't want to use the annotation on production code.

Describe the solution you'd like
I'd like to be able to use the macro only in test targets.

Describe alternatives you've considered
I tried multiple things like using a typealias or a second protocol refining the one I'd like to spy but I can't make it work.

[dafurman]

AFAIK, Macros can't be used to generate code in a specific location or target other than where their annotation lives. I don't believe there'd be a way to say, "I want to annotate this protocol living in target A, but have its generated code only available to the target B that tests it."

What could be done is to have the generated spy class be wrapped in an #if DEBUG, so that it at least can't be used in released builds. However, this would get in the way though if you're doing any sort of testing or using previews in release mode for whatever reason, so a restriction like this would probably have to be optional - possibly looking something like @Spyable(debugOnly: true).
Another option may be to allow users to specify their own preprocessor flag names to pass to wrap the code, like #if TESTING via @Spyable(preprocessorFlag: "TESTING")

[Alex293]

I understood that and it make sens but would there be a way to annotate (in the tests) an extension or a second protocol refining the first one ? That would be even better as production code wouldn’t even have the test annotation.

I agree that the if DEBUG is not the solution as one could do performance testing or production tests.

Anyway if this isn’t possible with swift now you can close this issue and I’ll post to the forum. New testing approaches are discussed and maybe this could get traction as mocks are imho one a the biggest blockers to tests in swift.

[dafurman]

Would there be a way to annotate (in the tests) an extension?

Unfortunately I don't think so, even if the extension lives side-by-side with the protocol, like this:

protocol MyProtocol {
    func doSomething()
}

@Spyable
extension MyProtocol {}

Inspecting the AST tree doesn't give any information about func doSomething() to work with, because all that's available is the code in the extension itself.

or a second protocol refining the first one ?
I may not be following you correctly on this, but I don't think that'd work either. It sounds like you'd want something like this

// Lives in target A
protocol MyThing {
    func doThing1()
    func doThing2()
}

// Lives in target ATests
protocol MyThingSubset: MyThing {
    func doThing1()
}

But same with the extension above, this would only be able to expose doThing1() and not doThing2().

I do think if there's a desire to limit where code can be run from, preprocessor macros may be the only tool available here.

I do agree about the appeal of this though, as you'd have confidence knowing you'd never have to watch out for coworkers accidentally using mocks in live code during a PR review, as the app archive would just totally fail if they did. (You could also set up a CI pipeline to make a release build if you wanted earlier erroring).
I may spend a little time looking into this when I can because now I'm curious 😄

[dafurman]

@Alex293

I created a proof of concept branch for this: https://github.com/dafurman/swift-spyable/tree/preprocessor_poc, but I don't think it'll ultimately work well.

This adds a flag parameter that lets you pass in a custom preprocessor macro that wraps the whole contents of the macro, like this:

This lets a protocol in one target to be defined and annotated with the macro, then for it to specify a flag to hide its spy behind. Test targets then can enable that flag by adding it via Active Compilation Conditions or Other Swift Flags. This works!

The bad news comes down to previews.

Unless devs were to define previews in separate targets with this flag enabled, they wouldn't be able to use spies in their previews. This recommendation would be a pretty strange deviation from standard practices, so it feels like a dealbreaker that'd prevent moving forward with this proof of concept. I don't know about others, but I personally use spies / mocks heavily to see how my views react to states, such as infinite loading, errors, etc. Without being able to include convenient use of previews in the story, my approach to optionally restrict spy use feels like it could end up being be more of an annoying impairment than a guard rail.

If we could only do something like this, the problem would be solved, but unfortunately I'm not aware of any option that lets you configure preprocessor macros inline like this:

#define SPIES_ENABLED true
#Preview {
    ContentView(myService: MyServiceSpy())
}
#define SPIES_ENABLED false

[Matejkob]

Hi everyone,

Thank you for your engagement on this topic. We've already delved into this matter in the discussion section of our repo, particularly in the post titled Spyable as part of test target. Additionally, this need has been echoed in other GitHub issues and on the Swift forums.

To encapsulate the core of our discussions:

Problem: A significant number of us have voiced concerns about including the @Spyable annotation and the spy classes it generates in the production code, especially when it's part of the main app target. The reservations are valid given the potential challenges and complexities this setup might introduce.

Key Points from Previous Discussions:

• The approach of leveraging preprocessor directives, such as #if DEBUG, was put forth. This would allow certain segments of code to be exclusive to debug builds. But, while it initially appears as a simple solution, it presents hurdles, notably when conducting tests or utilizing previews in the release mode.

• Annotating extensions or utilizing secondary protocols initially seemed like an elegant solution. However, the feasibility dwindles when we consider the inner workings of Swift Macros. They function based on the Abstract Syntax Tree (AST). Unfortunately, the AST doesn’t encompass all the necessary contextual information for this method to be viable. In simpler terms: macros only operate on the literal text of the entities they are attached to, without recognizing any external context.

---

I genuinely understand the appeal of maintaining a production codebase free from test annotations and the overarching importance of trust in the integrity of our production code. While we've charted multiple routes, the search for an uncomplicated solution continues, especially given our existing constraints.

First and foremost, I want to sincerely thank you @dafurman for the effort you've invested in creating the proof of concept. It's evident you've delved deep into this issue, and I genuinely appreciate your dedication to finding a solution. The POC you've shared looks promising!

Your point about the previews caught my attention:

Unless devs were to define previews in separate targets with this flag enabled, they wouldn't be able to use spies in their previews. This recommendation would be a pretty strange deviation from standard practices, so it feels like a dealbreaker that'd prevent moving forward with this proof of concept. I don't know about others, but I personally use spies / mocks heavily to see how my views react to states, such as infinite loading, errors, etc. Without being able to include convenient use of previews in the story, my approach to optionally restrict spy use feels like it could end up being be more of an annoying impairment than a guard rail.

I must admit, I'm not entirely following this part. Would the following solution not work?

#if MY_FLAG
#Preview {
    ContentView(myService: MyServiceSpy())
}
#endif

By wrapping the preview in the custom flag, wouldn't this allow developers the flexibility to toggle the spy use in previews based on the specified flag? What is more, if you have an intention to use the spies in one of the source targets, maybe hiding it behind a flag is not necessary?

Regarding the custom flag parameter, I completely agree. Offering the ability for users to specify any preprocessor flag is, in my view, one of the best approaches we can adopt right now. It provides a layer of flexibility that can be invaluable, allowing users to either define their own flag or simply utilize the default "DEBUG" flag in such a way:

@Spyable(flag: "DEBUG")
protocol ServiceProtocol { ... }

However, considering the broader user base and the varied ways in which they might be using swift-spyable, I believe it might be best to retain the default implementation without a specific flag. This way, we can ensure a smooth experience for most users while also providing the capability for customization to those who need or prefer it.

[dafurman]

Would the following solution not work?

#if MY_FLAG
#Preview {
    ContentView(myService: MyServiceSpy())
}
#endif

Unfortunately not if the preview is defined in a target that isn't compiled with the necessary flag. This is from a project I've got where I was trying out my PoC branch:

Let's say we've got some targets setup like this:

-- MyFeature (`SPIES_ENABLED = 0`)
---- MyFeatureTests (`SPIES_ENABLED = 1`)

And we have this protocol defined in MyFeature

@Spyable(flag: "SPIES_ENABLED")
protocol CoolService { ... }

If MyFeature uses previews, it won't be able to use CoolServiceSpy in them. The only way I've thought of to support that is to have a separate target just for previews, like this:

-- MyFeature (`SPIES_ENABLED = 0`)
---- MyFeatureTests (`SPIES_ENABLED = 1`)
---- MyFeaturePreviews (`SPIES_ENABLED = 1`)

So, we could go with my PoC, and this would work, but just with the understanding that the only way preview development would play nicely with these build flags would be to put previews in their own target. Devs would also probably end up changing a lot of things from internal to public in MyFeature just to support these previews in a different target. This is all what I meant by "This recommendation would be a pretty strange deviation from standard practices".
Creating a separate target just for previews seems like it'd be really obtrusive to workflows, as you'd have to hop back and forth between a view and its preview in different files when making UI changes.

Edit:

I believe it might be best to retain the default implementation without a specific flag. This way, we can ensure a smooth experience for most users while also providing the capability for customization to those who need or prefer it.

100% agreed on that too. The parameter would have to default to nil

[Matejkob]

Thank you for the clarification! Given the current state of macros, that's the best solution we can offer at the moment. I'm confident that developers who choose to use spies in previews or in source code outside of tests will be equipped to handle these limitations.

[Matejkob]

I've assigned both of us to the issue. If you'd like, you can refine your POC and submit a PR. If not, I can handle that in the coming days. Some enhancements I'd recommend include:

• Updating the README,
• Refreshing the examples package,
• Providing detailed documentation, with an example of a custom-defined flag as well as the DEBUG one.

[dafurman]

I've assigned both of us to the issue. If you'd like, you can refine your POC and submit a PR. If not, I can handle that in the coming days.

I'd love to, but I'll be fairly busy for about a week so take whatever you want from my branch. Shoot a PR my way whenever you're ready and I could probably give feedback at least! 😄