Feature request: add the latest version of lookout-sdk binary which is guaranteed to work

[vmarkovtsev]

We need to download the specific lookout-sdk binary which works with our client code for tests.
Our current approach is to take it from Gopkg.lock in src-d/lookout which works beautifully and reliably.
This is how it looks like: https://github.com/src-d/lookout-sdk-ml/blob/master/lookout/core/api/version.py

Since we want to eventually switch from artisanal homebrew to this project, I request the same feature here.

[carlosms]

The thing is, we haven't changed the protocol since we first released. So all versions should be compatible.
The lookout-sdk binary may have bug fixes or new options, but all of them should work with any server using any release of this repo.

Adding a variable somewhere in this repo will imply that:

• For any release of of the src-d/lookout repo we'll have to update this var and release also this repo, with the only change being this version number that will always be the latest one.
• This can be annoying for users of this repo, keeping up to date with our releases with no real changes in the release notes.
• We will probably forget this extra step at some point, and you'll use an outdated lookout-sdk binary.

What kind of compatibility issues are you trying to avoid?
We won't make any breaking change to the protocol. If we introduce any change, we'll try to make it backwards-compatible.
And if something changes we are probably talking about adding new features, in which case we'll put a disclaimer in the release notes pointing to the lowest src-d/lookout repo release that includes the new functionality. But you will still want to use the latest lookout-sdk bin, not just the one version fixed in this repo's release.

[vmarkovtsev]

@carlosms We seem to misunderstand each other here.

I am talking about versioning 101: lookout-sdk is a Python package which is intended to work with a specific range of versions of lookout-sdk binary from the other repository. The current range is 0-latest, as you noted. You will eventually change the protocol in the future, won't you?

Suppose that you add a feature. lookout-sdk binary is cached on our side, because we are not going to download them from GitHub Releases each time we run a test or a dependee script. Hence we need to check the version and if it is too old, download the recommended updated one.

Another example: timeouts this week. You decided to remove timeouts. All our stuff is currently supposing v0.5.0. And here is what happens next: Egor launches a production script which takes 12 hours to complete, it uses the cached outdated binary, we figure that out the following day after losing much time and eventually delaying the release.

Another example: you added the Babelfish proxy. We run our test suite with updated client code and it fails miserably because, again, it uses the old cached binary.

Another example: source{d} engine contains the hardcoded version numbers of all the components which are guaranteed to work correctly together.

Of course, we can continue tracking this dependency ourselves in lookout-sdk-ml. I just thought that "versioning 101" should be in the base package.

[smacker]

I think you inverted the responsibility here.
SDK shouldn’t define which version of server it can be used with. Server should define which SDK it’s compatible with.
Lookout-sdk binary is a lightweight version of lookoutd and it follows the same release process. We guarantee that the same version of this tool and the real server will provide the same features (like proxing bblfsh).
To remove this confusion we are going to rename lookout-sdk cli to something else that reflects it better.

P.S. in your example of engine. We don’t define engine version in components. We define components in engine.

[vmarkovtsev]

@smacker "Responsibility inversion" does not solve our problems, unfortunately. We continue listing lookout-sdk package in requirements.txt and getting an inconsistent state with the binary version.
What do you propose to solve the problems I described?

[smacker]

Sorry, I don't see in any of your examples where you had lookout-sdk package an inconsistent state with the binary version.

Any version since v0.0.4 sdk version and v0.1.5+ server are fully compatible with each other.

You are getting incompatibilities of your scripts / code with the testing utility, not the sdk.

I would propose to fix version of the testing utility somewhere in your test scripts. When you start using new server feature, you just bump version in the same commit that introduces changes to your codebase. If your scripts rely on some new fixes in the utilities you also bump version there.

Does it make sense for you?

We are really willing to help you to solve your problem. If the proposal above doesn't work for you please let us know more details and we will come up with some other better solution together.

[vmarkovtsev]

That's what we already do exactly: we point to the binary version in version.py. FWIW it's not only the tests but also research and benchmarks. OK, let me think if I can propose another solution for this.

[smacker]

If I understand you correctly you are looking not for exact version but for some automatization that would give you the latest binary which wouldn't break anything.

For that we can provide python script similar to this one https://github.com/src-d/lookout-sdk/blob/master/_tools/install-lookout-latest.sh + add caching there and when you call it you can specify a range like ^1.5.0.

The problem is it wouldn't solve your problem right now. Lookout is unstable 0.x.x now, so you can't rely on semver and have to track changes manually.

That's how semver works for unstable versions:

• when we add a new feature you would like to have automatically we would bump minor number
• when we introduce breaking change (change arguments of the binary for example) we would also bump minor number instead of major.

So you will have to fix minor version and then there is no much value in such script right now.