Is it possible to reference needs with intersphinx? / Add intersphinx support

[lassebn]

As the title says I'd like to know if it is possible to cross reference using intersphinx.

As a try out I tried adding the following to conf.py

extensions = [
  "sphinx.ext.intersphinx",
]

intersphinx_mapping = {
  'need': (
    'https://sphinxcontrib-needs.readthedocs.io/en/latest/',
     None
  )
}

and referencing using

:need:`:need:SPEC-1`

or

:need:`SPEC-1`

(SPEC-1 is defined on the front page of the documentation), but no such luck. Has anybody tried this?

[danwos]

No, I'm afraid this is not possible and I haven't worked with intersphinx up to now.

Okay, did just a fast training on intersphinx and how it works.
And Sphinx-Needs is definitely doing nothing to get its data into the object.inv.
Here is a link to the interpreted content of the object.inv file of Sphinx-Needs docs:
https://gist.github.com/danwos/8ac053396f449c52e569de2edcbe7712
No need-ids in it :(

I'm just not sure how to get own data into the object.inv file.
It looks like that Sphinx does this automatically for all registered Domains.

So one solution would be to register a domain for Sphinx-Needs.
But then we have to add automatically all needs to this domain.
Not sure if there will be any conflicts, as Sphinx-Needs is not using domains and there may be some default handling for domains, which we don't want... (module entry, needed representation in docs, ...)

Another solution would be to add our needs to the std:label domain.
But here the same as above, how and may there be any conflicts?

I'm willing to support this great feature, but I have to do some tests, to figure out how to do it.
So it will need some time...

If there is anybody who can help, maybe with a PR, you are welcome :)

[lassebn]

Thank you for your fast reply and willingness to support intersphinx in the future 🙌

I've never done any work on any extensions for Sphinx yet. At the moment I'm just using your excellent extension to write some documentation.

Having the ability to link needs across projects might become important enough for us at my work for me to get time allocated to work on it. Realistically that will be months out in the future though :(

[danwos]

The last days I have thought a lot about this feature and how to implement it.
My proposal is to not support intersphinx but to provide another mechanism, which allows the same behavior: Getting links to external needs into your document.

It would be great, if you could give me your opinion, if it covers your case.

Reason why not using intersphinx

1. I have no idea how to do it ;)
2. The intersphinx mechanism would not support other use cases, like getting additional data from external needs, showing external needs in tables or flows, using their data in dynamic functions...
3. We already have the needs builder for exporting data, so this part is already done.

Long-term goals of the new concept

1. Getting links to external needs
2. Showing external needs in tables and flows
   This includes being able to filter on their data!
3. Giving access to their data for dynamic functions and co.

Why not using needimport?

needimport creates a copy of external needs so that they become internal and are not linked to external resources anymore.

New concept

New conf-option needs_intersphinx

needs_intersphinx = [
  ('http://mydocs/my_project', 'http://mydocs/my_project/needs.json', '1.0'),
  ('http://mydocs/another_project/', 'http://mydocs/another_project/data/exports/needs.json', '2..5'),
  'http://docs.com/simple_project'
]

needs_intersphinx takes a list of tuples which define:

• An url/path to use as root-path for external links
• An url/path to an external needs.json file
• A specific version to read (if not give, latest is used)

The first option can also be given as single string. In this case the needs.json must be stored on root level and the version is automatically set to latest.

New conf-option needs_autoexport

needs_autoexport = True

If set to True, the functions to build a needs.json file will be called even if other builders are used, e.g. html.
In this case it is easy to always provide a needs.json file without caring about to call the needs builder.

Update of need-role

The need-role will search for a given ID in this order:

1. internal needs
2. external needs

If an id is found in the external needs, a link to the exact location in the external documentation is generated.

New need-value: doc_path

The needs.json does not store any information, in which document the need can be found.
This must be changed by adding a new, automatically calculated option to the internal need-dict: doc_path.
This value stores the path relative to the doc-root folder (conf.py location)

Suggested behavior

During a build, all need.json files given by needs_intersphinx will get downloaded and internally imported as standalone information (no link to the internal needs).
Beside the role :need: there will be no way to work with this data (for now)

Drawbacks

• IDs should be unique. This must cover the internal needs and all external needs.
• A build will always need an internet connection, if needs.json files needs to be imported.
  (But i think this drawback also intersphinx has.)
• Big needs.json files may need some time to be downloaded.

So, any ideas/feedback about this concept?
Will it work for your cases?

[lassebn]

Excellent writeup! ✨

This would cover our use cases and I think the benefits you've outlined definitely outweighs the drawbacks 👍

Nitpicking details
I'm a bit conflicted about the naming of the new conf-option needs_intersphinx. It does indeed link Sphinx documentation, but might people confuse it with support for intersphinx? needs_external_needs might be a better name?

Questions

• Would it be possible to support multiple needs with the same id at a later point? Maybe by providing a namespace for each of the external sources. All our needs are at the moment unique, but I can imagine that won't always hold for all use-cases. Maybe that is a symptom of too broad needs, but nevertheless. I guess it is just something to think about.
• Can it support getting the external need.json from a site which is not publicly available? Something like supporting basic http authentication like interspinx does.

[danwos]

Good points.

needs_external_needs is fine. Lets take this name.

For same id of multiple needs we could use a prefix, like need_import does

needs_external_needs = [
  ('http://mydocs/my_project', 'http://mydocs/my_project/needs.json', '1.0', 'ext_'),
  ('http://mydocs/another_project/', 'http://mydocs/another_project/data/exports/needs.json', '2..5', 'other_'),
  'http://docs.com/simple_project'
]

So the last parameter is a prefix, which is added to all imported needs ids. E.g. req_123 --> ext_req_123.
However, this modified id will then always be used as id for the need, also in tables and co.
So the user must know that he/she needs to search for req_123 in the external document, if he/shed reads something about ext_req_123 in the main doc. But a reader normally follows the links, this shouldn't be a big drawback.

And for authentication, I plan to use requests lib so at least BasicAuth should be possible. And the integration of other methods should be easy enough, as long as the method is supported by requests. But with other implementations I will wait for a use case :)

[lassebn]

👍 to all of it.

[bngoy]

+1 for the feature !

[twodrops]

+1 for the feature too

[twodrops]

@danwos I did a quick test of the feature branch. I couldn't wait long to test it out 😃
The feature is really cool 🥇

Two issues I quickly noticed:

Issue 1
The second build after a successful first build always fails with the following error:

A need with ID REQ_xxxxxxx already exists! This is not allowed.

A "clean" build or the "third build" succeeds.

Issue 2
The needs in the externel json appears correctly (for example, in a needtable). If the imported need is referred elsewhere, the need ID does not appear in the need that references it.

[danwos]

@twodrops Thanks for already testing it 👍
Issue 2 got solved.
Issue 1 I may need some time to reproduce it.

[twodrops]

@danwos Thanks for that.

I couldn't get the base_url working somehow. The generated URL somehow does not have the .html in the URL to point to the correct file. Maybe I am doing something wrong.