Is Search Code API reliable?

[FabioBis]

Select Topic Area

Question

Body

I'm talking about the standard search engine --not the new beta one--

Long story short: I'm working on a tool to keep track of python requirements for my team. This tool use the API to fetch all repositories that include a given requirement. Testing the tool I've seen that each run gave me different results and I figured out that the issue came from the API response.

Calling the following request many times over time give me different results:
GET https://api.github.com/search/code?q=<requirement_name> in:file org:<org_name> language:text filename:requirement filename:requirements&per_page=100
What I see is that at the beginning the "total_count" value vary over time (330, 227, 412, ...). Remark: always "incomplete_results": false.
After a while it seems to stablize at 426.
If I wait few minutes before a new run it will start to vary again (seem to me a sort of caching mechanism).

The same issue happens using the pagination mechanims (next Link on headers)

So I made an experiment with a (hugly) workaround that check for a stable total_count (426 in my case) and retry each request (page) if the total_count value does not match. That way I obtained less discrepancy between results: 2 or 3 over 229 repos against up to 50 without the workaround. Missing repos are not the same each run, look like a non-deterministic behavior.

So my questions are:

1. Is this an issue or it works as intended (best effort)?
2. Is there a way to get complete and deterministic results from that kind of search? (a workaround, best practice, or a different approach)
3. Will the new search engine provide APIs that solve this problem?

Thanks in advance for Your feedback and sorry for the long post.

[look]

@FabioBis sorry about that. The code search API should be treated as best effort right now. We are working on a new API which should be more reliable.

As a workaround, consider using the content API to fetch the file content from the repositories you're interested in.

[FabioBis]

@look thank You for your quick answer. Can't wait to try the new API.

[yanickrochon]

If there is a new search feature coming, that will be great, because I never find anything with the current search feature (top left search field). Entering a search query either A) does not return anything when it should, or B) return something that does not match the search terms.

In short, the current search feature is quite useless and broken.

Kinda reminds me of Bing.... oh...

[sdann]

Adding on our very similar problem with likely the same root cause. I'm trying a further reduced search from OP looking for a specific file (of which there are 1 or 2) within a specific repository. Example:

GET https://api.github.com/search/code?q=repo:<org_name>/<repo-name> filename:yarn.lock&per_page=100

And this is returning 0 results, consistently now, where it would return 2 results just 1 month ago in April 2023.

I understand from this thread that the current Search API is Best Effort, but it is still surprising given a specific repository constraint, that it can't find all file metadata. i.e. no content search, just tell me the existence of a file name in a repo.

[saddamsial]

We have switched the backend of the code search API to the new system, which does not index yarn.lock files because they are generated. Sorry to break your workflow, but it is unlikely we will change this in the near term.

Will you please elaborate I want to search some specific code search why your new api is giving results of 2012? I think there should b option to search code with in a specific time period.(like find healthhandler last week or last month etc)

[look]

@saddamsial time based sorting is no longer supported. Our Changelog announcement for this has some alternative approaches you can use.

[sdann]

We have switched the backend of the code search API to the new system, which does not index yarn.lock files because they are generated. Sorry to break your workflow, but it is unlikely we will change this in the near term.

@look Thanks for the explicit confirmation on behavior change. We'll update our workflow.

The generated vs non-generated files has some nuance. I understand why you're not including generated files in the index (save time/space), but something like yarn.lock is explicitly versioned as an input to the build pipeline so that build outputs are consistent. It absolutely belongs in version control along side the source code that's using the dependencies it locks to.

[look]

Agreed! We would like to index all textual files eventually but we had to start with some exclusion of generated and vendored files to keep the index size manageable.

[saddamsial]

I appreciate your response and I hope there will be some better solution soon. I think "path" qualifier should be for repositeries search too.(e.g search repositories that have this "path"). Thanks!