Why are my public repositories not found by GitHub search?

[0cjs]

Select Topic Area

Bug

Body

I have a repo https://github.com/cynic-net/dent with tags including docker, a description including 'Start a shell in arbitrary Docker containers,' a title at the start of the top-level README.md of 'dent: Create and Enter Docker Containers,' a file called bin/dent, and plenty more that kind of strongly indicates that 'dent' and 'docker' should be good search terms for this.

However, when I use the GitHub search functionality to search for 'dent' it finds about 22.7 thousand repos, and mine isn't in the first few pages of that, despite being named exactly dent. (The first page, as well as including a repo named Dent, includes repos named AlDente-Charge-Limiter, dentistsmile, denti, and so on.)

No problem; I know my repo is related to Docker so I'll search for 'dent docker'. This narrows things down quite a bit, to just 7 repos. But mine is not there. (Most are docker-dental, dentist-docker, and so on.)

Fine. My dent repo is under the cynic-net org: surely searching for 'dent cynic-net' will turn it up. But no, that produces no results at all.

I've seen this with some of my other public repos as well: GitHub search simply doesn't seem to admit that those repos even exist. What's going on with these repos, and how do I get them listed in GitHub search?

As a side note, DuckDuckGo, Google and Bing have no problem at all finding these repos: for the search 'dent docker' my repo is first in DuckDuckGo and Bing results, results and second in Google results.

[joemar25]

It's super annoying when you know your repo is public and should be showing up, but for some reason it's just not there in the search results.

First off, you can try to double check a couple things. Make sure your repo is set to public in the settings - I know it sounds obvious, but sometimes it's the little things, you know? Also, take a peek under the "Danger Zone" section in your repo settings and make sure the "Search engine indexing" box is checked. If it's not, GitHub won't include your repo in search results.

Another thing to check is that you've got a README file in there. GitHub tends to favor repos with nice README's in their search rankings. If you've got one, great! If not, try adding one and see if that helps.

Now, if you've checked all that and your repo still isn't showing up, it could just be that GitHub's search index is a bit behind. Sometimes it takes a while for changes to show up in their search. You could try making a small change to your repo (like editing the README) and pushing it to see if that nudges the search index to update.

If none of that works, it might be time to ping GitHub support. They should be able to take a look under the hood and figure out why your repo is being shy in the search results.

It's funny though - external search engines like Google and DuckDuckGo seem to have no problem finding your repo! They've got their own search systems that are separate from GitHub's. Guess that's why your repo is popping up first or second in their results.

Anyway, I hope some of those suggestions help! If not, definitely reach out to GitHub's support team. They're usually pretty helpful with this kind of thing. Best of luck getting your dent repo to show up where it belongs in those search results!

[0cjs]

@joemar25 Thanks for your reply.

1. Yes, it's definitely public. I always use the public download links (https://github.com/...) when cloning and later fetching from repos that are supposed to be public, so I'm constantly confirming that.

2. I have checked under the Settings / General tab (in both 'Danger Zone' and throughout the entire page) and I see no references to search engine indexing; doing 'Find' in the page finds 'search' only in the search bar and finds 'index' nowhere. (Perhaps this is only a setting for private repos or something like that?)

3. Yes, as mentioned in my original post, I have a README.md that is quite descriptive, including both keywords in the Markdown 'H1' level heading at the start of the README. Please feel free simply to click the link to my repo in the original post above and examine the rendered README that you see there.

4. The repo is not new and sees regular changes. The repo was renamed to dent and moved to the cynic-net org about seven months ago, and has had a couple of dozen changes in the four months after that.

I am, unfortunately, not exactly clear on how to reach out to GitHub's support team on this. That was what I was originally trying to do, but going through the various support pages eventually seemed to lead me here as the option I should choose. (I ended up at https://support.github.com/request and the only one of the fifteen options on that page even vaguely related to my problem is at the bottom, 'Other questions about Git or GitHub?', which links to here. I guess I'll try contacting the 'Account or Billing Issues' link and see if they can eventually get me to the right people to handle my request.

[Manjunathhs-0003]

Repository Visibility Settings: Ensure that your repository is set to be publicly visible. Sometimes, repositories might be set to private or restricted visibility unintentionally, which could affect their appearance in search results.

Repository Metadata: GitHub's search algorithm takes into account various metadata associated with repositories, such as the repository name, description, tags, and README content. Ensure that these elements accurately reflect the content and keywords relevant to your repository. From your description, it seems like you've already included relevant keywords like "dent" and "docker", which is a good practice.

README Optimization: The README file plays a crucial role in GitHub's search algorithm. Make sure your README contains detailed and relevant information about your project, including keywords and descriptions that potential users might search for. It's also helpful to include installation instructions, usage examples, and any other relevant details to make your repository more discoverable and user-friendly.

[0cjs]

Ensure that your repository is set to be publicly visible.

I clarified this in my response above your answer. If there's anything I've missed in checking that it's public, please let me know. (Also, you could just click on the link and see for yourself that it is public.)

README Optimization....

As I mentioned in my original post, I've got a good title at the start of the README, and so on. Again, perhaps you could just click on the link and have a look at the README yourself and tell me what you feel could be improved. (I've got pretty much everything you mentioned there.)

[look]

If you search by the terms in the description or use the exact name cynic-net/dent, the repository is returned, so it is in the index. I think the term dent is too common/short to give good results, unfortunately.

[0cjs]

Ok, well that's fair enough, I guess. (I don't like it that much, but that's a UI decision.)

But what's up with it refusing to find a repo if you include the actual repo name in the list of search words?

[look]

But what's up with it refusing to find a repo if you include the actual repo name in the list of search words?

I reviewed the query and it's using AND semantics. Since neither your repo description nor the name of the repo contain all the terms, they don't match. A simple workaround would be to rephrase your description to include the repo name, like "Dent starts a shell...".

I've passed my findings about the query on to the team that owns repository search for possible improvement.

[0cjs]

Well, the full name of the repo includes both cynic-net and dent, yet a search with those two terms doesn't find it. So I guess the name search is not searching on the name within the organisation and doesn't include the organisation name? That's something that you might want to consider adding, too.

At any rate, adding 'dent' to the description fixes this problem, more or less, though it still seems odd that my repo comes last in the list of eight returned by the search, despite being the only one that has those exact two words in the name and/or description. (The search really seems to give preference to the word 'dental' when I search for 'dent'.)

[look]

Well, the full name of the repo includes both cynic-net and dent, yet a search with those two terms doesn't find it. So I guess the name search is not searching on the name within the organisation and doesn't include the organisation name? That's something that you might want to consider adding, too.

Good suggestion. The full name with owner is indexed, but isn't searched by default. I think that would have helped here. I noted that to the team.

(The search really seems to give preference to the word 'dental' when I search for 'dent'.)

This is due to stemming. The search index doesn't know the original word was "dent". Since the results are considered equivalent in terms of relevance, other factors are applied to rank the results.

[0cjs]

Great to hear that we might get repository name and organisation included in that search. I think that would help a lot.

As for the stemming, I don't know a huge amount about that, but it looks to me as if it's done wrong for the word 'dent'. While there is apparently an engineering use of the word that is related to 'dental,' 'dentist,' etc. (from Middle French dental or Late Latin dentālis, in turn from dēns, "tooth"), it's rare enough that I'd never heard of it, nor have I ever heard it used. I think that 'dent' should be separately stemmed based on the etymology of its completely separate and much more common use (etymologically from Middle English dent, dente, dint, "a blow; strike; dent", Old English dynt , Proto-Germanic *duntiz), along with 'dented' and so on.

[rrrix]

@0cjs you can submit feedback for search result problems via the GitHub Code Search "provide feedback here" modal dialog box on the code search results page, found under the "Why wasn't my code found?" fold:

GitHub Code Search: cynic-net/dent README.md

Expand to see how to find that link (it's a bit hidden!)

Incidentally, I have a feeling there's a bug or a systemic underlying problem that you might have uncovered. Or maybe your repository was stuck in a dead letter queue or something.

Quite curiously, in the last 10 minutes of writing this post, code search and repository results changed between (approximately-ish) 2024-04-03 16:35:00+00:00 (UTC) and 16:42:00! This got me REALLY interested :)

Search Type	Search	Results Before	Results After
Code	cynic-net/dent README.md	0 Results	0 Results
Code	repo:cynic-net/dent README.md	0 Results † Banner message: "⚠️ This repository's code is being indexed right now. Try again in a few minutes."	Found! ✅
Code	cynic-net dent	0 Results	cynic-net/dent bin/dent is the top result
Code	dent	Lots of results, your repo not included ❌
Repository	cynic-net/dent	0 Results	Found! ✅
Repository	repo:cynic-net/dent	Found! ✅	Found! ✅
Repository	cynic-net dent	0 Results	Not Found ❌

† Banner message:

Maybe @look fixed something on the backend? :)

[0cjs]

Ah, you're doing a code search there, not a repo search, which has a different page that comes up. (The repo empty search results page has only notes on how to query across an organisation and save searches.)

Regardless, this was still useful since you pointing out that feedback linked prompted me to look around a bit more. There is indeed, a feedback link for repo search, too, but in the drop-down menu. So I've sent a query via that, mentioning this whole conversation here as well. (After several days I've still had no reply to the ticket I filed.)

Your experiments are interesting, though I'm not sure how relevant they are, since they all seem to be related to searching within the repo, whereas the lack of results that are causing a problem for me are just about searching for the repo itself based on the repo's title and description, rather than the files in it. (See my second reply to look above.) Though I certainly wouldn't mind if search for terms in the README could help bring it higher up in the results, or into the results period.