Skip to content

repo sync #4250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Mar 4, 2021
Merged
Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
6fa5bd6
add script to update internal links
sarahs Feb 24, 2021
2aa1347
add option to return cheerio object from renderContent instead of html
sarahs Feb 24, 2021
a687c39
lint
sarahs Feb 24, 2021
d747ab2
preserve any inline markup in link titles
sarahs Feb 24, 2021
50bbee7
lint
sarahs Feb 24, 2021
7fb5ab6
Update script/update-internal-links.js
sarahs Feb 24, 2021
d0e894f
Merge branch 'main' into check-for-outdated-links
sarahs Feb 24, 2021
030eb97
simplify boolean var
sarahs Feb 24, 2021
5f223ce
Merge branch 'check-for-outdated-links' of github.com:github/docs-int…
sarahs Feb 24, 2021
9bfb31c
Merge branch 'main' into check-for-outdated-links
sarahs Feb 26, 2021
9cc1350
Merge branch 'main' into check-for-outdated-links
chiedo Mar 2, 2021
02c2275
Merge branch 'main' of github.com:github/docs-internal into check-for…
sarahs Mar 2, 2021
4e75f03
handle currentVersion with spaces around it that prevent the AST pars…
sarahs Mar 2, 2021
a2a8437
Merge branch 'main' of github.com:github/docs-internal into check-for…
sarahs Mar 2, 2021
fc18a1b
fix the regex
sarahs Mar 2, 2021
86b39f8
script should not walk translations content dirs
sarahs Mar 2, 2021
27dc17f
Merge branch 'main' into check-for-outdated-links
sarahs Mar 2, 2021
67c90c6
lint
sarahs Mar 2, 2021
e4c696d
Merge branch 'check-for-outdated-links' of github.com:github/docs-int…
sarahs Mar 2, 2021
d5b99c8
Merge branch 'main' of github.com:github/docs-internal into check-for…
sarahs Mar 2, 2021
3911909
fixed broken links and images
mchammer01 Mar 3, 2021
e3ef829
fix versioning
mchammer01 Mar 3, 2021
78c759c
Merge branch 'main' into check-for-outdated-links
sarahs Mar 3, 2021
f328a83
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 3, 2021
95e5090
Merge branch 'main' of github.com:github/docs-internal into check-for…
sarahs Mar 3, 2021
39c6c0d
Merge branch 'main' into check-for-outdated-links
sarahs Mar 3, 2021
08d7c08
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 3, 2021
1b6180e
Merge branch 'main' into check-for-outdated-links
sarahs Mar 3, 2021
61f8bfa
Merge branch 'main' into check-for-outdated-links
sarahs Mar 3, 2021
cd2e02b
Merge branch 'main' into mchammer01/more-broken-link-fixes
JamesMGreene Mar 3, 2021
f8782bb
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 4, 2021
6579c24
Merge branch 'main' into mchammer01/more-broken-link-fixes
mchammer01 Mar 4, 2021
179f97b
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 4, 2021
cac92ef
Merge branch 'main' into mchammer01/broken-links
shati-patel Mar 4, 2021
012c39b
fix table formatting issue
mchammer01 Mar 4, 2021
be358ad
Merge branch 'mchammer01/more-broken-link-fixes' of github.com:github…
mchammer01 Mar 4, 2021
d63b83d
Merge branch 'main' into mchammer01/more-broken-link-fixes
mchammer01 Mar 4, 2021
43b02f0
Merge branch 'main' into mchammer01/more-broken-link-fixes
mchammer01 Mar 4, 2021
2e4a684
Update content/github/setting-up-and-managing-your-github-user-accoun…
mchammer01 Mar 4, 2021
eafc9ed
Merge branch 'main' into mchammer01/more-broken-link-fixes
mchammer01 Mar 4, 2021
178f221
repo sync
Octomerger Mar 4, 2021
d42a6a6
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 4, 2021
ec1ffe4
Merge pull request #18055 from github/mchammer01/more-broken-link-fixes
mchammer01 Mar 4, 2021
e1d2b62
Merge branch 'main' into mchammer01/broken-links
mchammer01 Mar 4, 2021
c889456
Merge pull request #18054 from github/mchammer01/broken-links
mchammer01 Mar 4, 2021
4ae3652
Branch was updated using the 'autoupdate branch' Actions workflow.
Octomerger Mar 4, 2021
612324e
repo sync
Octomerger Mar 4, 2021
c01b7fa
Remove Browser Tests until it works again (#18088)
chiedo Mar 4, 2021
05c3952
Update OpenAPI Descriptions
github-openapi-bot Mar 4, 2021
d9139cb
Branch was updated using the 'autoupdate branch' Actions workflow.
Octomerger Mar 4, 2021
6fec12b
Merge branch 'main' into check-for-outdated-links
sarahs Mar 4, 2021
e1cc751
repo sync
Octomerger Mar 4, 2021
f8c6a62
Merge branch 'main' into check-for-outdated-links
sarahs Mar 4, 2021
fb9103e
Merge pull request #17981 from github/check-for-outdated-links
sarahs Mar 4, 2021
4e7d920
Branch was updated using the 'autoupdate branch' Actions workflow.
Octomerger Mar 4, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/browser-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ name: Browser Tests

on:
workflow_dispatch:
push:
branches:
- main
pull_request:
# push:
# branches:
# - main
# pull_request:

jobs:
see_if_should_skip:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ You can search for the following information across all repositories you can acc
- [Discussions](/github/searching-for-information-on-github/searching-discussions){% endif %}
- [Code](/articles/searching-code)
- [Commits](/articles/searching-commits)
- [Users](/articles/searching-users){% if currentVersion == "free-pro-team@latest" or currentVersion == "github-ae@latest" or currentVersion ver_gt "enterprise-server@2.1" %}
- [Users](/articles/searching-users){% if currentVersion == "free-pro-team@latest" or currentVersion == "github-ae@latest" or currentVersion ver_gt "enterprise-server@2.21" %}
- [Packages](/github/searching-for-information-on-github/searching-for-packages){% endif %}
- [Wikis](/articles/searching-wikis)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ The repository owner has full control of the repository. In addition to the acti
| Delete the repository | "[Deleting a repository](/github/administering-a-repository/deleting-a-repository)" |
| Manage the repository's topics | "[Classifying your repository with topics](/github/administering-a-repository/classifying-your-repository-with-topics)" |{% if currentVersion == "free-pro-team@latest" %}
| Manage security and analysis settings for the repository | "[Managing security and analysis settings for your repository](/github/administering-a-repository/managing-security-and-analysis-settings-for-your-repository)" |{% endif %}{% if currentVersion == "free-pro-team@latest" %}
| Enable the dependency graph for a private repository | "[Exploring the dependencies of a repository](/github/visualizing-repository-data-with-graphs/exploring-the-dependencies-of-a-repository#enabling-and-disabling-the-dependency-graph-for-a-private-repository)" |{% endif %}
{% if currentVersion == "free-pro-team@latest" or currentVersion ver_gt "[email protected]" %}| Delete and restore packages | "[Deleting and restoring a package](/packages/learn-github-packages/deleting-and-restoring-a-package)" |{% elsif currentVersion ver_lt "[email protected]" or currentVersion == "github-ae@latest" %}| Delete packages | "[Deleting packages](/packages/learn-github-packages/deleting-a-package)" |{% endif %}
| Enable the dependency graph for a private repository | "[Exploring the dependencies of a repository](/github/visualizing-repository-data-with-graphs/exploring-the-dependencies-of-a-repository#enabling-and-disabling-the-dependency-graph-for-a-private-repository)" |{% endif %}{% if currentVersion == "free-pro-team@latest" or currentVersion ver_gt "[email protected]" %}
| Delete and restore packages | "[Deleting and restoring a package](/packages/learn-github-packages/deleting-and-restoring-a-package)" |{% endif %}{% if currentVersion == "[email protected]" or currentVersion == "[email protected]" or currentVersion == "github-ae@latest" %}
| Delete packages | "[Deleting packages](/packages/learn-github-packages/deleting-a-package)" |{% endif %}
| Customize the repository's social media preview | "[Customizing your repository's social media preview](/github/administering-a-repository/customizing-your-repositorys-social-media-preview)" |
| Create a template from the repository | "[Creating a template repository](/github/creating-cloning-and-archiving-repositories/creating-a-template-repository)" |{% if currentVersion == "free-pro-team@latest" or enterpriseServerVersions contains currentVersion %}
| Receive {% if currentVersion == "free-pro-team@latest" or currentVersion ver_gt "[email protected]" %}{% data variables.product.prodname_dependabot_alerts %}{% else %}security alerts{% endif %} for vulnerable dependencies | "[About alerts for vulnerable dependencies](/github/managing-security-vulnerabilities/about-alerts-for-vulnerable-dependencies)" |{% endif %}{% if currentVersion == "free-pro-team@latest" %}
Expand Down
2 changes: 1 addition & 1 deletion data/reusables/user-settings/oauth_apps.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
1. In the left sidebar, click **OAuth Apps**.
![OAuth Apps section](/assets/images/settings/developer-settings-oauth-apps.png)
![OAuth Apps section](/assets/images/help/settings/developer-settings-oauth-apps.png)
4 changes: 4 additions & 0 deletions lib/render-content/renderContent.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ module.exports = async function renderContent (
.trim()
}

if (options.cheerioObject) {
return cheerio.load(html, { xmlMode: true })
}

if (options.encodeEntities) html = entities.encode(html)

return html.trim()
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/api.github.com.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -139709,7 +139709,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -139719,6 +139719,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-2.18.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -79203,7 +79203,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -79213,6 +79213,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-2.19.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -82135,7 +82135,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -82145,6 +82145,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-2.20.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -84244,7 +84244,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -84254,6 +84254,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-2.21.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -95077,7 +95077,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -95087,6 +95087,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-2.22.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -121969,7 +121969,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -121979,6 +121979,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/ghes-3.0.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -127095,7 +127095,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -127105,6 +127105,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
3 changes: 2 additions & 1 deletion lib/rest/static/dereferenced/github.ae.deref.json
Original file line number Diff line number Diff line change
Expand Up @@ -109550,7 +109550,7 @@
"name",
"head_sha"
],
"anyOf": [
"oneOf": [
{
"properties": {
"status": {
Expand All @@ -109560,6 +109560,7 @@
}
},
"required": [
"status",
"conclusion"
],
"additionalProperties": true
Expand Down
197 changes: 197 additions & 0 deletions script/update-internal-links.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
#!/usr/bin/env node

const fs = require('fs')
const walk = require('walk-sync')
const path = require('path')
const astFromMarkdown = require('mdast-util-from-markdown')
const visit = require('unist-util-visit')
const { loadPages, loadPageMap } = require('../lib/pages')
const loadSiteData = require('../lib/site-data')
const loadRedirects = require('../lib/redirects/precompile')
const { getPathWithoutLanguage, getPathWithoutVersion } = require('../lib/path-utils')
const allVersions = Object.keys(require('../lib/all-versions'))
const frontmatter = require('../lib/read-frontmatter')
const renderContent = require('../lib/render-content')
const patterns = require('../lib/patterns')

const walkFiles = (pathToWalk) => {
return walk(path.posix.join(__dirname, '..', pathToWalk), { includeBasePath: true, directories: false })
.filter(file => file.endsWith('.md') && !file.endsWith('README.md'))
.filter(file => !file.includes('/early-access/')) // ignore EA for now
}

const allFiles = walkFiles('content').concat(walkFiles('data'))

// The script will throw an error if it finds any markup not represented here.
// Hacky but it captures the current rare edge cases.
const linkInlineMarkup = {
emphasis: '*',
strong: '**'
}

const currentVersionWithSpacesRegex = /\/enterprise\/{{ currentVersion }}/g
const currentVersionWithoutSpaces = '/enterprise/{{currentVersion}}'

// [start-readme]
//
// Run this script to find internal links in all content and data Markdown files, check if either the title or link
// (or both) are outdated, and automatically update them if so.
//
// Exceptions:
// * Links with fragments (e.g., [Bar](/foo#bar)) will get their root links updated if necessary, but the fragment
// and title will be unchanged (e.g., [Bar](/noo#bar)).
// * Links with hardcoded versions (e.g., [Foo](/enterprise-server/baz)) will get their root links updated if
// necessary, but the hardcoded versions will be preserved (e.g., [Foo](/enterprise-server/qux)).
// * Links with Liquid in the titles will have their root links updated if necessary, but the titles will be preserved.
//
// [end-readme]

main()

async function main () {
console.log('Working...')
const pageList = await loadPages()
const pageMap = await loadPageMap(pageList)
const redirects = await loadRedirects(pageList)
const site = await loadSiteData()

const context = {
pages: pageMap,
redirects,
site: site.en.site,
currentLanguage: 'en'
}

for (const file of allFiles) {
const { data, content } = frontmatter(fs.readFileSync(file, 'utf8'))
let newContent = content

// Do a blanket find-replace for /enterprise/{{ currentVersion }}/ to /enterprise/{{currentVersion}}/
// so that the AST parser recognizes the link as a link node. The spaces prevent it from doing so.
newContent = newContent.replace(currentVersionWithSpacesRegex, currentVersionWithoutSpaces)

const ast = astFromMarkdown(newContent)

// We can't do async functions within visit, so gather the nodes upfront
const nodesPerFile = []

visit(ast, node => {
if (node.type !== 'link') return
if (!node.url.startsWith('/')) return
if (node.url.startsWith('/assets')) return
if (node.url.startsWith('/public')) return
if (node.url.includes('/11.10.340/')) return
if (node.url.includes('/2.1/')) return
if (node.url === '/') return

nodesPerFile.push(node)
})

// For every Markdown link...
for (const node of nodesPerFile) {
const oldLink = node.url

// Find and preserve any inline markup in link titles, like [*Foo*](/foo)
let inlineMarkup = ''
if (node.children[0].children) {
inlineMarkup = linkInlineMarkup[node.children[0].type]

if (!inlineMarkup) {
console.error(`Cannot find an inline markup entry for ${node.children[0].type}!`)
process.exit(1)
}
}

const oldTitle = node.children[0].value || node.children[0].children[0].value
const oldMarkdownLink = `[${inlineMarkup}${oldTitle}${inlineMarkup}](${oldLink})`

// As a blanket rule, only update titles in links that begin with quotes. (Many links
// have punctuation before the closing quotes, so we'll only check for opening quotes.)
// Update: "[Foo](/foo)
// Do not update: [Bar](/bar)
const hasQuotesAroundLink = newContent.includes(`"${oldMarkdownLink}`)

let foundPage, fragmentMatch, versionMatch

// Run through all supported versions...
for (const version of allVersions) {
context.currentVersion = version
// Render the link for each version using the renderContent pipeline, which includes the rewrite-local-links plugin.
const $ = await renderContent(oldMarkdownLink, context, { cheerioObject: true })
let linkToCheck = $('a').attr('href')

// We need to preserve fragments and hardcoded versions if any are found.
fragmentMatch = oldLink.match(/(#.*$)/)
versionMatch = oldLink.match(/(enterprise-server(?:@.[^/]*?)?)\//)

// Remove the fragment for now.
linkToCheck = linkToCheck
.replace(/#.*$/, '')
.replace(patterns.trailingSlash, '$1')

// Try to find the rendered link in the set of pages!
foundPage = findPage(linkToCheck, pageMap, redirects)

// Once a page is found for a particular version, exit immediately; we don't need to check the other versions
// because all we care about is the page title and path.
if (foundPage) {
break
}
}

if (!foundPage) {
console.error(`Can't find link in pageMap! ${oldLink} in ${file.replace(process.cwd(), '')}`)
process.exit(1)
}

// If the original link includes a fragment OR the original title includes Liquid, do not change;
// otherwise, use the found page title. (We don't want to update the title if a fragment is found because
// the title likely points to the fragment section header, not the page title.)
const newTitle = fragmentMatch || oldTitle.includes('{%') || !hasQuotesAroundLink ? oldTitle : foundPage.title

// If the original link includes a fragment, append it to the found page path.
// Also remove the language code because Markdown links don't include language codes.
let newLink = getPathWithoutLanguage(fragmentMatch ? foundPage.path + fragmentMatch[1] : foundPage.path)

// If the original link includes a hardcoded version, preserve it; otherwise, remove versioning
// because Markdown links don't include versioning.
newLink = versionMatch ? `/${versionMatch[1]}${getPathWithoutVersion(newLink)}` : getPathWithoutVersion(newLink)

let newMarkdownLink = `[${inlineMarkup}${newTitle}${inlineMarkup}](${newLink})`

// Handle a few misplaced quotation marks.
if (oldMarkdownLink.includes('["')) {
newMarkdownLink = `"${newMarkdownLink}`
}

// Stream the results to console as we find them.
if (oldMarkdownLink !== newMarkdownLink) {
console.log('old link', oldMarkdownLink)
console.log('new link', newMarkdownLink)
console.log('-------')
}

newContent = newContent.replace(oldMarkdownLink, newMarkdownLink)
}

fs.writeFileSync(file, frontmatter.stringify(newContent, data, { lineWidth: 10000 }))
}

console.log('Done!')
}

function findPage (tryPath, pageMap, redirects) {
if (pageMap[tryPath]) {
return {
title: pageMap[tryPath].title,
path: tryPath
}
}

if (pageMap[redirects[tryPath]]) {
return {
title: pageMap[redirects[tryPath]].title,
path: redirects[tryPath]
}
}
}