Merge branch 'main' into wdqs-separate-concept-uri

Author: deepeshaburse

build/wdqs-frontend/README.md

@@ -2,6 +2,14 @@
 
 Frontend for the [Wikidata Query Service (WDQS)](https://www.mediawiki.org/wiki/Wikidata_Query_Service).
 
+To interact with the WDQS frontend, navigate to the URL corresponding to the port allocated for it. In the example below, the WDQS frontend is available at `http://localhost:8834`.
+
+When writing queries using the frontend interface, click "Code" to view the corresponding URL.
+
+For general instructions on using WDQS, building SPARQL queries, and additional resources, see:
+- [Wikidata Query Service User Manual](https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual)
+- [What is SPARQL](https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service)
+
 > 💡 This image is part of Wikibase Suite (WBS). [WBS Deploy](https://github.com/wmde/wikibase-release-pipeline/deploy/README.md) provides everything you need to self-host a Wikibase instance out of the box.
 
 ## Requirements
@@ -21,11 +29,11 @@ We suggest using the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wiki
 
 ### WDQS as server
 
-We suggest using the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wdqs).
+We suggest using the [WBS WDQS image](https://hub.docker.com/r/wikibase/wdqs).
 
 ### WDQS as updater
 
-We suggest using the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wdqs), the same as used for WDQS server. Check out the [documentation](https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service) to learn how to run it in updater mode.
+We suggest using the [WBS WDQS image](https://hub.docker.com/r/wikibase/wdqs), the same as used for WDQS server. Check out the [documentation](https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service) to learn how to run it in updater mode.
 
 ### Reverse proxy
 
@@ -185,7 +193,7 @@ volumes:
 
 Official releases of this image can be found on [Docker Hub wikibase/wdqs-frontend](https://hub.docker.com/r/wikibase/wdqs-frontend).
 
-## Tags and Versioning
+## Tags and versioning
 
 This WDQS Frontend image is using [semantic versioning](https://semver.org/spec/v2.0.0.html).
 
@@ -206,4 +214,4 @@ This image is built from this [Dockerfile](https://github.com/wmde/wikibase-rele
 
 This image is maintained by the Wikibase Suite Team at [Wikimedia Germany (WMDE)](https://wikimedia.de).
 
-If you have questions not listed above or need help, use this [bug report form](https://phabricator.wikimedia.org/maniphest/task/edit/form/129/) to start a conversation with the engineering team.
+If you have questions not covered above or need further help, use this [bug report form](https://phabricator.wikimedia.org/maniphest/task/edit/form/129/) to start a conversation with the engineering team.

build/wdqs/Dockerfile

@@ -54,6 +54,7 @@ RUN apt-get update && \
 COPY --from=fetcher --chown=blazegraph:blazegraph /tmp/wdqs-service /wdqs
 
 RUN mkdir /var/log/wdqs && chown blazegraph /var/log/wdqs
+RUN mkdir /wdqs/data && chown blazegraph /wdqs/data
 
 # Don't set a memory limit otherwise bad things happen (OOMs)
 ENV MEMORY=""\
@@ -77,4 +78,6 @@ COPY --chown=blazegraph:blazegraph RWStore.properties allowlist.txt logback.xml
 # TODO this shouldn't be needed, but CI currently doesnt check for the +x bit, which is why this line is here
 RUN chmod +x /wdqs/runUpdate.sh
 
+USER blazegraph
+
 ENTRYPOINT ["/entrypoint.sh"]

build/wdqs/README.md

@@ -23,6 +23,9 @@ We suggest using the [WBS Wikibase image](https://hub.docker.com/r/wikibase/wiki
 
 You'll need one instance of the image to execute the actual WDQS daemon started using `/runBlazegraph.sh`.
 
+You can send `GET` requests with your SPARQL query to the WDQS endpoint (following the example below):
+`http://wdqs:9999/bigdata/namespace/wdq/sparql?query={SPARQL}`
+
 ### WDQS as updater
 
 You'll need one instance of the image to execute the updater started using `/runUpdate.sh`. This polls changes from Wikibase.
@@ -213,7 +216,7 @@ In the Docker Compose example provided above, you might use the commands and ins
 docker compose stop wdqs-updater
 
 # Start an updater with force sync settings
-docker compose run --rm wdqs-updater bash '/wdqs/runUpdate.sh -h http://"$WDQS_HOST":"$WDQS_PORT" -- --wikibaseUrl "$WIKIBASE_SCHEME"://"$WIKIBASE_HOST" --conceptUri "$WIKIBASE_CONCEPT_URI" --entityNamespaces "$WDQS_ENTITY_NAMESPACES" --init --start $(date +%Y%m%d000000)'
+docker compose run --rm wdqs-updater /wdqs/runUpdate.sh -h http://\$WDQS_HOST:\$WDQS_PORT -- --wikibaseUrl \$WIKIBASE_SCHEME://\$WIKIBASE_HOST --conceptUri \$WIKIBASE_CONCEPT_URI --entityNamespaces \$WDQS_ENTITY_NAMESPACES --init --start $(date +%Y%m%d000000)
 
 # As soon as you see "Sleeping for 10 secs" in the logs, press CTRL-C to stop it again
 

build/wdqs/entrypoint.sh

@@ -4,11 +4,11 @@
 # Test if required environment variables have been set
 REQUIRED_VARIABLES=(WIKIBASE_HOST WDQS_HOST WDQS_PORT)
 for i in "${REQUIRED_VARIABLES[@]}"; do
-    eval THISSHOULDBESET=\$"$i"
-    if [ -z "$THISSHOULDBESET" ]; then
+  eval THISSHOULDBESET=\$"$i"
+  if [ -z "$THISSHOULDBESET" ]; then
     echo "$i is required but isn't set. You should pass it to docker. See: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file";
     exit 1;
-    fi
+  fi
 done
 
 set -eu
@@ -17,15 +17,5 @@ export BLAZEGRAPH_OPTS="${BLAZEGRAPH_EXTRA_OPTS} -DwikibaseHost=${WIKIBASE_HOST}
 export UPDATER_OPTS="-DwikibaseHost=${WIKIBASE_HOST} -DwikibaseMaxDaysBack=${WIKIBASE_MAX_DAYS_BACK}"
 
 envsubst < /templates/mwservices.json > /wdqs/mwservices.json
-chown blazegraph:blazegraph /wdqs/mwservices.json
 
-# The data directory should always be owned by the blazegraph user
-# This used to be owned by root (https://phabricator.wikimedia.org/T237248)
-if [ -d /wdqs/data/ ]; then
-  chown blazegraph:blazegraph -R /wdqs/data/
-fi
-
-# Start as the blazegraph user.
-# --preserve-environment does not preserve PATH, so we manually
-# set the PATH again. Java cannot be found otherwise.
-su --preserve-environment --command "export PATH=$PATH; $*" blazegraph
+exec $(echo "$@"| envsubst)

deploy/README.md

@@ -146,12 +146,13 @@ To back up your data, shut down the instance and dump the contents of all Docker
 docker compose down
 
 for v in \
-    wikibase-suite-wikibase_image-data \
-    wikibase-suite_mysql-data \
-    wikibase-suite_wdqs-data \
-    wikibase-suite_elasticsearch-data \
-    wikibase-suite_quickstatements-data \
-    wikibase-suite_traefik-letsencrypt-data; do
+    wbs-deploy_elasticsearch-data \
+    wbs-deploy_mysql-data \
+    wbs-deploy_quickstatements-data \
+    wbs-deploy_traefik-letsencrypt-data \
+    wbs-deploy_wdqs-data \
+    wbs-deploy_wikibase-image-data \
+    ; do
   docker run --rm --volume $v:/backup debian:12-slim tar cz backup > $v.tar.gz
 done
 ```
@@ -164,12 +165,13 @@ To restore the volume backups, ensure your instance has been shut down by runnin
 docker compose down
 
 for v in \
-    wikibase-suite-wikibase_image-data \
-    wikibase-suite_mysql-data \
-    wikibase-suite_wdqs-data \
-    wikibase-suite_elasticsearch-data \
-    wikibase-suite_quickstatements-data \
-    wikibase-suite_traefik-letsencrypt-data; do
+    wbs-deploy_elasticsearch-data \
+    wbs-deploy_mysql-data \
+    wbs-deploy_quickstatements-data \
+    wbs-deploy_traefik-letsencrypt-data \
+    wbs-deploy_wdqs-data \
+    wbs-deploy_wikibase-image-data \
+    ; do
   docker volume rm $v 2> /dev/null
   docker volume create $v
   docker run -i --rm --volume $v:/backup debian:12-slim tar xz < $v.tar.gz
@@ -303,6 +305,15 @@ Removing the `traefik-letsencrypt-data` volume will request a new certificate fr
 ```yml
       --certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
 ```
+
+## WDQS Frontend
+
+To interact with the WDQS frontend, navigate to the URL defined as `WDQS_FRONTEND_PUBLIC_HOST` in the `.env` file. By default, this is set to `wdqs-frontend.example`.
+
+Alternatively, send `GET` requests with your SPARQL query to the WDQS frontend endpoint:
+`https://wdqs-frontend.example.com/proxy/wdqs/bigdata/namespace/wdq/sparql?query={SPARQL}`
+
+
 ## FAQ
 
 ### Can I host WBS Deploy locally?

docs/conventional-commits.md

@@ -0,0 +1,87 @@
+# Conventional commits
+
+[Conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) are human and machine readable commit messages. We use them to automatically generate changelog items and [semantic version (SemVer)](https://semver.org/) bumps on release using [NX](https://nx.dev). Here is an example of a commit message describing a bug fix in a backup script. Releasing this change would bump the project's patch version number:
+
+```
+fix(backup-script): report error if no space left on device
+```
+
+On release of a project, NX collects all commit messages from git for the project's directory (e.g. `build/wdqs`) since last release (e.g. `[email protected]`). For this to work well we need to follow a certain procedure in our Pull requests:
+
+- Use conventional commit syntax for the PR title, then squash merge the PR, so that the squashed commit will contain the conventional commit message from the PR title on the target branch. The PR body will become the commit message body.
+- Or: Merge with merge commit so that all commits from the PR are retained on the target branch. All commits in the PR should follow conventional commit syntax then.
+
+## Supported types
+
+NX supports [a number of](https://github.com/nrwl/nx/blob/db10812da789cd48d3a722628a00feda9d0e3810/packages/nx/src/command-line/release/config/conventional-commits.ts) conventional commit types. We use the following for changelog generation as configured in our [nx.json](https://github.com/wmde/wikibase-release-pipeline/blob/main/nx.json):
+
+| Type     | SemVer Bump | Changelog Title  |
+| -------- | ----------- | ---------------- |
+| build    | patch       | 📦 Build         |
+| chore    | patch       | 🏡 Chore         |
+| ci       | patch       | 🤖 CI            |
+| docs     | patch       | 📖 Documentation |
+| feat     | minor       | 🚀 Features      |
+| fix      | patch       | 🩹 Fixes         |
+| perf     | patch       | 🔥 Performance   |
+| refactor | patch       | 💅 Refactors     |
+| style    | patch       | 🎨 Styles        |
+| test     | patch       | ✅ Tests         |
+
+## Examples
+
+### Feature
+
+A new feature implementation bumping the minor version:
+
+```
+feat: added support for cats
+```
+
+Will generate the following changelog entry:
+
+#### 🚀 Features
+
+- added support for cats
+
+### Documentation update
+
+Some update to the docs bumping a minor version:
+
+```
+docs: describe how to work around sleeping cats
+```
+
+Will generate the following changelog entry:
+
+#### 📖 Documentation
+
+- describe how to work around sleeping cats
+
+### Breaking change
+
+All types can be breaking changes. Here is an example for a breaking refactor, bumping a major version.
+
+```
+refactor!: remove support for sabre-toothed tiger
+```
+
+Will generate the following changelog entry:
+
+#### 💅 Refactors
+
+- remove support for sabre-toothed tiger
+
+### Performance change for a certain component
+
+Noting that a change is specific to a component or subsystem in a project.
+
+```
+pref(food-dispenser): improved speed
+```
+
+Will generate the following changelog entry:
+
+#### 🔥 Performance
+
+- food-dispenser: improved speed