[Demo] Topic Modeling using HDP for Augur Message Table (GSoC 2025)

.github/workflows/build_docker.yml

@@ -11,45 +11,68 @@ on:
   release:
     types:
       - published
+  workflow_dispatch:
 
 permissions: {}
 
 jobs:
   build:
+    name: Build image
     permissions:
       contents: read #  to fetch code (actions/checkout)
       packages: write #  to push docker image
-
-    name: Build image
+    strategy:
+      matrix:
+        image:
+          - backend
+          - database
+          - rabbitmq
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout main
-        uses: actions/checkout@v2
-      - name: Run the build
-        run: |
-          set -ex
-          # use that here since the variable are not present before start, so can't be in env
-          export LOGIN=$GITHUB_REPOSITORY_OWNER
+      - name: Checkout repository
+        uses: actions/checkout@v4
 
-          echo $PASSWORD | docker login $REGISTRY -u $LOGIN --password-stdin
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+        id: setup-buildx
 
-          for i in docker/* ; do
-              CONTAINER=$(basename $i)
-              echo "Building $CONTAINER"
-              export IMAGE=$LOGIN/augur_$CONTAINER
-              DOCKERFILE=${i}/Dockerfile
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        if: github.event_name != 'pull_request'
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
 
-              docker build . -f $DOCKERFILE --tag $REGISTRY/$IMAGE:latest
-              if [[ $GITHUB_EVENT_NAME == 'release' ]]; then
-                TAG=$(basename $GITHUB_REF)
-                docker tag $REGISTRY/$IMAGE:latest $REGISTRY/$IMAGE:$TAG
-                docker push $REGISTRY/$IMAGE:latest
-                docker push $REGISTRY/$IMAGE:$TAG
-              elif [[ $GITHUB_EVENT_NAME == 'push' ]]; then
-                docker tag $REGISTRY/$IMAGE:latest $REGISTRY/$IMAGE:devel-latest
-                docker push $REGISTRY/$IMAGE:devel-latest
-              fi
-          done
+      - name: Set container metadata
+        uses: docker/metadata-action@v5
+        id: meta
         env:
-          REGISTRY: ghcr.io
-          PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+          DOCKER_METADATA_ANNOTATIONS_LEVELS: index,manifest
+        with:
+          annotations: |
+            org.opencontainers.image.title=augur_${{ matrix.image}}
+          labels: |
+            org.opencontainers.image.title=augur_${{ matrix.image}}
+          images: ghcr.io/${{ github.repository_owner }}/augur_${{ matrix.image }}
+          # Pushes to the dev branch update the *:devel-latest tag
+          # Releases update the *:latest tag and the *:<version> tag
+          tags: |
+            type=raw,value=devel-latest,enable=${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.ref == 'refs/heads/dev' }}
+            type=raw,value=latest,enable=${{ github.event_name == 'release' }}
+            type=raw,value=${{ github.event.release.tag_name }},enable=${{ github.event_name == 'release' }}
+
+      - name: Build and push
+        id: push
+        uses: docker/build-push-action@v6
+        with:
+          annotations: ${{ steps.meta.outputs.annotations }}
+          context: .
+          file: ./docker/${{ matrix.image }}/Dockerfile
+          labels: ${{ steps.meta.outputs.labels }}
+          platforms: linux/amd64
+          # Only push if we've tagged the image in the metadata step
+          push: ${{ github.event_name != 'pull_request' && steps.meta.outputs.tags != '' }}
+          tags: ${{ steps.meta.outputs.tags }}
+          cache-from: type=gha,scope=container-${{ matrix.image }}
+          cache-to: type=gha,scope=container-${{ matrix.image }},mode=max

.gitignore

@@ -198,3 +198,4 @@ sendgrid.env
 *sendgrid*.env
 ./sendgrid.env
 sendgrid.env
+notebooks/topic_modeling/output/

CONTRIBUTING.md

@@ -92,7 +92,7 @@ git push origin master
 ### Augur
 - [Stable documentation (`main` branch)](https://oss-augur.readthedocs.io/en/main/)
 - [Nightly/developer build documentation (`dev` branch)](https://oss-augur.readthedocs.io/en/dev/) (warning: this is should be considered an unstable branch and should not be used for production)
-- [Live Augur demo](http://zephyr.osshealth.io/)
+- [Live Augur demo](https://ai.chaoss.io)
 
 ### CHAOSS
 - [Website](https://chaoss.community/)

README.md

@@ -1,4 +1,4 @@
-# Augur NEW Release v0.85.1
+# Augur NEW Release v0.86.0
 
 Augur is primarily a data engineering tool that makes it possible for data scientists to gather open source software community data - less data carpentry for everyone else! 
 The primary way of looking at Augur data is through [8Knot](https://github.com/oss-aspen/8knot), a public instance of 8Knot is available [here](https://metrix.chaoss.io) - this is tied to a public instance of [Augur](https://ai.chaoss.io). 
@@ -11,7 +11,7 @@ We follow the [First Timers Only](https://www.firsttimersonly.com/) philosophy o
 ## NEW RELEASE ALERT!
 **If you want to jump right in, the updated docker, docker-compose and bare metal installation instructions are available [here](docs/new-install.md)**.
 
-Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.85.1).
+Augur is now releasing a dramatically improved new version to the ```main``` branch. It is also available [here](https://github.com/chaoss/augur/releases/tag/v0.86.0).
 
 
 - The `main` branch is a stable version of our new architecture, which features:

augur/application/db/data_parse.py

@@ -291,7 +291,7 @@ def extract_pr_event_data(event: dict, pr_id: int, gh_src_id: int, platform_id:
         'action': event['event'],
         'action_commit_hash': None,
         'created_at': event['created_at'],
-        'issue_event_src_id': gh_src_id,
+        'issue_event_src_id': event["id"],
         'node_id': event['node_id'],
         'node_url': event['url'],
         'tool_source': tool_source,

augur/application/schema/alembic/versions/31_update_pr_events_unique.py

@@ -0,0 +1,83 @@
+"""Update pr events unique
+
+Revision ID: 31
+Revises: 30
+Create Date: 2025-03-08
+
+"""
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy import text
+from augur.application.db import create_database_engine, get_database_string
+
+
+# revision identifiers, used by Alembic.
+revision = '31'
+down_revision = '30'
+branch_labels = None
+depends_on = None
+
+
+    # conn = op.get_bind() 
+    # conn.execute(text("""
+    #     UPDATE pull_request_events
+    #     SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT;
+    # """))
+
+
+def upgrade():
+
+    connection_string = get_database_string()
+    engine = create_database_engine(connection_string)
+
+    with engine.connect() as conn:
+
+        result = conn.execute(text("SELECT COUNT(*) FROM pull_request_events WHERE issue_event_src_id=pr_platform_event_id"))
+        total_rows = result.scalar()
+        if total_rows != 0:
+            print(f"Rows needing updated: {total_rows}")
+            print(f"0.0% complete")
+            total_updated = 0
+
+            while True:
+                result = conn.execute(text("""
+                    WITH cte AS (
+                        SELECT pr_event_id 
+                        FROM pull_request_events 
+                        WHERE issue_event_src_id=pr_platform_event_id 
+                        LIMIT 250000
+                    )
+                    UPDATE pull_request_events
+                    SET issue_event_src_id = substring(node_url FROM '.*/([0-9]+)$')::BIGINT
+                    FROM cte
+                    WHERE pull_request_events.pr_event_id = cte.pr_event_id
+                    RETURNING 1;
+                """))
+
+                conn.commit()
+
+                rows_updated = result.rowcount
+                total_updated += rows_updated
+
+                if rows_updated == 0:
+                    print(f"Update complete")
+                    break
+
+                percentage_updated = (total_updated / total_rows) * 100
+
+                print(f"{percentage_updated:.1f}% complete ({total_rows-total_updated} rows left)")
+
+            print("Creating (repo_id, issue_event_src_id) index")
+    op.create_unique_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', ['repo_id', 'issue_event_src_id'], schema='augur_data')
+
+
+def downgrade():
+    op.drop_constraint('pr_events_repo_id_event_src_id_unique', 'pull_request_events', schema='augur_data', type_='unique')
+
+    print("Please run in background. This downgrade will take a very *very* long time")
+    conn = op.get_bind() 
+    conn.execute(text("""
+        UPDATE pull_request_events
+        SET issue_event_src_id = pr_platform_event_id
+        WHERE issue_event_src_id <> pr_platform_event_id;
+    """))

augur/tasks/data_analysis/clustering_worker/setup.py

@@ -26,7 +26,7 @@ def read(filename):
         'Flask-Login==0.5.0',
         'Flask-WTF==1.0.0',
         'requests==2.32.0',
-        'psycopg2-binary==2.9.3',
+        'psycopg2-binary==2.9.9',
         #'sklearn==0.0.0',
         'scikit-learn==1.5.0',
         'numpy==1.26.0',

augur/tasks/data_analysis/contributor_breadth_worker/setup.py

@@ -27,7 +27,7 @@ def read(filename):
         'Flask-Login==0.5.0',
         'Flask-WTF==1.0.0',
         'requests==2.32.0',
-        'psycopg2-binary==2.9.3'
+        'psycopg2-binary==2.9.9'
     ],
     entry_points={
         'console_scripts': [

augur/tasks/data_analysis/discourse_analysis/setup.py

@@ -26,7 +26,7 @@ def read(filename):
         'Flask-Login==0.5.0',
         'Flask-WTF==1.0.0',
         'requests==2.32.0',
-        'psycopg2-binary==2.9.3',
+        'psycopg2-binary==2.9.9',
         'click==8.0.3', 
         'scipy>=1.10.0',
         'nltk==3.6.6',

augur/tasks/data_analysis/insight_worker/setup.py

@@ -27,7 +27,7 @@ def read(filename):
         'Flask-Login==0.5.0',
         'Flask-WTF==1.0.0',
         'requests==2.32.0',
-        'psycopg2-binary==2.9.3',
+        'psycopg2-binary==2.9.9',
         'click==8.0.3',
         'scipy>=1.10.0',
         'sklearn==0.0',

augur/tasks/data_analysis/message_insights/setup.py

@@ -28,7 +28,7 @@ def read(filename):
         'Flask-Login==0.5.0',
         'Flask-WTF==1.0.0',
         'requests==2.32.0',
-        'psycopg2-binary==2.9.3',
+        'psycopg2-binary==2.9.9',
         'click==8.0.3',
         'scipy>=1.10.0',
         'scikit-learn==1.5.0', #0.24.2',