Sync dev staging may 12 (#685)

* Harvest dynamic configs pt. 2 (#635)

* Update ckanext-s3filestore branch

* Fix typo

* Update layer manager to use wri release

* Fix Dockerfile

* [ODP-235] Allow private and hidden teams (#537)

* update teams create and edit form

* update organization list api

* update schema

* update api

* remove extras

* fix type errors

* update user_list_wri to support private teams

* fix api call

* remove chained action

* add visibility

* fix breaking changes

* update env

* update env plugin

* fix test

* update test

* add visibility to org during dataset edit

* add test for private teams

* remove logs

* update private team functionality

* Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams

* Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams

* resolve conflict

* undo changes

* update organization patch and create

* add private attribute to fetched orgs

* update organization_list sql logic

* improve team visbility

* improve error message

* update private team test

* update failing test

* update test

* update settings

* update test

* update test

* update test

* update test

* update test

* update test

* update test

* update docker compose

* add tooltip

* update test

* add padlock to private teams

* add back auth plugin

* Harvest config feb 03 2025 (#637)

* Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app

* Add whoami output to start_ckan.sh

* [download_event.py] Add missing closing parenthesis

* Set supervisor home by variable

* Fix harvest config permissions

* Use CKAN_IMAGE variable instead of local image name

* Handle sed supervisor changes as root; Add start_ckan sub-script

* Revert sub-script; Always use user ckan for supervisor

* Fix sed commands

* Use root user with ckan home for supervisord

* Revert "Harvest config feb 03 2025 (#637)"

This reverts commit e68155a.

* Harvest dynamic configs pt. 3 (#638)

* Add supercronic for harvest run command (#640)

* Fix alignment

* Fix build

* Fix build

* [ODP-414] add email notification when download-flow fails (#641)

* add email notification when flow fails

* update error notification

* empty

* remove pgclient version

* update libpq

---------

Co-authored-by: Michael Polidori <[email protected]>

* Upgrade to CKAN 2.11 (#632)

* Update setup.py

* Update Members permission to team form + ODP-370 (#642)

* members can not

* update role: Editor can create sub-team and admin moving subteam from public to private should not throw error

* disabled Public option if parent is private

* Trigger CI

---------

Co-authored-by: Luccas Mateus <[email protected]>

* Change setup.py and setup.cfg

* Fix ckanext-wri install (#643)

* Improve resource location design (#646)

* Trigger CI

* [odp-426]: Add edit icon to dashboard entities (#648)

* [odp-426]: Add edit icon to dashboard entities

* update schema

* Add bulk purge collaborator issue patch (#645)

* Increase UWSGI timeouts; Extract URLs from markdown links in learn_more (#649)

* Resource location search improvements (#650)

* add authorized check to group/teams last node (#651)

* Trigger CI

* Rm global from DatafileLocation

* Trigger CI

* enable edit icon for collaborators (#652)

* Fix typo

* intgration test for collaborator permission (#654)

* enable edit icon for collaborators

* add intergration test  for dashboard collaborator permission

* Improve speed

* Improve js

* Fix prefetching

* Change color of area select tool

* Improvements map

* Remove edit icon on teams/topic page for editor (#656)

* enable edit icon for collaborators

* remove editor permision

* empty

* Fix tests

* Fix tests

* Fix tests

* Fix test

* Clear all session storage

* Fix tests

* Fix tests

* Fix private teams tests

* Fix cypress.config.js

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix whitespace wrap on text

* Trigger CI

* Add group admin email to workflow error mail (#663)

* enable edit icon for collaborators

* send workflow error to team admin

* removed email check

* E2etest (#657)

* enable edit icon for collaborators

* empty

* add check to ensure org is actually private

* update

* update

* fix test

* update test

* empty

* update

* update test

* update test

* update test

* update test

* update test

* update test to see if organization create api is truly working

* update test

* remove other test for now

* push update

* update env.example

* fix test

* empty

* Fix git vulnerability

* Fix git

* Add if condition (#664)

* [odp-449] Teams Implementation - Root parent Team Edit (#669)

* enable edit icon for collaborators

* admin should be able to edit root team

* ODP-447 (#668)

* ODP-447

* Fix build

* Fix tests

* Fix t ests

* Trigger CI

* [Dev] Increase proxy-read-timeout to avoid early external request disconnects (#665)

* Odp 448 (#670)

* ODP-448

* Add extra tests

* Fix session leakage

* Fix tests

* Forgot to commit

* Fix tests

* Add required indicator for non-sysadmins (#672)

* Change data stored in orttho

* Fix odp430 (#674)

* enable edit icon for collaborators

* fix odp-430

* Trigger CI

* Add better warning

* Add extras to list of defaultvalues (#676)

* enable edit icon for collaborators

* fix odp-430

* add extras to default value

* Form-id

* Trigger CI

* [Odp 458] create/edit dataset in sub-team public (#678)

* enable edit icon for collaborators

* extend capacity so editor can create or edit dataset in subteam

* editor should be able to see public subteam in organization_list_for_user api

* parent team admin should be able to edit subteam public dataset

* add missing visbility check: public dataset can not be assigned to private team

* reshape logic

* update patch

* remove log

* update test

* empty

* [ODP-453] Teams Implementation - Edit sub team (private or public) (#675)

* Add integration tests

* Bump ckanext-hierarchy commit: fix cascading roles

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Limit setting private Teams to public only to sysadmins (#679)

* Add docs for subpaths (#568)

* Add docs for subpaths

* Force rerun pipeline

* Allow Team Admins public to private permissions (#680)

* use sysadmin key to fetch dataset collaborators (#682)

* enable edit icon for collaborators

* use sysadmin key to fetch dataset collaborators

* Hide by default when no viz available

* fix: first publish

* Fix test

* Fix caution max-w

* Set create_unowned_dataset to False; Remove dev testing GitHub Actions file

* Remove duplicate form-id

---------

Co-authored-by: Michael Polidori <[email protected]>
Co-authored-by: Stephen Oni <[email protected]>

Author: 3 people

ckan-backend-dev/.env.example

@@ -59,7 +59,7 @@ CKAN_SMTP_PASSWORD=testpassword
 CKAN_SMTP_MAIL_FROM=ckan@localhost
 TZ=UTC
 CKAN_INI=/srv/app/production.ini
-CKAN__AUTH__CREATE_UNOWNED_DATASET=True
+CKAN__AUTH__CREATE_UNOWNED_DATASET=False
 CKAN__AUTH__ALLOW_DATASET_COLLABORATORS=True
 CKAN__AUTH__ALLOW_ADMIN_COLLABORATORS=True
 CKAN__AUTH__ALLOW_COLLABORATORS_TO_CHANGE_OWNER_ORG=True

ckan-backend-dev/ckan/Dockerfile.dev

@@ -34,7 +34,7 @@ RUN pip3 install -e 'git+https://github.com/ckan/ckanext-scheming.git@5ce30cf285
     pip3 install httpretty && \
     pip3 install -e 'git+https://github.com/datopian/ckanext-auth.git@okta#egg=ckanext-auth' && \
     pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-auth/okta/requirements.txt' && \
-    pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@wri#egg=ckanext-hierarchy' && \
+    pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@5d069a5d032ba384f43d266fc397a0aa0a4b444c#egg=ckanext-hierarchy' && \
     pip3 install -e 'git+https://github.com/datopian/[email protected]#egg=ckanext-issues' && \
     # Required by ckanext-issues, but it's no longer included in any of the requirements files or the subdependencies
     pip3 install mock && \

ckan-backend-dev/ckan/patches/ckan/01_roles.patch

@@ -7,7 +7,7 @@ diff --git a/ckan/ckan/authz.py b/ckan/ckan/authz.py
      if _has_user_permission_for_groups(user_id, permission, [group_id]):
          return True
 -    capacities = check_config_permission('roles_that_cascade_to_sub_groups')
-+    capacities = ['admin']
++    capacities = ['admin', 'editor']
      assert isinstance(capacities, list)
      # Handle when permissions cascade. Check the user's roles on groups higher
      # in the group hierarchy for permission.

ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py

@@ -1007,11 +1007,9 @@ def organization_list_wri(context: Context, data_dict: DataDict):
     user_orgs = None
     if not user:
        private_orgs = get_private_organizations(context)
-       log.warning(f"Private orgs: {private_orgs} \n\n {orgs}")
 
     if user:
         user_orgs = orgs
-        log.warning(f"User orgs: {user_orgs} \n\n {orgs}")
     results = get_hierarchy_group(context, orgs, "organization", q, private_orgs, user_orgs)
     return results
 
@@ -1409,24 +1407,49 @@ def organization_list_for_user(context: Context,
             .filter(model.Member.state == 'active') \
             .join(model.Group)
 
-        group_ids: set[str] = set()
+        
         roles_that_cascade = cast(
             "list[str]",
             authz.check_config_permission('roles_that_cascade_to_sub_groups')
         )
+        group_extra_alias = aliased(model.GroupExtra)
+        public_groups_query = (
+            model.Session.query(model.Group.id)
+            .outerjoin(group_extra_alias, model.Group.id == group_extra_alias.group_id)
+            .filter(
+                _or_(
+                    _and_(
+                        group_extra_alias.key == 'visibility',
+                        group_extra_alias.value == 'public'
+                    ),
+                    group_extra_alias.key == None
+                )
+            )
+        )
+        public_group_ids = {gid for gid, in public_groups_query.all()}
+        group_ids: set[str] = set()
         group_ids_to_capacities: dict[str, str] = {}
         for member, group in q.all():
+            group_ids.add(group.id)
+            group_ids_to_capacities[group.id] = member.capacity
+            children_group_ids = [
+                grp_tuple[0] for grp_tuple
+                in group.get_children_group_hierarchy(type='organization')
+            ]
             if member.capacity in roles_that_cascade:
-                children_group_ids = [
-                    grp_tuple[0] for grp_tuple
-                    in group.get_children_group_hierarchy(type='organization')
-                ]
+                
                 for group_id in children_group_ids:
+                    group_ids.add(group_id)
                     group_ids_to_capacities[group_id] = member.capacity
-                group_ids |= set(children_group_ids)
+            else:
+                for group_id in children_group_ids:
+                    if group_id in public_group_ids:
+                        group_ids.add(group_id)
+                        group_ids_to_capacities[group_id] = member.capacity
 
-            group_ids_to_capacities[group.id] = member.capacity
-            group_ids.add(group.id)
+
+            # group_ids_to_capacities[group.id] = member.capacity
+            # group_ids.add(group.id)
 
         if not group_ids:
             return []
@@ -1449,6 +1472,8 @@ def organization_list_for_user(context: Context,
 
 
 
+
+
 @logic.side_effect_free
 def organization_list(context: Context,
                       data_dict: DataDict) -> ActionResult.OrganizationList:
@@ -1782,21 +1807,10 @@ def organization_patch(context, data_dict):
     if not isSysadmin:
         temp_context = {"model": context["model"], "session": context["session"], "user": context["user"]}
         old_org = get_action("organization_show")(temp_context, data_dict)
-        old_parent = old_org.get("groups", [])
-        if len(old_parent) == 0 or data_dict.get("parent") is None:
-            raise ValidationError({"message": _("You can't edit or create a Parent Team")})
         if old_org.get("visibility", "public") == "private" and visibility == "public":
             raise ValidationError({"message": 
-                                   _("Team has private visibility and cannot be updated; Only sysadmin can update private team")
+                                   _("Team has private visibility and cannot be updated; Only sysadmin can switch private to public")
                                    })
-
-        if visibility == "private":
-            users = old_org.get("users", [])
-            if users:
-                c_user = str(user)
-                user_capacity = [user.get("capacity") for user in users if user.get("name") == c_user]
-                if "admin" not in user_capacity:
-                    raise ValidationError({"message": _("You can't update visibility of a team")})
 
 
     if visibility == "public":
@@ -1824,22 +1838,12 @@ def validate_visibility(context, data_dict):
 
     visibility = data_dict.get('visibility_type', "public")
     owner_org = data_dict.get('owner_org', None)
-    id = data_dict.get('id', None)
-    if visibility in ["public", "internal"] and id:
-        package_show = get_action("package_show")(context, {"id": id})
-        id_org = package_show.get("owner_org", None)
-        if id_org:
-            org = get_action("organization_show")(context, {"id": id_org})
-            org_visibility = org.get("visibility", "public")
-            if org_visibility == "public":
-                raise ValidationError({"message": _("Organization has private visibility and cannot be made public")})
-        
-    elif visibility in ["public", "internal"] and owner_org:
+    if visibility in ["public", "internal"] and  owner_org:
         org = get_action("organization_show")(context, {"id": owner_org})
         org_visibility = org.get("visibility", "public")
         if org_visibility == "private":
             raise ValidationError({"message": _("Organization has private visibility and cannot create public datasets")})
-
+        
 
 
 

ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/update.py

@@ -468,6 +468,7 @@ def old_package_patch(context: Context, data_dict: DataDict) -> ActionResult.Pac
     You must be authorized to edit the dataset and the groups that it belongs
     to.
     """
+    validate_visibility(context, data_dict)
     _before_dataset_create_or_update(context, data_dict)
     _check_access("package_patch", context, data_dict)
 

ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/auth/auth.py

@@ -106,11 +106,15 @@ def package_update(up_func, context, data_dict):
             # if there is an owner org then we must have update_dataset
             # permission for that organization
             if authz.users_role_for_group_or_org(package.owner_org, user) != "admin":
-                return {
-                    "success": False,
-                    "msg": _("User %s not authorized to edit package %s")
-                    % (str(user), package.id),
-                }
+                 if not  authz.has_user_permission_for_group_or_org(
+                    package.owner_org, user, "admin"
+                ):
+                    return {
+                        "success": False,
+                        "msg": _("User %s not authorized to edit package %s")
+                        % (str(user), package.id),
+                    }
+
         else:
             if authz.check_config_permission("allow_dataset_collaborators"):
                 # if org-level auth failed, check dataset-level auth

deployment/ckan/Dockerfile

@@ -14,7 +14,7 @@ RUN pip3 install -e 'git+https://github.com/ckan/ckanext-scheming.git@5ce30cf285
     pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/signed-url/dev-requirements.txt' && \
     pip3 install -e 'git+https://github.com/datopian/ckanext-auth.git@a9e42702b797eaf3ffe967040867519219d32224#egg=ckanext-auth' && \
     pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-auth/okta/requirements.txt' && \
-    pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@wri#egg=ckanext-hierarchy' && \
+    pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@5d069a5d032ba384f43d266fc397a0aa0a4b444c#egg=ckanext-hierarchy' && \
     pip3 install -e 'git+https://github.com/datopian/ckanext-issues.git@ea2b3cbace924a162de009fd8e5626052ebc99e1#egg=ckanext-issues' && \
     # Required by ckanext-issues, but it's no longer included in any of the requirements files or the subdependencies
     pip3 install mock && \
@@ -93,6 +93,7 @@ RUN ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
 RUN ckan config-tool ${CKAN_INI} "ckan.auth.create_user_via_web = false"
 RUN ckan config-tool ${CKAN_INI} "ckan.root_path = /private-admin/{{LANG}}"
 RUN ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}"
+RUN ckan config-tool ${CKAN_INI} "ckan.auth.create_unowned_dataset = false"
 
 RUN if [ "$GITHUB_ACTIONS" = "true" ]; then ckan config-tool ${CKAN_INI} "ckan.cors.origin_allow_all = True"; fi
 

deployment/ckan/patches/ckan/01_roles.patch

@@ -7,7 +7,7 @@ diff --git a/ckan/ckan/authz.py b/ckan/ckan/authz.py
      if _has_user_permission_for_groups(user_id, permission, [group_id]):
          return True
 -    capacities = check_config_permission('roles_that_cascade_to_sub_groups')
-+    capacities = ['admin']
++    capacities = ['admin', 'editor']
      assert isinstance(capacities, list)
      # Handle when permissions cascade. Check the user's roles on groups higher
      # in the group hierarchy for permission.

deployment/frontend/public/sitemap-0.xml

@@ -1,18 +1,18 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
-<url><loc>http://localhost:3000</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/applications</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/applications/404</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/auth/password-reset</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/datasets/404</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/search</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/teams</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/teams/404</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/topics</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/topics/404</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/user-guide</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/user-guide/basic-definition</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/user-guide/dataset-view-pages</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/user-guide/search-page</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
-<url><loc>http://localhost:3000/user-guide/teams-topics</loc><lastmod>2025-03-23T20:54:07.812Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/applications</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/applications/404</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/auth/password-reset</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/datasets/404</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/search</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/teams</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/teams/404</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/topics</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/topics/404</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/user-guide</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/user-guide/basic-definition</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/user-guide/dataset-view-pages</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/user-guide/search-page</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
+<url><loc>http://localhost:3000/user-guide/teams-topics</loc><lastmod>2025-04-10T14:12:02.675Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
 </urlset>

deployment/frontend/src/components/_shared/SimpleSelect.tsx

@@ -37,7 +37,8 @@ export default function SimpleSelect<T extends FieldValues, V extends Object>({
     name,
     id,
     onChange: _onChange = (val) => {},
-}: SimpleSelectProps<T, V> & { onChange?: (val: any) => void }) {
+    disabled,
+}: SimpleSelectProps<T, V> & { onChange?: (val: any) => void, disabled?: boolean }) {
     const { control } = formObj ?? useForm()
     return (
         <Controller
@@ -59,6 +60,7 @@ export default function SimpleSelect<T extends FieldValues, V extends Object>({
                         }
                         setSelected(e)
                     }}
+                    disabled={disabled}
                 >
                     {({ open }) => (
                         <>