Skip to content

Commit dd973fc

Browse files
mpolidoristeveoniluccasmmgMuhammadIsmailShahzad
authored
Sync staging prod 2025 03 27 (#660)
* [sitemap]: remove diallow all from sitemap * Add harvest configs and supervisorctl * Force rerun deployment * Force rerun deployment * empty * Fix supervisord file permissions * Add harvester to initial plugins * Trigger CI * Searchcard (#605) * [sitemap]: remove diallow all from sitemap * update searchcard style * add aria-label to the footer * update test * update * update dataset card * Fix supervisord.conf path * Fix supervisord file path in Dockerfile.dev * Revert "Merge branch 'dev' of https://github.com/wri/wri-odp into dev" (#608) This reverts commit 09e0761, reversing changes made to c54ef01. Co-authored-by: Luccas Mateus <[email protected]> * Re apply multiple applications (#609) * Revert "Merge branch 'dev' of https://github.com/wri/wri-odp into dev" This reverts commit 09e0761, reversing changes made to c54ef01. * Reapply "Merge branch 'dev' of https://github.com/wri/wri-odp into dev" This reverts commit 9399810. --------- Co-authored-by: Luccas Mateus <[email protected]> * Force rerun deployment * Add log output after unit tests * Move error logs in GitHub Actions * Improve group package count api call (#611) * [sitemap]: remove diallow all from sitemap * empty * improve groups package count fetch * Move log print; Fix timeout * update auto-url field in dataset form * ODP-399 (#614) * ODP-399 * Fix build * scroll to the top on state change (#615) * scroll to the top on stte change * update pagination * Swap back to Azure SSO from Okta SSO * ODP-397 (#617) * Fix requesteds by Nesma * ODP-380 (#620) * Allow CORS in tests * Remove unnecessary build section from docker-compose.test * Update docs * update application and others (#624) * update images and doc (#623) * update userdoc search page * Applications docs * [ODP-400] Homepage search bar improvements (#612) * update search field style * update search field * update color contrast * update cta button (#627) * Fix build * User info (#628) * Progress * Only misiing csv export * Almost finished * Add docs * Tests * Fix test * Orttho * Rm poetry * QA Fixes * Fix ortto signup (Only when user accepts) * Fixes onSuccess mutation download batch * Downgrade git * QA Fixes * Delete test thats no longer relevant * Fix test cost splitting * Fix test cost splitting * Rm console.log * Change wording for button * Marketing + Vulnerability * fix issue with weird scrolling behaviour (#629) * fix issue with weird scrolling bahaviour * add height for mobile scroll * Update to have timestamp * Fix vulnerability * [run_unit_tests.sh] Give verbose output * fix search input alignment * update search input fields * Kim comments * [main.yml] Temp. stop build logs * Fix vulnerability * [main.yml] Fix broken build * [main.yml] Fix broken build * [main.yml] Typo fix * [main.yml] Fix image name * Fix alignment recent added/updated * Fix broken link * ODP-397 Kim request * update dataset edit url placeholder (#631) * [docker-compose.test.yml] Temp. use redis 3 * [main.yml] Run build * [main.yml] Disable build * [docker-compose.test.yml] Add redis retry * [docker-compose.test.yml] Dont run frontend * [main.yml] Don't run frontend docker * [docker-compose.test.yml] Remove redis health check * [docker-compose.test.yml] Remove datapusher temporarily * [docker-compose.test.yml] Fix redis container name * [run_unit_tests.sh] Run traceback as long * [run_unit_tests.sh] Update test command * Small UI Fixes * Fix font size * [supervisor.harvest.conf] Temp. run harvester as root * [main.yml] Run the build and push the images * [docker-compose.test.yml] Revert changes * [main.yml] Run tests * [supervisor.*.conf] Use env variable to dynamically set user based on user running CKAN * update filter scrolltop (#633) * Kim requests(Download id, new fomratting date) * Fix fonts * Fix timezone * Ortto forms * Fix timezone * Fix font size * Fix typo * Trigger CI * Fix build * Sync dev staging 02 17 25 (#639) * Harvest dynamic configs pt. 2 (#635) * Update ckanext-s3filestore branch * Fix typo * Update layer manager to use wri release * Fix Dockerfile * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) --------- Co-authored-by: Luccas Mateus <[email protected]> Co-authored-by: Stephen Oni <[email protected]> * Sync dev staging 03 03 25 (#644) * Sync dev to Staging Mar 3, 2025 * Fix typo * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) * Fix alignment * Fix build * Fix build * [ODP-414] add email notification when download-flow fails (#641) * add email notification when flow fails * update error notification * empty * remove pgclient version * update libpq --------- Co-authored-by: Michael Polidori <[email protected]> * Update Members permission to team form + ODP-370 (#642) * members can not * update role: Editor can create sub-team and admin moving subteam from public to private should not throw error * disabled Public option if parent is private * Trigger CI --------- Co-authored-by: Luccas Mateus <[email protected]> * Fix shapely imports --------- Co-authored-by: Luccas Mateus <[email protected]> Co-authored-by: Stephen Oni <[email protected]> * Sync dev staging mar 19 (#653) * Harvest dynamic configs pt. 2 (#635) * Update ckanext-s3filestore branch * Fix typo * Update layer manager to use wri release * Fix Dockerfile * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) * Fix alignment * Fix build * Fix build * [ODP-414] add email notification when download-flow fails (#641) * add email notification when flow fails * update error notification * empty * remove pgclient version * update libpq --------- Co-authored-by: Michael Polidori <[email protected]> * Upgrade to CKAN 2.11 (#632) * Update setup.py * Update Members permission to team form + ODP-370 (#642) * members can not * update role: Editor can create sub-team and admin moving subteam from public to private should not throw error * disabled Public option if parent is private * Trigger CI --------- Co-authored-by: Luccas Mateus <[email protected]> * Change setup.py and setup.cfg * Fix ckanext-wri install (#643) * Improve resource location design (#646) * Trigger CI * [odp-426]: Add edit icon to dashboard entities (#648) * [odp-426]: Add edit icon to dashboard entities * update schema * Add bulk purge collaborator issue patch (#645) * Increase UWSGI timeouts; Extract URLs from markdown links in learn_more (#649) * Resource location search improvements (#650) * add authorized check to group/teams last node (#651) * Trigger CI * Rm global from DatafileLocation * Trigger CI * enable edit icon for collaborators (#652) * Fix typo * Remove dev helm config values and GH Actions file --------- Co-authored-by: Michael Polidori <[email protected]> Co-authored-by: Stephen Oni <[email protected]> * Sync dev staging mar 26 (#659) * Harvest dynamic configs pt. 2 (#635) * Update ckanext-s3filestore branch * Fix typo * Update layer manager to use wri release * Fix Dockerfile * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) * Fix alignment * Fix build * Fix build * [ODP-414] add email notification when download-flow fails (#641) * add email notification when flow fails * update error notification * empty * remove pgclient version * update libpq --------- Co-authored-by: Michael Polidori <[email protected]> * Upgrade to CKAN 2.11 (#632) * Update setup.py * Update Members permission to team form + ODP-370 (#642) * members can not * update role: Editor can create sub-team and admin moving subteam from public to private should not throw error * disabled Public option if parent is private * Trigger CI --------- Co-authored-by: Luccas Mateus <[email protected]> * Change setup.py and setup.cfg * Fix ckanext-wri install (#643) * Improve resource location design (#646) * Trigger CI * [odp-426]: Add edit icon to dashboard entities (#648) * [odp-426]: Add edit icon to dashboard entities * update schema * Add bulk purge collaborator issue patch (#645) * Increase UWSGI timeouts; Extract URLs from markdown links in learn_more (#649) * Resource location search improvements (#650) * add authorized check to group/teams last node (#651) * Trigger CI * Rm global from DatafileLocation * Trigger CI * enable edit icon for collaborators (#652) * Fix typo * intgration test for collaborator permission (#654) * enable edit icon for collaborators * add intergration test for dashboard collaborator permission * Improve speed * Improve js * Fix prefetching * Change color of area select tool * Improvements map * Remove edit icon on teams/topic page for editor (#656) * enable edit icon for collaborators * remove editor permision * empty * Fix tests * Fix tests * Fix tests * Fix test * Clear all session storage * Fix tests * Fix tests * Fix private teams tests * Fix cypress.config.js * Fix tests * Fix tests * Fix tests * Fix tests * Rm test-main --------- Co-authored-by: Michael Polidori <[email protected]> Co-authored-by: Stephen Oni <[email protected]> * Fix merge conflict leftovers --------- Co-authored-by: steveoni <[email protected]> Co-authored-by: Luccas Mateus <[email protected]> Co-authored-by: Muhammad Ismail Shahzad <[email protected]>
1 parent 2cb1de6 commit dd973fc

File tree

99 files changed

+27516
-25705
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

99 files changed

+27516
-25705
lines changed

‎.github/workflows/main.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -114,8 +114,6 @@ jobs:
114114
run: |
115115
docker compose -f docker-compose.test.yml --env-file .env.example up --build -d
116116
working-directory: ./ckan-backend-dev
117-
- name: Initialize the Extensions
118-
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
119117
- name: Cypress Install and CKAN setup
120118
if: ${{ env.ENABLE_INT_TESTS == 'true' }}
121119
uses: cypress-io/github-action@v6
@@ -133,8 +131,6 @@ jobs:
133131
CKAN_IMAGE: '${{ steps.login-ecr.outputs.registry }}/${{ secrets.ECR_CKAN_REPO }}:prod-${{ github.sha }}'
134132
run: docker compose -f docker-compose.test.yml --env-file .env.example up --build -d frontend
135133
working-directory: ./ckan-backend-dev
136-
- name: Initialize the Extensions
137-
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
138134
- name: Print Logs
139135
if: ${{ env.ENABLE_DEBUG_LOGS == 'true' }}
140136
env:
@@ -150,6 +146,8 @@ jobs:
150146
node-version: 18
151147
runTests: false
152148
working-directory: ./e2e-tests
149+
- name: Initialize the Extensions
150+
run: bash ./ckan-backend-dev/ckan/scripts/init-extensions.sh
153151
- name: Run Integration tests 🧪
154152
if: ${{ env.ENABLE_INT_TESTS == 'true' }}
155153
uses: cypress-io/github-action@v6
@@ -197,6 +195,8 @@ jobs:
197195
working-directory: ./ckan-backend-dev
198196
- name: Initialize the pending datasets table
199197
run: docker exec ckan-wri sh -c "ckan -c production.ini pendingdatasetsdb"
198+
- name: Initialize the issues table
199+
run: docker exec ckan-wri sh -c "ckan -c production.ini issuesdb"
200200
- name: Run Unit Tests 🧪
201201
env:
202202
CKAN_IMAGE: '${{ steps.login-ecr.outputs.registry }}/${{ secrets.ECR_CKAN_REPO }}:${{ github.sha }}'

‎ckan-backend-dev/.env.example

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ TEST_CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore_test
4343
TEST_CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore_test
4444

4545
# CKAN core
46-
CKAN_VERSION=2.10.0
46+
CKAN_VERSION=2.11.0
4747
CKAN_SITE_ID=default
4848
CKAN_SITE_URL=http://ckan-dev:5000
4949
CKAN_PORT=5000
@@ -84,8 +84,8 @@ DATAPUSHER_REWRITE_RESOURCES=True
8484
DATAPUSHER_REWRITE_URL=http://ckan-dev:5000
8585

8686
# Extensions
87-
CKAN__PLUGINS=image_view text_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups harvest wri wri_harvester auth hierarchy_display hierarchy_form hierarchy_group_form issues envvars
88-
CKAN__VIEWS__DEFAULT_VIEWS=image_view text_view webpage_view datatables_view
87+
#CKAN__PLUGINS='text_view image_view webpage_view resource_proxy datatables_view datastore datapusher activity s3filestore scheming_datasets scheming_organizations scheming_groups harvest wri wri_harvester auth hierarchy_display hierarchy_form hierarchy_group_form envvars'
88+
#CKAN__VIEWS__DEFAULT_VIEWS="image_view text_view webpage_view datatables_view"
8989
CKAN__HARVEST__MQ__TYPE=redis
9090
CKAN__HARVEST__MQ__HOSTNAME=redis
9191
CKAN__HARVEST__MQ__PORT=6379

‎ckan-backend-dev/.env.frontend

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Minio
2+
MINIO_ROOT_USER=minioadmin
3+
MINIO_ROOT_PASSWORD=minioadmin
4+
NEXTAUTH_SECRET=secret
5+
NEXTAUTH_URL=http://localhost:3000
6+
CKAN_URL=http://ckan-dev:5000/private-admin/en
7+
S3_ACCESS_KEY_ID=${MINIO_ROOT_USER}
8+
S3_SECRET_KEY_ID=${MINIO_ROOT_PASSWORD}
9+
S3_BUCKET_NAME=ckan
10+
S3_BUCKET_REGION=us-east-1
11+
SYS_ADMIN_API_KEY=CKAN_API_TOKEN
12+
SMTP_SERVER="smtp.ssss.com"
13+
SMTP_PORT="587"
14+
SMTP_USER="SMTP_Injection"
15+
SMTP_PASSWORD="b07f"
16+
SMTP_FROM="[email protected]"
17+
OSANO_URL="https://cmp.osano.com/AzZuApUOBG9lEDOds/e8610315-1e7f-4f15-a759-e4124cab69a9/osano.js"
18+
NEXT_PUBLIC_WRI_PRIVACY_POLICY_URL="https://www.wri.org/about/privacy-policy?sitename=WRI%20Data%20Explorer&osanoid=c2a89d08-4931-4ad0-99cb-8d3aa022aaec"
19+
NEXT_PUBLIC_DEPLOYMENT_TYPE="dev"

‎ckan-backend-dev/ckan/Dockerfile.dev

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM viderum/ckan:v2.10.3-alpine-3.18
1+
FROM viderum/ckan:v2.11.0
22

33
# Set up environment variables
44
ENV APP_DIR=/srv/app
@@ -22,16 +22,22 @@ ENV SRC_EXTENSIONS_DIR=${APP_DIR}/src_extensions
2222
# to get them mounted in this image at runtime
2323
USER root
2424
RUN pip3 install --upgrade setuptools==70.0.0
25-
# Echo the version of setuptools to check if it is installed correctly
26-
RUN pip3 install -e 'git+https://github.com/datopian/ckanext-scheming.git@0983735ca163f906693c46f5315ee5abf68d2742#egg=ckanext-scheming' && \
27-
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-scheming/ckan-2.10/test-requirements.txt' && \
25+
26+
RUN pip3 install -e 'git+https://github.com/ckan/ckanext-scheming.git@5ce30cf2856aee97fa7b49de46847f8119dad355#egg=ckanext-scheming' && \
27+
pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-scheming/5ce30cf2856aee97fa7b49de46847f8119dad355/test-requirements.txt' && \
2828
pip3 install -e 'git+https://github.com/datopian/ckanext-s3filestore.git@wri/cost-splitting-orgs#egg=ckanext-s3filestore' && \
2929
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/requirements.txt' && \
30-
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/dev-requirements.txt' && \
30+
#pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-s3filestore/wri/cost-splitting-orgs/dev-requirements.txt' && \
31+
# Use latest versions for now due to build errors. TODO: Update to specific newer versions (hopefully these are compatible)
32+
pip3 install moto && \
33+
pip3 install ckanapi && \
34+
pip3 install httpretty && \
3135
pip3 install -e 'git+https://github.com/datopian/ckanext-auth.git@okta#egg=ckanext-auth' && \
3236
pip3 install -r 'https://raw.githubusercontent.com/datopian/ckanext-auth/okta/requirements.txt' && \
33-
pip3 install -e 'git+https://github.com/ckan/ckanext-hierarchy.git@master#egg=ckanext-hierarchy' && \
37+
pip3 install -e 'git+https://github.com/datopian/ckanext-hierarchy.git@wri#egg=ckanext-hierarchy' && \
3438
pip3 install -e 'git+https://github.com/datopian/[email protected]#egg=ckanext-issues' && \
39+
# Required by ckanext-issues, but it's no longer included in any of the requirements files or the subdependencies
40+
pip3 install mock && \
3541
pip3 install -e 'git+https://github.com/ckan/[email protected]#egg=ckanext-harvest' && \
3642
pip3 install -r 'https://raw.githubusercontent.com/ckan/ckanext-harvest/refs/tags/v1.6.0/requirements.txt'
3743

@@ -40,8 +46,8 @@ RUN apk --update add build-base libxslt-dev python3-dev
4046

4147
# Fixing security vulnerabilities
4248
RUN apk --update add git
43-
RUN apk add postgresql15-client=15.10-r0
44-
RUN apk add libpq=15.10-r0
49+
RUN apk add postgresql15-client
50+
RUN apk add libpq
4551
RUN pip install --force-reinstall -v "Pillow==11.0.0"
4652

4753
RUN apk add --virtual .build-deps \
@@ -106,6 +112,9 @@ RUN chown ckan:ckan ${APP_DIR}/start_ckan_development.sh
106112

107113
RUN apk --no-cache add openssl
108114

115+
RUN cat ${APP_DIR}/src/ckan/requirements.txt
116+
RUN pip3 install -r /srv/app/src/ckan/requirements.txt
117+
109118
RUN openssl genpkey -algorithm RSA -out ${APP_DIR}/jwtRS256.key && \
110119
openssl rsa -in ${APP_DIR}/jwtRS256.key -pubout -outform PEM -out ${APP_DIR}/jwtRS256.key.pub && \
111120
chown ckan:ckan ${APP_DIR}/jwtRS256.key && \
@@ -120,6 +129,7 @@ COPY docker-entrypoint.d/* /docker-entrypoint.d/
120129

121130
RUN ckan config-tool ${CKAN_INI} "ckan.cors.origin_allow_all = True"
122131
RUN ckan config-tool ${CKAN_INI} "ckan.root_path = /private-admin/{{LANG}}"
132+
RUN ckan config-tool ${CKAN_INI} -s app:main "ckan.datapusher.api_token = xxxx-xxxx-xxxx-xxxx"
123133

124134
COPY setup/who.ini ${APP_DIR}/src/ckan/who.ini
125135

‎ckan-backend-dev/ckan/patches/ckan/02_tests_reset_db.patch

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,16 @@
1-
diff --git a/ckan/ckan/model/__init__.py b/ckan/ckan/model/__init__.py
2-
--- ckan/ckan/model/__init__.py
3-
+++ ckan/ckan/model/__init__.py
4-
@@ -219,8 +219,13 @@
5-
with warnings.catch_warnings():
1+
diff --git a/ckan/model/__init__.py b/ckan/model/__init__.py
2+
index 595ae3876..a03e50c25 100644
3+
--- a/ckan/model/__init__.py
4+
+++ b/ckan/model/__init__.py
5+
@@ -237,6 +237,11 @@ class Repository():
66
warnings.filterwarnings('ignore', '.*(reflection|tsvector).*')
7-
meta.metadata.reflect()
7+
meta.metadata.reflect(engine)
88

99
+ tables = meta.metadata.sorted_tables
1010
+ for table in tables:
1111
+ if table.name == "spatial_ref_sys":
1212
+ meta.metadata.remove(table)
1313
+
14-
meta.metadata.drop_all()
15-
self.tables_created_and_initialised = False
16-
log.info('Database tables dropped')
14+
with engine.begin() as conn:
15+
meta.metadata.drop_all(conn)
1716

18-

‎ckan-backend-dev/ckan/patches/ckan/03_fix_dataset_purge_collaborators.patch

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
diff --git a/ckan/logic/action/delete.py b/ckan/logic/action/delete.py
2+
index dc3ce8ec2..1828effca 100644
3+
--- a/ckan/logic/action/delete.py
4+
+++ b/ckan/logic/action/delete.py
5+
@@ -152,6 +152,14 @@ def dataset_purge(context: Context, data_dict: DataDict) -> ActionResult.Dataset
6+
model.PackageRelationship.object_package_id == pkg.id)).all():
7+
r.purge()
8+
9+
+ dataset_collaborators = (
10+
+ model.Session.query(model.PackageMember)
11+
+ .filter(model.PackageMember.package_id == id)
12+
+ .all()
13+
+ )
14+
+ for collaborator in dataset_collaborators:
15+
+ collaborator.delete()
16+
+
17+
pkg = model.Package.get(id)
18+
assert pkg
19+
pkg.purge()

‎ckan-backend-dev/ckan/scripts/init-extensions.sh

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
21
# Initialize Issues and Notification DB
32
docker exec ckan-wri sh -c "ckan -c production.ini issuesdb"
43
docker exec ckan-wri sh -c "ckan -c production.ini downloadeventdb"

‎ckan-backend-dev/ckan/setup/prerun.py.override

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -93,20 +93,24 @@ def check_solr_connection(retry=None):
9393

9494

9595
def init_db():
96+
# Output the contents of the CKAN_INI file
97+
print("[prerun] CKAN_INI file contents:")
98+
with open(ckan_ini, "r") as f:
99+
print(f.read())
96100

97101
db_command = ["ckan", "-c", ckan_ini, "db", "init"]
98102
print("[prerun] Initializing or upgrading db - start")
99103
try:
100104
subprocess.check_output(db_command, stderr=subprocess.STDOUT)
101105
print("[prerun] Initializing or upgrading db - end")
102106
except subprocess.CalledProcessError as e:
103-
if "OperationalError" in e.output:
104-
print(e.output)
107+
if "OperationalError" in e.output.decode("utf-8"):
108+
print(e.output.decode("utf-8"))
105109
print("[prerun] Database not ready, waiting a bit before exit...")
106110
time.sleep(5)
107111
sys.exit(1)
108112
else:
109-
print(e.output)
113+
print(e.output.decode("utf-8"))
110114
raise e
111115

112116

@@ -144,13 +148,13 @@ def init_datastore_db():
144148
print(str(e))
145149

146150
except subprocess.CalledProcessError as e:
147-
if "OperationalError" in e.output:
148-
print(e.output)
151+
if "OperationalError" in e.output.decode("utf-8"):
152+
print(e.output.decode("utf-8"))
149153
print("[prerun] Database not ready, waiting a bit before exit...")
150154
time.sleep(5)
151155
sys.exit(1)
152156
else:
153-
print(e.output)
157+
print(e.output.decode("utf-8"))
154158
raise e
155159
finally:
156160
cursor.close()

‎ckan-backend-dev/ckan/setup/start_ckan.sh.override

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,11 @@ echo "From GitHub Actions: $GITHUB_ACTIONS"
5151
# Check if we are running in GitHub Actions
5252
if [ "$GITHUB_ACTIONS" = "true" ]; then
5353
UWSGI_OPTS="$UWSGI_OPTS --single-interpreter --manage-script-name --mount /private-admin=/srv/app/wsgi.py"
54+
SUPERVISOR_USER='root'
55+
SUPERVISOR_HOME='/tmp'
56+
else
57+
SUPERVISOR_USER='ckan'
58+
SUPERVISOR_HOME='/srv/app'
5459
fi
5560
echo "Starting CKAN with the following uwsgi options: $UWSGI_OPTS"
5661

@@ -59,7 +64,17 @@ export SCRIPT_NAME="/private-admin"
5964
if [ $? -eq 0 ]
6065
then
6166
# Start supervisord
62-
export ENV_SUPERVISOR_USER=$(whoami)
67+
echo "==================================="
68+
echo "SUPERVISOR_USER: $SUPERVISOR_USER"
69+
echo "SUPERVISOR_HOME: $SUPERVISOR_HOME"
70+
echo "==================================="
71+
72+
sed -i "s|SUPERVISOR_USER|$SUPERVISOR_USER|g" /etc/supervisord.d/harvest.conf
73+
sed -i "s|SUPERVISOR_HOME|$SUPERVISOR_HOME|g" /etc/supervisord.d/harvest.conf
74+
sed -i "s|SUPERVISOR_USER|$SUPERVISOR_USER|g" /etc/supervisord.d/worker.conf
75+
sed -i "s|SUPERVISOR_HOME|$SUPERVISOR_HOME|g" /etc/supervisord.d/worker.conf
76+
sed -i "s|SUPERVISOR_HOME|$SUPERVISOR_HOME|g" /etc/supervisord.conf
77+
6378
supervisord --configuration /etc/supervisord.conf &
6479
# Start uwsgi
6580
sudo -u ckan -EH uwsgi $UWSGI_OPTS

‎ckan-backend-dev/ckan/setup/start_ckan_development.sh.override

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
echo "Looking for local extensions to install..."
55
echo "Extension dir contents:"
66
ls -la $SRC_EXTENSIONS_DIR
7+
78
for i in $SRC_EXTENSIONS_DIR/*
89
do
910
if [ -d $i ];
@@ -46,7 +47,7 @@ echo "Enabling debug mode"
4647
ckan config-tool $CKAN_INI -s DEFAULT "debug = true"
4748

4849
# Add ckan.datapusher.api_token to the CKAN config file (updated with corrected value later)
49-
ckan config-tool $CKAN_INI ckan.datapusher.api_token=xxx
50+
#ckan config-tool $CKAN_INI ckan.datapusher.api_token=xxx
5051

5152
# Set up the Secret key used by Beaker and Flask
5253
# This can be overriden using a CKAN___BEAKER__SESSION__SECRET env var
@@ -60,10 +61,6 @@ then
6061
ckan config-tool $CKAN_INI "api_token.jwt.decode.secret=${JWT_SECRET}"
6162
fi
6263

63-
# Update the plugins setting in the ini file with the values defined in the env var
64-
echo "Loading the following plugins: $CKAN__PLUGINS"
65-
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
66-
6764
# Update test-core.ini DB, SOLR & Redis settings
6865
echo "Loading test settings into test-core.ini"
6966
ckan config-tool $SRC_DIR/ckan/test-core.ini \
@@ -73,13 +70,30 @@ ckan config-tool $SRC_DIR/ckan/test-core.ini \
7370
"solr_url = $TEST_CKAN_SOLR_URL" \
7471
"ckan.redis.url = $TEST_CKAN_REDIS_URL"
7572

73+
ckan config-tool $CKAN_INI "ckan.plugins = datastore envvars"
74+
ckan config-tool $CKAN_INI "ckan.views.default_views ="
75+
76+
DEFAULT_VIEWS=$CKAN__VIEWS__DEFAULT_VIEWS
77+
PLUGINS=$CKAN__PLUGINS
78+
79+
unset CKAN__PLUGINS
80+
unset CKAN__VIEWS__DEFAULT_VIEWS
81+
7682
# Run the prerun script to init CKAN and create the default admin user
7783
python3 prerun.py
7884

85+
export CKAN__VIEWS__DEFAULT_VIEWS="$DEFAULT_VIEWS"
86+
export CKAN__PLUGINS="$PLUGINS"
87+
7988
unset CKAN__DATAPUSHER__API_TOKEN
8089
echo "Set up ckan.datapusher.api_token in the CKAN config file"
8190
ckan config-tool $CKAN_INI "ckan.datapusher.api_token=$(ckan -c $CKAN_INI user token add ckan_admin datapusher | tail -n 1 | tr -d '\t')"
8291

92+
# Update the plugins setting in the ini file with the values defined in the env var
93+
echo "Loading the following plugins: $CKAN__PLUGINS"
94+
ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS"
95+
ckan config-tool $CKAN_INI "ckan.views.default_views = $CKAN__VIEWS__DEFAULT_VIEWS"
96+
8397
# Run any startup scripts provided by images extending this one
8498
if [[ -d "/docker-entrypoint.d" ]]
8599
then

‎ckan-backend-dev/ckan/setup/supervisor.harvest.conf

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ stdout_logfile=/dev/stdout
88
stdout_logfile_maxbytes=0
99
stderr_logfile=/dev/stdout
1010
stderr_logfile_maxbytes=0
11-
user=%(ENV_SUPERVISOR_USER)s
12-
environment=HOME="/srv/app",USER="%(ENV_SUPERVISOR_USER)s"
11+
user=SUPERVISOR_USER
12+
environment=HOME="SUPERVISOR_HOME",USER="SUPERVISOR_USER"
1313

1414

1515
[program:ckan-gather]
@@ -22,5 +22,5 @@ stdout_logfile=/dev/stdout
2222
stdout_logfile_maxbytes=0
2323
stderr_logfile=/dev/stdout
2424
stderr_logfile_maxbytes=0
25-
user=%(ENV_SUPERVISOR_USER)s
26-
environment=HOME="/srv/app",USER="%(ENV_SUPERVISOR_USER)s"
25+
user=SUPERVISOR_USER
26+
environment=HOME="SUPERVISOR_HOME",USER="SUPERVISOR_USER"

‎ckan-backend-dev/ckan/setup/supervisor.worker.conf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@ stdout_logfile=/dev/stdout
88
stdout_logfile_maxbytes=0
99
stderr_logfile=/dev/stdout
1010
stderr_logfile_maxbytes=0
11-
user=%(ENV_SUPERVISOR_USER)s
12-
environment=HOME="/srv/app",USER="%(ENV_SUPERVISOR_USER)s"
11+
user=SUPERVISOR_USER
12+
environment=HOME="SUPERVISOR_HOME",USER="SUPERVISOR_USER"

‎ckan-backend-dev/ckan/setup/supervisord.conf

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
11
[unix_http_server]
2-
file = /tmp/supervisor.sock
2+
file = SUPERVISOR_HOME/supervisor.sock
33
chmod = 0777
44
chown = ckan:ckan
55

66
[supervisord]
7-
logfile = /tmp/supervisord.log
7+
logfile = SUPERVISOR_HOME/supervisord.log
88
logfile_maxbytes = 50MB
99
logfile_backups=10
1010
loglevel = info
11-
pidfile = /tmp/supervisord.pid
11+
pidfile = SUPERVISOR_HOME/supervisord.pid
1212
nodaemon = true
1313
umask = 022
1414
identifier = supervisor
1515

1616
[supervisorctl]
17-
serverurl = unix:///tmp/supervisor.sock
17+
serverurl = unix://SUPERVISOR_HOME/supervisor.sock
1818

1919
[rpcinterface:supervisor]
2020
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

0 commit comments

Comments
 (0)