Skip to content

Commit 3a2baac

Browse files
luccasmmgmpolidoristeveoni
authored
Sync dev staging apr 14 (#671)
* Harvest dynamic configs pt. 2 (#635) * Update ckanext-s3filestore branch * Fix typo * Update layer manager to use wri release * Fix Dockerfile * [ODP-235] Allow private and hidden teams (#537) * update teams create and edit form * update organization list api * update schema * update api * remove extras * fix type errors * update user_list_wri to support private teams * fix api call * remove chained action * add visibility * fix breaking changes * update env * update env plugin * fix test * update test * add visibility to org during dataset edit * add test for private teams * remove logs * update private team functionality * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * Merge branch 'priveateams' of https://github.com/wri/wri-odp into priveateams * resolve conflict * undo changes * update organization patch and create * add private attribute to fetched orgs * update organization_list sql logic * improve team visbility * improve error message * update private team test * update failing test * update test * update settings * update test * update test * update test * update test * update test * update test * update test * update docker compose * add tooltip * update test * add padlock to private teams * add back auth plugin * Harvest config feb 03 2025 (#637) * Fix variable replacement in harvest worker configs and switch from /tmp path to /srv/app * Add whoami output to start_ckan.sh * [download_event.py] Add missing closing parenthesis * Set supervisor home by variable * Fix harvest config permissions * Use CKAN_IMAGE variable instead of local image name * Handle sed supervisor changes as root; Add start_ckan sub-script * Revert sub-script; Always use user ckan for supervisor * Fix sed commands * Use root user with ckan home for supervisord * Revert "Harvest config feb 03 2025 (#637)" This reverts commit e68155a. * Harvest dynamic configs pt. 3 (#638) * Add supercronic for harvest run command (#640) * Fix alignment * Fix build * Fix build * [ODP-414] add email notification when download-flow fails (#641) * add email notification when flow fails * update error notification * empty * remove pgclient version * update libpq --------- Co-authored-by: Michael Polidori <[email protected]> * Upgrade to CKAN 2.11 (#632) * Update setup.py * Update Members permission to team form + ODP-370 (#642) * members can not * update role: Editor can create sub-team and admin moving subteam from public to private should not throw error * disabled Public option if parent is private * Trigger CI --------- Co-authored-by: Luccas Mateus <[email protected]> * Change setup.py and setup.cfg * Fix ckanext-wri install (#643) * Improve resource location design (#646) * Trigger CI * [odp-426]: Add edit icon to dashboard entities (#648) * [odp-426]: Add edit icon to dashboard entities * update schema * Add bulk purge collaborator issue patch (#645) * Increase UWSGI timeouts; Extract URLs from markdown links in learn_more (#649) * Resource location search improvements (#650) * add authorized check to group/teams last node (#651) * Trigger CI * Rm global from DatafileLocation * Trigger CI * enable edit icon for collaborators (#652) * Fix typo * intgration test for collaborator permission (#654) * enable edit icon for collaborators * add intergration test for dashboard collaborator permission * Improve speed * Improve js * Fix prefetching * Change color of area select tool * Improvements map * Remove edit icon on teams/topic page for editor (#656) * enable edit icon for collaborators * remove editor permision * empty * Fix tests * Fix tests * Fix tests * Fix test * Clear all session storage * Fix tests * Fix tests * Fix private teams tests * Fix cypress.config.js * Fix tests * Fix tests * Fix tests * Fix tests * Fix whitespace wrap on text * Trigger CI * Add group admin email to workflow error mail (#663) * enable edit icon for collaborators * send workflow error to team admin * removed email check * E2etest (#657) * enable edit icon for collaborators * empty * add check to ensure org is actually private * update * update * fix test * update test * empty * update * update test * update test * update test * update test * update test * update test to see if organization create api is truly working * update test * remove other test for now * push update * update env.example * fix test * empty * Fix git vulnerability * Fix git * Add if condition (#664) * [odp-449] Teams Implementation - Root parent Team Edit (#669) * enable edit icon for collaborators * admin should be able to edit root team * ODP-447 (#668) * ODP-447 * Fix build * Fix tests * Fix t ests * Trigger CI * [Dev] Increase proxy-read-timeout to avoid early external request disconnects (#665) * Revert "ODP-447 (#668)" This reverts commit 7c0e465. * Revert "[odp-449] Teams Implementation - Root parent Team Edit (#669)" This reverts commit 7d61dcd. * Rm test-main * Revert ingress changes --------- Co-authored-by: Michael Polidori <[email protected]> Co-authored-by: Stephen Oni <[email protected]>
1 parent 5b29a5e commit 3a2baac

File tree

13 files changed

+106
-49
lines changed

13 files changed

+106
-49
lines changed

‎ckan-backend-dev/.env.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ CKANEXT__S3FILESTORE__HOST_NAME=http://minio:9000
101101

102102
# scheming
103103
CKAN___SCHEMING__DATASET_SCHEMAS=ckanext.wri.schema:ckan_dataset.yaml
104-
CKAN___SCHEMING__ORGANIZATION_SCHEMAS=ckanext.scheming:custom_org_with_address.json
104+
CKAN___SCHEMING__ORGANIZATION_SCHEMAS=ckanext.wri.schema:custom_org_with_address.json
105105
CKAN___SCHEMING__GROUP_SCHEMAS=ckanext.wri.schema:wri_application.json
106106
CKAN___SCHEMING__PRESETS=ckanext.wri.schema:presets.json
107107

‎ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/datapusher_download.py

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
import ckanext.s3filestore.uploader as uploader
77
from ckan.lib.mailer import mail_recipient
88
from ckan.common import config
9-
from .datapusher_download_zip import fetch_dataset_name
9+
from .datapusher_download_zip import fetch_dataset_name, get_admin_emails_for_dataset
1010

1111
import datetime
1212
import requests
@@ -75,8 +75,21 @@ def download_request(context: Context, data_dict: dict[str, Any]):
7575
"error": "{}",
7676
}
7777

78+
dataset_id = fetch_dataset_name({
79+
"entity_id": res_id,
80+
"entity_type": "resource"
81+
})
82+
admin_email= get_admin_emails_for_dataset(dataset_id)
83+
84+
value = {}
85+
if admin_email:
86+
value["admin_emails"] = admin_email
87+
7888
if email:
79-
task["value"] = json.dumps({"emails": [email]})
89+
value["emails"] = [email]
90+
91+
task["value"] = json.dumps(value)
92+
8093

8194
try:
8295
existing_task = p.toolkit.get_action("task_status_show")(
@@ -105,6 +118,7 @@ def download_request(context: Context, data_dict: dict[str, Any]):
105118

106119
if update_emails:
107120
existing_task_values['emails'] = existing_task_emails
121+
existing_task_values["admin_emails"] = admin_email
108122
existing_task['value'] = json.dumps(existing_task_values)
109123
p.toolkit.get_action("task_status_update")({ "ignore_auth": True }, existing_task)
110124

@@ -182,7 +196,7 @@ def download_request(context: Context, data_dict: dict[str, Any]):
182196
"entity_id": res_id,
183197
"entity_type": "resource"
184198
})
185-
send_error([email], resource_title, dataset_name)
199+
send_error([email]+admin_email, resource_title, dataset_name)
186200
raise p.toolkit.ValidationError(error)
187201

188202
try:
@@ -208,11 +222,11 @@ def download_request(context: Context, data_dict: dict[str, Any]):
208222
"entity_id": res_id,
209223
"entity_type": "resource"
210224
})
211-
send_error([email], resource_title, dataset_name)
225+
send_error([email]+admin_email, resource_title, dataset_name)
212226
raise p.toolkit.ValidationError(error)
213227

214228
value = {"job_id": r.json()["id"]}
215-
229+
value["admin_emails"] = admin_email
216230
if email:
217231
value["emails"] = [email]
218232

@@ -251,6 +265,7 @@ def download_callback(context: Context, data_dict: dict[str, Any]):
251265

252266
value = json.loads(task["value"])
253267
emails = value.get("emails", [])
268+
admin_email = value.get("admin_emails", [])
254269
download_filename = value.get("download_filename")
255270

256271
log.info("Preparing to send email...")
@@ -263,7 +278,7 @@ def download_callback(context: Context, data_dict: dict[str, Any]):
263278
"entity_id": entity_id,
264279
"entity_type": "resource"
265280
})
266-
send_error(emails, download_filename, dataset_name)
281+
send_error(emails+admin_email, download_filename, dataset_name)
267282
log.error(error)
268283

269284

‎ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/datapusher_download_subset.py

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
import ckanext.s3filestore.uploader as uploader
77
from ckan.lib.mailer import mail_recipient
88
from ckan.common import config
9-
from .datapusher_download_zip import fetch_dataset_name
9+
from .datapusher_download_zip import fetch_dataset_name, get_admin_emails_for_dataset
1010

1111
import datetime
1212
import requests
@@ -143,8 +143,15 @@ def subset_download_request(context: Context, data_dict: dict[str, Any]):
143143
"error": "{}",
144144
}
145145

146+
admin_email= get_admin_emails_for_dataset(dataset_id)
147+
value = {}
148+
if admin_email:
149+
value["admin_emails"] = admin_email
150+
146151
if email:
147-
task["value"] = json.dumps({"emails": [email]})
152+
value["emails"] = [email]
153+
154+
task["value"] = json.dumps(value)
148155

149156
try:
150157
existing_task = p.toolkit.get_action("task_status_show")(
@@ -176,6 +183,7 @@ def subset_download_request(context: Context, data_dict: dict[str, Any]):
176183

177184
if update_emails:
178185
existing_task_values["emails"] = existing_task_emails
186+
existing_task_values["admin_emails"] = admin_email
179187
existing_task["value"] = json.dumps(existing_task_values)
180188
p.toolkit.get_action("task_status_update")(
181189
{"ignore_auth": True}, existing_task
@@ -260,7 +268,7 @@ def subset_download_request(context: Context, data_dict: dict[str, Any]):
260268
"entity_id": id if provider == "datastore" else dataset_id,
261269
"entity_type": "resource" if provider == "datastore" else "dataset"
262270
})
263-
send_error([email], "Subset of data", dataset_name)
271+
send_error([email]+admin_email, "Subset of data", dataset_name)
264272
raise p.toolkit.ValidationError(error)
265273

266274
try:
@@ -286,11 +294,11 @@ def subset_download_request(context: Context, data_dict: dict[str, Any]):
286294
"entity_id": id if provider == "datastore" else dataset_id,
287295
"entity_type": "resource" if provider == "datastore" else "dataset"
288296
})
289-
send_error([email], "Subset of data", dataset_name)
297+
send_error([email]+admin_email, "Subset of data", dataset_name)
290298
raise p.toolkit.ValidationError(error)
291299

292300
value = {"job_id": r.json()["id"]}
293-
301+
value["admin_emails"] = admin_email
294302
if email:
295303
value["emails"] = [email]
296304

@@ -325,6 +333,7 @@ def subset_download_callback(context: Context, data_dict: dict[str, Any]):
325333

326334
value = json.loads(task["value"])
327335
emails = value.get("emails", [])
336+
admin_emails = value.get("admin_emails", [])
328337
download_filename = value.get("download_filename")
329338

330339

@@ -336,7 +345,7 @@ def subset_download_callback(context: Context, data_dict: dict[str, Any]):
336345
"entity_id": entity_id,
337346
"entity_type": data_dict.get("entity_type", "resource")
338347
})
339-
send_error(emails, download_filename, dataset_name)
348+
send_error(emails + admin_emails, download_filename, dataset_name)
340349
log.error(error)
341350

342351

‎ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/datapusher_download_zip.py

Lines changed: 50 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,31 @@ def build_download_filename(dataset_id: str, context) -> str:
6666
return False
6767

6868

69+
70+
def get_admin_emails_for_dataset(dataset_id: str) -> list[str]:
71+
package = p.toolkit.get_action("package_show")(
72+
{"ignore_auth": True}, {"id": dataset_id}
73+
)
74+
organization = package.get("organization", None)
75+
admin_email = []
76+
if organization:
77+
organization = organization.get("name")
78+
org = p.toolkit.get_action("organization_show")(
79+
{"ignore_auth": True}, {"id": organization, "include_users": True}
80+
)
81+
users = org.get("users", [])
82+
if users:
83+
for user in users:
84+
if user.get("capacity") == "admin":
85+
users_obj = p.toolkit.get_action("user_show")(
86+
{"ignore_auth": True}, {"id": user.get("id")}
87+
)
88+
if users_obj.get("email", False):
89+
admin_email.append(users_obj.get("email"))
90+
return admin_email
91+
92+
93+
6994
def zipped_download_request(context: Context, data_dict: dict[str, Any]):
7095
prefect_url: str = config.get("ckanext.wri.prefect_url")
7196
deployment_name: str = config.get("ckanext.wri.datapusher_deployment_name")
@@ -93,11 +118,21 @@ def zipped_download_request(context: Context, data_dict: dict[str, Any]):
93118
"state": "submitting",
94119
"key": filename,
95120
"value": "{}",
96-
"error": "{}",
121+
"error": "{}",
97122
}
98123

124+
admin_email= get_admin_emails_for_dataset(dataset_id)
125+
log.error(f"admin_email: {admin_email}")
126+
127+
128+
value = {}
129+
if admin_email:
130+
value["admin_emails"] = admin_email
131+
99132
if email:
100-
task["value"] = json.dumps({"emails": [email]})
133+
value["emails"] = [email]
134+
135+
task["value"] = json.dumps(value)
101136

102137
try:
103138
existing_task = p.toolkit.get_action("task_status_show")(
@@ -129,6 +164,8 @@ def zipped_download_request(context: Context, data_dict: dict[str, Any]):
129164

130165
if update_emails:
131166
existing_task_values["emails"] = existing_task_emails
167+
if admin_email:
168+
existing_task_values["admin_emails"] = admin_email
132169
existing_task["value"] = json.dumps(existing_task_values)
133170
p.toolkit.get_action("task_status_update")(
134171
{"ignore_auth": True}, existing_task
@@ -204,7 +241,7 @@ def zipped_download_request(context: Context, data_dict: dict[str, Any]):
204241
task["state"] = "error"
205242
task["last_updated"] = (str(datetime.datetime.utcnow()),)
206243
p.toolkit.get_action("task_status_update")(context, task)
207-
send_error([email], "Zipped data")
244+
send_error([email]+admin_email, "Zipped data")
208245
raise p.toolkit.ValidationError(error)
209246

210247
try:
@@ -226,14 +263,17 @@ def zipped_download_request(context: Context, data_dict: dict[str, Any]):
226263
task["state"] = "error"
227264
task["last_updated"] = (str(datetime.datetime.utcnow()),)
228265
p.toolkit.get_action("task_status_update")(context, task)
229-
send_error([email], "Zipped data")
266+
send_error([email] + admin_email, "Zipped data")
230267
raise p.toolkit.ValidationError(error)
231268

232269
value = {"job_id": r.json()["id"]}
233270

234271
if email:
235272
value["emails"] = [email]
236273

274+
if admin_email:
275+
value["admin_emails"] = admin_email
276+
237277
value["download_filename"] = download_filename
238278

239279
task["value"] = json.dumps(value)
@@ -265,28 +305,32 @@ def zipped_download_callback(context: Context, data_dict: dict[str, Any]):
265305

266306
value = json.loads(task["value"])
267307
emails = value.get("emails", [])
308+
admin_email = value.get("admin_emails", [])
268309
download_filename = value.get("download_filename")
269310

270311
if state == "complete":
271312
url = data_dict.get("url")
272313
send_email(emails, url, download_filename)
273314
else:
274-
send_error(emails, download_filename)
315+
send_error(emails+admin_email, download_filename)
275316
log.error(error)
276317

277318
def send_error_callback(context: Context, data_dict: dict[str, Any]):
278319
entity_id = data_dict.get("entity_id")
320+
task_type = data_dict.get("task_type")
279321
key = data_dict.get("key")
280322
task = p.toolkit.get_action("task_status_show")(
281323
context,
282-
{"entity_id": entity_id, "task_type": "download_zipped", "key": key},
324+
{"entity_id": entity_id, "task_type": task_type, "key": key},
283325
)
284326

285327
if not task:
286328
raise logic.NotFound("Task not found")
287329

288330
value = json.loads(task["value"])
289331
emails = value.get("emails", [])
332+
admin_emails = value.get("admin_emails", [])
333+
emails += admin_emails
290334
download_filename = value.get("download_filename")
291335
dataset_name = fetch_dataset_name({
292336
"entity_id": entity_id,

‎ckan-backend-dev/src/ckanext-wri/ckanext/wri/logic/action/get.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1848,7 +1848,6 @@ def organization_show(context, data_dict):
18481848
data_dict = old_organization_show(context, data_dict)
18491849
user = context.get("user")
18501850

1851-
18521851
if not authz.is_sysadmin(user):
18531852
is_authorized = get_action("organization_list")(context, {"q": data_dict.get("name")})
18541853
if not is_authorized:

‎datapusher/main.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,6 @@ def convert_store_to_file(
131131
try:
132132
logger = get_run_logger()
133133
ckan_url = config.get("CKAN_URL")
134-
135134
logger.info("Fetching data...")
136135
data = query_datastore(
137136
api_key, ckan_url, sql, provider, rw_id, carto_account, format
@@ -169,7 +168,8 @@ def convert_store_to_file(
169168
"prefect_send_error_callback",
170169
{
171170
"task_id": task_id,
172-
"url": url,
171+
"url": "",
172+
"task_type": "download",
173173
"state": "complete",
174174
"entity_id": resource_id,
175175
"entity_type": "resource",
@@ -195,7 +195,6 @@ async def download_subset_of_data(
195195
try:
196196
logger = get_run_logger()
197197
ckan_url = config.get("CKAN_URL")
198-
199198
logger.info("Fetching data...")
200199
data = []
201200
if provider == "datastore":
@@ -237,6 +236,7 @@ async def download_subset_of_data(
237236
"prefect_send_error_callback",
238237
{
239238
"task_id": task_id,
239+
"task_type": "download_subset",
240240
"url": "",
241241
"state": "failed",
242242
"entity_id": id if provider == "datastore" else dataset_id,
@@ -260,7 +260,6 @@ async def download_resources_zipped(
260260
try:
261261
logger = get_run_logger()
262262
ckan_url = config.get("CKAN_URL")
263-
print("Filename", filename)
264263
with tempfile.TemporaryDirectory() as temp_dir:
265264
tasks = await download_keys(keys, filename, temp_dir)
266265
data = list(tasks)
@@ -295,6 +294,7 @@ async def download_resources_zipped(
295294
{
296295
"task_id": task_id,
297296
"url": "",
297+
"task_type": "download_zipped",
298298
"state": "failed",
299299
"entity_id": dataset_id,
300300
"entity_type": "dataset",

‎deployment/ckan/Dockerfile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ RUN chmod 777 ${APP_DIR}/src/ckanext-s3filestore
3232
# Required by shapely dependency
3333
RUN apk --update add build-base libxslt-dev python3-dev
3434
RUN apk update
35+
RUN apk add git=2.40.4-r0
3536
RUN apk add postgresql15-client=15.11-r0
3637

3738
RUN pip install --force-reinstall -v "Pillow==11.0.0"

‎deployment/frontend/src/components/search/DatasetHorizontalCard.tsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ export default function DatasetHorizontalCard({
8888
</div>
8989
</div>
9090

91-
<p className="font-acumin text-[1rem] leading-[22px] font-light text-[#1A1919] line-clamp-4 mt-[12px] w-full h-[88px] lg:w-[733.783px] mb-[21px] ">
91+
<p className="font-acumin text-[1rem] leading-[22px] font-light text-[#1A1919] mt-[12px] w-full min-h-[88px] mb-[21px] ">
9292
{dataset.short_description ?? ''}
9393
</p>
9494

‎deployment/frontend/src/server/api/routers/teams.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,6 +156,7 @@ export const teamRouter = createTRPCRouter({
156156
},
157157
}
158158
)
159+
159160
const team: CkanResponse<
160161
WriOrganization & {
161162
groups: Organization[]

‎deployment/helm-templates/templates/wri-staging-ingress-fe-internal-api.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ metadata:
1717
nginx.ingress.kubernetes.io/limit-rps: "50"
1818
nginx.ingress.kubernetes.io/proxy-body-size: 1000M
1919
nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
20-
nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
20+
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
2121
nginx.ingress.kubernetes.io/rewrite-target: /api/action/$1
2222
nginx.ingress.kubernetes.io/use-regex: "true"
2323
labels:

0 commit comments

Comments
 (0)