Changed install scripts to be consistent with new python installation method #1417
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewer's GuideThis pull request refactors the Python installation process by introducing a centralized shell function, Sequence diagram for Python installation processsequenceDiagram
participant CS as Calling Script
participant IPF as install_python function
participant APT as System Package Manager (apt)
participant PPA as PPA (deadsnakes)
CS->>IPF: Execute install_python
IPF->>IPF: Check for .python_installed marker
alt Marker not found
IPF->>APT: apt update
APT-->>IPF: Done
IPF->>APT: apt install software-properties-common
APT-->>IPF: Done
IPF->>PPA: add-apt-repository ppa:deadsnakes/ppa
PPA-->>IPF: Repository added
IPF->>APT: apt update
APT-->>IPF: Done
IPF->>APT: apt install python${DATAFED_PYTHON_VERSION}...
APT-->>IPF: Python installed
IPF->>IPF: touch .python_installed marker
end
Flow diagram for the new install_python functiongraph TD
A[Start] --> B{Python already installed? Check marker file};
B -- Yes --> F[End];
B -- No --> C[apt update];
C --> D[apt install software-properties-common];
D --> E[add-apt-repository ppa:deadsnakes/ppa];
E --> G[apt update];
G --> H[apt install python, python-dev, python-venv, python-distutils];
H --> I[Create .python_installed marker file];
I --> F;
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
JoshuaSBrown
reviewed
May 7, 2025
| @@ -64,6 +64,18 @@ else | |||
| fi | |||
| fi | |||
|
|
|||
| install_python() { | |||
| if [ ! -e "${DATAFED_DEPENDENCIES_INSTALL_PATH}/.python_installed-${DATAFED_PYTHON_VERSION}" ]; then | |||
| apt update | |||
There was a problem hiding this comment.
Suggested change
| apt update | |
| "$SUDO_CMD" apt update |
There was a problem hiding this comment.
This SUDO piece needs to be added to all of the apt calls.
AronPerez
reviewed
May 10, 2025
| # Install the configured Python version | ||
| RUN apt update | ||
| RUN apt install -y software-properties-common | ||
| RUN add-apt-repository ppa:deadsnakes/ppa |
There was a problem hiding this comment.
What was the motivation for using the deadsnakes PPA for Python installation rather than the default Ubuntu repositories?
Comment on lines
+67
to
+78
| install_python() { | ||
| if [ ! -e "${DATAFED_DEPENDENCIES_INSTALL_PATH}/.python_installed-${DATAFED_PYTHON_VERSION}" ]; then | ||
| "$SUDO_CMD" apt update | ||
| "$SUDO_CMD" apt install -y software-properties-common | ||
| "$SUDO_CMD" add-apt-repository ppa:deadsnakes/ppa | ||
| "$SUDO_CMD" apt update | ||
| "$SUDO_CMD" apt install -y "python${DATAFED_PYTHON_VERSION}" "python${DATAFED_PYTHON_VERSION}-dev" "python${DATAFED_PYTHON_VERSION}-venv" "python${DATAFED_PYTHON_VERSION}-distutils" | ||
|
|
||
| touch "${DATAFED_DEPENDENCIES_INSTALL_PATH}/.python_installed-${DATAFED_PYTHON_VERSION}" | ||
| fi | ||
| } | ||
|
|
JoshuaSBrown
added a commit
that referenced
this pull request
May 13, 2025
* updated requirements * updated requirements * updated protobuf submodule * update unit test ci version * updated protobuf version * updated setuptools version * fixed condition * changed where packages are installed * used python3.11 to create the virtual environment instead of latest * reverted protobuf version * testing specificying version * removed pyvenv in ci job for consistency * installed python3.11 in dependencies docker file * added missing -y flag * removed unification of install scripts * added missing python3.11 install * updated ci job to install python3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * missed version * updated includes for newer gcc versions * updated versions * updated nlohmann json * updated json schema version * changed library search path * hopefully fixed json dep issues * updating function signature to match newer version of base class * updated protobuf * updated protobuf and cmake * downgraded cmake * fixed protobuf version * fixed protobuf version * 1398 pin python version (#1405) * Pin python version * fixed incorrect path * Switch to using .tar file install of libsodium (#1414) * 1413 libsodium build refactor (#1415) * Switch to using .tar file install of libsodium * Switch bad option in wget command from -C to -P * Update scripts/dependency_install_functions.sh * Update scripts/dependency_install_functions.sh * Update dependency_install_functions.sh Libsodium folder version number. * Update dependency_install_functions.sh Make paths explicit. * cleaned up comments * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown <[email protected]> * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown <[email protected]> * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown <[email protected]> * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown <[email protected]> * add changes from review * Changed install scripts to be consistent with new python installation method (#1417) * changed install scripts to be consistent * added sudo check * add apt sources check --------- Co-authored-by: Blake Nedved <[email protected]> Co-authored-by: nedvedba <[email protected]>
JoshuaSBrown
added a commit
that referenced
this pull request
Jun 12, 2025
* Address formatting of jquery file * add more descriptive posix path error message * Remove redundant password file creation command * Add eslint plugin for jsdoc to github action * Update esling.config.js to error if invalid jsdoc * Add missing eslint jsdoc plugin * Remove non existant rule * Fix jsdoc issues in data_router * Fix JSDOc issues in process.js fiel * Fix JSDOc issues in repo_router.js * Address JSDoc in tasks.js * Address JSDoc linting issues * Switch to ignores statement * Address more eslint jsdoc items * Be more selective about eslint rules * Remove requrie description because seems to be faulty * Address finicky jsdoc linter * Fix js formatting * Feature DLT 1109 foxx new token data (#1194) * Stub logic for updating related edge in router function for token/set * Include logic for conditionally updating token data and pushing data to edge. * Change token_key to remove illegal characters * Fix object attribute call * Formally add query params to function. Extract query params near start of action. Check both exist before taking action on edge. Update comments, notes, TODOs * Stub testing for user_router * Fix naming of user_router test so it will be picked up with sane specification. Add test to CMakeLists * Add some fixtures for creating related user to allow for integration testing * Fix manifest json. Hard code user collection check. Set time variable. * Use correct path for router to user endpoints. Extract some query params for validation. * Remove intentionally failing test. Break now working test into logical sections. Update comments. * Add test for new features * Add further validation of no unexpected side-effects * Remove extraneous user document checks and comments * More testing for branches in introduced logic. Additional check on newly inserted globus_coll * assert -> expect * Work around multiple access and ordering issue by utilizing more test user fixtures. * Rearrange key for token edge, add suggested fields to documents * Pull user_doc from existing logic. Use object destructuring. Add error condition. Introduce protobuf enum to backend. Add parser for other_token_data string. Update document creation functions * Extract fixture build call to separate script. Add script as fixture in cmake * Remove some extraneous logic, remove password file overwrite. * Remove unused zeromq flag * Adjust testing to include enum, expect error on invalid input combination, typo, naming * Adjust scope string in user_router test. Add validations to parseOtherTokenData. Add tests for positive and error cases of parseOtherTokenData. * Change chai expects in support.test.js to use callable. Update perms on fixture setup script. Use URL encoded pipe character in user_router.test.js * Group OR in user_router.js * Adjust user_router.test.js integration tests to access deconstructed other_token_data * Add baseline jsdoc for parse method. Remove some unwanted data from globus_token doc insert * Add GLOBUS_DEFAULT to AccessTokenType enum. Add default to user_router endpoint param. Update logic to switch based on enum. Test new change by specifying default type and other_token_data. Fix name of test. * Move query parameter validations earlier in function. Decrease nesting of switch logic. * Cleanup some whitespace changes, add desired comments on assumptions. * Formatting * Fix naming for support test so they will run. * Update CHANGELOG.md with changes in minor feature. Update comments on user_router.js for saving a new globus_coll * Add comment about more fields in support.js::parseOtherTokenData * Fix issue for formatter * Add expiration check * Commit for CI * Formatting --------- Co-authored-by: Anthony Ramirez <[email protected]> * Add ci jobs to improve security posture * Change to unit stage for testing * Remove sast job * Attempt to fix semgrep sast job failure * Debug jobs * Add test stage for sast template job * Add missing test stage * Add dependency scanning * Split sast into separate test stage * Remove commented out line * Remove extra comment that is unused * Debugging log failure * Address jsdoc and typo in description * Remove outdated comments * Make comments JSDoc compliant * Address formatting inconsistency * Account for multiple images returned in log * Update .gitlab/common.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Debugging log container matching * Fix typo in gitlab anchor * Debug escape anchor * Debug * Debug anchor 2 * Debug anchor 3 * Debug anchor 4 * Debug anchor 5 * Debug 6 * Debug 7 * Debug 8 * Fix names of containers from gcs-authz to gcs * Cleanup add comment back in * Update common.yml * Remove Exports * Update data_router.js Remove nested docs. * Update repo_router.js * Update task_router.js * Update eslint.config.js * Update eslint.config.js * fix jsdoc comment in posix path * Address JSDOc complaints * Address JSDoc errors * Address linter * Address linter errors * Fix order of parameters * Apply formatting * Add missing argument form strategy method call * Apply prettier * Update posix_path.js * Address ai suggestion about better error message * Debugging log failure * Account for multiple images returned in log * Update .gitlab/common.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Debugging log container matching * Fix typo in gitlab anchor * Debug escape anchor * Debug * Debug anchor 2 * Debug anchor 3 * Debug anchor 4 * Debug anchor 5 * Debug 6 * Debug 7 * Debug 8 * Fix names of containers from gcs-authz to gcs * Cleanup add comment back in * Update common.yml * Remove Exports * Cleanup comments and apply formatting * Update nlhoman json version * Update protobuf versions * Add tests for authz router * Specify ubuntu 20.04 explicitly * Revert dependency versions * Fix problems with path consistent method * Cleanup javascript * Format javascript * Feature DLT 1127 relay new token data (#1217) * Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API. * Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility * Conditionally add other token options param when calling dbGetRaw. * Pass token type along to Database API. * Pass token type from message. * Extract logic for building URL from dbPost and dbGetRaw into separate function for easier testing and debugging * Extract url building logic from dbGet * Add some comments for moved code, address a TODO, add new TODO for AccessToken Message type field * Address default on token type, formatting * Add new AccessTokenType entries for a sentinel value and Globus default in SDMS.proto, use as desired in DatabaseAPI.cpp * Change query param to match backend changes * Restore formatting of SDMS.proto * Make built URL const where appropriate in DatabaseAPI.cpp and DatabaseAPI.hpp * Change default AccessTokenType for UserSetAccessTokenRequest in SDMS_Auth.proto. * Remove logging of potentially senitive data * Adjust AccessTokenType ACCESS_SENTINEL to have value of uint8 max. --------- Co-authored-by: Anthony Ramirez <[email protected]> * [DLT-1110] Update files with prettier [DLT-1110] Add controller unit tests [DLT-1110] Remove debug [DLT-1110] Correct import statements, endpoint list [DLT-1110] Refactor out into MVC [DLT-1110] Add debug [DLT-1110] Functioning modal, needs refactoring [DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see * [DLT-1110] Split branches * [DLT-1110] Update model, make logic private, remove controller form transfer request when update or get * [DLT-1110] Add commnets to logic * [DLT-1110] Correct HTML escape, francy tree init * [DLT-1110] Pull out template HTML * [DLT-1110] Fix update bug * [DLT-1110] Dependency injection * [DLT-1110] Update if statements * [DLT-1110] Update style * Revert "[DLT-1110] Mapped Collection Endpoint Browse (1/4)" * Address eslint * Address remaining eslint items * Fix bug on task_router.js abort function (#1234) Co-authored-by: Anthony Ramirez <[email protected]> * Address Aaron feedback * applied potential fix to deprecation warning * Reorder createRecord * Apply formatting * Throw error if OpenStack error code returned from API * Add dependency between logs * Add dependencies on end to end jobs as well * Job rule is not supported * Remove always run log * add end to end signal job * Fix tag of signal * Feature DLT 1120 retrieve transfer token (#1228) * Add conditional switch and stub some logic for handling other resources in callbacks * Fix order of logging. Add redirect * Debug logging for web service. Attempt to set token. * Remove incorrect method call to data, use data attribute. * Add some TODOs and logging * Give setAccessToken ability to accept additional params to pass to protobuf * Add additional params to DatabaseAPI::userSetAccessToken to pass message with addtional info on to database API. * Add convenience method with old type signature of DatabaseAPI::userSetAccessToken for compatibility * Conditionally add other token options param when calling dbGetRaw. * Pass token type along to Database API. * Pass token type from message. * Stub logic for updating related edge in router function for token/set * Include logic for conditionally updating token data and pushing data to edge. * Change token_key to remove illegal characters * Fix object attribute call * Prettier formatting for datafed-ws.js * Roll back changes to DatabaseAPI.hpp and DatabaseAPI.cpp that are covered in #1127 * Refactor new inclusions in datafed-ws.js to use as much existing code as possible, only diverge where necessary. * datafed-ws.js change scopes to scope * datafed-ws.js log user ID array * datafed-ws.js clarify uid assignment at ui/authn endpoint, add note on error cases, remove incorrect comment about uid, add appropriate error if collection_id not present * datafed-ws.js Extract AccessTokenType enum, add comment about fetching from protobuf; remove unnecessary commented code * datafed-ws.js Extract transfer set logic when calling setAccessToken to improve readability of router. * datafed-ws.js Create function for resolving token type; create function to handle logic in building optional_data for setAccessToken; reduce unneccessary nesting in ui/authn endpoint * datafed-ws.js Fix spacing, fix typo in variable name * datafed-ws.js Fix bug referring to request session * datafed-ws.js Nesting was necessary. * datafed-ws.js Address some TODOs, set token type with more context * datafed-ws.js Remove some additional extraneous logging * datafed-ws.js Formatting * Add comment about token set * datafed-ws.js Update deprecated substring method, Fix order for error case when new user receives transfer token * TokenHandler.js Refactor token handling logic out of web server main. datafed-ws.js User new OAuthTokenHandler class to handle token logic. * TokenHandler.js Implement token validator; move OAuthTransferToken def to top. TokenHandler.test.js Minimal testing of token handler * TokenHandler.test.js Add tests for case when resource server is auth * TokenHandler.test.js Add testing for Globus transfer resource server * TokenHandler.js Implement validation for existence of required keys. TokenHandler.test.js Adjust other_tokens fixtures to be nested according to incoming data; formatting. * datafed-ws.js formatting * TokenHandler.js Update error messages, remove unnecessary code in getTokenType, returns in resolveTokenType * datafed-ws.js Add error handling around token_handler construction; add errors and handling for setAccessToken, JSDoc. * datafed-ws.js Remove hard coded collection_id. * datafed-ws.js Remove thrown errors in nested functions; single redirect. --------- Co-authored-by: Anthony Ramirez <[email protected]> * Bug daps 1243 datafed web logging (#1251) * datafed-ws.js Reverse order of LogLevel object properties * datafed-ws.js Throw error so stack trace is adequately populated * Revert line number * TokenHandler.js Fix linting errors. --------- Co-authored-by: Anthony Ramirez <[email protected]> * [DLT-1110] Mapped Collection Endpoint Browse (1/4) (#1240) * [DLT-1110] Update style [DLT-1110] Update if statements [DLT-1110] Dependency injection [DLT-1110] Fix update bug [DLT-1110] Pull out template HTML [DLT-1110] Correct HTML escape, francy tree init [DLT-1110] Add commnets to logic [DLT-1110] Update model, make logic private, remove controller form transfer request when update or get [DLT-1110] Split branches [DLT-1110] Update files with prettier [DLT-1110] Add controller unit tests [DLT-1110] Remove debug [DLT-1110] Correct import statements, endpoint list [DLT-1110] Refactor out into MVC [DLT-1110] Add debug [DLT-1110] Functioning modal, needs refactoring [DLT-1110] Refactor dlg start transfer using OOO programming. Should be MVC cased on what I'm seeing but we'll see * [DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207) * [DLT-1110] Add tests * [DLT-1110] Update chai with mocha, attempt usage of mock-import * [DLT-1110] Add mocha config, correct prettier * [DLT-1110] Update tests, correct logic * [DLT-1110] Update tests, add fixtures, update setup, lock in packages * [DLT-1110] Update test * [DLT-1110] Config * [DLT-1110] Upgrade node v14 -> v23 * [DLT-1110] Upgrade node v23 -> v18, better jump * [DLT-1110] Correct ver .sh file * Revert "[DLT-1110] Correct ver .sh file" This reverts commit 5c78c59. * Revert "[DLT-1110] Upgrade node v23 -> v18, better jump" This reverts commit 9d66847. * Revert "[DLT-1110] Upgrade node v14 -> v23" This reverts commit fb8b93d. * Revert "[DLT-1110] Config" This reverts commit 82715f0. * Revert "[DLT-1110] Mapped Collection Endpoint Browse Tests (2/4) (#1207)" This reverts commit 22545fa. * added updated version t0 requirements.txt * Fix edge case in GridFTP Authz where '/' is used * cpp-py-formatter * Add changelog comment * Address prettier formatting * Add method for testing config, fix bug in global config, fix redundant log statements * Make error detection clearer in bool * Add mock core server * Add integration and, liveness test for mock * Address mock integration setup and consolidate * Apply formatting * Remove unused timeout variable from mock * Address JSDoc complaints * Apply prettier * Fix formatting and eslint prettier issues * Address codacy feedback * Add changelog comment * Clean up grammar in comment * Add note in Config.h * Apply clang formatting * Address codacy issues * Add env var * Update Arangodb packaged version to using 3.12.4 * Fix confusion in key uses in end to end tests * Add change log comment * Add changelog comment * corrected imported member * Add compose entries for foxx tests * Update the schema documentation to improve the usability. * Add example schema with a reference * Add notebook for linting jupyter * Using working directory option * Update .github/workflows/jupyternotebook-lint.yml Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Update python notebooks * Attempt to fix jupyternote book linting errors * Format with black * Specify line length with black to be consistent with flake8 * Add a Changelog comment * Change line length to 88 * Add black formatters * Make formatting consistent across project * Fix linter * Temporarily keep line length in black to 79 and address in separate issue * Change line length to 88 for now : * updated changelog * Fix bug in CI scripts associated with repo and gcs image build * Merge hotfixes into devel (#1326) * [DAPS-DEPS] Update cpp-py-formatter to 0.3.0 * [DAPS-DEPS] Update cpp-py-formatter to 0.3.1 * [DAPS-DEPS] Update ver, add push * updated versions for release * Update project.rst * Update format-check.yml * Pull in version numbers appropriately * Avoid build cache * Aggressive prune * Trigger rebuild * generate_repo_config.sh Accept env variable for datafed server from repo service instead of hard coding * Allow core config thread options to be read in from env * Add changelog comment * Adjust deployment version number * Fix Version month * Update CHANGELOG.md Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> --------- Co-authored-by: Aaron Perez <[email protected]> Co-authored-by: Blake Nedved <[email protected]> Co-authored-by: Joshua S Brown <[email protected]> Co-authored-by: JoshuaSBrown <[email protected]> Co-authored-by: Anthony Ramirez <[email protected]> Co-authored-by: nedvedba <[email protected]> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Feature DAPS 1215 foxx UserGetAccessTokenRequest mapped collection support (#1284) * Begin prototyping required code for mapped collection token retrieval. * SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens * user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments. * user_router.js Add logic for finding token based on collection; add token match and missing match cases * Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes. * user_router.js Add check for existence of Globus collection * user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code * datafed-ws.js Temporary handling of need consent response. * user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails * datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent * datafed-ws.js No redirect, show error directly in list. * datafed-ws.js Early return to prevent further requests * support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib * user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment. * user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods. * CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention. * CMakeLists.txt Fix variable get. user_token.js Fix export. * user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix. * user_router.js Formatting; Throw error if more than one token matches, add note about token type. * ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field. * user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values. * user_router.js Clean up some comments for clarity. datafed-ws.js Formatting. * ClientWorker.cpp Rewrite comment regarding mapped token refresh. * user_token.js Formatting. * CHANGELOG.md update for feature * user_router.js Re-introduce accidentally removed `globus_collection` object. * CMakeLists.txt Remove comment about necessity * Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture. * SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field. * user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent. * Formatting changes. --------- Co-authored-by: Anthony Ramirez <[email protected]> * [DLT-1110] Implement Consent Required Action (3/4 & 4/4) (#1242) * [DLT-1110] Refactor browse into component and use CSS * [DAPS-1110] Update * [DAPS-1110] Update web server * [DAPS-1110] add refactors * [DAPS-1110] Prettier * [DAPS-1110] Refactor tests * [DAPS-1110] Remove .. in ep ls, remove logs * [DAPS-1110] Add sinon, remove dead code * [DAPS-1110] Address transfer start button bug * [DAPS-1110] eslint, update api.test.js to reflect usage * [DAPS-1110] format * [DAPS-1110] Correct bug * JSON parse error fix (#1328) * DatabaseAPI.cpp Pull in json dependency, use json object to serialize request payload * Replace json serialization in more locations where parseSearchRequest is being called. * Add comments around locations that will need json serialization * Convert missed body append to serialization format * typo * Refactoring DatabaseAPI.cpp to incorporate json serialization * More serialization work * Pull in changes from 1214 for dataPut * Bring in changes from 1214 for dataGet * DatabaseAPI.cpp Fix error with missing curly brace. Some formatting that is likely wrong. Serialize up to original point. Need to validate. * Add missing scope resolution * More typos * More typos. Scoping * Remove non-existant vars * declare body * Formatting * Formatting and verification of updates * Replace most places * Formatting * Remove redeclaration * DatabaseAPI.cpp Remove some comments. * DatabaseAPI.cpp Add missing push_back * Adding braces * Prevent Double Escaping string, by using nlohmann json parse to read in value. * remove escapeJSON calls, redundant with nlohmann json dump * Allow parsing for metric data to be run in parallel for now, output messages when old serialization is used. * Ensure ASCII setting. * Upgrade debug to warning. --------- Co-authored-by: Anthony Ramirez <[email protected]> Co-authored-by: Austin Hampton <[email protected]> Co-authored-by: Joshua S Brown <[email protected]> * Feature - Mapped Collection Token Support (#1334) * Begin prototyping required code for mapped collection token retrieval. * SDMS_Auth.proto Add optional fields to UserGetAccessTokenRequest and UserAccessTokenReply to support finding correct tokens * user_router.js Add baseline query params and checks. DatabaseAPI.cpp & DatabaseAPI.hpp Use updated names and types. ClientWorker.cpp Update comments. * user_router.js Add logic for finding token based on collection; add token match and missing match cases * Return necessary data from Database to refresh tokens. Mock data on dir/list to test out changes. * user_router.js Add check for existence of Globus collection * user_router.js Remove existence check, move to filter method; Add notes on covered cases; add notes on code * datafed-ws.js Temporary handling of need consent response. * user_router.js Add queryParam validation for new params via joilib. ClientWorker.cpp Add refresh token try/catch to send through flow if refresh fails * datafed-ws.js Use correct converted casing for protobuf messages; redirect to error when needs_consent * datafed-ws.js No redirect, show error directly in list. * datafed-ws.js Early return to prevent further requests * support.js Update AccessTokenType enum. user_token.js New class introduced for handling logic for validating params when getting tokens and building GetAccessToken responses. user_router.js Replace logic with calls to new lib * user_token.js Add jsdoc definition for return type and format function. user_router.js Remove unnecessary comment. * user_token.js More jsdoc definition; Add validation for collection token document required fields. user_token.test.js Add preliminary unit tests for static methods. * CMakeLists.txt Add new unit tests to build. user_token.test.js Modify naming to match convention. * CMakeLists.txt Fix variable get. user_token.js Fix export. * user_fixture.js Add new user for get/token tests. user_router.test.js Add simple tests for get/token endpoint. user_token.js Fix naming for token document scopes field. user_token.test.js Changes for scope field bugfix. * user_router.js Formatting; Throw error if more than one token matches, add note about token type. * ClientWorker.cpp Address some TODOs and extraneous comments. DatabaseAPI.cpp Remove TODOs. SDMS_Auth.proto Remove unused field. * user_token.js Update formatUserToken to always return object values. DatabaseAPI.cpp Add note on possibly missing field. user_token.test.js Update tests to check for new expected values. * user_router.js Clean up some comments for clarity. datafed-ws.js Formatting. * ClientWorker.cpp Rewrite comment regarding mapped token refresh. * user_token.js Formatting. * CHANGELOG.md update for feature * user_router.js Re-introduce accidentally removed `globus_collection` object. * CMakeLists.txt Remove comment about necessity * Add collection_type and collection_id to dat/put task state * support.js Update getAccessToken method to work based on collection information, JSDoc. tasks.js Include context required in call to getAccessToken, add comment on refresh. TaskWorker.cpp Identify where changes will be reflected. * Update token lookup logic to use `byExample`, add test case for missing Globus collection. Add globus_coll fixture. * SDMS_Auth.proto Change UserAccessTokenReply needs_consent to optional field. * user_router.js Update token/get to default to start consent flow when collection tokens are present. user_router.test.js Update test from error to needs_consent. * Formatting changes. * SDMS_Auth.proto Add new fields to DataGetRequest * Begin refactor of token decision logic. * Address bugs in fetching and mapping data, make unit tests pass. * globus_token.js Match other models for get * tasks.js Pull in new logic for determining tokens, replace tokens; Needs to be verified. user_token.js Add new format function for compatibility with transfer doc replacement. * DatabaseAPI.cpp Sane serialization of json body for taskInitDataPut. TaskWorker.cpp Comments around work items. * DatabaseAPI.cpp Sane serialization of json body for taskInitDataGet. * ClientWorker.cpp Stub out required changes for consent flow when creating tasks * ClientWorker.cpp More thorough consent flow for procDataPutRequest, stub on needs_consent * tasks.js Throw error when initializing data put * tasks.js Format error throwing. ClientWorker.cpp comment out potentially unnecessary code. * user_token.js Add exists method to abstract check. tasks.js User exists method when checking token. * user_token.js JSDoc typing * tasks.js Add more required fields to taskRunDataPut for refresh in TaskWorker. * tasks.js Add collection_id to params in taskRunDataPut for refresh in TaskWorker. TaskWorker.cpp Refresh conditionals * DatabaseAPI.cpp Update merged devel changes to address some leftover code. * DatabaseAPI.cpp Add collection specification to get request. * DatabaseAPI.cpp whitespace. ClientWorker.cpp Clean up unused code. user_router.js Switch to checking token existence through standard API. * user_token.js Add comment about type return. data_router.js Do validations at router level for user collection token relationship. tasks.js Remove validation from logic level; make dataGet mirror dataPut; Updates to naming and params structure. * ClientWorker.cpp Revert to minimize diff. TaskWorker.cpp Remove unnecessary TODO; move set client for refresh in transfer tasks. * models/ JSDocs, formatting, make token fetches follow defined model. user_token.js JSDocs and typing. * user_token.js Cover case where collection does not exist. * Small changes for UI detection of collection type * fix epview call for UI changes, add list view change to react appropriately to needing consent according to current code * extract ep view logic to function for either path to utilize * small changes and notes for UI updates * UI search based on path, not ID. camelCasing * TaskWorker.cpp small fix to only enforce requirement of additional refresh fields when token is transfer token. * add params for dat/get on UI * task.js Fix bug where client is incorrectly referenced. * Add user model tests. Extract DataFedOAuthToken.js from user_token.js for reusability. * Add globus collection model tests. * Add globus token tests. Add fixtures to support globus token testing. Make all model properties static for iteration purposes. Utilize constructors for typing. * Add user token tests. Make non-existent user throw an error when building user tokens. * Change error to be not found * Formatting changes * Remove unused changes in support.js getAccessToken * Changelog update, update comments, remove log. --------- Co-authored-by: Anthony Ramirez <[email protected]> * Add small fix for storeCollectionId (#1341) * Add small fix for storeCollectionId * move todo comment --------- Co-authored-by: Anthony Ramirez <[email protected]> * Fix bug with playwright (#1343) * Fix bug with playwright install for end to end web test --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * [DAPS-1330] Address Download Extension Transfer (#1340) * [DAPS-1337] Refactor GET and Put mode * [DAPS-1337] Clear consent issue * [DAPS-1337] Correct bug * [DAPS-1337] Add Dir mode path logic update * Parse md json strings (#1347) * Parse md json strings * parse execQuery params * move to direct only * make params an object * remove console log --------- Co-authored-by: Anthony Ramirez <[email protected]> * [DAPS-1349] Provenance Capture Update Bug (#1350) * [DAPS-1349] Return logic to original check * [DAPS-1349] version * [Release] February 2025 (#1352) * updated versions for release * Trigger rebuild * Adjust deployment version number * Fix Version month * Update Version.cmake * removed leading zero --------- Co-authored-by: JoshuaSBrown <[email protected]> Co-authored-by: Joshua S Brown <[email protected]> * Add .venv/ to gitignore (#1335) * [DAPS-1388] Add GCP Support (#1392) * [DAPS-1388] Add endpoint model, Add endpoint model, unify seperate collection logic, endpoint validation check, Add endpoint query validations, next is token * [DAPS-1388] Add tests for model * [DAPS-1388] PR comments * Update web/static/components/transfer/transfer-endpoint-manager.js Co-authored-by: Joshua S Brown <[email protected]> --------- Co-authored-by: Joshua S Brown <[email protected]> * Add missing python module (#1404) * Set reasonable default that will help with filling in repo form for core t… (#1400) * Set reasonable default that will help filling in repo form for core to repo communication in compose file * Update scripts/compose_generate_env.sh Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Set appropriate default when building compose image --------- Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> * Add documentation describing how to test the foxx microservices unit … (#1393) * Add documentation describing how to test the foxx microservices unit tests * Update testing.rst * Note on database version, variable definition and usage when running arangodb version described, and steps to run unit test numbered. * Fix ci build (#1390) * updated requirements * updated requirements * updated protobuf submodule * update unit test ci version * updated protobuf version * updated setuptools version * fixed condition * changed where packages are installed * used python3.11 to create the virtual environment instead of latest * reverted protobuf version * testing specificying version * removed pyvenv in ci job for consistency * installed python3.11 in dependencies docker file * added missing -y flag * removed unification of install scripts * added missing python3.11 install * updated ci job to install python3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * changed installed python version to 3.9 since protobuf does not officially support 3.11 * missed version * updated includes for newer gcc versions * updated versions * updated nlohmann json * updated json schema version * changed library search path * hopefully fixed json dep issues * updating function signature to match newer version of base class * updated protobuf * updated protobuf and cmake * downgraded cmake * fixed protobuf version * fixed protobuf version * 1398 pin python version (#1405) * Pin python version * fixed incorrect path * Switch to using .tar file install of libsodium (#1414) * 1413 libsodium build refactor (#1415) * Switch to using .tar file install of libsodium * Switch bad option in wget command from -C to -P * Update scripts/dependency_install_functions.sh * Update scripts/dependency_install_functions.sh * Update dependency_install_functions.sh Libsodium folder version number. * Update dependency_install_functions.sh Make paths explicit. * cleaned up comments * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown <[email protected]> * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown <[email protected]> * Update scripts/dependency_install_functions.sh Co-authored-by: Joshua S Brown <[email protected]> * Update cmake/JSONSchema.cmake Co-authored-by: Joshua S Brown <[email protected]> * add changes from review * Changed install scripts to be consistent with new python installation method (#1417) * changed install scripts to be consistent * added sudo check * add apt sources check --------- Co-authored-by: Blake Nedved <[email protected]> Co-authored-by: nedvedba <[email protected]> * Fix regression (#1428) * Fix docker logs output in end-to-end tests (#1426) * Improve serialization randomization (#1427) * Increase serialization randomization pause heuristic - 6 seconds is not long enough to register running pipelines. * improve serialization * Upgrade formatter (#1440) * Update action for CPP and Python formatter * 1437 fix zmq inproc bind order (#1438) * Reorder zmq bind and connect calls to prevent undefined behavior when using INPROC * Make sure zmq messages are closed when exception is encountered * Make correction to socket name * Initialize cmd call * Add missing include * Allow looping over containers if there is more than one. (#1446) * 1445 docker logs multiple containers (#1447) * Allow looping over containers if there is more than one. * Fix multipline comment * Fix conflicting dependency install flags (#1444) * [DAPS-1408] Provenance Visual Management Lint (1/2) (#1431) * [DAPS-14xx] Move files over and move out svg * [DAPS-1408] Provenance Visual Management (2/2) (#1419) * [DAPS-14xx] Base * [DAPS-14xx] Add styling and tooltips * [DAPS-14xx] Add node and label customization * [DAPS-14xx] Add editor modal on right-click. Draggable model * [DAPS-14xx] Revert * [DAPS-14xx] Seperate files, update styles * [DAPS-14xx] simplify feat * [DAPS-14xx] Expand mocha tests, prettier * [DAPS-14xx] Update checkout version * [DAPS-14xx] Add todo * [DAPS-14xx] Update tests, consts, fix tooltip * [DAPS-14xx] Base: [DAPS-14xx] Address collapse, expand, hide * [DAPS-14xx] THEME, address comments * [DAPS-14xx] Zoom * [DAPS-14xx] JSDOC * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown <[email protected]> * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown <[email protected]> * Update web/static/jquery-ui-dark/datafed.css Co-authored-by: Joshua S Brown <[email protected]> --------- Co-authored-by: Joshua S Brown <[email protected]> * [DAPS-1395] Address Memory Leaks in Core Server (#1424) * Add missing virtual destructors * Fix ZeroMQ INPROC and TCP context * Add memory tests to CMakeLists.txt as an option * Add protobuf library shutdown to unit tests for memory cleanup * [TASK] Update tests and fixture (#1454) * [DAPS-1331] Collection ID Updated on restart (#1468) * [DAPS-1331] Modified the script to only delete and recreate guest collections when the base path actually changes * [DAPS-1331] Add base path logic * [DAPS-1331] Update guest collection file * [DAPS-1331] Fix base path caching logic in should_recreate_guest_collection function to recreate by default when no cached path is available * [DAPS-1331] Add tests * [DAPS-1331] Revert Gitlab CI * [DAPS-RELEASE] * [TASK] Remove icons from labels (#1489) --------- Co-authored-by: Anthony <[email protected]> Co-authored-by: Anthony Ramirez <[email protected]> Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com> Co-authored-by: Aaron Perez <[email protected]> Co-authored-by: AronPerez <[email protected]> Co-authored-by: Blake Nedved <[email protected]> Co-authored-by: par-hermes <[email protected]> Co-authored-by: nedvedba <[email protected]> Co-authored-by: Austin Hampton <[email protected]> Co-authored-by: Theo Beers <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Update installation scripts to consistently install Python using a new method across different dependency installation scripts
New Features:
install_python()function in dependency installation scriptsEnhancements:
Chores: