Skip to content

Commit 789ee83

Browse files
myxiemyxie
authored
Merge pull request #311 from ICRAR/DLG_PyPI_Deployment
Add workflow to deploy DALiuGE to PyPI on Release
2 parents 989f07e + b64f7cc commit 789ee83

20 files changed

+925
-48
lines changed

‎.github/workflows/publish_to_pypi.yml

+163
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
name: Publish Python distribution to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
build-common:
9+
name: Build distribution 📦
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- uses: actions/checkout@v4
14+
with:
15+
persist-credentials: false
16+
- name: Set up Python
17+
uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.10"
20+
- name: Install pypa/build
21+
run: >-
22+
python3 -m
23+
pip install
24+
build
25+
--user
26+
- name: Build a binary wheel and a source tarball
27+
run: |
28+
cd daliuge-common
29+
python3 -m build --sdist && python -m build --wheel
30+
- name: Store the distribution packages
31+
uses: actions/upload-artifact@v4
32+
with:
33+
name: daliuge-common-dist
34+
path: daliuge-common/dist/
35+
36+
build-engine:
37+
name: Build daluge-engine
38+
runs-on: ubuntu-latest
39+
40+
steps:
41+
- uses: actions/checkout@v4
42+
with:
43+
persist-credentials: false
44+
- name: Set up Python
45+
uses: actions/setup-python@v5
46+
with:
47+
python-version: "3.10"
48+
- name: Install pypa/build
49+
run: >-
50+
python3 -m
51+
pip install
52+
build
53+
--user
54+
- name: Build a binary wheel and a source tarball
55+
run: |
56+
cd daliuge-engine
57+
python3 -m build --sdist && python -m build --wheel
58+
- name: Store the distribution packages
59+
uses: actions/upload-artifact@v4
60+
with:
61+
name: daliuge-engine-dist
62+
path: daliuge-engine/dist/
63+
64+
build-translator:
65+
name: Build daluge-translator
66+
runs-on: ubuntu-latest
67+
68+
steps:
69+
- uses: actions/checkout@v4
70+
with:
71+
persist-credentials: false
72+
- name: Set up Python
73+
uses: actions/setup-python@v5
74+
with:
75+
python-version: "3.10"
76+
- name: Install pypa/build
77+
run: >-
78+
python3 -m
79+
pip install
80+
build
81+
--user
82+
- name: Build a binary wheel and a source tarball
83+
run: |
84+
cd daliuge-translator
85+
python3 -m build --sdist && python -m build --wheel
86+
- name: Store the distribution packages
87+
uses: actions/upload-artifact@v4
88+
with:
89+
name: daliuge-translator-dist
90+
path: daliuge-translator/dist/
91+
92+
publish-common:
93+
name: Publish daliuge-common
94+
needs:
95+
- build-common
96+
runs-on: ubuntu-latest
97+
98+
environment:
99+
name: pypi
100+
url: https://pypi.org/p/daliuge-common
101+
102+
permissions:
103+
id-token: write # IMPORTANT: mandatory for trusted publishing
104+
105+
steps:
106+
- name: Download all the dists
107+
uses: actions/download-artifact@v4
108+
with:
109+
name: daliuge-common-dist
110+
path: dist/
111+
- name: Publish distribution 📦 to TestPyPI
112+
uses: pypa/gh-action-pypi-publish@release/v1
113+
with:
114+
verbose: true
115+
116+
publish-engine:
117+
name: Publish daliuge-engine
118+
needs:
119+
- build-engine
120+
runs-on: ubuntu-latest
121+
122+
environment:
123+
name: pypi
124+
url: https://pypi.org/p/daliuge-engine
125+
126+
permissions:
127+
id-token: write # IMPORTANT: mandatory for trusted publishing
128+
129+
steps:
130+
- name: Download all the dists
131+
uses: actions/download-artifact@v4
132+
with:
133+
name: daliuge-engine-dist
134+
path: dist/
135+
- name: Publish distribution 📦 to TestPyPI
136+
uses: pypa/gh-action-pypi-publish@release/v1
137+
with:
138+
verbose: true
139+
140+
publish-translator:
141+
name: Publish daliuge-translator
142+
needs:
143+
- build-translator
144+
runs-on: ubuntu-latest
145+
146+
environment:
147+
name: pypi
148+
url: https://pypi.org/p/daliuge-translator
149+
150+
permissions:
151+
id-token: write # IMPORTANT: mandatory for trusted publishing
152+
153+
steps:
154+
- name: Download all the dists
155+
uses: actions/download-artifact@v4
156+
with:
157+
name: daliuge-translator-dist
158+
path: dist/
159+
- name: Publish distribution 📦 to TestPyPI
160+
uses: pypa/gh-action-pypi-publish@release/v1
161+
with:
162+
verbose: true
163+

‎.github/workflows/run-unit-tests.yml

-1
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,6 @@ jobs:
6565
COVFILES=" daliuge-translator/.coverage"
6666
echo "COVFILES=$COVFILES" >> $GITHUB_ENV
6767
cd daliuge-translator
68-
pip install -r test-requirements.txt
6968
py.test --cov --show-capture=no
7069
7170
- name: Run daliuge-engine tests

‎.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,6 @@ daliuge.iml
2929
*.xml
3030
__pycache__
3131
.DS_Store
32-
VERSION
3332
DALIUGE
3433
docs/DALIUGE
3534
lalo

‎Makefile

+3-1
Original file line numberDiff line numberDiff line change
@@ -89,8 +89,10 @@ release: ## Create a new tag for release.
8989
@read -p "Version? (provide the next x.y.z semver) : " TAG
9090
@if ! grep -q "v$${TAG}" CHANGELOG.md; then echo "TAG version number must be added to CHANGELOG.md before committing." && exit; fi
9191
@echo "v$${TAG}" > daliuge-common/VERSION
92+
@echo "v$${TAG}" > daliuge-engine/VERSION
93+
@echo "v$${TAG}" > daliuge-translator/VERSION
9294
@git add daliuge-common/VERSION CHANGELOG.md
93-
@git commit -m "release: version v$${TAG} 🚀"
95+
@git commit -m "Release: version v$${TAG} 🚀"
9496
@echo "creating git tag : v$${TAG}"
9597
@git tag v$${TAG}
9698
@git push -u origin HEAD --tags

‎daliuge-common/VERSION

+1-1
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
v4.8.0
1+
v4.8.1

‎daliuge-common/dlg/version_helper.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ def get_git_version():
3030

3131
def get_version_info(version, is_release):
3232
git_version = "Unknown"
33-
if os.path.exists(".git"):
33+
if os.path.exists("../.git"):
3434
git_version = get_git_version()
3535
full_version = version
3636
if not is_release:

‎daliuge-common/setup.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ def do_versioning():
6464

6565
install_requires = [
6666
"gputil>=1.4.0",
67-
"merklelib@git+https://github.com/pritchardn/merklelib",
67+
"np-merklelib",
6868
"pyzmq==25.1.1",
6969
"pydantic>=2.5",
7070
"boto3",
@@ -106,6 +106,7 @@ def do_versioning():
106106
],
107107
license="LGPLv2+",
108108
packages=find_packages(),
109+
package_data={"":["VERSION"]},
109110
test_suite="test",
110111
entry_points={
111112
"console_scripts": ["dlg=dlg.common.tool:run"]

‎daliuge-engine/VERSION

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
v4.8.1

‎daliuge-engine/dlg/apps/pyfunc.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,6 @@
3131
import logging
3232
import os
3333
import pickle
34-
import pyext
3534
import re
3635

3736
from typing import Callable
@@ -59,6 +58,7 @@
5958
dlg_batch_output,
6059
dlg_streaming_input,
6160
)
61+
from dlg.pyext import pyext
6262

6363
logger = logging.getLogger(__name__)
6464

‎daliuge-engine/dlg/named_port_utils.py

+11-2
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
import dlg.drop_loaders as drop_loaders
77
from typing import Tuple
88

9+
from dlg.data.drops import DataDROP
10+
911
logger = logging.getLogger(__name__)
1012

1113

@@ -398,15 +400,22 @@ def get_port_reader_function(input_parser: DropParser):
398400
if input_parser is DropParser.PICKLE:
399401
# all_contents = lambda x: pickle.loads(droputils.allDropContents(x))
400402
reader = drop_loaders.load_pickle
401-
elif input_parser is DropParser.EVAL or input_parser is DropParser.UTF8:
403+
elif input_parser is DropParser.EVAL:
402404

403405
def optionalEval(x):
404406
# Null and Empty Drops will return an empty byte string
405407
# which should propogate back to None
406408
content: str = droputils.allDropContents(x).decode("utf-8")
407409
return ast.literal_eval(content) if len(content) > 0 else None
408-
409410
reader = optionalEval
411+
elif input_parser is DropParser.UTF8:
412+
def utf8decode(drop: "DataDROP"):
413+
"""
414+
Decode utf8
415+
Not stored in drop_loaders to avoid cyclic imports
416+
"""
417+
return droputils.allDropContents(drop).decode("utf-8")
418+
reader = utf8decode
410419
elif input_parser is DropParser.NPY:
411420
reader = drop_loaders.load_npy
412421
elif input_parser is DropParser.PATH:

‎daliuge-engine/dlg/pyext/LICENSE

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
Copyright (c) 2015 Ryan Gonzalez
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of
4+
this software and associated documentation files (the "Software"), to deal in
5+
the Software without restriction, including without limitation the rights to
6+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
7+
the Software, and to permit persons to whom the Software is furnished to do so,
8+
subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in all
11+
copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
15+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
16+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
17+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
18+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎daliuge-engine/dlg/pyext/__init__.py

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#
2+
# ICRAR - International Centre for Radio Astronomy Research
3+
# (c) UWA - The University of Western Australia, 2020
4+
# Copyright by UWA (in the framework of the ICRAR)
5+
# All rights reserved
6+
#
7+
# This library is free software; you can redistribute it and/or
8+
# modify it under the terms of the GNU Lesser General Public
9+
# License as published by the Free Software Foundation; either
10+
# version 2.1 of the License, or (at your option) any later version.
11+
#
12+
# This library is distributed in the hope that it will be useful,
13+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
15+
# Lesser General Public License for more details.
16+
#
17+
# You should have received a copy of the GNU Lesser General Public
18+
# License along with this library; if not, write to the Free Software
19+
# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
20+
# MA 02111-1307 USA
21+
#
22+
23+
# Declaring this as a namespace package
24+
__path__ = __import__("pkgutil").extend_path(__path__, __name__) # @ReservedAssignment
25+
# set the version
26+
try:
27+
from dlg.common import version
28+
29+
__version__ = version.full_version
30+
except:
31+
# This can happen when running from source
32+
__version__ = "unknown"

0 commit comments

Comments
 (0)