Skip to content

Doc for using raw container in the map task #984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Doc for using raw container in the map task #984

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
66fcd8f
fixed test
pingsutw Apr 9, 2023
4612435
Doc for using raw container in the map task
pingsutw Apr 10, 2023
e921280
nit
pingsutw Apr 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions cookbook/core/containerization/raw_container.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
Refer to the raw protocol to understand how to leverage this.
"""
import logging
import sys

from flytekit import ContainerTask, kwtypes, task, workflow

Expand Down Expand Up @@ -193,3 +194,25 @@ def wf(a: float, b: float):
# ^^^^^^^^^^^^^^^^^^^^^^^^^
# .. literalinclude:: ../../../../core/containerization/raw-containers-supporting-files/per-language/julia/calculate-ellipse-area.jl
# :language: julia

# %%
# Use Raw Container in the map task
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Use Raw Container in the map task
# Use Raw Container in map tasks

# =================================
#
# To use raw container in the map task, users have to
# 1. Manually parse the input data. The input is string representation of a list, so we need to convert it to python list.
# 2. Get array index (BATCH_JOB_ARRAY_INDEX_VAR_NAME), and use it to get the element in the list.
Comment on lines +202 to +204
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# To use raw container in the map task, users have to
# 1. Manually parse the input data. The input is string representation of a list, so we need to convert it to python list.
# 2. Get array index (BATCH_JOB_ARRAY_INDEX_VAR_NAME), and use it to get the element in the list.
# To use raw container in the map task, users have to:
#
# 1. Manually parse the input data. The input is string representation of a list, so we need to convert it to python list.
# 2. Get array index ``(BATCH_JOB_ARRAY_INDEX_VAR_NAME)``, and use it to get the element in the list.

def main(a: str, output_dir):
# convert a to list
res = eval(a)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is eval(a)? is this a [] literal? could we get an example of it? is there any other way of getting the list object like loading it through json.loads or something?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah maybe we don't use eval in our examples... or at least json.loads

# get the job index
index = int(os.environ.get(os.environ.get("BATCH_JOB_ARRAY_INDEX_VAR_NAME")))
...


if __name__ == "__main__":
a = sys.argv[1]
b = sys.argv[2]
output_dir = sys.argv[3]

main(a, output_dir)