Loops
When writing Workflows, you may want to reuse a single template over a set of inputs. Argo exposes two mechanisms for this looping, which are “with items” and “with param”. These mechanisms function in exactly the same way, but as the name suggests, “with param” lets you use a parameter to loop over, while “with items” is generally for a hard-coded list of items. When using loops in Argo, the template will run in parallel for all the items; the items will be launched sequentially but the running times may overlap. If you do not want to loop over the items in parallel, you should use a Synchronization mechanism; see the Sequential Steps example.
Loops in Hera
In pure Argo YAML specification, withItems
and withParam
are single values or JSON objects. In Hera, we can pass any
Parameter
or serializable object, plus, with_items
and with_params
work exactly the same for hard-coded values.
A Simple with_items
Example
Consider the Hello World example:
@script()
def echo(message: str):
print(message)
with Workflow(
generate_name="hello-world-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
echo(arguments={"message": "Hello world!"})
We can easily convert this to call echo
for multiple strings; the only changes we need to make are in the function
call. First, specify the list of items you want to echo in the with_items
kwarg:
echo(
arguments={"message": "Hello world!"},
with_items=["Hello world!", "I'm looping!", "Goodbye world!"],
)
Now, we need to replace the value of the message
argument. In Argo, you would use the {{item}}
expression syntax,
which is also what we use in Hera (within a string):
echo(
arguments={"message": "{{item}}"},
with_items=["Hello world!", "I'm looping!", "Goodbye world!"],
)
When running this Workflow, each value of with_items
is passed to the {{item}}
expression and runs in an independent
instance of the echo
script container. Your Workflow logs should look something like:
hello-world-9cf9j-echo-3186990983: Hello world!
hello-world-9cf9j-echo-4182774221: I'm looping!
hello-world-9cf9j-echo-1812072106: Goodbye world!
with_items
Using a Dictionary
We mentioned we can use any serializable object, so let’s see how dictionaries are handled.
The {{item}}
syntax represents the whole “item” passed in to the argument, so in the “hello world” example above, that
translates to just a string. For dictionaries, this {{item}}
would translate to the whole dictionary. If instead we
want to pass values from the item dictionary to the function arguments, we provide them with Argo’s key access syntax:
{{item.key}}
.
Let’s create a workflow to process everyone’s favorite bubble tea orders!
First, a function that takes the customer’s name, the drink flavor, ice level and sugar level:
@script()
def make_bubble_tea(
name: str,
flavor: str,
ice_level: float,
sugar_level: float,
):
print(
f"Making {name}'s {flavor} bubble tea with {ice_level:.0%} ice and {sugar_level:.0%} sugar."
)
Now, a Workflow with a Steps
context:
with Workflow(
generate_name="make-drinks-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
make_bubble_tea(...)
And now for each argument of make_bubble_tea
, we can let Hera infer from the values in with_item
! We just need to
pass a list of dictionaries, with the keys matching the make_bubble_tea
arguments: “name”, “flavor”, “ice_level” and
“sugar_level”:
with Workflow(
generate_name="make-drinks-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
make_bubble_tea(
with_items=[
{
"name": "Elliot",
"flavor": "Taro Milk Tea",
"ice_level": 0.25,
"sugar_level": 0.75,
},
{
"name": "Flaviu",
"flavor": "Brown Sugar Milk Tea",
"ice_level": 1.00,
"sugar_level": 0.5,
},
{
"name": "Sambhav",
"flavor": "Green Tea",
"ice_level": 0.5,
"sugar_level": 0.25,
},
],
)
Running this Workflow, in the UI we’ll see a fanout of three nodes, and the logs for “All” containers will show:
make-drinks-h2qgq-make-bubble-tea-3759662853: Making Elliot's Taro Milk Tea bubble tea with 25% ice and 75% sugar.
make-drinks-h2qgq-make-bubble-tea-470512305: Making Sambhav's Green Tea bubble tea with 50% ice and 25% sugar.
make-drinks-h2qgq-make-bubble-tea-615962639: Making Flaviu's Brown Sugar Milk Tea bubble tea with 100% ice and 50% sugar.
Remember in the above example, we could swap out with_item
for with_param
and get the same output. with_param
is
useful for passing dynamically generated lists and fanning out to process the list which we’ll learn in the next section.
Dynamic Fanout Using with_param
Let’s improve our bubble tea maker by generating a dynamic list of orders! To do this, we’ll need a new create_orders
function. We’re going to make use of the script’s result
output parameter by dumping the orders to stdout.
Let’s make our order randomizer:
@script()
def create_orders():
import json
import random
names = ["Elliot", "Flaviu", "Sambhav"]
flavors = ["Brown Sugar Milk Tea", "Green Tea", "Taro Milk Tea"]
levels = [0, 0.25, 0.5, 0.75, 1.0]
orders = []
for _ in range(random.randint(4, 7)):
orders.append(
{
"name": random.choice(names),
"flavor": random.choice(flavors),
"ice_level": random.choice(levels),
"sugar_level": random.choice(levels),
}
)
print(json.dumps(orders, indent=4)) # indent is just used here for nice human-readable logs
Note: we must import any modules used within the function itself, as Hera currently only passes the source lines of the function to Argo. If you need to import modules not in the standard Python image, use a custom image as described in the
script
decorator section, or see the experimental callable script example.
Now we can construct a Workflow that calls create_orders
, and passes its result
to make_bubble_tea
. We’ll need to
hold onto the Step
returned from the create_orders
call, and change with_items
to with_param
to use .result
.
We’ll keep the arguments
the same, as the .result
will be a json-encoded list of dictionaries!
with Workflow(
generate_name="make-drinks-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
orders = create_orders()
make_bubble_tea(with_param=orders.result)
Click to expand for logs. A Workflow run will look something like this. Remember, it's all random!
make-drinks-t49mm-create-orders-628494701: [
make-drinks-t49mm-create-orders-628494701: {
make-drinks-t49mm-create-orders-628494701: "name": "Flaviu",
make-drinks-t49mm-create-orders-628494701: "flavor": "Brown Sugar Milk Tea",
make-drinks-t49mm-create-orders-628494701: "ice_level": 1.0,
make-drinks-t49mm-create-orders-628494701: "sugar_level": 1.0
make-drinks-t49mm-create-orders-628494701: },
make-drinks-t49mm-create-orders-628494701: {
make-drinks-t49mm-create-orders-628494701: "name": "Elliot",
make-drinks-t49mm-create-orders-628494701: "flavor": "Green Tea",
make-drinks-t49mm-create-orders-628494701: "ice_level": 0,
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.5
make-drinks-t49mm-create-orders-628494701: },
make-drinks-t49mm-create-orders-628494701: {
make-drinks-t49mm-create-orders-628494701: "name": "Sambhav",
make-drinks-t49mm-create-orders-628494701: "flavor": "Green Tea",
make-drinks-t49mm-create-orders-628494701: "ice_level": 0,
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.25
make-drinks-t49mm-create-orders-628494701: },
make-drinks-t49mm-create-orders-628494701: {
make-drinks-t49mm-create-orders-628494701: "name": "Flaviu",
make-drinks-t49mm-create-orders-628494701: "flavor": "Taro Milk Tea",
make-drinks-t49mm-create-orders-628494701: "ice_level": 0.5,
make-drinks-t49mm-create-orders-628494701: "sugar_level": 0.5
make-drinks-t49mm-create-orders-628494701: }
make-drinks-t49mm-create-orders-628494701: ]
make-drinks-t49mm-make-bubble-tea-3020754075: Making Flaviu's Brown Sugar Milk Tea bubble tea with 100% ice and 100% sugar.
make-drinks-t49mm-make-bubble-tea-2627605331: Making Elliot's Green Tea bubble tea with 0% ice and 50% sugar.
make-drinks-t49mm-make-bubble-tea-3584623812: Making Sambhav's Green Tea bubble tea with 0% ice and 25% sugar.
make-drinks-t49mm-make-bubble-tea-1040507004: Making Flaviu's Taro Milk Tea bubble tea with 50% ice and 50% sugar.
Aggregating Fan Out Results (Fan In)
Okay, we’ve made all these drinks, now we need to serve them up together!
For this, we can again use the result
output parameter, but as we will use it on the make_bubble_tea
step, it
expects JSON objects to aggregate them together.
Let’s edit our make_bubble_tea
function to dump a JSON object:
@script()
def make_bubble_tea(
name: str,
flavor: str,
ice_level: float,
sugar_level: float,
):
import json
print(json.dumps({"name": name, "status": "Completed"}))
And now let’s write a function to call out “Serving N orders” and the names attached to the orders:
@script()
def serve_orders(orders: List[Dict[str, str]]):
names = list(set([order["name"] for order in orders]))
print(f"Serving {len(orders)} orders for {', '.join(names[:-1])} and {names[-1]}!")
In our Workflow, we can now link these scripts together with each Step’s result
:
with Workflow(
generate_name="make-drinks-",
entrypoint="steps",
) as w:
with Steps(name="steps"):
orders = create_orders()
teas = make_bubble_tea(with_param=orders.result)
serve_orders(arguments={"orders": teas.result})
The logs will look something like this (click to expand).
make-drinks-xk4hm-create-orders-2830038274: [
make-drinks-xk4hm-create-orders-2830038274: {
make-drinks-xk4hm-create-orders-2830038274: "name": "Elliot",
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.5
make-drinks-xk4hm-create-orders-2830038274: },
make-drinks-xk4hm-create-orders-2830038274: {
make-drinks-xk4hm-create-orders-2830038274: "name": "Elliot",
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Taro Milk Tea",
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.5,
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 1.0
make-drinks-xk4hm-create-orders-2830038274: },
make-drinks-xk4hm-create-orders-2830038274: {
make-drinks-xk4hm-create-orders-2830038274: "name": "Sambhav",
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 1.0
make-drinks-xk4hm-create-orders-2830038274: },
make-drinks-xk4hm-create-orders-2830038274: {
make-drinks-xk4hm-create-orders-2830038274: "name": "Sambhav",
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Taro Milk Tea",
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.5,
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.75
make-drinks-xk4hm-create-orders-2830038274: },
make-drinks-xk4hm-create-orders-2830038274: {
make-drinks-xk4hm-create-orders-2830038274: "name": "Flaviu",
make-drinks-xk4hm-create-orders-2830038274: "flavor": "Green Tea",
make-drinks-xk4hm-create-orders-2830038274: "ice_level": 0.25,
make-drinks-xk4hm-create-orders-2830038274: "sugar_level": 0.75
make-drinks-xk4hm-create-orders-2830038274: }
make-drinks-xk4hm-create-orders-2830038274: ]
make-drinks-xk4hm-make-bubble-tea-2143417526: {"name": "Elliot", "status": "Completed"}
make-drinks-xk4hm-make-bubble-tea-2058639815: {"name": "Elliot", "status": "Completed"}
make-drinks-xk4hm-make-bubble-tea-316598325: {"name": "Sambhav", "status": "Completed"}
make-drinks-xk4hm-make-bubble-tea-4190830807: {"name": "Sambhav", "status": "Completed"}
make-drinks-xk4hm-make-bubble-tea-3301714217: {"name": "Flaviu", "status": "Completed"}
make-drinks-xk4hm-serve-orders-974975305: Serving 5 orders for Sambhav, Elliot and Flaviu!