Skip to content

Conversation

@SimonHeybrock
Copy link
Member

@SimonHeybrock SimonHeybrock commented Mar 12, 2024

What is this?

See scipp/sciline#141 for background / context.

What can we do with it?

In Sciline:

Reviewing

Do not review the implementation yet, as I am just doing a complete refactor, feel free to play with it though!

Try it out

Get Sciline with the cyclebane branch.

Define a workflow:

import sciline
from typing import NewType

Bank = NewType("Bank", str)
Counts = NewType("Counts", int)
Result = NewType("Result", float)
Param = NewType("Param", float)
Normalized = NewType("Normalized", float)


def load(bank: Bank) -> Counts:
    return Counts(123 if bank == "bank1" else 456)


def process(counts: Counts, param: Param) -> Result:
    return Result(counts * param)


def normalize(result: Result) -> Normalized:
    return Normalized(result / 10)


pl = sciline.Pipeline((load, process, normalize))
pl[Param] = Param(0.5)
pl[Bank] = Bank("bank1")
graph = pl.get(Normalized)
graph.visualize(graph_attr={"rankdir": "LR"})

Use CycleBane.
!!! This is not the proposed API, there will be wrappers in Sciline, please try to read between the lines. This is just for showing how it will work. !!!

import cyclebane as cb
import pandas as pd

df = pd.DataFrame({Bank: ["bank1", "bank2"]}).rename_axis(index="bank")
params = pd.DataFrame({Param: [0.5, 0.7, 0.9]}).rename_axis(index="param")


def mean(*args):
    return sum(args) / len(args)


transformed = (
    cb.Graph(graph.to_networkx())
    .map(df)
    .map(params)
    .reduce(Normalized, name='Normalized', attrs={'reduce': mean}, index='bank')
    .reduce('Normalized', name='Normalized2', attrs={'reduce': mean}, index='param')
)
transformed
graph2 = sciline.task_graph.TaskGraph.from_networkx(transformed.graph)
graph2.visualize(graph_attr={"rankdir": "LR"})
graph2.compute('Normalized2')

@SimonHeybrock SimonHeybrock marked this pull request as draft March 13, 2024 07:37
@SimonHeybrock
Copy link
Member Author

I will merge this as there is little point in a review at this point. A lot will probably be refactored, we are still experimenting. I will open a PR for review purposes some time in the future.

@SimonHeybrock SimonHeybrock marked this pull request as ready for review March 21, 2024 09:17
@SimonHeybrock SimonHeybrock merged commit 1099b1d into main Mar 21, 2024
@SimonHeybrock SimonHeybrock deleted the map-reduce branch March 21, 2024 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants