Jaal vs inference orchestration

A plain loop, checked by the compiler

Ray asks for .remote() futures, LangGraph for Send and edges, Triton for ensemble configs, KServe for YAML graphs. In Jaal you write a loop and mark it parallel; the compiler checks the iterations are independent and refuses the mark by name if not.

A model that has answered is never called again

Every model call is a recorded step: after a crash, or in a replay, the recorded answer stands in. Ray keeps no execution journal; LangGraph's time travel re-executes nodes from a checkpoint.

Every engine in its own process (being built)

vLLM, Triton, llama.cpp and plain CUDA as services of one orchestrator (being built; the CUDA core runs today on our CUDA simulator). A GPU server reached as its own endpoint cannot take the orchestrator down with it. The model servers are partners, not rivals.

Where others are ahead

Serving itself: NVIDIA Dynamo's disaggregated prefill and decode and KV-aware routing, Ray's scale-out, LangGraph's ecosystem. Jaal's inference orchestration is in progress; the four-engine demo is being built.

LangGraph: declare the graph

class State(TypedDict):
    photos: list
    notes: Annotated[list, operator.add]
    summary: str

def fan_out(state):
    return [Send("describe", {"photo": p})
            for p in state["photos"]]

def describe(item):
    return {"notes": [describe_photo(item["photo"])]}

def summarize(state):
    return {"summary": write_summary(state["notes"])}

builder = StateGraph(State)
builder.add_node("describe", describe)
builder.add_node("summarize", summarize)
builder.add_conditional_edges(START, fan_out, ["describe"])
builder.add_edge("describe", "summarize")
builder.add_edge("summarize", END)
graph = builder.compile()

Jaal: write the loop, mark it parallel, the graph is derived

import jaal.x.Parallel;

public class Album extends jaal.x.Service<Photos, Summary> {
    private jaal.x.Service<Photo, Note> describe;
    private jaal.x.Service<Notes, Summary> summarize;

    @Override public Summary invoke(Photos in) {
        int n = in.urls().length;
        String[] notes = new String[n];
        for (@Parallel int i = 0; i < n; i++) {
            Note note = describe.invoke(PhotoBuilder.photo().url(in.urls()[i]).build());
            notes[i] = note.text();
        }
        return summarize.invoke(NotesBuilder.notes().texts(notes).build());
    }
}

Every call recorded, a crash finished by another process, the graph drawn for you. Or ask the assistant for it in a sentence (in progress).