Scores - Nelkap

A run can complete and still do a poor job: an AI agent answers, but off the point, or a support flow ends with a human taking over. Status alone can’t tell you that. A score can. Record a number or a yes or no from your workflow, and Nelkap charts it over time, so you see right away whether a change made your runs better or worse.

Record a score

Call step.score with a name and a value:

export async function answerTicket(ticketId: string) {
  const reply = await nelkap.step.run('reply', () =>
    draftReply(ticketId),
  );

  await nelkap.step.score(
    'resolved without a human',
    reply.resolved,
  );
  await nelkap.step.score(
    'confidence',
    reply.confidence,
  );
}

A number, like a relevance from 0 to 1 or a rating from 1 to 5, is charted as an average.
A yes or no, like whether a ticket was solved, is charted as a pass rate: the share of runs that recorded true.

The name is up to 128 characters. Runs that record the same name share a chart, so use the same name across workflows to compare them. A score is saved like a step. When the run picks up later, the score is already recorded and isn’t counted twice.

Score sources

Score what you can measure at that point of the run:

A value your own code computes, like whether an order shipped on time.
A judgment from a model, like the relevance of an answer graded by a second model call, made in a step so it’s saved.
A person’s feedback, received with step.wait, like a thumbs up on a support reply.

Charts

Each score name, with its kind of value, gets its own chart, for the period and project you choose: from the last hour, minute by minute, to the last 30 days, day by day. A drop the day after a deploy points to that change. Each score also shows on its run’s page, next to the step that recorded it.

⌘I