Scoring

Functions for evaluating ALHENA annotations when independent ground-truth labels are available.

align_truth() puts known labels in the order of the annotated cells, score() returns classification metrics with both baselines, chance_floor() computes the commonest-class baseline from the known labels alone, and confidence_sweep() evaluates performance across confidence thresholds. score() and confidence_sweep() take the AnnotationResult itself.

Tutorial 6 shows them on an example.

alhena.metrics.align_truth

align_truth(result, known)

Put known labels in the order of the annotated cells, matching them by cell identifier.

Identifiers are matched as they are written, never converted, so an integer and its text do not meet. Every annotated cell needs a known label. A table holds TRUTH_COLUMNS, a Series is indexed by identifier, and a mapping goes from identifier to label.

Parameters:

Name	Type	Description	Default
result	AnnotationResult	Annotations whose cells order the labels.	required
known	KnownLabels	The known labels: a table, a Series, a mapping, or the path of a .csv or .parquet table.	required

Returns:

Type	Description
ndarray	One known label per annotated cell, in the order of the result, as score and
ndarray	confidence_sweep take them.

Raises:

Type	Description
AlhenaValueError	If the labels cannot be read, or do not name each cell once.

alhena.metrics.score

score(predicted, truth, class_order=None, macro_min_support=0)

Score predicted cell types against known ones.

Cells without a prediction stay in the population as errors and enter no column of the confusion matrix. The macro average runs over the classes the population carries, less those under macro_min_support, which moves the denominator and removes no cell. Both baselines are scored on the same cells under the same floor.

Parameters:

Name	Type	Description	Default
predicted	AnnotationResult | LabelInput	The annotations, which carry their class order; or predicted class names or codes indexing class_order, "" or -1 marking a cell without a prediction.	required
truth	LabelInput	Known class names, or codes indexing class_order, one per cell, as align_truth orders them. Every entry must be a class of the axis.	required
class_order	Sequence[str] | None	Classes of the axis, in the order that fixes their codes and the rows and columns of the confusion matrix. Read from annotations when None.	None
macro_min_support	int	Cells a class needs in this population to enter the macro average; zero keeps every class it carries.	0

Returns:

Type	Description
ScoreReport	The agreement between the two label sets, with the commonest-class and frequency-matched
ScoreReport	baselines of the same population.

Raises:

Type	Description
AlhenaValueError	If the class order or the labels are malformed or do not match.

alhena.metrics.chance_floor

chance_floor(truth, class_order, macro_min_support=0)

Score the commonest-class baseline, which gives every cell the commonest class.

Read against a headline from score on the same cells, it takes the same macro_min_support. Under a macro average it scores zero on every class but the one it predicts.

Parameters:

Name	Type	Description	Default
truth	LabelInput	Known class names, or codes indexing class_order.	required
class_order	Sequence[str]	Classes of the axis, in the order that fixes their codes. A tie between two equally common classes resolves to the lower code.	required
macro_min_support	int	Cells a class must carry in this population to enter the macro average, matching the value the headline was scored with.	0

Returns:

Type	Description
ScoreReport	The baseline's own report, whose floor_macro_f1 and floor_accuracy are its scores.

Raises:

Type	Description
AlhenaValueError	If the class order is malformed, or a truth label is not of the axis.

alhena.metrics.confidence_sweep

confidence_sweep(result, truth, thresholds=DEFAULT_CONFIDENCE_THRESHOLDS)

Score the annotations that survive each of a grid of confidence thresholds.

A cell is retained when it is annotated and its confidence reaches the threshold; each row is score on the retained cells, pooled, so no support floor applies. A class whose errors are confident loses recall as the threshold rises.

Parameters:

Name	Type	Description	Default
result	AnnotationResult	Annotations to sweep, supplying the class order, the labels, the confidence and the per-cell validity.	required
truth	LabelInput	Known class names, or codes indexing the result's class order, one per cell.	required
thresholds	Iterable[float]	Confidence thresholds, reported in the order given.	DEFAULT_CONFIDENCE_THRESHOLDS

Returns:

Type	Description
DataFrame	One row per threshold: the retained count and fraction, accuracy, macro F1, balanced
DataFrame	accuracy, the realized macro denominator, both baselines of the retained cells, and the
DataFrame	support and recall of every class. attrs["missing_classes"] names the classes the
DataFrame	reference held no cell of, as the annotations record them.

Raises:

Type	Description
AlhenaValueError	If the grid is empty or not finite, or the labels do not match the cells.

alhena.metrics.NO_CALL_LABEL module-attribute

NO_CALL_LABEL = ''

Label a cell without a prediction carries.

alhena.metrics.DEFAULT_CONFIDENCE_THRESHOLDS module-attribute

DEFAULT_CONFIDENCE_THRESHOLDS = (0.0, *(round(0.3 + 0.05 * step, 2) for step in range(14)))

Default grid of confidence_sweep; its first entry retains every annotated cell.

alhena.metrics.LabelInput module-attribute

LabelInput = 'Sequence[str] | Sequence[int] | np.ndarray'

Labels as class names or as codes indexing the class order.

alhena.metrics.KnownLabels module-attribute

KnownLabels = 'pd.DataFrame | pd.Series | Mapping[Any, Any] | str | Path'

Known labels by cell identifier: a table, a Series, a mapping, or the path of a table.

alhena.metrics.TRUTH_COLUMNS module-attribute

TRUTH_COLUMNS = ('cell_id', 'label')

Columns a table of known labels must carry.

---

Next: SpatialData