2. Cell embeddings¶
This tutorial computes image-derived embeddings for individual cells.
This tutorial teaches
- Task: compute cell-level representations
- Required inputs: an H&E image source and nuclei
- Output:
CellEmbeddings- Example used here: the slide from Tutorial 1 and nucleus polygons in the same pixel coordinate system
- Key concept: pooling image tokens over each nucleus
image source + nuclei → encode_cells() → CellEmbeddings
ALHENA accepts several nucleus representations, including polygons, label maps, masks, and centroids. See Nuclei and Coordinates.
ALHENA does not perform nucleus segmentation. If no segmentation is available yet, see If you have no nuclei yet.
Example outputs. The outputs shown in this tutorial come from the slide and the region of Tutorial 1, the H&E image of the 10x Genomics FFPE Human Breast with Custom Add-on Panel dataset. In this example,
nuclei.geojsoncontains the nucleus outlines associated with the H&E image.Source: FFPE Human Breast with Custom Add-on Panel, 10x Genomics, CC BY 4.0.
from pathlib import Path
import numpy as np
import alhena
from alhena import io
# Optional model configuration, as in Tutorial 1.
BACKBONE = None
BUNDLE = None
DEVICE = None
# H&E slide and nucleus polygons in full-resolution slide pixels.
SLIDE = Path("slide.ome.tif")
NUCLEI = Path("nuclei.geojson")
# The same region as Tutorial 1, so that the tiles and the cells describe the same tissue.
REGION = (30912, 37184, 41664, 47936)
OUT = Path("alhena_out")
OUT.mkdir(exist_ok=True)
print(alhena.__version__)
0.1.0
Read the slide and nuclei¶
The tile source is prepared as in Tutorial 1. read_polygons() then loads the nucleus outlines and
uses tiles=source to associate them with the image tiles.
Here the outlines are written in the slide's full-resolution pixels, so they need no conversion.
Outlines written in microns need units="um", and outlines segmented on another image need
transform=. See Coordinates in microns and
Coordinates of another image.
source = io.read_slide(SLIDE, region=REGION)
nuclei = io.read_polygons(NUCLEI, tiles=source)
print(len(source), "tiles |", len(nuclei["cell_ids"]), "nuclei")
576 tiles | 36804 nuclei
Encode the cells¶
encode_cells() validates the image and nucleus geometry, encodes the required image tiles, and
pools the resulting image tokens over each nucleus.
The pooling weights reflect how many nucleus pixels overlap each image token. See Cell pooling.
encoder = alhena.load_encoder(backbone=BACKBONE, bundle=BUNDLE, device=DEVICE)
cells = encoder.encode_cells(source, nuclei)
print(len(cells), "cells |", int(cells.valid.sum()), "can be annotated")
print(cells.n_cells_without_tokens, "nuclei reached no token")
36804 cells | 36804 can be annotated 0 nuclei reached no token
Other ways to supply nuclei (optional)¶
Nucleus outlines are not the only input ALHENA accepts. Some segmentation tools produce label maps, where each nucleus occupies a set of pixels with the same integer identifier. Others provide centroids and an approximate nucleus radius.
The next example uses a small four-tile region from the same slide. Only the way the nuclei are read
changes: the same encode_cells() method accepts both representations.
Label maps preserve the segmented nucleus pixels; centroids approximate each nucleus as a disk. Their resulting cell embeddings need not be identical, and tile-local label maps may count nuclei crossing tile borders separately.
See Nuclei for the other supported forms.
import warnings
# Four tiles of the example region: their top-left corners in full-resolution pixels.
BLOCK = np.array([[32256, 37632], [32704, 37632], [32256, 38080], [32704, 38080]])
LABEL_MAPS = Path("label_maps") # one integer label map per tile, named after the tile
CENTROIDS = Path("centroids.csv") # cell_id, x, y and radius_px, in full-resolution pixels
small_source = io.read_slide(SLIDE, coords=BLOCK)
map_nuclei = io.read_label_maps(LABEL_MAPS, tiles=small_source)
map_cells = encoder.encode_cells(small_source, map_nuclei)
centroid_nuclei = io.read_centroids(CENTROIDS, tiles=small_source, frame="origins")
with warnings.catch_warnings(record=True) as raised:
warnings.simplefilter("always")
centroid_cells = encoder.encode_cells(small_source, centroid_nuclei)
for entry in raised:
print(type(entry.message).__name__, entry.message)
for name, encoded in (("label maps", map_cells), ("centroids", centroid_cells)):
print(f"{name}:", len(encoded), "cells | membership_approximate:", encoded.membership_approximate)
GeometryWarning Nucleus shapes were approximated as circles around their centroids, so cell pooling uses approximate boundaries. If segmentation masks or polygons are available, use them for more faithful pooling. label maps: 475 cells | membership_approximate: False centroids: 421 cells | membership_approximate: True
Inspect the cell representations¶
Each row of CellEmbeddings represents one cell and stores two related embeddings:
backbone/X_alhena_backbone, 1536 dimensions: the pre-projection image-derived cell embedding used for general downstream analysis;cell_space/X_alhena_cell, 512 dimensions: the projected representation used for comparison with single-cell references during annotation.
The second representation is obtained by projecting the first into the shared cell space; they are not independent embeddings.
The result also retains cell identifiers, tile provenance, encoding metadata, and the
cell_space_id required to check compatibility with references.
print("pre-projection", cells.backbone.shape, "| shared cell space", cells.cell_space.shape)
print("identifiers", cells.cell_id[:3])
pre-projection (36804, 1536) | shared cell space (36804, 512) identifiers ['ablknega-1' 'abllcdad-1' 'ablljpoe-1']
Positions on the slide¶
Because the nucleus outlines are expressed in the same pixel coordinate system as the slide, ALHENA can retain a spatial position for each cell.
centroids_frame identifies the coordinate system and centroids_mpp gives its physical scale.
These positions use the same slide frame as the tile results from Tutorial 1, which allows the two
scales to be related in Tutorial 4.
print("frame:", cells.centroids_frame, "| micron per unit:", cells.centroids_mpp)
print(cells.centroids[:3])
frame: slide.ome.tif, level 0 | micron per unit: 0.2125 [[31036.50130387 40334.04519861] [31039.84374677 40382.45659919] [31070.12102593 40384.7771145 ]]
Tile context generated during cell encoding¶
cells.tiles contains the tile embeddings computed during the same forward pass, with
purpose="cell_context".
By default, these are only the tiles required by the supplied nuclei, so they should not be
interpreted as a systematic tile-level analysis of the whole slide. Use encode_tiles() for that
purpose.
context = cells.tiles
print(len(context), "tiles read, of", len(source), "| purpose:", context.purpose)
561 tiles read, of 576 | purpose: cell_context
Save and reload¶
write_h5ad() stores both cell representations, spatial coordinates, and provenance in an AnnData
file. read_cells() reconstructs the CellEmbeddings, allowing the cells to be annotated later
without rerunning the image encoder.
See As an AnnData table.
cells.write_h5ad(OUT / "cells.h5ad")
reloaded = io.read_cells(OUT / "cells.h5ad")
print(reloaded.to_anndata())
AnnData object with n_obs × n_vars = 36804 × 0
obs: 'cell_id', 'alhena_tile_index', 'alhena_nucleus_px', 'alhena_n_candidate_tiles', 'alhena_embedding_valid'
uns: 'alhena_cells'
obsm: 'X_alhena_backbone', 'X_alhena_cell', 'spatial'
layers: None (.X)
Where next¶
The cell embeddings are now saved in OUT.
Tutorial 3 annotates these cells with a single-cell reference without rerunning the image encoder. Tutorial 4 explores the pre-projection cell embeddings and combines them with tile-level results.
See Cell embeddings for the complete result format.
Next: 3. Cell annotation