Cancer label prediction

[LukasHats]

Thanks for the awesome model, we made it run without any problems, which is insane compared to many methods!

One general question. How does the model predict cancer If it does not have any context on what tissue it processes? The cancer cell identity depends on the tissue; it could be plasma cells in myeloma (could argue about IRF4?), it can be epithelial in many cancers, it can be Reed-Sternberg (CD30) in lymphoma, melanocytes in melanoma (melanA, dox10 etc) etc. pp. There is no universal cancer marker and also if one uses panCK (+ki67?) this is not accurate for a lot of datasets, and also panCK stains healthy epithelial cells. So, how does that work? Just very curious.

Thanks for the fast replies!

[rossbar]

Also didn't we decide to stop using codecov after https://about.codecov.io/security-update/?

This is a great question/observation! I'm certainly no expert - @xuefei-wang 's take would be much more valuable - but that said I can give my perspective.

In practice, I consider the "Tumor" as a result (or perhaps artifact) of the limited amount of training data available. We know that training models of this type takes a lot of data, so in the initial development of the model, the approach was very much to try to make use of every bit of annotated training data we could find. Of course, a lot of multiplexed images are not of healthy tissue, and quite often that "unhealthy" state meant cancerous. Thus a lot of the available training data had some or another form of "cancer"/"tumor" in the labels. For the initial exploration in the development of this model, these data were kept in the training data. There's nothing wrong in principle with this treatment - in fact, given training data that sufficiently samples tissues, disease states, and marker panels, I'd hypothesize¹ that an approach like the one proposed here would be capable predicting disease state as part of the same inference pipeline.

Of course, at this stage we're nowhere near this level of coverage in the training data to achieve such properties in the model. And I think your observation is entirely correct - because the "features" expressed by the cells labeled "tumor" in the training data show wide variability across tissue types and often tend to express common markers that are indicative of other things in healthy tissue (e.g. PanCK) - I suspect that the "tumor" category would be one of the lowest-precision (in the ML sense of precision-recall) labels in new tissue. So much so that, if I were applying the model to a completely unseen dataset (i.e. new tissue-type/modality/disease state) I would be inclined to interpret the "tumor" category as more of an "over-expresses certain markers" label than truly being indicative of a cancerous state.

Anyways - that's my perspective. Hopefully @xuefei-wang and others who are much more knowledgeable than I can weigh in too!

Footnotes

1. Note the use of the word hypothesis - I would personally love to be able to test this claim. In my estimation, the number/scale of existing public datasets is the current bottleneck to being able to do so. Other ideas such as synthetic data or different approaches to language embedding are, IMO, promising avenues to explore as well. ↩

[LukasHats]

I see, thanks for your insights. I assume if one already uses an LLM to give some context to markers, one could also input the tissue/disease as context and thereby 'rendering' some markers as cancer markers (like panCK turns into a cancer marker when we have lung cancer tissue for example).

[xuefei-wang]

The model learns to identify "Tumor" cells by associating specific marker expression patterns from our training data with that label. Its knowledge is therefore constrained by the examples it has seen. Because our training data includes various cancers (e.g., epithelial, melanoma), the model has learned to associate certain marker combinations like PanCK+/Ki67+ or Melan-A+ with the "Tumor" label.

On a relevant note, we added a feature in the latest model that might help mitigate this issue. We have a flexible classification head using contrastive learning, which allows you to define the set of possible cell types at prediction time. So, for a dataset you know is from healthy tissue, you can simply remove "Tumor" from the list of potential labels. This forces the model to choose from the remaining, biologically plausible cell types.

We don't have this "rendering" function or the "disease" context yet, but this idea of conditional modeling—where the model's interpretation of markers changes based on context—is definitely an interesting concept! Thank you for the question!