Auto3DSeg: pairing scalar data with training images

[pwrightkcl]

Is it possible to include scalar data as a training input to auto3dseg? My use case is that I am training a model to segment ischaemic lesions on CT brain images. The appearance of the lesion changes qualitatively, especially in the first hours. I know the times of symptom onset and scan acquisition. I would like to include time to scan in the model training, in the hope that the model can learn to expect different lesions signs over time. This is a pretty standard thing to do in old school GLM analyses, like having a covariate alongside a group comparison (not even sure if anyone in this forum is old enough to know what I'm on about!) As a noob I have no idea how to approach this in auto3dseg and would appreciate the commmunity's advice.

A naive approach would be to create a new image volume and set every voxel to the value I'm interested in (in this case time from onset to scan). There are several problems implementing this. The first is that I do not know if auto3dseg will even accept multi-channel data, and if so how.

The second issue is around scaling. Since my modality is CT, auto3dseg will clip the voxel values to absolute min and max worked out during data analysis. I don't want it to clip my time value image, and I don't want the time value image to alter the range used for clipping. I think I could take the 0.5-99.5 centile range from a previous run and limit my time value image to within this range. I could use the full range, or a proportion of the range around the centre. I would imagine this might affect the weighting of the time value data but that is just a guess.

I understand a more proper way to introduce the time values would be to modify the generated training code so it inserts time value at a later point in training, but I do not have the expertise for this. I do not know if it is too detailed a job to ask the community to explain it but if anyone wants to tackle it I would welcome the input. I am using the segresnet algo, as this has consistently performed best on my data.

[diazandr3s]

Hi @pwrightkcl,

I assume this is for a segmentation task, right?
For multimodality tasks, I'd suggest working directly with the SegResNet network instead of using Auto3DSeg. You should be able to add a layer that accepts tabular data to the netwrok.

Happy to chat more about this :)

[pwrightkcl]

I figured I'd have to drill down into the segresnet code (that's the best performing model for my dataset anyway). At the moment that's beyond my expertise and was hoping for some guidance, but I'll look for help locally too and do some reading.