latents generation

[shengzhang90]

Hi, MAISI team

When I trained the maisi model with my own private data, the generation effect is as follow:

I doubt that the issue occurs at the latent generation, could you please help me? I will appreciate your help very much.

Best wishes

[Can-Zhao]

Could you help give more information on how you generated latents?
Did you use MAISI VAE or your own VAE?
How did you do image preprocessing especially intensity transform?

[Can-Zhao]

If GT latents can decode correctly, could you compare GT latents and generated latents? If they look similar, then issue may happen at post intensity processing

[shengzhang90]

Yes, I used MAISI VAE.

I used the default transform in the file of "scripts/diff_model_create_training_data.py".

def create_transforms(dim: tuple = None) -> Compose:
"""
Create a set of MONAI transforms for preprocessing.

Args:
    dim (tuple, optional): New dimensions for resizing. Defaults to None.

Returns:
    Compose: Composed MONAI transforms.
"""
if dim:
    return Compose(
        [
            monai.transforms.LoadImaged(keys="image"),
            monai.transforms.EnsureChannelFirstd(keys="image"),
            monai.transforms.Orientationd(keys="image", axcodes="RAS"),
            monai.transforms.EnsureTyped(keys="image", dtype=torch.float32),
            monai.transforms.ScaleIntensityRanged(
                keys="image", a_min=-1000, a_max=1000, b_min=0, b_max=1, clip=True
            ),
            monai.transforms.Resized(keys="image", spatial_size=dim, mode="trilinear"),
        ]
    )
else:
    return Compose(
        [
            monai.transforms.LoadImaged(keys="image"),
            monai.transforms.EnsureChannelFirstd(keys="image"),
            monai.transforms.Orientationd(keys="image", axcodes="RAS"),
        ]
    )

You means that GT latents are from the output of VAE, and the generated latents are from the diffusion model?

Thanks a lot.