The proposed method, Synthetic Modality Diffusion – SynthModDiff, is based on a fundamental implementation of DDPMs, which represent a class of generative models that have been proven for their ability to generate high-quality data by leveraging probabilistic frameworks. These models operate by systematically introducing noise to data during a forward diffusion process and then training a deep learning network to reverse this process. SynthModDiff learns to iteratively remove noise in a stepwise, residual manner, ultimately reconstructing the original data distribution. By employing this structured approach, our model demonstrates exceptional performance in generating complex data distributions, making it a powerful tool for a wide range of applications in generative modelling.

Figure 1 illustrates the detailed workflow of SynthModDiff - the proposed methodology that is structured into 2 sequential stages.

As illustrated in Fig. 1, the diffusion model employed in SynthModDiff comprises two sequential stages described above with three core components: the Forward Process, a Deep Learning network, and the Reverse Process. Each of these elements plays a distinct and critical role in the overall generative process.

The evaluation of the proposed method was conducted on images corrupted by two distinct types of noise to assess its performance under varying conditions. First, the method was tested on images contaminated with Gaussian noise, which aligns with the noise distribution used during the training phase of SynthModDiff. This ensures the capacity to efficiently handle the type of noise for which the model was explicitly trained. Second, the evaluation extended to images affected by Poisson noise, which is widely recognized as a more realistic representation of noise typically faced in X-ray imaging. By integrating Poisson noise into the evaluation, the study aimed to assess the model’s generalizability and robustness when applied to scenarios corresponding to real-world clinical settings. This dual approach to noise evaluation provides a comprehensive analysis of the method’s capability to adapt to both synthetic and clinically-relevant noise characteristics and highlights its potential for practical applications in medical imaging.

The Forward Process is responsible for corrupting progressively an input image, denoted as X 0 ∈ R H × W , by using Additive White Gaussian Noise over a sequence of timesteps t ∈ [ 0 , T ] to create noise corrupted image X t for timestep t using hyperparameters β t . This forward diffusion process incrementally degrades the clean image and transitions it to a state of pure noise. The mathematical formulation of this process is defined in equation (1). (1) q ( X t | X t − 1 ) = N ( X t ; 1 − β t X t − 1 , β t I ) .

The Forward Process is executed in a single step for any arbitrary t as follows: (2) q ( X 0 : t | X 0 ) = q ( X 1 , X 2 , … , X t | X 0 ) = ∏ t = 1 T q ( X t | X t − 1 ) = N ( X t ; α ¯ t X 0 , ( 1 − α ¯ t ) I ) , where α t = 1 − β t and α ¯ t = ∏ s = 0 t α s . The parameter β t can be updated using any differentiable function that guarantees α ¯ T ≈ 0. The linear function: β t = ( β 1 − β 2 ) t T + β 2 is adopted with β 1 = 0.02 and β 2 = 10 − 6 . The value of β 2 was chosen to ensure that the resulting evaluation noise levels correspond to realistic X-ray dose levels.

To perform diffusion in inference, the Forward Process is executed from t = t ′ to t = T, illustrated in Fig. 1, where t ′ represents the denoising timestep determined based on an estimate of the image’s noise level. The method used to compute t ′ depends on the probabilistic noise model employed. In this study, two noise models are considered: Gaussian noise, for which the Deep Learning Network has been explicitly trained; and Poisson noise, which arises naturally in X-ray imaging due to its quantum characteristics. Gaussian noise is simulated using equation (2) for a given timestep t, leading to t ′ = t. Conversely, Poisson noise is modelled using equation (3), which incorporates a minor Gaussian noise component η ∼ N ( 0 , I ), scaled by σ 2 = 10 to account for electronic noise effects. (3) Y = Poisson ( I exp ( − X ) ) + σ 2 η I , where X is noisy input image and I = ∫ I 0 ( ϵ ) d ϵ corresponds to the flood image. Here, the parameter I denotes the effective photon dose (or intensity level) of the X-ray acquisition, defined as I = ∫ I 0 ( ϵ ) d ϵ, where I 0 ( ϵ ) represents the incident photon spectrum over energy ϵ. The parameter I directly controls the signal-dependent variance of the Poisson noise, with lower values of I corresponding to higher noise levels. In simulated experiments, I is fixed to predefined dose levels, while for real clinical images, it can be estimated from acquisition parameters or via noise estimation methods such as Turajlić and Karahodzic (2017). Owing to the signal-dependent characteristics of Poisson noise, the denoising timestep t ′ is systematically determined by evaluating the maximum noise variance present in the image, as formulated by: (4) X = I N ∑ i = 1 N Percentile 95 ( max ( X i ) ) . Here, N denotes the size of the training dataset. A percentile-based approach is employed to mitigate the influence of high-intensity artificial details in the images, such as medical annotations. The denoising timestep t ′ is then estimated as follows: (5) t ′ = 1 − α ′ ¯ t ≈ Variance ( − log ( Y ) ) .

The inferred timestep t ′ is defined as the timestep in the Gaussian diffusion process whose noise variance matches the estimated variance of the observed Poisson-corrupted image. Specifically, since the variance of the forward diffusion process at timestep t is given by 1 − α ¯ t , the timestep t ′ is selected such that 1 − α ¯ t ′ ≈ Variance ( − log ( Y ) ). This establishes an explicit correspondence between the Poisson noise level and the Gaussian diffusion noise schedule.

Determining the timestep t ′ using equation (5) ensures that the model effectively removes noise with the highest variance. It is important to note that computing Y requires prior knowledge of the dose I. For real noisy images, I can be estimated based on the X-ray acquisition parameters or through noise estimation techniques, such as those proposed by Turajlić and Karahodzic (2017). To achieve an equivalent noise level between Gaussian and Poisson noise, Gaussian noise was simulated by selecting the timestep t from equation (2) to match the denoising timestep t ′ estimated for Poisson noise. During inference, the forward diffusion process is initialized at timestep t = t ′ , and the reverse process is subsequently applied from t ′ down to t = 0, ensuring consistency between the estimated noise level of the input image and the diffusion model’s noise schedule.

The reverse process reconstructs an image by iteratively inverting the forward diffusion process. Given that the inverse of a Gaussian diffusion process follows a Gaussian distribution, the probability density function of the original data can be recovered by integrating over the sequence of state transitions in the associated Markov chain: (6) p θ ( X 0 ) = ∫ p θ ( X 0 : T ) d X 1 : T = ∫ p ( X T ) ∏ t = 0 T − 1 p θ ( X t − 1 | X t ) d X 1 : T .

Let θ denote the set of parameters governing the deep learning model. The prior distribution is defined as p ( X T ) ∼ N ( 0 , I ), while the transition probability in the reverse diffusion process is formulated as: (7) p θ ( X t − 1 ∣ X t ) = N ( X t − 1 ; μ θ ( X t , t ) , β ¯ t ( X t ) ) .

Here, the mean function μ θ is approximated by X t − η θ , where η θ represents the Gaussian noise estimated by the neural network. Additionally, to maintain numerical stability during the Reverse Process, the predicted mean μ 0 is clipped within the intensity range [ − 1 , 1 ].

The U-Net architecture utilized in this study is structured hierarchically, with a configuration defined by the sequence [ 3 , 3 , 3 , 3 , 3 , 3 , 3 ], where each numerical value denotes the number of layers in the corresponding network stage. This design integrates two residual layers with ReLU activation function, alongside a Transformer layer that employs 8 attention heads per layer. The incorporation of the Transformer mechanism enables the model to effectively capture long-range dependencies and contextual relationships, thereby improving its capacity to process complex image data.

The network employs an approach to upsampling and downsampling across multiple blocks, adhering to a predefined scaling ratio of [ 1 , 2 , 4 , 8 , 4 , 2 , 1 ]. This scaling process progressively reduces the spatial resolution of feature maps and then symmetrically restores it, thereby enabling multi-scale feature extraction. In particular, the network initially processes input data at a resolution of 128 channels, which is incrementally expanded to a maximum of 1024 channels at the bottleneck stage, then reduced symmetrically back to 128. These transformations are implemented using convolutional operations with a kernel size of 3 × 3, to ensure efficient feature representation while maintaining spatial coherence throughout the network. With this hierarchical framework, the U-Net model effectively captures both fine-grained local details and broader global structures, hence becoming highly compatible with image reconstruction applications.

The performance of the proposed method was assessed with equivalent Poisson and Gaussian noise. The noise parameters were selected to ensure both physical relevance to X-ray imaging and statistical consistency within the diffusion framework. Specifically, the Poisson intensity I was chosen to represent clinically meaningful dose levels, with I = 5 × 10 4 for a high-dose setting and I = 9 × 10 3 for a low-dose setting, reflecting typical diagnostic and dose-reduction scenarios in X-ray imaging.

To enable a fair comparison between Poisson and Gaussian noise, Gaussian noise levels were selected to achieve equivalent noise variance in the diffusion process, governed by 1 − α ¯ t . Accordingly, timesteps t = 3 and t = 9 were chosen for the high- and low-dose cases, resulting in noise levels comparable to those induced by the corresponding Poisson settings. The denoising timestep t ′ for Poisson noise was estimated using the variance-based criterion in equation (5), ensuring alignment between the signal-dependent Poisson noise and the diffusion noise level.

In this research, the proposed method was evaluated using 2 widely used medical image datasets: BraTS2020 (Zhang et al., 2020) and IXI (Ionescu et al., 2017). The BraTS2020 dataset is a proven guideline for brain tumour segmentation in magnetic resonance imaging (MRI) and contains multi-modal MRI scans, including T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and Fluid Attenuated Inversion Recovery (FLAIR) sequences, as shown in Table 1. The IXI (Information eXtraction from Images) dataset is a public collection of multimodal brain imaging data, containing 600 MRI scans from healthy subjects. These images were acquired using 4 different imaging protocols: T1-weighted, T2-weighted, Proton Density (PD), and Magnetic Resonance Angiography (MRA), as summarized in Table 1.

In multi-domain MRI synthesis, statistical bias toward certain contrasts may arise when the training data distribution is imbalanced across modalities. To alleviate this issue, we evaluate SynthModDiff on two complementary public datasets, BraTS2020 and IXI, which together provide a diverse and relatively balanced set of MRI contrasts. This design choice reduces the risk of the model overfitting to a dominant modality, such as T1-weighted images. Although explicit contrast-wise reweighting is not applied in the current framework, potential modality bias can be quantified through per-contrast evaluation metrics or conditional error analysis, which we identify as a promising direction for future work.

Building on prior research by Han et al. (2024), the Mask Vector for Training on Multiple Datasets was employed to integrate the two datasets during training. Furthermore, the images were downsampled from 1024 × 1024 to 512 × 512, subjected to horizontal flipping as a transformation, and normalized to an intensity range of [ − 1 , 1 ], following the methodology by Matviychuk et al. (2016).

Since some of the compared methods are specifically designed for Gaussian noise, the Anscombe transform (Makitalo and Foi, 2012) was applied to ensure a fair comparison by converting Poisson noise into a Gaussian distribution with a variance of 1. Additionally, the images were normalized to the [ 0 , 1 ] intensity range, following the approach described by Bodduna and Weickert (2019).

To simulate realistic MRI noise characteristics, Rician noise is introduced during data preprocessing. All images are first normalized to the range [ 0 , 1 ]. Complex-valued Gaussian noise is independently added to the real and imaginary components of the clean signal, with noise standard deviation σ. The final magnitude image is computed as (8) I rician = ( I + n r ) 2 + n i 2 , n r , n i ∼ N ( 0 , σ 2 ) .

The noise level is controlled through the signal-to-noise ratio (SNR), defined as SNR = μ signal / σ. In our experiments, SNR values are uniformly sampled from [ 5 , 15 ] dB for low-dose settings and [ 20 , 30 ] dB for high-dose settings. Unless otherwise specified, Rician noise is applied only during evaluation to assess robustness under non-Gaussian noise conditions.

All experiments were conducted on an Ubuntu 20.04 system using the PyTorch 1.12.1 framework, Python 3.10.4, and an NVIDIA DGX A100 GPU with CUDA 12.1, powered by an Intel Xeon Platinum 8470Q CPU.

The model was trained for 100 epochs using mixed-precision arithmetic to improve computational efficiency without compromising numerical accuracy. A learning rate of 10 − 4 was employed in conjunction with the AdamW optimizer and a cosine learning rate schedule, enabling rapid initial convergence followed by stable refinement. During training, the diffusion timestep t was sampled uniformly from the interval [ 1 , T ], ensuring that the network learns to estimate noise across the full range of diffusion levels. Optimization was performed using the mean squared error loss L = E [ ‖ η − η θ ( x t , t ) ‖ 2 ], which directly supervises the prediction of Gaussian noise and is consistent with the theoretical objective of denoising diffusion probabilistic models.

The diffusion process was configured with T = 1000 timesteps and a linear noise schedule β t ∈ [ 10 − 6 , 0.02 ]. This configuration ensures a smooth and gradual corruption of the input data, providing sufficient discretization of noise levels to support stable training and reliable reverse process. All input images were normalized to the intensity range [ − 1 , 1 ]. During inference, the reverse diffusion process was initialized from the estimated timestep t ′ and iteratively performed until t = 0. To enhance numerical stability and prevent intensity outliers, the predicted mean μ 0 was clipped to the same normalized range.

The U-Net backbone employed convolutional layers with 3 × 3 kernels were used throughout all residual blocks. In addition, Transformer layers with 8 attention heads were incorporated to effectively capture long-range dependencies while maintaining computational efficiency.

To evaluate the practical utility of SynthModDiff in a clinically relevant scenario, we conducted a downstream tumour classification task using synthetic multi-contrast MRI images. Specifically, synthetic target modalities generated by different synthesis methods were used to train a tumour classification model, which was then evaluated on real MRI images.

For the classification network, we employed a ResNet-18 architecture initialized with ImageNet pre-trained weights. The input to the classifier consists of concatenated multi-contrast MRI slices (T1, T2, T1CE, and FLAIR), which are stacked along the channel dimension and resized to 224 × 224. All classifiers were trained from scratch on synthetic images generated by each method, while the test set remained identical and consisted exclusively of real MRI data to ensure a fair comparison.

The training pipeline follows three stages:

This design ensures that performance differences in the classification task can be directly attributed to the quality of the synthesized images. The classifier was trained using the cross-entropy loss with the Adam optimizer, a learning rate of 1 × 10 − 4 , and a batch size of 32. Standard data augmentations, including random flipping and rotation, were applied during training.

In this study, the proposed model is systematically compared with a diverse set of existing models, including Pix2pix (Isola et al., 2017), StarGAN (Choi et al., 2018), ResUNet (Diakogiannis et al., 2020), Multi-cycle GAN (Liu et al., 2021), Improved DDPM (Nichol and Dhariwal, 2021), Guided DDPM (Dhariwal and Nichol, 2021), Hieu-Model (Han et al., 2024), and Diffusion Mamba (Wang et al., 2024). This comparative analysis aims to assess the performance, effectiveness, and robustness of the proposed approach relative to these established methodologies in the field.

The evaluation of model performance was conducted using three widely recognized metrics in image denoising: Peak Signal-to-Noise Ratio (PSNR), normalized mean absolute error (NMAE), and the Structural Similarity Index Measure (SSIM). PSNR is typically used to quantify the reconstruction quality of an image by measuring the ratio between the maximum possible signal power and the power of the noise that distorts the image. A higher PSNR value generally indicates better visual quality and lower distortion. NMAE, on the other hand, is a normalized measure of the absolute differences between the predicted and ground truth images, hence creating an alternative assessment of visual accuracy. Lastly, SSIM evaluates the structural fidelity of the denoised images by comparing luminance, contrast, and texture similarities with the original image. With these metrics together, a comprehensive assessment of the model’s ability is provided to preserve image quality while effectively removing noise.