1 Introduction
When considering any image, beyond seeing it as a container of objects, among other things, it is natural for a human being to give it meaning or to infer the explanation of some event of interest captured in it, but how can such an inference be reached through artificial intelligence? Causal inference can be applied in many areas of science and technology, such as economics, epidemiology, image processing, and autonomous driving, which are areas where it is crucial to make accurate decisions. Currently, there are widely studied methods that, through correlation, recognize and classify objects using datasets such as (Deng et al., 2009) which has sufficient size and information to ensure high accuracy in such tasks (Zeiler and Fergus, 2014). However, in the last decade, as pointed out by Saeed and Omlin (2021), explainable AI (XAI) has been proposed to respond to the need raised by important contributions in artificial intelligence, which have led to an increasing complexity of algorithms and lack of transparency of models, and to advance the adoption of AI in critical domains. Then, to obtain the explanation we are looking for about an event captured in an image, we would have to consider causal relationships that can either be inferred through expert knowledge (Martin, 2018) or intervene such data sets through experimentation, as indicated in He and Geng (2008), taking into account that, in probabilistic language, not having a way to distinguish between giving value to a variable and observing it, prevents modelling cause and effect relationships (Perry, 2003). Thus, taking modelling as an essential step to achieve causal inference, Xin et al. (2022) discusses the role of causal inference to improve the interpretability and robustness of machine learning methods, and highlights opportunities in the development of machine learning models with causal capacity adapted for the analysis of mobility considering images or sequential data. In the punctual case on causal inference applied to images, Lopez-Paz et al. (2017) propose to use neural causality coefficients (NCCs) that are calculated by applying convolutional neural networks (CNNs) to the pixels of an image, so that the appearance of causality between variables suggests that there is a causal link between the real-world entities themselves, Lebeda et al. (2015) have proposed a statistical approach – transfer entropy – to discover and quantify the relationship between camera motion and the motion of a tracked object to predict the location of the tracked object, Fire and Zhu (2013) presented a Bayesian grammar (C-AOG) model for human-perceived causal relationships that can be learned from a video, and Pickup et al. (2014) use the causality method, supplemented with computer vision and machine learning techniques, to determine whether a video is playing forward or backward by observing the “arrow of time” in a temporal sequence.
This paper consists of four sections, in Section 1 we state the motivation of the study, present some antecedents that have made important contributions in the area of causal inference applied to images and define our contribution as a starting point to address a problem area already detected by several authors. In Section 2, we present the method used to generate the data, define the causal model and validate it with the NOTEARS algorithm, and then query the model by means of interventions. In Section 3, we analyse the results obtained in the applied causal discovery and causal inference processes and, finally, in Section 4, we conclude that the graphical representation of a causal model makes it simpler to understand the problem, although for the validation using NOTEARS we had to make restrictions based on expert knowledge. Likewise, we recognize the importance of the structure of the dataset for causal inference in contrast to the structure of a machine learning dataset and, finally, thanks to the interventions and queries of the causal model, it was possible to deduce, with a high level of certainty, the cause of the shadow projection.
1.1 Related Work
Regarding the explainability of events or phenomena captured in an image or video, taking modelling as an essential step to achieve causal inference, Xin et al. (2022) discusses the role of causal inference to improve the interpretability and robustness of machine learning methods, and highlights opportunities in the development of machine learning models with causal capability adapted for mobility analysis considering images or sequential data. In the punctual case on causal inference applied to image analysis, Lopez-Paz et al. (2017) propose to use neural causality coefficients (NCCs) that are calculated by applying convolutional neural networks (CNNs) to the pixels of an image, so that the appearance of causality between variables suggests that there is a causal link between the real-world entities themselves, Lebeda et al. (2015) have proposed a statistical approach – transfer entropy – to discover and quantify the relationship between camera motion and the motion of a tracked object to predict the location of the tracked object, Fire and Zhu (2013) presented a Bayesian grammar (C-AOG) model for human-perceived causal relationships that can be learned from a video, and Pickup et al. (2014) use the causality method, supplemented with computer vision and machine learning techniques, to determine whether a video is playing forward or backward by observing the “arrow of time” in a temporal sequence.
1.2 Our Contribution
By taking into account the underlying causes of shadow formation, causal inference can provide more accurate predictions and improve the overall realism of virtual environments. By virtue of this, given the relevance of this topic and the need for experimentation on specific cases that would potentially be contributory to evolving fields such as 3D graphics where shadow detection is an area where causal inference can be applied to improve accuracy and efficiency in this process, as opposed to traditional techniques such as ray tracing which is computationally expensive in terms of handling complex scenes with many objects (Levoy, 1990), this challenge is equally recognized in novel solutions such as that of Li et al. (2018) which computes derivatives of scalar functions on a rendered image with respect to arbitrary scene parameters such as camera location, scene geometry, materials and lighting parameters using an edge sampling algorithm. Our study provides a benchmark for addressing one aspect of this problem using causal inference to detect and deduce the cause of a shadow cast on the surface of a 3D scene.
2 Materials and Method
Our objective was to explain, by means of causal inference, the appearance of a shadow cast on the surface defined on the lower face of a 3D scene in which, in addition to being illuminated, a sphere is present. For this we established 3 steps; in the first one we generated $1\times {10^{6}}$ scenes (images) in which 4 features can be observed: light, sphere, surface and shadow. There are numerous libraries that facilitate both statistical and causal estimation and inference operations, therefore, we chose to use pgmpy (Ankan and Panda, 2015), causalinference (Wong, 2016) and causalnex (Beaumont et al., 2021) for the development of the experiment, thus, in the second step we define a structural causal model (SCM) taking into account the relationships between these features, also identifying the independence of the features, and then, through Causal Discovery, structurally validate the model by applying the NOTEARS algorithm devloped by Zheng et al. (2018) which differs significantly from other search approaches in the discrete space of directed acyclic graphs (DAG). In the third, through conditional probability distributions (CPDs), it was possible to calculate the probabilities of each variation in the observations and thus validate the hypothesis that the appearance of the shadow is due to the presence of the sphere in an illuminated scenario with a defined surface, all by means of “What if...” queries to the model, as indicated in level 2 of the causal scale (DOING), which is reached when one can predict the effect(s) of deliberate alterations of the environment and choose among these alterations to produce a desired outcome (Pearl and Mackenzie, 2018). Then, we intervened the model by removing the sphere and looked at, for a 95% confidence interval, the average treatment effect (ATE) and p-value to have an estimate of how far our hypothesis was from being null. Finally, we compare the performance of our experiment with two conventional shadow detection techniques.
2.1 Data
As already stated, we considered 4 observable features for the confirmation of the dataset, Table 1 shows the label used for each of them and the values they can take, with 1 indicating that the feature is present in the scene and 0, otherwise.
Table 1
Labellingof the observed features.
| Feature | Label | Posible values |
| Light | A | $[1,0]$ |
| Sphere | B | $[1,0]$ |
| Surface | C | $[1,0]$ |
| Shadow | Y | $[1,0]$ |
Figure 1 shows the dataset based on a hypothetical scenario similar to that used by Pearl and Mackenzie (2018) to demonstrate the importance of probabilities emphasizing that a causal model involves more than drawing arrows, for behind these are probabilities.
Thus, by means of Algorithm 1, the data of $1\times {10^{6}}$ scenes were synthetically generated, of which all features are perceived in 99% (0), while in 70% of the remaining 1% illumination and sphere are observed, but neither surface nor shadow (1); in half of the images (15%) of the remaining 30% no feature can be observed (2), while, in the second half, only illumination and surface are observed, but neither sphere nor shadow (3).
2.2 Structured Causal Model SCM
According to Pearl et al. (2019), an SCM is a way of describing the relevant features of the world and how they interact with each other. Specifically, a structural causal model describes how nature assigns values to variables of interest. Causal inference usually requires specialized knowledge and untestable hypotheses about the causal network linking treatment, outcome, and other variables. By summarizing knowledge and hypotheses in an intuitive way, graphs help clarify conceptual issues and improve communication between researchers (Hernán and Robins, 2020). Accordingly, we model an SCM in which each node corresponds to each observable feature and all edges point to a single collider node -Y-. Figure 2 shows the SCM and the independence of the variables that comprise it.
2.3 Causal Inference
Once the model is built, we calculate the conditional probability distributions (CPD) which are defined for a set of discrete and mutually dependent random variables to show the conditional probabilities of a single variable with respect to the others (Murphy, 2012). These are calculated by applying the chain rule as illustrated in (1), where ${X_{i}}$ is an event and N is the number of events considered in the model, hence $0\lt i\lt N$.
As shown in Fig. 3, we calculate the probability of each possible value of each variable knowing the values taken by the other variables.
Then, to strengthen our hypothesis, we asked the model what would happen if no sphere had been detected, in other words, we intervened the model by not detecting the sphere in order to obtain the probability of detecting the shadow. To provide clarity on what role the SCM variables play in the causal inference process we follow, among others, Chiappa and Isaac (2019), Guo et al. (2020) and Dague and Lahey (2019) who explain that a causal inference process is determined by a treatment, a set of confounders and an outcome. Table 2 shows the role of each MCS variable in the causal inference process.
Table 2
Role of SCM variables in the Causal Inference process.
| Label | Variable |
| Treatment | B |
| Confounders | [A, C] |
| Outcome | Y |
Subsequently, considering this intervention, we calculated for the whole set of cases $(N)$ the treatment effect to measure the average difference between the cases in which the treatment was applied (Y1) and the control cases (Y2) by applying (2).
Finally, to contrast the treatment results and thus obtain the estimate of how far the hypothesis was from being null, i.e. that there was no relationship between the sphere and the shadow, for a 95% confidence interval, we calculated a table of z-scores by applying (3) where ${X_{i}}$ is each outcome, μ is the mean and σ is the standard deviation, and then calculated the bilateral tailed p-value and a significance level $\alpha =0.05$.
2.4 Shadow Detection
Within the realm of shadow detection in images, prominent methods include adaptive thresholding, threshold segmentation (Bradski, 2000), and clustering-based segmentation (Felzenszwalb) (Van der Walt et al., 2014). We extended the latter by integrating a causal inference module. This advancement not only facilitated shadow detection but also enabled the identification of their root causes within specific scenarios. This comprehensive approach demanded parameters encompassing a comprehensive record of perceptible elements within the image (light, sphere, surface, and shadow), alongside the causal inference model.
Illustrating this process, Algorithm 2 outlines the intricate interplay between shadow detection and causal inference within the shadow phenomenon.
3 Results
3.1 Causal Model
The structural causal model (SCM) was designed based on expert knowledge as Hernán and Robins (2020) pointed out, but validated in two attempts by means of causal Discovery using NOTEARS. In the first attempt, the algorithm took almost 5 minutes to generate the model shown in Fig. 4(a), which we consider quite long for the size of the dataset, resulting in a model that was not very coherent according to the expert knowledge. On the other hand, in the second attempt, we added a constraint to the algorithm to consider that A, B and C are independent as already shown in Fig. 2. The algorithm, as can be seen in Fig. 4(b), generated the model in less than 10 seconds and with the expected consistency.
3.2 Causal Inference
From the conditional probability distribution (CPD) it was possible to query the model under the hypothesis formulated. In Table 3, it can be seen that by eliminating the sphere there would be a 99.5% probability that no shadow would be cast; furthermore, it can be seen that the hypothesis gains strength by obtaining a p-value of less than 0.05 (the default threshold value) indicating that the null hypothesis is false, and a positive average treatment effect (ATE) value suggests that $P(Y|B)\gt P(Y)$, indicating that the presence of the sphere increases the probability of detecting a shadow cast on the surface or, in other words, it was possible to infer that the sphere is the most likely cause of the shadow cast on the surface.
3.3 Shadow Detection and Causality
To establish a contrast, we employed an identical image and introduced a confounding element by aligning the background colour with the shade’s hue projected onto the surface. Subsequent execution encompassed the Felzenszwalb method integrated with the causal inference module, as well as the adaptive thresholding and threshold segmentation techniques. The outcomes of this comprehensive approach are visualized in Fig. 5.
In the context of shadow detection, the outcomes are evident. Among the approaches, the combination of the Felzenszwalb method with causal inference (refer to Fig. 5(a)) showcased the most promising results. It achieved an acceptable shadow detection accuracy of $51.5\% $. Following closely, the adaptive threshold method achieved an accuracy of $51.7\% $ (refer to Fig. 5(c)), while the threshold segmentation method achieved $55.9\% $ accuracy (refer to Fig. 5(b)).
It’s important to emphasize that the presence of confounding factors significantly influenced the accuracy of the detection results. However, when considering the determination of the shadow’s causality, the impact of confounding factors became negligible. Notably, only the Felzenszwalb method (refer to Fig. 5(a)) yielded a substantial result in this regard.
4 Conclusions and Future Work
We’ve shown how to employ causal inference to strengthen a hypothesis and, as a result, deduce the cause of a shadow phenomenon with high certainty. This is accomplished by utilizing interventions and inquiries within the causal model. We start with a set of photos from a 3D scenario in which four occurrences were examined as part of a structural causal model validated with the NOTEARS algorithm for causal detection. By contrasting their performance, we also demonstrated that adding a causal inference module to a shadow detection approach is feasible and advantageous. This opens the door for similar connections in other diverse and complex ways. A causal model’s visual representation improves understanding of the problem and the roles that events play in its resolution. Despite testing the causal model with NOTEARS, there was some worry about the need to set limits based on expert knowledge. A dataset with a more intricate structure is required for causal inference when compared to typical datasets utilized for machine learning applications. Confounding factors had a considerable impact on the detection method’s accuracy but not on the causal inference model. In the future, we hope to create a second version of this project. We intend to improve causal inference in this iteration by incorporating machine learning techniques. This combined approach will determine the origin of shadows sensed in complex graphical settings.
