Physics Informed Neural Networks (PINNs) is a promising application of the techniques of deep learning neural networks in modeling physical processes (Raissi et al., 2019). PINNs can be used for solving the governing equations and for developing parametric solutions. PINNs can be also used for solving inverse problems and for metamodeling. However, current implementations of PINNs for flow in heterogeneous porous media are facing convergence issues (Zhang et al. 2023). Indeed, automatic differentiation is used for PINNs to evaluate spatial derivatives. Automatic differentiation cannot be applied for evaluating the spatial derivatives of the hydraulic conductivity field as this field can be discontinuous. The main objective of this work is to develop an accurate implementation of PINNs that avoids this problem.