For real-world understanding, it is essential to perceive in all-day conditions including cases which are not suitable for RGB sensors, especially at night. Beyond limitations, we innovate multispectral solution as depth estimation from illumination invariant thermal sensor without an additional depth sensor. Based on the analysis of multispectral properties and relevance of a depth prediction, we propose the efficient and novel multi-task framework called Multispectral Transfer Network (MTN) to estimate depth image from a single thermal image. By exploiting geometric priors and chromaticity, our model can generate the pixel-wise depth image in unsupervised manners. Moreover,we propose a new type of multitask module called Interleaver as a way to incorporate the chromaticity and fine details of skip-connections into depth estimation framework without sharing feature layers. Lastly, we explain a novel technical approach for stably training and covering a large disparity and extending the thermal image to data-driven methods for all-day conditions. In experiments, we demonstrate better performance and generalization ability in depth estimation through our proposed multispectral stereo dataset, including various driving conditions.