Segmentation procedures of concrete microscopic images and standard test methods devoted to the spacing factor calculation for the freeze-thaw resistance assessment of concrete are time-consuming and skill-dependent. Moreover, manual color treatment and careful image examination are often needed. Within the past few years, Convolutional neural networks (CNN) have proved unpreceded performances in image segmentation and object detection tasks, though they often showed limited reusability and modularity. This study introduces an open-source modular deep learning segmentation algorithm of concrete microscopic images. The algorithm is based on two CNN models dedicated to air voids and aggregates detection. The algorithm performances have been calculated using various concrete, mortar, and cement paste samples. The Protected Paste Volume (PPV) and distance-to-air-void have been computed and agreed well with the experimental spacing factor. Moreover, a better correlation between PPV and scaling was found than between experimentally measured spacing factors and scaling, highlighting a critical spacing factor interval from 200 µm to 300 µm.