Monitoring the technical condition of infrastructure is a crucial element of its maintenance. Although there are many deep learning models intended for this purpose, they are severely limited in their application due to labour-intensive gathering of new datasets and high demand for computing power during model training. To overcome these limiting factors we propose a KrakN framework. It enables end-to-end development of unique infrastructure defect detectors on digital images, achieving an accuracy of above 90%. The framework also supports the semi-automatic creation of new datasets and has modest computing power requirements. It can be used to immediately implement deep learning in the process of infrastructure management and, due to its architecture based on transfer learning, allows low metric loss when used across different datasets. We also demonstrate that thanks to its scalability and modular structure, the presented framework is easily modifiable, and can be used in realistic scenarios.