Background and Objective The global population has been heavily impacted by the COVID-19 pandemic of coronavirus. Infections are spreading quickly around the world, and new spikes (Delta, Delta Plus, and Omicron) are still being made. The real-time reverse transcription-polymerase chain reaction (RT-PCR) is the method most often used to find viral RNA in a nasopharyngeal swab. However, these diagnostic approaches require human involvement and consume more time per prediction. Moreover, the existing conventional test mainly suffers from false negatives, so there is a chance for the virus to spread quickly. Therefore, a rapid and early diagnosis of COVID-19 patients is needed to overcome these problems. Methods Existing approaches based on deep learning for COVID detection are suffering from unbalanced datasets, poor performance, and gradient vanishing problems. A customized skip connection-based network with a feature union approach has been developed in this work to overcome some of the issues mentioned above. Gradient information from chest X-ray (CXR) images to subsequent layers is bypassed through skip connections. In the script’s title, “SCovNet” refers to a skip-connection-based feature union network for detecting COVID-19 in a short notation. The performance of the proposed model was tested with two publicly available CXR image databases, including balanced and unbalanced datasets. Results A modified skip connection-based CNN model was suggested for a small unbalanced dataset (Kaggle) and achieved remarkable performance. In addition, the proposed model was also tested with a large GitHub database of CXR images and obtained an overall best accuracy of 98.67% with an impressive low false-negative rate of 0.0074. Conclusions The results of the experiments show that the proposed method works better than current methods at finding early signs of COVID-19. As an additional point of interest, we must mention the innovative hierarchical classification strategy provided for this work, which considered both balanced and unbalanced datasets to get the best COVID-19 identification rate.