In medical image segmentation tasks, Convolutional Neural Networks (CNNs) have become an efficient and successful solution, although they have limitations in explicitly modeling long-term dependencies. The Transformer neural network has recently demonstrated its capabilities in image segmentation, although a large amount of data is required for training. In this study, we present a hybrid architecture, UTNetPara, that integrates the Transformer into a U-shaped CNN to improve segmentation accuracy on a medium-sized dataset. Self-attention modules are applied in both the encoder and decoder to enhance the ability to capture long-term dependencies at different scales. Efficient self-attention mechanisms with relative position encoding are employed to reduce the computational cost accordingly. A fully annotated dataset consisting of whole slide images scanned from periodic acid-Schiff stained mouse kidney tissue is used for evaluation. The proposed method is trained to segment the main renal structures: glomerular tuft, glomerulus including Bowman's capsule, tubules, arteries, arterial lumina, and veins. Our experimental results indicate that the UTNetPara has a better segmentation performance than other state-of-the-art models.