We present APR-CNN, a novel class of convolutional neural networks designed for efficient and scalable three-dimensional microscopy image analysis. APR-CNNs operate natively on a sparse, multi-resolution image representation known as the Adaptive Particle Representation (APR). This significantly reduces memory and compute requirements compared to traditional pixel-based CNNs. We introduce APR-native layers for convolution, pooling, and upsampling, along with hybrid architectures that combine APR and pixel layers to balance accuracy and computational efficiency. We show in benchmarks that APR-CNNs achieve comparable segmentation accuracy to pixel-based CNNs while drastically reducing memory usage and inference time. We further showcase the potential of APR-CNNs in large-scale volumetric image analysis, reducing inference times from weeks to days. This opens up new avenues for applying deep learning to large, high-resolution, three-dimensional biomedical datasets with constrained computational resources.