Wearable devices enable remote, long-term, and unobtrusive monitoring of patients in their everyday living and working environments. Remote health monitoring often involves monitoring physical and cardiac activities (exertions) to establish correlations between the two. With recent advances in sensor technologies and machine learning, the efficiency with which these activities can be recognized has been steadily improving. In this article, we apply convolutional neural networks (CNNs) to measurements taken with wireless electrocardiograms (ECGs) and inertial sensors for human activity recognition (HAR). Experimental results confirm that our approach can recognize a wide range of everyday activities with a high degree of accuracy. Specifically, activities such as jumping, running, and sitting could be recognized with an accuracy exceeding 99%, while activities such as bending over, walking, standing up, and climbing stairs could be recognized with an accuracy exceeding 90%. Overall, the results suggest that the combined use of inertial sensors and ECG leads to better recognition accuracy. Likewise, this article closely examines the contributions of individual sensors and if and to what extent their placement affects recognition accuracy.