Medical digital twins are computational models of human biology relevant to a given medical condition, which can be tailored to an individual patient, thereby predicting the course of disease and individualized treatments, an important goal of personalized medicine. The immune system, which has a central role in many diseases, is highly heterogeneous between individuals, and thus poses a major challenge for this technology. If medical digital twins are to faithfully capture the characteristics of a patient's immune system, we need to answer many questions, such as: What do we need to know about the immune system to build mathematical models that reflect features of an individual? What data do we need to collect across the different scales of immune system action? What are the right modeling paradigms to properly capture immune system complexity? In February 2023, an international group of experts convened in Lake Nona, FL for two days to discuss these and other questions related to digital twins of the immune system. The group consisted of clinicians, immunologists, biologists, and mathematical modelers, representative of the interdisciplinary nature of medical digital twin development. A video recording of the entire event is available. This paper presents a synopsis of the discussions, brief descriptions of ongoing digital twin projects at different stages of progress. It also proposes a 5-year action plan for further developing this technology. The main recommendations are to identify and pursue a small number of promising use cases, to develop stimulation-specific assays of immune function in a clinical setting, and to develop a database of existing computational immune models, as well as advanced modeling technology and infrastructure.