Biliary tract cancer (BTC) refers to a heterogenous group of epithelial malignancies arising along the biliary tree. The highly aggressive nature combined with its silent presentation contribute to the dismal prognosis of this tumor. Tumor-infiltrating immune cells (TIICs) are frequently present in BTC and there is growing evidence regarding their role as therapeutic targets. In this study, we analyzed the immune cell infiltration in BTC and developed a promising immune signature score to predict prognosis in BTC. Immunohistochemistry (IHC) was carried out on tissue microarray sections from 45 patients with resectable cholangiocarcinoma for the detection of 6-sulfoLacNAc+ monocytes (slanMo), BDCA-2+ plasmacytoid dendritic cells (pDC), CD8+ or CD4+T-lymphocytes, CD103+ cells, GATA3+ cells, Toll-like receptor (TLR) 3, 7 and 9-expressing cells as well as programmed cell death protein 1 and programmed cell death ligand 1 positive cells. Data from the IHC staining were analyzed and correlated with clinicopathological and survival data. High expression of TLR7, TLR9, and GATA3 was associated with improved overall survival (OS, Log-rank p < 0.05). In addition, TLR9 was associated with better disease-free survival (Log-rank p < 0.05). In the multivariate Cox proportional-hazards model for OS, the TLR/TLR9/GATA3 score was found to be an independent prognostic factor for OS (“Score 2” vs. “Score 0”: HR 11.17 95% CI 2.27–54.95, p < 0.01).