A retrospective cohort study including 112 patients suffering from esophageal squamous cell carcinoma (ESCC) was performed to investigate the expression of B7-H4 in ESCC and determine its association with patient's clinicopathological parameters and survival. Expression levels of B7-H4 on tumor cells and densities of tumor infiltrating lymphocytes (TILs) in the surgical specimens of ESCC tissues were characterized using immunohistochemical assays. Uni- and multivariate analyses were performed to evaluate the prognostic value of B7-H4 expression levels and densities of TILs in tumor sections. Positive B7-H4 immunostaining was observed in 107 of 112 (95.5%) of ESCC tissue sections. We further divided all patients into two major subgroups, a lower B7-H4 expression group with 46 patients and a higher B7-H4 expression group with 66 patients. We found that expression levels of B7-H4 on tumor cells were significantly correlated with patient's gender (P = 0.0288), distant metastasis (P = 0.0500), and TNM stage (P = 0.0258). Moreover, tumor cell B7-H4 expression was inversely correlated with densities of CD3(+) T cells in tumor nest (P = 0.0424) and CD8(+) T cells in tumor stroma (P = 0.0229). The overall survival rate of the patients with higher B7-H4 expression was significantly worse than that of the patients with lower B7-H4 expression (P = 0.0105, Hazard Ratio: 1.854, 95%CI:1.152-2.902). Markers of cell-mediated immune responses such as CD3, CD8, and T-bet were associated with better patient survival. The present study demonstrated that B7-H4 expression in human ESCC is associated with cancer progression, reduced tumor immunosurveillance and worse patient outcomes. B7-H4 can serve as a novel prognostic predictor for human ESCC and a potential target for the immune therapy against this malignancy.