Background The aim of the present study was to assess the efficacy of concurrent chemoradiotherapy (CRT) or radiotherapy alone (RT-alone) in elderly patients with esophageal squamous cell carcinoma (ESCC). Methods The clinical data of patients with ESCC treated with RT-alone or CRT were collected and retrospectively reviewed. The 1-, 3- and 5-year overall survival (OS) rates and the clinical characteristics correlated with survival were analyzed statistically. Propensity score matching (PSM) analyses were used to compensate for differences in baseline characteristics between the CRT and RT-alone groups to confirm the survival difference. Results A total of 729 patients fulfilling the inclusion criteria were reviewed. Diabetes, primary tumor volume (pTV), primary tumor location (pTLo), clinical T stage,(cT) clinical N stage (cN), clinical M stage (cM) and short-term response to RT were independent factors influencing OS (P = 0.002-0.044). The 5-year OS rate was 26.6, 26.0 and 30.1% in the whole cohort, RT-alone and CRT groups, respectively. The survival difference between RT alone and CRT was not significant before or following PSM. Compared with the corresponding subgroups treated with RT alone, CRT significantly benefited patients with diabetes (P = 0.003), cT4 (P = 0.030) and cN0 (P = 0.049), whereas no benefit was identified between CRT and RT alone in the other subgroups, including cT1-3, cN1, cM, pTLo, pTV, age and gender. Conclusions CRT with the current chemotherapy regimens may not improve the survival of elderly ESCC patients compared to RT-alone, except in patients with cT4 stage, cN0 stage or diabetes. However, due to the limitation of the retrospective nature of the current study, further clinical trials are required for confirmation.