Background: Previous studies suggested that radiation therapy (RT) for breast cancer (BC) can induce cardiac arrhythmias and conduction disorders. However, the association with mean heart dose and specific cardiac substructures doses was less studied. Materials and methods: We conducted a nested case-control study based on French BC patients, enrolled in the European MEDIRAD-BRACE study (https://clinicaltrials.gov, Identifier: NCT03211442), who underwent three-dimensional conformal radiation therapy (3D-CRT) between 2009 and 2013 and were retrospectively followed until 2019. Cases were incident cases of cardiac arrhythmia. Controls without arrhythmia were selected with propensity-scored matching by age, duration of follow-up, chemotherapy, hypertension, and diabetes (ratio 1:4 or 5). Doses to the whole heart (WH), left and right atria (LA and RA), and left and right ventricles (LV and RV) were obtained after delineation with multi-atlas-based automatic segmentation. Results: The study included 116 patients (21 cases and 95 controls). Mean age at RT was 64 ± 10 years, mean follow-up was 7.0 ± 1.3 years, and mean interval from RT to arrhythmia was 4.3 ± 2.1 years. None of the results on association between arrhythmia and cardiac doses reached statistical significance. However, the proportion of right-sided BC was higher among patients with arrhythmia than among controls (57% vs. 51%, OR = 1.18, p = 0.73). Neither mean WH dose, nor LV, RV, and LA doses were associated with an increased risk of arrhythmia (OR = 1.00, p > 0.90). In contrast, the RA dose was slightly higher for cases compared to controls [interquartile range (0.61-1.46 Gy) vs. (0.49-1.31 Gy), p = 0.44], and a non-significant trend toward a potentially higher risk of arrhythmia with increasing RA dose was observed (OR = 1.19, p = 0.60). Subanalysis according to BC laterality showed that the association with RA dose was reinforced specifically for left-sided BC (OR = 1.76, p = 0.75), while for right-sided BC, the ratio of mean RA/WH doses may better predict arrhythmia (OR = 2.39, p = 0.35). Conclusion: Despite non-significant results, our exploratory investigation on BC patients treated with RT is the first study to suggest that right-sided BC patients and the right atrium irradiation may require special attention regarding the risk of cardiac arrhythmia and conduction disorders. Further studies are needed to expand on this topic.