Objective To evaluate the prevalence of seven social factors using physician notes as compared to claims and structured electronic health records (EHRs) data and the resulting association with 30‐day readmissions. Study Setting A multihospital academic health system in southeastern Massachusetts. Study Design An observational study of 49,319 patients with cardiovascular disease admitted from January 1, 2011, to December 31, 2013, using multivariable logistic regression to adjust for patient characteristics. Data Collection/Extraction Methods All‐payer claims, EHR data, and physician notes extracted from a centralized clinical registry. Principal Findings All seven social characteristics were identified at the highest rates in physician notes. For example, we identified 14,872 patient admissions with poor social support in physician notes, increasing the prevalence from 0.4 percent using ICD‐9 codes and structured EHR data to 16.0 percent. Compared to an 18.6 percent baseline readmission rate, risk‐adjusted analysis showed higher readmission risk for patients with housing instability (readmission rate 24.5 percent; p < .001), depression (20.6 percent; p < .001), drug abuse (20.2 percent; p = .01), and poor social support (20.0 percent; p = .01). Conclusions The seven social risk factors studied are substantially more prevalent than represented in administrative data. Automated methods for analyzing physician notes may enable better identification of patients with social needs.