We proposed a framework to detect and quantify local tumor morphological changes due to chemo-radiotherapy (CRT) using Jacobian map and to extract quantitative radiomic features from the Jacobian map to predict the pathologic tumor response in locally advanced esophageal cancer patients. In 20 patients who underwent CRT, a multi-resolution BSpline deformable registration was performed to register the follow-up (post-CRT) CT to the baseline CT image. Jacobian map (J) was computed as the determinant of the gradient of the Deformation Vector Field. Jacobian map measured the ratio of local tumor volume change where J < 1 indicated tumor shrinkage and J > 1 denoted expansion. The tumor was manually delineated and corresponding anatomical landmarks were generated on the baseline and follow-up images. Intensity, texture and geometry features were then extracted from the Jacobian map of the tumor to quantify tumor morphological changes. The importance of each Jacobian feature in predicting pathologic tumor response was evaluated by both univariate and multivariate analysis. We constructed a multivariate prediction model by using a support vector machine (SVM) classifier coupled with a least absolute shrinkage and selection operator (LASSO) for feature selection. The SVM-LASSO model was evaluated using ten-times repeated 10-fold cross-validation (10×10-fold CV). After registration, the average Target Registration Error was 4.30±1.09mm (LR:1.63mm AP:1.59mm SI:3.05mm) indicating registration error was within two voxels and close to 4mm slice thickness. Visually, Jacobian map showed smoothly-varying local shrinkage and expansion regions in a tumor. Quantitatively, the average Median Jacobian was 0.80±0.10 and 1.05±0.15 for responder and non-responder tumors, respectively. These indicated that on average responder tumors had 20% median volume shrinkage while non-responder tumors had 5% median volume expansion. In univariate analysis, Minimum Jacobian (p=0.009, AUC=0.98) and Median Jacobian (p=0.004, AUC=0.95) were the most significant predictors. The SVM-LASSO model achieved the highest accuracy when these two features were selected (Sensitivity=94.4%, Specificity=91.8%, AUC=0.94). Novel features extracted from the Jacobian map quantified local tumor morphological changes using only baseline tumor contour without post-treatment tumor segmentation. The SVM-LASSO model using Median Jacobian and Minimum Jacobian achieved high accuracy in predicting pathologic tumor response. Jacobian map showed great potential for longitudinal evaluation of tumor response.