Many adaptive tracking-by-detection methods have been proposed to track object with slowly changing appearance. However, most of those methods are designed for single object tracking. This paper proposes a method for adaptively tracking multiple objects based on a modified structured Support Vector Machine (SVM). The method utilizes the inter-object constraints and the layout information, which are frequently present in multiple object tracking. Moreover, our approach detects the existences of objects by adding binary constraints in the structured SVM formulation, and therefore can handle frequent occlusions in multiple object tracking. In contrast, the original structured SVM assumes continual existence of the tracked object, making it susceptible to drift. Experimental results show the proposed method works better than existing adaptive trackingby-detection method, as well as non-adaptive association-based multiple object tracking approach.