A central element of high throughput screens for chemical effect assessment using zebrafish is the assessment and quantification of phenotypic changes. By application of an automated and more unbiased analysis of these changes using image analysis, patterns of phenotypes may be associated with the mode of action (MoA) of the exposure chemical. The aim of our study was to explore to what extent compounds can be grouped according to their anticipated toxicological or pharmacological mode of action using an automated quantitative multi-endpoint zebrafish test. Chemical-response signatures for 30 endpoints, covering phenotypic and functional features, were generated for 25 chemicals assigned to 8 broad MoA classes. Unsupervised clustering of the profiling data demonstrated that chemicals were partially grouped by their main MoA. Analysis with a supervised clustering technique such as a partial least squares discriminant analysis (PLS-DA) allowed to identify markers with a strong potential to discriminate between MoAs such as mandibular arch malformation observed for compounds interfering with retinoic acid signaling. The capacity for discriminating MoAs was also benchmarked to an available battery of in vitro toxicity data obtained from ToxCast library indicating a partially similar performance. Further, we discussed to which extent the collected dataset indicated indeed differences for compounds with presumably similar MoA or whether other factors such as toxicokinetic differences could have an important impact on the determined response patterns.