Web applications are the public-facing components of information systems, which makes them an easy entry point for various types of attacks. While it is often the responsibility of web developers to implement the proper security controls, it remains a challenge for them to develop a good understanding of the whole attack surface.This paper aims to understand the developers' familiarity with a number of web attack and defense mechanisms. In particular, we conduct two different experiments: First, we employ a questionnaire to understand the perceived attack surface and the types of security controls that are often considered. Second, we design a Capture the Flag challenge that aims to push the participants to discover as many attack points as possible on a given web application. Among several other observations, we find that one third of developers are not aware of the client's ability to intercept and modify all parts of an HTTP request. Moreover, developers' attack awareness focus on a limited set of attacks (such as Crosssite scripting and SQL injection), overlooking a large part of the attack surface.