Mapping the expansion of Google's serving infrastructure

Abstract

Modern content-distribution networks both provide bulk content and act as "serving infrastructure" for web services in order to reduce user-perceived latency. Serving infrastructures such as Google's are now critical to the online economy, making it imperative to understand their size, geographic distribution, and growth strategies. To this end, we develop techniques that enumerate IP addresses of servers in these infrastructures, find their geographic location, and identify the association between clients and clusters of servers. While general techniques for server enumeration and geolocation can exhibit large error, our techniques exploit the design and mechanisms of serving infrastructure to improve accuracy. We use the EDNS-client-subnet DNS extension to measure which clients a service maps to which of its serving sites. We devise a novel technique that uses this mapping to geolocate servers by combining noisy information about client locations with speed-of-light constraints. We demonstrate that this technique substantially improves geolocation accuracy relative to existing approaches. We also cluster server IP addresses into physical sites by measuring RTTs and adapting the cluster thresholds dynamically. Google's serving infrastructure has grown dramatically in the ten months, and we use our methods to chart its growth and understand its content serving strategy. We find that the number of Google serving sites has increased more than sevenfold, and most of the growth has occurred by placing servers in large and small ISPs across the world, not by expanding Google's backbone.

On the benefits of using a large IXP as an internet vantage point

“…The end effect of such developments is a clear trend towards more heterogeneous eyeball ISP networks by virtue of such ASes hosting more servers from an increasing number of interested third-party networks. In view of similar announcements from key companies such as Google [51,25,34], Amazon [2], or Facebook [4], the challenges of studying, leave alone controlling, such increasingly intertwined networks and traffic are quickly becoming daunting. As an example, consider a large Web hosting company (AS36351), for which we identified more than 40K server IPs belonging to a total more than 350 different organizations (highlighted in Figure 6(c) with a square).…”

Cloud storage service benchmarking: Methodologies and experimentations

“…Secondly, each server hostname is resolved querying more than 1,800 open DNS servers scattered around the globe (located in more than 100 countries, and involving 500 ISPs). This allows us to i) discover all front-end IP addresses; and ii) check if loadbalancing techniques are implemented to route customers from different places to different IP addresses [7]. At last, geographical localization of server IPs is performed.…”

Back-Office Web Traffic on The Internet

“…Back-end servers either host the content in data centers or are closer to the origin content server, depending on the CDN's deployment and operation strategy [46,35,57,37,18,5,3]. If the front-end does not have a requested object available locally, it fetches the object from another front-end, a back-end, or the origin server.…”

“…This data set contains a list of IPs, i.e., servers, that are responsive to GET requests on port 80 (HTTP) and SSL services on port 443 (HTTPS), spanning the time period from October 2013 to January 2014. In addition, we also make use of the data made public by the authors of [18,56] that disclose the set of IPs used by the Google infrastructure. When combining Google IPs with one of our packet traces, we use the snapshot of the Google IPs that corresponds to the last day of the trace.…”

“…3 We then check the origin AS of the corresponding subnet, and manually inspect if the WHOIS information reveals that the address is registered to a known CP or CDN. Since this check is not sufficient to reveal cases in which front-end servers and caches are embedded in ISP networks and use the address space registered to those networks, e.g., Akamai and Google Global Cache, we also check for DNS host-names and use the techniques reported in [18,56] to attribute IPs to content providers and CDNs.…”