Census and survey of the visible internet

Abstract

Prior measurement studies of the Internet have explored traffic and topology, but have largely ignored edge hosts. While the number of Internet hosts is very large, and many are hidden behind firewalls or in private address space, there is much to be learned from examining the population of visible hosts, those with public unicast addresses that respond to messages. In this paper we introduce two new approaches to explore the visible Internet. Applying statistical population sampling, we use censuses to walk the entire Internet address space, and surveys to probe frequently a fraction of that space. We then use these tools to evaluate address usage, where we find that only 3.6% of allocated addresses are actually occupied by visible hosts, and that occupancy is unevenly distributed, with a quarter of responsive /24 address blocks (subnets) less than 5% full, and only 9% of blocks more than half full. We show about 34 million addresses are very stable and visible to our probes (about 16% of responsive addresses), and we project from this up to 60 million stable Internet-accessible computers. The remainder of allocated addresses are used intermittently, with a median occupancy of 81 minutes. Finally, we show that many firewalls are visible, measuring significant diversity in the distribution of firewalled block size. To our knowledge, we are the first to take a census of edge hosts in the visible Internet since 1982, to evaluate the accuracy of active probing for address census and survey, and to quantify these aspects of the Internet.

No Need for Black Chambers: Testing TLS in the E-mail Ecosystem at Large

“…RELATED WORK 1) Large-scale Scanning: In 2008 Heidemann et al [35] started an Internet wide discovery scan of edge hosts. They were the first to census the IPv4 address space, but were limited to host discovery, not containing protocol or service specific results.…”

Towards Validation of the Internet Census 2012

“…In [6], Heidemann et al presented a study of the active Internet over the period [2003][2004][2005][2006][2007][2008]. The presented census highlighted anomalies in the un-allocated address space and indicates the percentage of usage for allocated network blocks.…”

“…Active scans on the Internet are certainly an important source of information, as several studies have demonstrated that they can help reveal new kinds of vulnerabilities, monitor deployment of mitigation, and highlight hidden distributed ecosystems [5,6,7,8,11]. However, the IC results were published anonymously, and the methodology only partially described, which, as pointed out by the Cooperative Association for Internet Data Analysis (CAIDA) [1], leads to some important questions, such as: how does one know that the IC scan actually happened, and if it did, how does one know that the resulting data is correct?…”

Enabling High-Performance Internet-Wide Measurements on Windows

“…aim to shed light on its structure [2], [13], user behavior [14], [17], host availability [6], [11], and web content [7], [10]; however, accurately capturing Internet-wide metrics has long remained a challenging research problem. The main tradeoff involves the amount of available hardware and the delay the user is willing to tolerate.…”

“…Sending such a large number of small packets presents a problem for Winsock and limits the duration of the measurement to days instead of minutes. A slightly different, but related, measurement goal that requires high pps sending rates is discovery of open services using horizontal scanning [1], [3], [6], [11], where each IP address in the IANA (3.3B destinations) or BGP (2.1B) space is probed with a packet on a given port. Instead of using months to scan the Internet as in prior work [1], [3], [6], [11], our goal in another ongoing project is to accomplish this activity in several hours/days.…”