Abstract-We describe the real-time monitoring infrastructure of the smart-grid pilot on the EPFL campus. We experimentally validate the concept of a real-time state-estimation for a 20 kV active distribution network. We designed and put into operation the whole infrastructure composed by the following main elements: (1) dedicated PMUs connected on the medium-voltage side of the network secondary substations by means of specific current/voltage transducers; (2) a dedicated communication network engineered to support stringent time limits and (3) an innovative state estimation process for real-time monitoring that incorporates phasor-data concentration and state estimation processes. Special care was taken to make the whole chain resilient to cyber-attacks, equipment failures and power outages. The achieved latency is within 65ms. The refresh rate of the estimated state is 20ms. The real-time visualization of the state estimator output is made publicly available, as well as the historical data (PMU measurements and estimated states). To the best of our knowledge, the work presented here is the first operational system that provides low-latency real-time stateestimation by using PMU measurements of a real active distribution network.

We compute bounds on end-to-end worst-case latency and on nodal backlog size for a per-class deterministic network that implements Credit Based Shaper (CBS) and Asynchronous Traffic Shaping (ATS), as proposed by the Time-Sensitive Networking (TSN) standardization group. ATS is an implementation of the Interleaved Regulator, which reshapes traffic in the network before admitting it into a CBS buffer, thus avoiding burstiness cascades. Due to the interleaved regulator, traffic is reshaped at every switch, which allows for the computation of explicit delay and backlog bounds. Furthermore, we obtain a novel, tight per-flow bound for the response time of CBS, when the input is regulated, which is smaller than existing network calculus bounds. We also compute a per-flow bound on the response time of the interleaved regulator. Based on all the above results, we compute bounds on the per-class backlogs. Then, we use the newly computed delay bounds along with recent results on interleaved regulators from literature to derive tight end-to-end latency bounds and show that these are less than the sums of per-switch delay bounds.

Abstract-We consider real-time control systems that consist of a controller that computes and sends setpoints to be implemented in physical processes through process agents. We focus on systems that use commercial off-the-shelf hardware and software components. Setpoints of these systems have strict real-time constraints: Implementing a setpoint after its deadline, or not receiving setpoints within a deadline, can cause failure. In this paper, we address delay faults: faults that cause setpoints to violate their real-time constraints. We present Axo, a fault-tolerance protocol that guarantees safety and improves availability for a class of such systems that exhibit two main properties: the setpoints must have a known validity horizon, and process agents must be capable of handling duplicate setpoints. To reason about delay faults, and consequently design Axo, we present an abstraction of a controller; the abstraction applies to a wide range of real-time control systems. We prove guarantees of safety and availability. Finally, we present an implementation of Axo and the results of the tests performed with Commelec, a real-time control system for electric grids.

Abstract-Real-time control systems (RTCSs) tolerate delay and crash faults by replicating the controller. Each replica computes and issues setpoints to actuators over a network that might drop or delay messages. Hence, the actuators might receive an inconsistent set of setpoints. Such inconsistency is avoided either by having a single primary replica compute and issue setpoints (in passive replication) or a consensus algorithm select one sendingreplica (in active replication). However, due to the impossibility of a perfect failure-detector, passive-replication schemes can have multiple primaries, causing inconsistency, especially in the presence of intermittent delay faults. Furthermore, the impossibility of bounded-latency consensus causes both schemes to have poor real-time performance. We identified three properties of RTCSs that enable active-replication schemes to agree on the measurements before computing, instead of using traditional consensus. As all computing replicas compute with the same state, the resulting setpoints are guaranteed to be consistent. We present the design of Quarts, an agreement solution for active replication that guarantees consistency and bounded latency-overhead. We prove the guarantees and compare the performance of Quarts with existing solutions through simulation. We show that Quarts provides an availability higher than existing solutions, and that the availability improvement is up to 10x with two replicas.

Abstract-Reliable packet delivery within stringent delay constraints is of primal importance to industrial processes with hard real-time constraints, such as electrical grid monitoring. Because retransmission and coding techniques counteract the delay requirements, reliability is achieved through replication over multiple fail-independent paths. Existing solutions such as parallel redundancy protocol (PRP) replicate all packets at the MAC layer over parallel paths. PRP works best in local area networks, e.g., sub-station networks. However, it is not viable for IP layer wide area networks which are a part of emerging smart grids. Such a limitation on scalability, coupled with lack of security, and diagnostic inability, renders it unsuitable for reliable data delivery in smart grids. To address this issue, we present a transport-layer design: IP parallel redundancy protocol (iPRP). Designing iPRP poses non-trivial challenges in the form of selective packet replication, soft-state and multicast support. Besides unicast, iPRP supports multicast, which is widely using in smart grid networks. It duplicates only time-critical UDP traffic. iPRP only requires a simple software installation on the end-devices. No other modification to the existing monitoring application, end-device operating system or intermediate network devices is needed. iPRP has a set of diagnostic tools for network debugging. With our implementation of iPRP in Linux, we show that iPRP supports multiple flows with minimal processing and delay overhead. It is being installed in our campus smart grid network and is publicly available. I. INTRODUCTIONSpecific time-critical applications (found for example in electrical networks) have such strict communication-delay constraints that retransmissions following packet loss can be both detrimental and superfluous. In smart grids, critical control applications require reliable information about the network state in quasi-real time, within hard delay constraints of the order of approximately 10 ms. Measurements are streamed periodically (every 20 ms for 50 Hz systems) by phasor measurement units (PMUs) to phasor data concentrators (PDCs). In such settings, retransmissions can introduce delays for successive, more recent data that in any case supersede older ones. Also, IP multicast is typically used for delivering the measurements to several PDCs. Hence, UDP is preferred over TCP, despite its best-effort delivery approach. Increasing the reliability of such unidirectional (multicast) UDP flows is a major challenge.The parallel redundancy protocol (PRP, IEC standard [1]) was proposed as a solution for deployments inside a local area network (LAN) where there are no routers. Communicating devices need to be connected to two cloned (disjoint) bridged networks. The sender tags MAC frames with a sequence number and replicates it over its two interfaces. The receiver discards redundant frames based on sequence numbers.PRP works well in controlled environments, like a substation LAN, where network setup is entirely up ...

Abstract-Applications performing streaming of phasormeasurement data require low latency and losses from the communication network. Traditionally, such requirements are realized through wired infrastructure. Recently, wireless infrastructure has gained attention due to its low-cost and ease of deployment, but its poor quality-of-service is a strong deterrent for use in mission-critical applications. Recent studies have used measurements to explore the use of packet replication over redundant Wi-Fi paths, for obtaining the desired loss performance without hampering the end-to-end latency. However, these studies are done in a controlled, laboratory environment and do not reflect the real, in-field performance. In this paper, we perform extensive measurements using two co-located directional Wi-Fi links in a real-life setting, to experimentally validate the use of packet replication over Wi-Fi for streaming phasor data. In the setting that we evaluated, we find that the two channels are not fail-independent but the performance achieved with replication is very close to what it would be if they were to be independent. From the loss and latency statistics after replication, we conclude that replicating the phasor data over redundant Wi-Fi paths is a viable option for achieving the desired quality-of-service.

Abstract-Reliable packet delivery within stringent delayconstraints is of paramount importance to mission-critical computer applications with hard real-time constraints. Because retransmission and coding techniques counteract the delay requirements, reliability may be achieved through replication over multiple fail-independent paths. Existing solutions, such as the parallel redundancy protocol (PRP), replicate all packets at the MAC layer over parallel paths. PRP works best in local area networks. However, it is not viable for IP networks that are a key element of emerging mission-critical systems. This limitation, coupled with diagnostic inability and lack of security, renders PRP unsuitable for reliable data-delivery in these IP networks. To address this issue, we present a transport-layer solution: the IP parallel redundancy protocol (iPRP). Designing iPRP poses non-trivial challenges in the form of selective packetreplication, soft-state and multicast support. iPRP replicates only time-critical unicast or multicast UDP traffic. iPRP requires no modifications to the existing monitoring application, end-device operating system or to the intermediate network devices. It only requires a simple software installation on the end-devices. iPRP has a set of diagnostic tools for network debugging. With our implementation of iPRP in Linux, we show that iPRP supports multiple flows with minimal processing-and-delay overhead. It is being installed in our campus smart-grid network and is publicly available. I. INTRODUCTIONSpecific mission-critical computer applications have hard delay-constraints. Failure to satisfy these constraints can result in economic losses or, even worse, human lives can be endangered in cases when these failures affect safety mechanisms. Notable examples of such applications (often built on top of cyber-physical systems) are process-control and emergencymanagement applications in the oil and gas industry Reliable and timely packet delivery, even in the order of 10 ms, is of utmost importance in satisfying the hard-delay constraints. The classic approaches to reliable communication through coding and retransmission are not compatible with the hard delay-constraints. An alternative is to achieve reliability through replication over multiple fail-independent paths, which is the focus of this paper. More precisely, we present a solution for packet-replication over multiple paths in IP networks. Indeed, as we discuss next, existing solutions apply to MAClayer networks and cannot be transposed to IP networks that are a requirement for many of the aforementioned applications.

We consider cyber-physical systems (CPSs) comprising a central controller that might be replicated for highreliability, and one or more process agents. The controller receives measurements from process agents, causing it to compute and issue setpoints that are sent back to process agents. The implementation of these setpoints causes a change in the state of the controlled physical process, and the new state is communicated to the controllers through resulting measurements. To ensure correct operation, the process agents must implement only those setpoints that were caused by their most recent measurements. However, in the presence of replication of the controller, network or computation delays, setpoints and measurements do not necessarily succeed in causing the intended behavior. To capture the dependencies among events associated with measurements and setpoints, we introduce the intentionality relation among such events in a CPS and illustrate its differences with respect to the happened-before relation. We propose a mechanism, intentionality clocks, and the design of controllers and process agents that can be used to guarantee the strong clock-consistency condition under the intentionality relation. Moreover, we prove that our design ensures correct operation despite crash, delay, and network faults. We also demonstrate the practical application of our abstraction through an illustration with a real-world CPS for electrical vehicles.