A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor

Brändli, Christian; Berner, Raphael; Yang, Minhao; Liu, Shih-Chii; Delbrück, Tobias

doi:10.1109/jssc.2014.2342715

Cited by 845 publications

(716 citation statements)

References 14 publications

Supporting

Mentioning

658

Contrasting

Unclassified

Order By: Relevance

“…The DAVIS [8] is a novel vision sensor combining a conventional frame-based camera and a DVS in the same array of pixels. The global-shutter frames provide absolute illumination on demand, whereas the event sensor responds asynchronously to pixel-level brightness changes, independently for each pixel.…”

Section: The Dynamic and Active-pixel Vision Sensormentioning

confidence: 99%

See 1 more Smart Citation

Low-latency visual odometry using event-based feature tracks

Kueng

Mueggler

Gallego

et al. 2016

2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

154

163

View full text Add to dashboard Cite

New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of eventbased sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. Low-Latency Visual Odometry using Event-based Feature Tracks Beat Kueng, Elias Mueggler, Guillermo Gallego and Davide ScaramuzzaAbstract-New vision sensors, such as the Dynamic and Active-pixel Vision sensor (DAVIS), incorporate a conventional camera and an event-based sensor in the same pixel array. These sensors have great potential for robotics because they allow us to combine the benefits of conventional cameras with those of event-based sensors: low latency, high temporal resolution, and high dynamic range. However, new algorithms are required to exploit the sensor characteristics and cope with its unconventional output, which consists of a stream of asynchronous brightness changes (called "events") and synchronous grayscale frames. In this paper, we present a lowlatency visual odometry algorithm for the DAVIS sensor using event-based feature tracks. Features are first detected in the grayscale frames and then tracked asynchronously using the stream of events. The features are then fed to an event-based visual odometry algorithm that tightly interleaves robust pose optimization and probabilistic mapping. We show that our method successfully tracks the 6-DOF motion of the sensor in natural scenes. This is the first work on event-based visual odometry with the DAVIS sensor using feature tracks. MULTIMEDIA MATERIALA video attachment to this work is available on the authors' webpage.

show abstract

Section: The Dynamic and Active-pixel Vision Sensormentioning

confidence: 99%

“…The Dynamic and Active-pixel Vision Sensor (DAVIS) [8] has been introduced very recently (2014). It is an integrated sensor comprising a conventional frame-based camera and an asynchronous event sensor.…”

Section: Introductionmentioning

confidence: 99%

Low-latency visual odometry using event-based feature tracks

Kueng

Mueggler

Gallego

et al. 2016

2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

154

163

View full text Add to dashboard Cite

show abstract

“…The former strategy minimizes transmission latency at the cost of higher loss rates, which limit maximum throughput to 18% of output channel's capacity; the latter offers 95% capacity with higher latencies of a few % of the inter-spike intervals [132]. If arbitration is only performed in one dimension of the array, the total delay of transfers can be minimized by using burst-mode schemes like [133,134], which allow transmitting selected row/column-events at the time the next cluster is being selected. Since N pix > 256 pixels are not foreseen at this stage of research, and considering substantial event clustering is expected in one of the dimensions of the array for the target applications of Sec.…”

Section: Motivation and Design Proposalmentioning

confidence: 99%

Low-Power CMOS Digital Pixel Imagers for High-Speed Uncooled PbSe IR Applications

Margarit¹

2017

Springer Theses

View full text Add to dashboard Cite

Certifiquen ResumLa present tesi doctoral descriu la recerca i el desenvolupament d'una nova tecnologia monolítica d'imatgeria infraroja de longitud d'ona mitja (MWIR), no refrigerada i de baix cost, per a usos industrials d'alta velocitat. El treball pren el relleu dels últims avenços assolits pel soci industrial NIT S.L. en el camp dels detectors MWIR de PbSe depositats en fase vapor (VPD), afegint-hi coneixement fonamental en la investigació de noves tècniques de disseny de circuits VLSI analògics i mixtes pel desenvolupament del dispositiu integrat de lectura unit al detector pixelat.Es parteix de la hipòtesi que, mitjançant l'ús de les esmentades tècniques de disseny, les tecnologies CMOS estàndard satisfan tots els requeriments operacionals del detector VPD PbSe respecte a connectivitat, fiabilitat, funcionalitat i escalabilitat per integrar de forma econòmica el dispositiu. La càmera PbSe-CMOS resultant ha de consumir molt baixa potència, operar a freqüències de kHz, exhibir bona uniformitat, i encabir els píxels actius CMOS de lectura en el pitch compacte del pla focal de la imatge, tot atenent a les particulars característiques del detector: altes relacions de corrent d'obscuritat a senyal, elevats valors de capacitat paràsita a l'entrada i dispersions importants en el procés de fabricació.Amb la finalitat de complir amb els requisits previs, es proposen arquitectures de sensors de visió de molt baix acoblament interpíxel basades en l'ús d'una matriu de pla focal (FPA) de píxels actius exclusivament digitals. Cada píxel sensor digital (DPS) està equipat amb mòduls de comunicació d'alta velocitat, autopolarització, cancel·lació de l'offset, conversió analògica-digital (ADC) i correcció del soroll de patró fixe (FPN). El consum en cada cel·la es minimitza fent un ús exhaustiu del MOSFET operant en subllindar.L'objectiu últim és potenciar la integració de les tecnologies de sensat d'imatge infraroja (IR) basades en PbSe per expandir-ne el seu ús, no només a diferents escenaris, sinó també en diferents estadis de maduresa de la integració PbSe-CMOS. En aquest sentit, es proposa investigar un conjunt complet de blocs funcionals distribuïts en dos enfocs paral·lels: viii• Dispositius d'imatgeria MWIR "Smart" basats en frames utilitzant noves topologies de circuit DPS amb correcció de l'FPN en guany i offset. Aquesta línia de recerca exprimeix el pitch del detector per oferir una programabilitat completament digital a nivell de píxel i plena funcionalitat amb compensació de la capacitat paràsita d'entrada i memòria interna de fotograma.• Dispositius de visió MWIR "Compact"-pitch "frame-free" en base a un novedós esquema d'integració analògica en el DPS i diferenciació temporal configurable, combinats amb protocols de comunicació asíncrons dins del pla focal. Aquesta estratègia es concep per permetre una alta compactació del pitch i un increment de la velocitat de lectura, mitjançant la supressió del filtrat digital intern i l'assignació dinàmica de l'ample de banda a cada píxel de l'FPA. Aquest treball ha cond...

show abstract

“…An additional advantage of the DVS is its high dynamic range of 120 dB (compared to 60 dB of high quality traditional image sensors). Current research efforts [8] are being carried towards increasing the spatial resolution of the sensor as well as offering the possibility to return the absolute pixel brightness (at standard frame rates) in addition to the events.…”

Section: Dynamic Vision Sensor (Dvs)mentioning

confidence: 99%

Continuous-Time Trajectory Estimation for Event-based Vision Sensors

Mueggler¹,

Gallego²,

Scaramuzza³

2015

Robotics: Science and Systems XI

View full text Add to dashboard Cite

Abstract-Event-based vision sensors, such as the Dynamic Vision Sensor (DVS), do not output a sequence of video frames like standard cameras, but a stream of asynchronous events. An event is triggered when a pixel detects a change of brightness in the scene. An event contains the location, sign, and precise timestamp of the change. The high dynamic range and temporal resolution of the DVS, which is in the order of micro-seconds, make this a very promising sensor for high-speed applications, such as robotics and wearable computing. However, due to the fundamentally different structure of the sensor's output, new algorithms that exploit the high temporal resolution and the asynchronous nature of the sensor are required. In this paper, we address ego-motion estimation for an event-based vision sensor using a continuous-time framework to directly integrate the information conveyed by the sensor. The DVS pose trajectory is approximated by a smooth curve in the space of rigid-body motions using cubic splines and it is optimized according to the observed events. We evaluate our method using datasets acquired from sensor-in-the-loop simulations and onboard a quadrotor performing flips. The results are compared to the ground truth, showing the good performance of the proposed technique.

show abstract

A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision Sensor

Cited by 845 publications

References 14 publications

Low-latency visual odometry using event-based feature tracks

Low-latency visual odometry using event-based feature tracks

Low-Power CMOS Digital Pixel Imagers for High-Speed Uncooled PbSe IR Applications

Continuous-Time Trajectory Estimation for Event-based Vision Sensors

Contact Info

Product

Resources

About