Lifting freehand concept sketches into 3D

Abstract

We present the first algorithm capable of automatically lifting real-world, vector-format, industrial design sketches into 3D. Targeting real-world sketches raises numerous challenges due to inaccuracies, use of overdrawn strokes, and construction lines. In particular, while construction lines convey important 3D information, they add significant clutter and introduce multiple accidental 2D intersections. Our algorithm exploits the geometric cues provided by the construction lines and lifts them to 3D by computing their intended 3D intersections and depths. Once lifted to 3D, these lines provide valuable geometric constraints that we leverage to infer the 3D shape of other artist drawn strokes. The core challenge we address is inferring the 3D connectivity of construction and other lines from their 2D projections by separating 2D intersections into 3D intersections and accidental occlusions. We efficiently address this complex combinatorial problem using a dedicated search algorithm that leverages observations about designer drawing pREFERENCES, and uses those to explore only the most likely solutions of the 3D intersection detection problem. We demonstrate that our separator outputs are of comparable quality to human annotations, and that the 3D structures we recover enable a range of design editing and visualization applications, including novel view synthesis and 3D-aware scaling of the depicted shape.

Sketch-Based Modeling in Mechanical Engineering Design: Current Status and Opportunities

“…Azariadis et al built on the Cross-Section Criterion of Ros and Thomas [RT05] to check the realizability of the sketched pictorial drawings [KAS11] [AKS13] and a recent contribution on finding hidden lines in a natural line-drawing is due to Bonnici and Camilleri [BC16]. Other recent contributions use alternative inflation methods which do not rely exclusively on projective geometry, but on perceptual rules [GHL20]. Annotations are used extensively in industrial settings to document the design of products.…”

Free2CAD

“…In the last two decades, a number of algorithms and user interfaces have been proposed to lift 2D drawings to 3D [Bae et al 2008;Gryaditskaya et al 2020;Li et al 2017;Lipson and Shpitalni 1996;Orbay and Kara 2012;Schmidt et al 2009;Xu et al 2014]. Most of these algorithms cast the problem of 3D reconstruction as an optimization where the depths of the pen stroke are treated as variables, and geometric relationships between the strokes -such as parallelism and orthogonality -act as constraints.…”

StrokeStrip

“…Sketch Processing. User demand has led to the emergence of numerous methods and tools for processing of vector-format artistdrawn sketches, including colorization [Adobe 2020; Orzan et al 2008], shading [Finch et al 2011;Shao et al 2012], style transfer [Freeman et al 2003], and editing [Igarashi et al 2005], and has lead to increased use of free-hand vector sketches as inputs for high-level tasks such as 3D modeling [Bessmeltsev et al 2016;Gryaditskaya et al 2020;Lipson and Shpitalni 1996;Xu et al 2014]. These methods typically target clean drawings where each stroke corresponds to a complete meaningful geometric curve and cannot be applied as-is to typical raw artist drawings which depict such curves using clusters of overdrawn strokes [Liu et al 2018[Liu et al , 2015Yan et al 2020]; thus processing overdrawn sketches requires consolidating them first.…”