Generate Optimized Android Layouts Automatically Directly from an Image
<ConstraintLayout layout_width="match_parent" layout_height="match_parent"> <Button id="@+id/signup" layout_width="360dp" layout_height="5dp" constraintLeft_toLeftOf="parent" constraintTop_toBottomOf="@+id/view4" fancy:fb_radius = 10dp fancy:fb_borderWidth = 0dp android:text = "Sign Up" /> ... </ConstraintLayout>
Creating a user interface typically involves a collaboration between a designer, who draws an image of how the user interface should look like, and a developer that implements the design on the desired platform. InferUI aims at automating this process for the developer, by using program synthesis and machine learning to generate the corresponding code automatically.
As a first step, all the UI components and their location on the screen is identified. To achieve, one can either extract this information for an UI prototyping tool (such as Figma, Sketch or AdobeXD) or train state-of-the-art object detection models trained on thousands of existing applications.
Given the components and their screen position we synthesize the latest and most efficient Android ConstraintLayout that renders them according to this specification. To synthesize layout that is natural and likely to be written by a developer, the synthesis is guided by a probabilistic model trained on large corpus of existing layouts. Furthermore, our synthesis ensures that the synthesized layout works well on a range of devices with different screen dimensions, beyond the single example provided as input.Robust Synthesis (OOPSLA'18) Guiding Synthesis (ICLR'20)
<Button id="connect" layout_constraintLeft_toLeftOf="parent" layout_constraintRight_toRightOf="parent" layout_constraintBottom_toTopOf="signin" layout_marginBottom="10dp" ... />
<Button border_radius="10dp" border_width="0dp" text="SIGN UP" ... />
<Button border_radius="20dp" border_width="4dp" text="Login" ... />
In addition to synthesizing layouts we provide a pixel accurate technique for predicting component attributes useful for re-creating the user interface. As an example, for a Button component we can infer from the image that the corner radius is 10dp and it has no border.Learning to Infer Attributes