From Images to Layouts

Generate Optimized Android Layouts Automatically Directly from an Image

InferUI


<ConstraintLayout
  layout_width="match_parent"
  layout_height="match_parent">

  <Button
    id="@+id/signup"
    layout_width="360dp"
    layout_height="5dp"
    constraintLeft_toLeftOf="parent"
    constraintTop_toBottomOf="@+id/view4"
    fancy:fb_radius = 10dp
    fancy:fb_borderWidth = 0dp
    android:text = "Sign Up"
  />

  ...

</ConstraintLayout>
Discover More

How it Works

Helping Developers to be More Productive

Creating a user interface typically involves a collaboration between a designer, who draws an image of how the user interface should look like, and a developer that implements the design on the desired platform. InferUI aims at automating this process for the developer, by using program synthesis and machine learning to generate the corresponding code automatically.

Component Detection

As a first step, all the UI components and their location on the screen is identified. To achieve, one can either extract this information for an UI prototyping tool (such as Figma, Sketch or AdobeXD) or train state-of-the-art object detection models trained on thousands of existing applications.

Layout Synthesis

Given the components and their screen position we synthesize the latest and most efficient Android ConstraintLayout that renders them according to this specification. To synthesize layout that is natural and likely to be written by a developer, the synthesis is guided by a probabilistic model trained on large corpus of existing layouts. Furthermore, our synthesis ensures that the synthesized layout works well on a range of devices with different screen dimensions, beyond the single example provided as input.

Robust Synthesis (OOPSLA'18) Guiding Synthesis (ICLR'20) 

<Button
  id="connect"
  layout_constraintLeft_toLeftOf="parent"
  layout_constraintRight_toRightOf="parent"
  layout_constraintBottom_toTopOf="signin"
  layout_marginBottom="10dp"
  ...
/>

<Button
  border_radius="10dp"
  border_width="0dp"
  text="SIGN UP"
  ...
/>

<Button
  border_radius="20dp"
  border_width="4dp"
  text="Login"
  ...
/>

Attribute Prediction

In addition to synthesizing layouts we provide a pixel accurate technique for predicting component attributes useful for re-creating the user interface. As an example, for a Button component we can infer from the image that the corner radius is 10dp and it has no border.

Learning to Infer Attributes
To Top