In this demonstration, Digit starts out knowing there is trash on the floor and bins are used for recycling/trash. We use a voice command “clean up this mess” to have Digit help us. Digit hears the command and uses an LLM to interpret how best to achieve the stated goal with its existing physical capabilities. At no point is Digit instructed on how to clean or what a mess is. This is an example of bridging the conversational nature of Chat GPT and other LLMS to generate real-world physical action.
————————————————————————-
At Agility, we make robots that are made for work. Our robot Digit works alongside us in spaces designed for people. Digit handles the boring and repetitive tasks that are meant for a machine, which allows companies and their people to focus on the work that requires the human element.
it knows that there are several items on
the floor with semantic information
hey digit clean up this mess
when I walk on screen I issue a voice
command to digit which it recognizes
based on hearing its name
at this point digit makes a prompt to an
llm that solves for what it can do to
satisfy this voice command given its
physical intelligence around how it can
interact with the world
The Prompt is specially constructed so
that digit can interpret the response of
the llm as a set of actions it can take
in the world
the command is interpreted to mean that
digits should pick up the trash on the
floor and put it in the bins
this interpretation is not directly
encoded in any part of the prompter
software digits llm generates this from
its own semantic interpretations of a
mass trash bins and clean
once this response is decoded the digit
begins performing the actions it has
determined will satisfy the command
in other situations digits llm can
decide to do other actions like move
around or gestures such as waving
the entire demo shown here from concept
to execution was completed within two
weeks using an unmodified digit platform
to recap what we just saw after hearing
a voice command digit successfully moved
all the trash on the floor into the
appropriate bins for each item digits
started out knowing that there are
different types of items and different
bins used for recycling or Garbage but
at no point is digit instructed on how
to clean or what a mess is digit uses an
llm to interpret how best to achieve
this goal with its physical capabilities