Merge pull request #30 from Roboy/devel

midterm WS17/18
Roboy · Jan 25, 2018 · 59f6f99 · 59f6f99
2 parents 356f811 + fad53e7
commit 59f6f99
Show file tree

Hide file tree

Showing 734 changed files with 19,690 additions and 3,353 deletions.
diff --git a/.gitignore b/.gitignore
@@ -14,4 +14,6 @@ local.properties
 *.idea
 
 
-.target
+.target
+
+resources/word2vec_toy_data_and_model
diff --git a/.gitmodules b/.gitmodules
@@ -0,0 +1,3 @@
+[submodule "roboy_parser"]
+	path = roboy_parser
+	url = git://github.com/Roboy/roboy_parser.git
diff --git a/README.md b/README.md
@@ -1,19 +1,19 @@
 # DialogSystem
 [![Documentation Status](https://readthedocs.org/projects/roboydialog/badge/?version=master)](http://roboydialog.readthedocs.io/en/master/?badge=master)
 
-## What is it?
+## What is it 
 
 This repository contains a dialog system developed for the humanoid robot Roboy (roboy.org). 
 
-## How does it work?
+## How does it work
 
 The basic NLP architecture is designed as a pipeline. An input device (derived from de.roboy.io.InputDevice) is producing text, which is passed to a variety of linguistic analyzers (derived from de.roboy.linguistics.sentenceanalysis.Analyzer). This currently consists of a Tokenizer and a POS tagger (both in de.roboy.linguistics.sentenceanalysis.SentenceAnalyzer) but could in the future be accompanied by named entity recognition, a syntactical and semantical analysis, an interpretation of the sentence type or other tools. The results of all these linguistics analyzers are collected together with the original text (de.roboy.linguistics.sentenceanalysis.Interpretation) and passed on to a state machine (states are derived from de.roboy.dialog.personality.states.State) within a personality class (derived from de.roboy.dialog.personality.Personality) that decides how to react to the utterance. In the future, the intentions (de.roboy.logic.Intention) determined by the state machine will then formulated into proper sentences or other actions (de.roboy.dialog.action.Action) by a module called Verbalizer. Currently, these actions are still directly created in the personality class. Finally, the created actions are sent to the corresponding output device (de.roboy.io.OutputDevice).
 
 There are interfaces for each step in the processing pipeline to enable an easy exchange of elements. The goal would be to easily exchange personalities based on the occasion.
 
 The implementation of the pipeline is in Java. Integrations with tools in other languages, like C++ RealSense stuff, should be wrapped in a module in the pipeline.
 
-## How to run it?
+## How to run it
 
 The repository contains a project that can be readily imported into Eclipse. Best use the EGit Eclipse Plugin to check it out. The code can be executed by running de.roboy.dialog.DialogSystem.
 
@@ -28,7 +28,7 @@ To execute:
 mvn exec:java -Dexec.mainClass="roboy.dialog.DialogSystem"
 ```
 
-## How to extend it?
+## How to extend it
 
 Pick the corresponding interface, depending on which part of the system you want to extend. If you want to add new devices go for the input or output device interfaces. If you want to extend the linguistic analysis implement the Analyzer interface or extend the SentenceAnalyzer class. If you are happy with input, linguistics and output and just want to create more dialog, implement the Personality interface.
 

diff --git a/config.properties b/config.properties
@@ -1 +1,4 @@
-ROS_HOSTNAME: 10.183.122.142
+ROS_HOSTNAME: 127.0.0.1
+PARSER_PORT: 5000
+DEMO_GUI: false
+PERSONALITY_FILE: "resources/personalityFiles/ExamplePersonality.json"
diff --git a/docs/Architecture/0_deployment.rst b/docs/Architecture/0_deployment.rst
diff --git a/docs/Architecture/1_sequence.rst b/docs/Architecture/1_sequence.rst
diff --git a/docs/Architecture/2_buildingblock.rst b/docs/Architecture/2_buildingblock.rst
diff --git a/docs/Architecture/images/buildingblock.png b/docs/Architecture/images/buildingblock.png
diff --git a/docs/Architecture/images/deployment.png b/docs/Architecture/images/deployment.png
diff --git a/docs/Architecture/images/sequence.png b/docs/Architecture/images/sequence.png
diff --git a/docs/Doxyfile b/docs/Doxyfile
diff --git a/docs/ScopeContext/6_API.rst b/docs/ScopeContext/6_API.rst
diff --git a/docs/Usage/0_installation.rst b/docs/Usage/0_installation.rst
diff --git a/docs/Usage/1_getting_started.rst b/docs/Usage/1_getting_started.rst
diff --git a/docs/Usage/2_personality_and_states.rst b/docs/Usage/2_personality_and_states.rst
@@ -12,21 +12,41 @@ During one run-through, the Dialog System uses a single Personality instance (``
 
 The current primary personality is the SmallTalkPersonality (``de.roboy.dialog.personality.SmallTalkPersonality``).
 
-State
------
+A new personality (``roboy.newDialog.StateBasedPersonality``) is currently being implemented. Similarly to the SmallTalkPersonality, it is built on top of a state machine. The new personality is designed to be more generic one and allows to load the behaviour from a personality file. The file stores a representation of the state machine. Additionally, it is still possible to define the dialog structure directly from code (as it was done in previous implementations).
+
+Legacy State
+------------
 
 A state's activity can be divided into two stages. When the state is entered, the initial action from the ``act()`` method is carried out, which is expected to trigger a response from the person. After Roboy has received and analyzed the response, the ``react()`` method completes the current state's actions and Roboy moves on to the next state.
 
 The AbstractBooleanState describes a special case where the state's reaction depends on whether the ``act()`` method resulted in successful interaction. States which implement AbstractBooleanState can respond differently move on into different stages according to their ``determineSuccess()`` method.
 
-For example, the intial action of ``de.roboy.dialog.personality.states.IntroductionState`` is to ask the user's name. Then the response is analyzed externally and when the state's determineSuccess() method is called, it checks whether a name was extracted. If this is the case, then the system outputs predefined sentences with the extracted name embedded into them. Otherwise, fallback sentences are used which do not include a name.
+For example, the initial action of ``de.roboy.dialog.personality.states.IntroductionState`` is to ask the user's name. Then the response is analyzed externally and when the state's determineSuccess() method is called, it checks whether a name was extracted. If this is the case, then the system outputs predefined sentences with the extracted name embedded into them. Otherwise, fallback sentences are used which do not include a name.
+
+New State
+---------
+
+Currently, we are improving the state system. Old state implementations will be replaced with newer ones. The functionality of the AbstractBooleanState will be improved to allow arbitrary conditional transitions in every new state. Nested states will be replaced with the fallback concept.
+
+A state's activity can be divided into two stages. When the state is entered, the initial action from the ``act()`` method is carried out, which is expected to trigger a response from the person. After Roboy has received and analyzed the response, the ``react()`` method completes the current state's actions and Roboy picks a transition to the next state defined by the ``getNextState()`` method of the current state.
+
+It is possible to remain in the same state for many cycles. In this case the ``getNextState()`` method just returns a reference to the current state (``this``) and the ``act()`` and ``react()`` methods are carried out again.
+
+A state can have any number of transitions to other states. Every transition has a name (like "next" or "errorState"). When changing states, the following state can be selected based on internal conditions of the current state. For example, a state can expect a "yes/no" answer and have tree outgoing transitions: "gotYes", "gotNo" and "askAgain" (if the reply is not "yes/no"). 
+
+When designing a new state, only the transition names are defined. The following states are defined in the personality file later. At run time the state machine loads the file and initializes the transitions to point it correct states. The destination state can be retrieved by the transition name using ``getTransition(transitionName)``.
+
+A fallback can be attached to a state. In the case this state doesn't know how to react to an utterance, it can return ``null`` from the ``react()`` function. The state machine will query the fallback in this case. This concept helps to decouple the states and reduce the dependencies between them. When implementing the ``react()`` function of a new state, it is sufficient to detect unknown input and return ``null``. 
+
+
+
 
-State machine structure
------------------------
+Legacy State machine structure
+------------------------------
 
-Every state defines at least one successor state, and more complex hierarchies can be realized - for example as a fallback system for cases when a single state cannot respond in a meaningful manner. The following is an example from the documentation of SmallTalkPersonality:
+Every state defines at least one successor state, and more complex hierarchies can be realized - for example as a fallback system for cases when a single state cannot respond in a meaningful manner. The fallback system implemented using nested states in the legacy state machine and will be improved in the newer implementation. The following is an example from the documentation of SmallTalkPersonality:
 
-The current state machine looks like this:
+The current legacy state machine looks like this:
 
 Greeting state
       |
@@ -39,6 +59,6 @@ Greeting state
     |_Segue state
       |_Wild talk state
 
- The Question Randomizer, Question Answering, Segue and Wilk talk states are stacked. If one cannot give an appropriate reaction to the given utterance, the utterance is passed on to the next one. The Wild talk state will always answer.
+ The Question Randomizer, Question Answering, Segue and Wilk talk states are nested. If one cannot give an appropriate reaction to the given utterance, the utterance is passed on to the next one. The Wild talk state will always answer.
 
 
diff --git a/docs/Usage/3_memory.rst b/docs/Usage/3_memory.rst
@@ -25,6 +25,6 @@ Roboy's Dialog System interactions with the Memory module are based on ROS messa
 | CypherMemoryQuery  | For more complex queries (future)                |
 +--------------------+--------------------------------------------------+
 
-The messages received from Memory are in JSON format. To enable flexible high-level handling of Memory information, two classes were created to incorporate the node structures and logic inside the Dialog System. The ``de.roboy.memory.nodes.MemoryNodeModel`` contains the labels, properties and relations in a format which can be directly parsed from and into JSON. For this, Dialog is using the GSON parsing methods which enable direct translation of a JSON String into its respective Java class representation.
+The messages received from Memory are in JSON format. To enable flexible high-level handling of Memory information, two classes were created to incorporate the node structures and logic inside the Dialog System. The ``de.roboy.memory.nodes.MemoryNodeModel`` contains the labels, properties and relationships in a format which can be directly parsed from and into JSON. For this, Dialog is using the GSON parsing methods which enable direct translation of a JSON String into its respective Java class representation.
 
 Methods such as ``getRelation()`` or ``setProperties()`` were implemented to allow intuitive handling of the MemoryNodeModel instances. A separate class, ``de.roboy.memory.nodes.Interlocutor``, encapsulates a MemoryNodeModel and is intended to further ease saving information about the current conversation partner of Roboy. Interlocutor goes one step further by also abstracting the actual calls to memory, such that adding the name of the conversant performs an automatic lookup in the memory with subsequent updating of the person-related information. This is then available in all subsequent interactions, such that Roboy can refrain from asking questions twice, or refer to information he rememberes from earlier conversations.
diff --git a/docs/Usage/4_configuration.rst b/docs/Usage/4_configuration.rst
@@ -30,6 +30,12 @@ Profiles
 |                    | profile includes all restrictions of NOROS and also does not    |
 |                    | call DBPedia.                                                   |
 +--------------------+-----------------------------------------------------------------+
+| MEMORY-ONLY        | To be used during Memory development, when no other ROS services|
+|                    | are running. Only Neo4j-related ROS calls will be made.         |
++--------------------+-----------------------------------------------------------------+
+| DEBUG              | With this setting, DM will run like DEFAULT but not shut down   |
+|                    | when ROS failures are encountered.                              |
++--------------------+-----------------------------------------------------------------+
 
 Extending
 ---------

diff --git a/docs/Usage/5_context.rst b/docs/Usage/5_context.rst
@@ -0,0 +1,55 @@
+*******
+Context
+*******
+
+The goal of Context is to collect information about Roboy's environment and state. This information can be used by the DM classes and also to react upon situations that match certain conditions, such as turning the head of Roboy when the Interlocutor moves.
+
+Architecture
+============
+
+.. figure:: ../images/context.png
+:alt: Context architecture
+
+The Context supports storing data as a ``Value`` or ``ValueHistory``. A ``Value`` only stores the latest data object that was pushed to it. A ``ValueHistory`` stores every value it receives and assigns each a unique key, thus the values can be ordered by their adding time.
+
+
+How to add Values?
+==================
+
+Here we describe how a new Value can be created and added to the Context. Sample implementations can be found inside ``roboy.context.contextObjects`` packge.
+
+1. Consider what type of data will be stored in the Value. For this example, we chose ``String``.
+2. In the ``contextObjects`` directory, create a new class which inherits from the Value class. The final signature should look similar to: ``public class SampleValue extends Value<String>`` (replacing String with your type).
+3. Add the new Value in the main class ``Context.java``:
+    1. In the standard constructor, initialize the Value object and add it to the builder of the value map: ``put(SampleValue.class, sampleValue)``
+    2. Make the value available over the enum ``Values`` within the Context class by adding a new element with your class name and stored data type. For example: ``SAMPLE_VALUE(SampleValue.class, String.class);``
+
+How to add ValueHistories?
+==========================
+
+ValueHistories extend the functionality of Values by storing all data objects sent to them. Over the ``getNLastValues(int n)`` method, a map with several most recent data objects can be retrieved, including their ordering.
+
+Adding a ``ValueHistory`` is very much alike to adding a ``Value``, just make sure to:
+
+1. extend ``ValueHistory<>`` instead of ``Value<>``,
+
+2. in ``Context.java``, add the new object to the Builder of ``valueHistories`` instead of ``values``, and to the enum ``ValueHistories`` instead of ``Values``.
+
+How to add Updaters?
+====================
+
+New values can only be added to the Context over an ``Updater`` instance. Internal updaters can be used by DM classes to actively add new values. External updaters run in separate threads and seek out new values, for example over a ROS connections to the Vision module.
+
+Adding an External Updater
+""""""""""""""""""""""""""
+Currently, the only implementation of an external updater is the ``IntervalUpdater`` abstract class. Usage:
+
+1. Create a class extending ``IntervalUpdater`` and implement its ``update()`` method. It should retrieve the values and finally add them over the ``target.updateValue(value)`` method call.
+2. Add the updater to ``externalUpdaters`` in the ``Context.java`` constructor, setting its ``target`` parameter with the ``Value`` or ``ValueHistory`` object created in the same constructor.
+
+Adding a new Internal Updater
+"""""""""""""""""""""""""""""
+1. Create a class extending InternalUpdater<*targetClass*, *valueType*>. The class and data type of the target ``Value`` or ``ValueHistory`` are the generic parameters for the updater.
+2. A constructor is required for the class, simply match the InternalUpdater constructor and call ``super(target)`` within. An example is in the ``DialogTopicsUpdater`` class.
+3. In the ``Context`` class constructor, initialize the updater and add it to the ``internalUpdaters`` map.
+4. Add an entry to the ``Updaters`` enum, similarly as discussed above for ``Values``.
diff --git a/docs/Usage/6_semantic_parser.rst b/docs/Usage/6_semantic_parser.rst
@@ -0,0 +1,76 @@
+Semantic Parser
+===============
+
+Semantic parser is used to translate text representation into formal language representation. The aim is to be able to process user utterances and react upon them.
+
+roboy_parser is based on `SEMPRE <http://nlp.stanford.edu/software/sempre/`. It is currently being modified to fulfill Roboy Dialog system needs.
+
+
+Installation
+------------
+
+In order to use semantic parser, you need to:
+
+- clone ``roboy_parser`` repository::
+
+	git clone http://github.com/Roboy/roboy_parser
+
+- navigate to created repository::
+
+	cd roboy_parser
+
+- build it::
+
+	ant freebase
+
+- run it::
+
+	./quick_start [options]
+
+Architecture
+------------
+
+Semantic parser is based on the language model and NLP algorithms that then apply rules to the utterance to translate it. Language model consists of:
+- set of grammar rules,
+- lexicon,
+- training dataset.
+
+General architecture can be seen on the diagram below.
+
+.. figure:: ../images/parser.png
+:alt: Semantic parser general architecture
+
+Implementation
+--------------
+
+roboy_parser is a separate Java project and is communicating using ``WebSocket``. Dialog system has a client implemented in ``SemanticParserAnalyzer.java`` class. It is therefore part of Natural Language Understanding unit.
+
+Functionalities
+"""""""""""""""
+
+Roboy parser currently has currently following functionalities:
+
+.. csv-table:: Semantic Parser algorithms used
+  :header: "Functionality", "Software used", "Summary"
+  :widths: 20, 20, 60
+
+  "POS Tags", "OpenNLP", "Tagging tokens as part of speech"
+	"NER Tags", "OpenNLP", "Tool used to tag named entities like PERSON, NUMBER, ORGANIZATION"
+  "Triple extraction", "OpenIE", "Tool used to extract triples from sentences in form ``(Subject,Predicate,Object)``"
+
+Usage
+-----
+
+In order to run the parser, you need to run **roboy_parser** first - see instructions on `project Github <http://github.com/Roboy/roboy_parser` and then run Dialog System.
+
+Configurations
+""""""""""""""
+
+To test parser, you can run following configurations using ``quick_start.sh`` script. For more information refer to `project documentation <http://github.com/Roboy/roboy_parser`
+
+.. csv-table:: Possible parser configurations
+  :header: "Command", "Options"
+  :widths: 20, 40
+
+  "./quick_start", "Default configuration. Using custom Roboy grammar and lexicons"
+	"./quick_start freebase", "Example setup to test Freebase functionality"
diff --git a/docs/doxyxml/_abstract_boolean_state_8java.xml b/docs/doxyxml/_abstract_boolean_state_8java.xml
@@ -69,6 +69,6 @@
 <codeline lineno="83"><highlight class="normal"></highlight></codeline>
 <codeline lineno="84"><highlight class="normal">}</highlight></codeline>
     </programlisting>
-    <location file="/home/roboy/workspace/Roboy/src/roboy_dialog_system/src/main/java/roboy/dialog/personality/states/AbstractBooleanState.java"/>
+    <location file="/home/emilka/ros/Roboy/src/roboy_dialog_system/roboy_dialog/src/main/java/roboy/dialog/personality/states/AbstractBooleanState.java"/>
   </compounddef>
 </doxygen>
-Original file line number
+Diff line change
@@ Expand Up / @@ -14,4 +14,6 @@ local.properties @@
     *.idea
-    .target
+    .target
+    resources/word2vec_toy_data_and_model