Merge branch 'main' of https://github.com/mff-uk/simpipes-components …

…into main
mff-uk · Jan 11, 2021 · 6bf89ec · 6bf89ec
2 parents c3f8155 + e97a683
commit 6bf89ec
Show file tree

Hide file tree

Showing 51 changed files with 100 additions and 27 deletions.
diff --git a/README.md b/README.md
@@ -5,13 +5,16 @@ The repository is structured according to conceptual component types:
 
 - [Extractors](extractors)
   - [Extract external knowledge](extractors/extract-external-knowledge)
-  - [Extract metadata](extractors/extract-metadata-descriptor) 
-  - Extract content (empty at the moment)
+  - [Extract metadata descriptor](extractors/extract-metadata-descriptor)
+  - [Extract content descriptor](extractors/extract-content-descriptor) (empty at the moment)
 - [Processors](processors)
   - [Refine descriptor](processors/refine-descriptor)
-  - Refine external knowledge (empty at the moment)
+  - [Refine external knowledge](processors/refine-external-knowledge) (empty at the moment)
   - [Refine mapping](processors/refine-mapping)
-  - [Compute similarities](processors/compute-similarities)
-  - Multimodal fusion (empty at the moment)
+  - [Compute similarity](processors/compute-similarity)
+  - [Fuse similarities](processors/fuse-similarities) (empty at the moment)
   - [Map dataset to knowledge](processors/map-dataset-to-knowledge)
-- [Presenters](presenters) (empty at the moment)
+- [Presenters](presenters)
+  - [Similarity evaluation](presenters/similarity-evaluation)
+  - [Similarity exploration](presenters/similarity-exploration) (empty at the moment)
+  - [Similarity search](presenters/similarity-search) (empty at the moment)
diff --git a/extractors/README.md b/extractors/README.md
@@ -4,5 +4,5 @@ An extractor is a component which performs an operation of extracting data from
 The folder is structured according to extractor types:
 
 - [Extract external knowledge](extract-external-knowledge)
-- [Extract metadata](extract-metadata-descriptor) 
-- Extract content (empty at the moment)
+- [Extract metadata descriptor](extract-metadata-descriptor)
+- [Extract content descriptor](extract-content-descriptor) (empty at the moment)
diff --git a/extractors/extract-content-descriptor/README.md b/extractors/extract-content-descriptor/README.md
@@ -0,0 +1,6 @@
+# ExtractContentDescriptor
+*ExtractContentDescriptor* represents components performing
+
+<code><i>extract</i><sub><i>desc</i></sub>(DatasetSource,Metadata<sup>+</sup>): Descriptor<sup>+</sup></code>
+
+which extracts from a dataset source a descriptor for each dataset with a provided metadata descriptor.
diff --git a/extractors/extract-external-knowledge/README.md b/extractors/extract-external-knowledge/README.md
@@ -1,2 +1,6 @@
-# Knowledge graph extractors
-Knowledge graph extractors are components, which ...
+# ExtractExternalKnowledge
+*ExtractExternalKnowledge* represents components performing
+
+<code><i>extract</i><sub><i>k</i></sub>(KnowledgeSource): Knowledge</code>
+
+which extracts an external knowledge from a given external knowledge source.
diff --git a/extractors/extract-metadata-descriptor/README.md b/extractors/extract-metadata-descriptor/README.md
@@ -1,2 +1,6 @@
-# Metadata extractors
-Metadata extractors are components, which ...
+# ExtractContentDescriptor
+*ExtractContentDescriptor* represents components performing
+
+<code><i>extract</i><sub><i>meta</i></sub>(DataCatalogSource): Metadata<sup>+</sup></code>
+
+which extracts metadata about datasets from a given data catalog source.
diff --git a/presenters/README.md b/presenters/README.md
@@ -3,4 +3,6 @@ A presenter is a component which uses the products of dataset similarity pipelin
 
 The folder is structured according to presenter types:
 
-(Empty)
+- [Similarity evaluation](similarity-evaluation) (empty at the moment)
+- [Similarity exploration](similarity-exploration) (empty at the moment)
+- [Similarity search](similarity-search) (empty at the moment)
diff --git a/presenters/similarity-evaluation/README.md b/presenters/similarity-evaluation/README.md
@@ -0,0 +1,7 @@
+# SimilarityEvaluation
+*SimilarityEvaluation* represents components performing
+
+<code><i>present</i><sub><i>eval</i></sub>(Similarity<sup>+</sup>, Descriptor<sup>+</sup>)</code>
+
+which presents dataset similarities to a human user who evaluates the similarities.
+It uses input descriptors of the datasets for presenting the datasets.
diff --git a/...imilarity-evaluators/exact-size/README.md → ...imilarity-evaluation/exact-size/README.md b/...imilarity-evaluators/exact-size/README.md → ...imilarity-evaluation/exact-size/README.md
diff --git a/...ors/exact-size/input-sample/baseline.json → ...ion/exact-size/input-sample/baseline.json b/...ors/exact-size/input-sample/baseline.json → ...ion/exact-size/input-sample/baseline.json
diff --git a/...tors/exact-size/input-sample/distance.csv → ...tion/exact-size/input-sample/distance.csv b/...tors/exact-size/input-sample/distance.csv → ...tion/exact-size/input-sample/distance.csv
diff --git a/...ators/exact-size/input-sample/reduced.csv → ...ation/exact-size/input-sample/reduced.csv b/...ators/exact-size/input-sample/reduced.csv → ...ation/exact-size/input-sample/reduced.csv
diff --git a/...arity-evaluators/exact-size/validation.py → ...arity-evaluation/exact-size/validation.py b/...arity-evaluators/exact-size/validation.py → ...arity-evaluation/exact-size/validation.py
diff --git a/.../similarity-evaluators/pr-curve/README.md → .../similarity-evaluation/pr-curve/README.md b/.../similarity-evaluators/pr-curve/README.md → .../similarity-evaluation/pr-curve/README.md
diff --git a/...ators/pr-curve/input-sample/baseline.json → ...ation/pr-curve/input-sample/baseline.json b/...ators/pr-curve/input-sample/baseline.json → ...ation/pr-curve/input-sample/baseline.json
diff --git a/...uators/pr-curve/input-sample/distance.csv → ...uation/pr-curve/input-sample/distance.csv b/...uators/pr-curve/input-sample/distance.csv → ...uation/pr-curve/input-sample/distance.csv
diff --git a/...luators/pr-curve/input-sample/reduced.csv → ...luation/pr-curve/input-sample/reduced.csv b/...luators/pr-curve/input-sample/reduced.csv → ...luation/pr-curve/input-sample/reduced.csv
diff --git a/...ilarity-evaluators/pr-curve/validation.py → ...ilarity-evaluation/pr-curve/validation.py b/...ilarity-evaluators/pr-curve/validation.py → ...ilarity-evaluation/pr-curve/validation.py
diff --git a/...ors/similarity-evaluators/top-k/README.md → ...ers/similarity-evaluation/top-k/README.md b/...ors/similarity-evaluators/top-k/README.md → ...ers/similarity-evaluation/top-k/README.md
diff --git a/...aluators/top-k/input-sample/baseline.json → ...aluation/top-k/input-sample/baseline.json b/...aluators/top-k/input-sample/baseline.json → ...aluation/top-k/input-sample/baseline.json
diff --git a/...valuators/top-k/input-sample/distance.csv → ...valuation/top-k/input-sample/distance.csv b/...valuators/top-k/input-sample/distance.csv → ...valuation/top-k/input-sample/distance.csv
diff --git a/...evaluators/top-k/input-sample/reduced.csv → ...evaluation/top-k/input-sample/reduced.csv b/...evaluators/top-k/input-sample/reduced.csv → ...evaluation/top-k/input-sample/reduced.csv
diff --git a/...similarity-evaluators/top-k/validation.py → ...similarity-evaluation/top-k/validation.py b/...similarity-evaluators/top-k/validation.py → ...similarity-evaluation/top-k/validation.py
diff --git a/presenters/similarity-exploration/README.md b/presenters/similarity-exploration/README.md
@@ -0,0 +1,9 @@
+# SimilarityExploration
+*SimilarityExploration* represents components performing
+
+<code><i>present</i><sub><i>expl</i></sub>(Similarity<sup>+</sup>, Descriptor<sup>+</sup>, Knowledge<sup>*</sup>)</code>
+
+which explains similarities between datasets to a human user.
+For the explanation, it uses the input similarity, dataset descriptors and optionally also external knowledge of different kinds.
+Among the input descriptors there are also mappings of the datasets to the external knowledge if provided on the input.
+Descriptors may be used for presentation as well as explanation purposes.
diff --git a/presenters/similarity-search/README.md b/presenters/similarity-search/README.md
@@ -0,0 +1,7 @@
+# SimilaritySearch
+*SimilaritySearch* represents components performing
+
+<code><i>present</i><sub><i>sear</i></sub>(Similarity<sup>+</sup>, Descriptor<sup>+</sup>)</code>
+
+which enables a human user to choose a dataset and then shows datasets similar to the chosen one on the base of the input similarities.
+It uses input descriptors of the datasets for presenting the datasets.
diff --git a/processors/README.md b/processors/README.md
@@ -3,8 +3,8 @@ A processor is a component which performs an operaton for processing input entit
 
 The folder is structured according to processor types:
 - [Refine descriptor](refine-descriptor)
-- Refine external knowledge (empty at the moment)
+- [Refine external knowledge](refine-external-knowledge) (empty at the moment)
 - [Refine mapping](refine-mapping)
-- [Compute similarities](compute-similarities)
-- Multimodal fusion (empty at the moment)
+- [Compute similarity](compute-similarity)
+- [Fuse similarities](fuse-similarities) (empty at the moment)
 - [Map dataset to knowledge](map-dataset-to-knowledge)
diff --git a/processors/compute-similarities/README.md b/processors/compute-similarities/README.md
diff --git a/processors/compute-similarity/README.md b/processors/compute-similarity/README.md
@@ -0,0 +1,7 @@
+# ComputeSimilarity
+*ComputeSimilarity* represents components performing
+
+<code><i>process</i><sub><i>sim</i></sub>(Descriptor<sup>+</sup>): Similarity<sup>+</sup></code>
+
+which computes a similarity of each dataset described by an input descriptor with other datasets described by the other input descriptors.
+The component uses the input descriptors to compute the similarity.
diff --git a/...sors/compute-similarities/basic/README.md → ...essors/compute-similarity/basic/README.md b/...sors/compute-similarities/basic/README.md → ...essors/compute-similarity/basic/README.md
diff --git a/...ut-sample/nkod-keywords.concat.reduce.csv → ...ut-sample/nkod-keywords.concat.reduce.csv b/...ut-sample/nkod-keywords.concat.reduce.csv → ...ut-sample/nkod-keywords.concat.reduce.csv
diff --git a/...pute-similarities/basic/linda/__init__.py → ...ompute-similarity/basic/linda/__init__.py b/...pute-similarities/basic/linda/__init__.py → ...ompute-similarity/basic/linda/__init__.py
diff --git a/...e-similarities/basic/linda/descriptors.py → ...ute-similarity/basic/linda/descriptors.py b/...e-similarities/basic/linda/descriptors.py → ...ute-similarity/basic/linda/descriptors.py
diff --git a/...ute-similarities/basic/linda/distances.py → ...mpute-similarity/basic/linda/distances.py b/...ute-similarities/basic/linda/distances.py → ...mpute-similarity/basic/linda/distances.py
diff --git a/...-similarities/basic/linda/similarities.py → ...te-similarity/basic/linda/similarities.py b/...-similarities/basic/linda/similarities.py → ...te-similarity/basic/linda/similarities.py
diff --git a/.../compute-similarities/basic/linda/tlsh.py → ...rs/compute-similarity/basic/linda/tlsh.py b/.../compute-similarities/basic/linda/tlsh.py → ...rs/compute-similarity/basic/linda/tlsh.py
diff --git a/...sors/compute-similarities/basic/matrix.py → ...essors/compute-similarity/basic/matrix.py b/...sors/compute-similarities/basic/matrix.py → ...essors/compute-similarity/basic/matrix.py
diff --git a/...d-keywords.concat.reduce.sets.jaccard.csv → ...d-keywords.concat.reduce.sets.jaccard.csv b/...d-keywords.concat.reduce.sets.jaccard.csv → ...d-keywords.concat.reduce.sets.jaccard.csv
diff --git a/.../compute-similarities/hausdorff/README.md → ...rs/compute-similarity/hausdorff/README.md b/.../compute-similarities/hausdorff/README.md → ...rs/compute-similarity/hausdorff/README.md
diff --git a/...mpute-similarities/hausdorff/hausdorff.py → ...compute-similarity/hausdorff/hausdorff.py b/...mpute-similarities/hausdorff/hausdorff.py → ...compute-similarity/hausdorff/hausdorff.py
diff --git a/...mple/nkod-description.udpipe-f.reduce.csv → ...mple/nkod-description.udpipe-f.reduce.csv b/...mple/nkod-description.udpipe-f.reduce.csv → ...mple/nkod-description.udpipe-f.reduce.csv
diff --git a/...-similarities/hausdorff/linda/__init__.py → ...te-similarity/hausdorff/linda/__init__.py b/...-similarities/hausdorff/linda/__init__.py → ...te-similarity/hausdorff/linda/__init__.py
diff --git a/...milarities/hausdorff/linda/descriptors.py → ...similarity/hausdorff/linda/descriptors.py b/...milarities/hausdorff/linda/descriptors.py → ...similarity/hausdorff/linda/descriptors.py
diff --git a/...similarities/hausdorff/linda/distances.py → ...e-similarity/hausdorff/linda/distances.py b/...similarities/hausdorff/linda/distances.py → ...e-similarity/hausdorff/linda/distances.py
diff --git a/...ilarities/hausdorff/linda/similarities.py → ...imilarity/hausdorff/linda/similarities.py b/...ilarities/hausdorff/linda/similarities.py → ...imilarity/hausdorff/linda/similarities.py
diff --git a/...pute-similarities/hausdorff/linda/tlsh.py → ...ompute-similarity/hausdorff/linda/tlsh.py b/...pute-similarities/hausdorff/linda/tlsh.py → ...ompute-similarity/hausdorff/linda/tlsh.py
diff --git a/...on.udpipe-f.reduce.set.hausdorff[law].csv → ...on.udpipe-f.reduce.set.hausdorff[law].csv b/...on.udpipe-f.reduce.set.hausdorff[law].csv → ...on.udpipe-f.reduce.set.hausdorff[law].csv
diff --git a/processors/fuse-similarities/README.md b/processors/fuse-similarities/README.md
@@ -0,0 +1,7 @@
+# FuseSimilarities
+*FuseSimilarities* represents components performing
+
+<code><i>process</i><sub><i>fuse</i></sub>(Similarity<sup>+</sup>): Similarity<sup>+</sup></code>
+
+which performs multimodal fusion of two or more input similarities for a dataset to a single similarity.
+For each dataset with specified similarities on the input an output similarity is produced.
diff --git a/processors/map-dataset-to-knowledge/README.md b/processors/map-dataset-to-knowledge/README.md
@@ -1,2 +1,7 @@
-# Dataset to knowledge graph mappers
-Dataset to knowledge graph mappers are components, which ...
+# MapDatasetToKnowledge
+*MapDatasetToKnowledge* represents components performing
+
+<code><i>process</i><sub><i>map</i></sub>(Descriptor<sup>+</sup>, Knowledge): Mapping<sup>+</sup></code>
+
+which maps a set of datasets described by the input descriptors to an input external knowledge.
+The resulting output mapping is created on the base of the input descriptors and it maps each dataset to the external knowledge.
diff --git a/processors/refine-descriptor/README.md b/processors/refine-descriptor/README.md
@@ -1,3 +1,9 @@
-# Descriptor refiners
+# RefineDescriptor
+*RefineDescriptor* represents components performing
 
-Transform one descriptor into another one. For example from text into vectors.
+<code><i>process</i><sub><i>refd</i></sub>(Descriptor<sup>+</sup>): Descriptor<sup>+</sup></code>
+
+which transforms input descriptors to output descriptors.
+The input descriptors must be of the same kind.
+The output descriptors must be also of the same kind which can be different from the input descriptors.
+The output descriptors must describe the same datasets as the input descriptors.
diff --git a/processors/refine-external-knowledge/README.md b/processors/refine-external-knowledge/README.md
@@ -0,0 +1,6 @@
+# RefineExternalKnowledge
+*RefineExternalKnowledge* represents components performing
+
+<code><i>process</i><sub><i>refk</i></sub>(Knowledge): Knowledge</code>
+
+which transforms an external knowledge to another external knowledge, e.g. transforming literal objects of literal edges to other literal objects in a knowledge graph, removing some object edges from a knowledge graph or applying some vector operation on the vectors in an embedding.
diff --git a/processors/refine-mapping/README.md b/processors/refine-mapping/README.md
@@ -1,2 +1,7 @@
-# Mapping refiners
-Mapping refiners are components, which ...
+# RefineMapping
+*RefineMapping* represents components performing
+
+<code><i>process</i><sub><i>refm</i></sub>(Mapping<sup>+</sup>, Descriptor<sup>*</sup>, Knowledge<sup>?</sup>): Mapping<sup>+</sup></code>
+
+which transforms an input mapping of datasets to an external knowledge to another mapping of the same datasets to the same external knowledge or another external knowledge specified as an optional input.
+The new mapping is created using the optional input descriptors of the datasets.
diff --git a/processors/similarity-evaluators/README.md b/processors/similarity-evaluators/README.md