Home

Rated Ranking Evaluator

The Rated Ranking Evaluator (RRE) is a search quality evaluation tool which, as the name suggests, evaluates the quality of results coming from a search infrastructure.

It is something which helps a Search Engineer in his daily job. Are you a Search Engineer? Are you tuning/implementing/changing/configuring a search infrastructure? Do you want to have something that gives you an evidence about the improvements between changes? So you are in the right place.

There's more: RRE formalises how well a search system satisfies the user information needs, at "technical" level, combining a rich tree-like domain model with several evaluation measures, but also at "functional" level, providing human-readable outputs that could be act as deliverables for the business stakeholders.

It encourages an incremental/iterative/immutable approach when developing and evolving a search system: assuming we're starting from version 1.0, when it's time to apply some relevant change to the configuration, instead of changing that version, is better to clone it and apply those changes to the new version.

In this way, the evaluation process will happen for all available versions and RRE will provide the delta/trend between each subsequent versions, so you can immediately get a fine-grained picture about where the system is going, in terms of relevance.

The following sections highlight the most important features of the project.

Set of Search Quality Evaluation tools

RRE is not a monolithic piece of software, it has been built keeping in mind a fundamental aspect: modularity and extensibility.
You will find a lot of modules within this repository. All of them round around a central thing called rre-core which provides, as the name suggests, the core evaluation functionalities.

If from one side, the runtime perspective of the rre-core is a library which can be used programmatically in your code, the ultimate goal is to provide several runtime containers for running the evaluation process within some managed environment (e.g. Apache Maven, Gradle, Apache Solr, Jenkins). One of these containers, the Apache Maven binding, is already available in the current release.

Search Quality Evaluation Framework

We usually don't hard-code things in RRE. The "open-close" approach is required in these kind of tools because:

• it's not possible to create in advance a binding for all available search platforms
• for the same reasons, we couldn't create an implementation for all available evaluation measures

So as consequence of that, we provided, where it makes sense, a level of abstraction which allows to bind new things in RRE.

Multi (Search) Platform

RRE doesn't target a specific search platform. It provides, as mentioned before, an abstraction layer which defines through interfaces, the expected lifecycle methods that a search platform must have in order to be integrated in a search quality evaluation process.
As part of the current release we provide two bindings: Apache Solr and Elasticsearch (see the supported versions in this Wiki).

Written in Java but it can be used also in non-Java projects

RRE is entirely written in Java but it can be used also in non Java projects, by means of the RRE Maven archetype or (not part of the current release) RRE CLI.
That means your project can be written in any language, and in order to run RRE you must have

• a JVM (8 or later) and Apache Maven (for using the Maven archetype)
• a JVM (8 or later) for the command line client (not part of the current release)