A general overview
This page (i) describes the idea behind the Alpha Testbed and (ii) provides a few pointers on how work with it. If you are interested in using ATB, this should be the first page to read.
The Alpha Testbed (ATB) is a tool for benchmarking trust models. It can evaluate a trust model, while
- not requiring that its owner (i.e. agent that is using it) makes any decisions (such as with whom to interact next, or who to ask for opinions), and by
- not requiring that the tested trust model estimates trust on a predefined domain (such as for instance the interval [0, 1]; ATB, however, does require that trust is estimated with comparable values, i.e. that one can say that it trust agent A more than agent B).
ATB is not a multi-agent system nor a agent-based simulation. Instead, it is a simulator that builds on the notion of a data generator, a trust estimator, and a set of metrics that evaluate the accuracy of the estimated trust. In ATB, the data generator is called a scenario, the trust estimator a trust model and the metrics are called, well, metrics.
The scenario, being the data generator, represents the entire surroundings of the agent that uses the tested trust model. (We term this agent Alpha -- thus came the name for the testbed.) The scenario therefore controls the world around agent Alpha: it controls the population of other agents, it controls how agents provide services, how they provide opinions, etc. Ultimately, the scenario generates data (such as experiences and opinions).
The trust model, being the trust estimator, uses the data that the scenario generated to estimate trust. The values, with which a trust model estimates trust, need not be numeric, but they have to belong to an ordinal scale; i.e. if a trust model estimates trust towards agent A as X and trust towards agent B as Y, ATB needs to know whether X is equal, greater or lower than Y. While it is not required that agent Alpha makes any decisions, ATB can also accommodate such cases.
Finally, the metrics (accuracy, utility, opinion cost) evaluate the estimated trust with respect to the actual state of the world as as given by the scenario.
The mechanism that connects the scenario, the trust model and the metrics, is called the Evaluation Protocol. This component is described in more detail below. The following diagram shows the relationship between these concepts.
Evaluation protocols connect scenarios, trust models and metrics; they determine what types of scenarios, trust models, and metrics can be legally combined. Currently, ATB provides three Evaluation Protocols:
-
No decisions. In this protocol the scenario determines (i) with whom does agent Alpha interact and (ii) from whom and about whom does Alpha receive opinions. All that Alpha does is that it estimates trust from received information. The estimated trust is then evaluated with the Accuracy metric.
-
Decisions mode A. While in this protocol the scenario still determines from whom and about whom does Alpha receive opinions, Alpha gets to select its interaction partners; that is, it decides with whom to interact. Therefore this evaluation protocol evaluates the Accuracy of computed trust and the Utility that Alpha obtains in interactions.
-
Decisions mode B. This evaluation protocol extends the Decision mode A by requiring that agent Alpha not only selects interaction partners but also decides who (and about whom) to ask for opinions. Besides measuring the Accuracy of the estimated trust and the Utility of partner selections, this protocol also measures the cost of obtaining opinions with the Opinion cost metric.
ATB is open for extensions, therefore new Evaluation protocols can be added. For instance, if a trust model uses additional information sources, such as information from virtual organizations or social-network information, appropriate Evaluation protocols can be implemented.
In ATB, scenarios, trust models and metrics are implemented as plug-ins. The plug-in approach allows ATB to be flexible; if you would like to implement your own trust model and benchmark it against the set of existing scenarios, you simply implement the model as a plug-in.
Similarly, if you would like to see how trust models react to a certain type of an attack, you can implement these attacks in a scenario and then see how trust models fare in it. The same holds for new metrics: if you have an idea on how to evaluate the computed trust values (or any other action of the agent Alpha), you implement a metric plug-in.
A tutorial for writing plug-ins is comming soon (TODO).
While the text on this page tries to give an overview of the ATB, you are strongly encouraged to check the following two papers that describe ATB in more detail.
-
David Jelenc, Ramón Hermoso, Jordi Sabater-Mir, and Denis Trček. Decision Making Matters: A Better Way to Evaluate Trust Models. Knowledge-Based Systems, 2013. The most up-to-date paper that describes ATB the most accurately. However, the paper is rather long and if you are only interested in the functioning of ATB you can only read sections 1, 3 and 5.
-
David Jelenc, Ramón Hermoso, Sascha Ossowski, and Denis Trček. Alpha Test-bed: A New Approach for Evaluating Trust Models. In Proceedings of The Third International Workshop on Infrastructures and Tools for Multiagent Systems (ITMAS), Valencia, Spain, 2012. Although a bit out-dated, the paper describes the idea behind the ATB's design (data generators, trust estimators, and metrics).
The principal designer of the Alpha Testbed is David Jelenc. The testbed formed a part of his PhD thesis. However, the Alpha testbed and the surrounding research is a result of joint work between David, Ramón Hermoso, Jordi Sabater-Mir, and David's PhD advisor Denis Trček.
Special thanks must also go to Sascha Ossowski that hosted David on a short term scientific mission in the scope of ICT COST Action IC0801: Action Agreement Technologies, where the initial idea was conceived.