Synthetic File System Technical Documentation
To be independent from a specific platform and to make sure that every team member has the same run-time environment, we decided to use docker. The choice of programming language is python, because it has simplified syntax and it is easy to learn and use. Furthermore, the focus of the project from the industry partner is to prototype the synthetic file system. Hence the most suitable language to achieve that is python. PostgreSQL and GraphQL are integral parts of Metadatahub’s service. Because the the synthetic file system must be in sync with the Metadatahub, PostgreSQL and GraphQL are chosen to store and retrieve metadata. It was clear that we not only want to integrate automated tests, but also to use these to guide the development (Test Driven Development). Eventhough pytest is more popular than unit-test, we decide to go for unit-test, because it is usaully the first testing framework someone learns when he or she starts learning python. To summarize mentioned technologies which will be used to develop the synthetic file system, please see the table.
The Synthetic File System can be browsed as a normal file system. For example a command like “ls” only involves metadata so the Metadata-Module queries the relevant data from the Metadatahub and displays it in regular files and folders. If the user wants to access such data for example with “open” the Data-Module fetches the data from the data storage. Even though the data is stored in different file systems and object stores, they get exposed to the user in a unified way. Which means that the Synthetic File System displays the different data formats under a shared/ common namespace.
