Archived distributed Systems Design

Pi Gazing Distributed Computing

Nodes in the Pi Gazing project, particularly those attached to cameras, are capable of stand-alone operation. They run their own internal web interfaces and provide API access to their data. Some functionality, however, requires that we automatically replicate or otherwise export some or all of these data to other nodes. This document describes the approach we plan to take to this class of requirements.

Requirements

To support the distributed computing requirements of Pi Gazing we have the following requirements:

• Must support Camera nodes which are on the ‘wrong’ side of a firewall and / or NAT, and which cannot be exposed explicitly on the public internet. This seems likely to be the case; even where schools have the requisite level of IT support to actually do fixed IP and port forwarding they’re likely to have policies in place which prevent us making use of this. The Camera nodes therefore must be able to push data, ideally over HTTP/HTTPS, reducing the requirement on the installation to allowing external access on those ports, something they probably already allow (although some kind of firewall / whitelist mechanism might exist with which we’d need to coexist)
• Must be secure, specifically it must prevent unauthorised data push to non-terminal nodes.
• Must be configurable, for both data quality and load optimisations, in terms of what it exports and to where.
• Should be efficient, in particular avoiding transmitting substantial redundant, unwanted or duplicate data. We’re not too worried about out and out performance but bandwidth is important.
• Should be heterogenous, maintaining simplicity by running an identical software stack on Camera and intermediate nodes. Any differences in behaviour should be handled by configuration, ideally through the web interface now we have role based permission management.

Export of Events and FileRecords

The export process can be broken down into two distinct stages. In the first, entities (henceforth used to generally describe both Event and FileRecord instances) are selected for export to a given destination. In the second, this queue of pending exports is processed to actually export the data.

Entity selection

We already have a mechanism to identify sets of both Events and FileRecords, vis. the query interfaces in the database and server packages. Additionally we have an existing facility to serialise these queries into a packed string form, currently used in the API to transmit queries on the wire from the Python and web clients to the REST server. This makes it straightforward to persist queries in the database, and to use such queries to specify a subset of all available entities to export. To support this we need to:

1. Introduce a new export table t_export to the database, used to track entities which are tagged for export. The table needs to track:
   • The entity itself (by FK relation)
   • The target, specified as the URL of the receiving import service
   • A timestamp, initially set to time at which the entity was tagged for export
   • A boolean flag to indicate whether the export is live (we may wish to make this an enumeration to cope with potentially more detailed reporting on the export lifecycle)
2. Introduce a table t_exportTask to track export tasks, comprising:
   • The endpoint to which entities matching a search should be exported
   • The type of entity to export, currently this can be FileRecord, Event or CameraStatus
   • The search defining entities to export. Note that this search must exclude entities which already have an entry for this endpoint in t_export, this allows us to naively repeat the same query without worrying about scheduling entities multiple times for the same target (although we would have multiple tasks when exporting an entity to multiple destinations)
   • The username / password pair required by the endpoint
   • Optionally we might want to set some kind of timing information, this could be in the form of an interval and / or a range of times within the day during which export operations would run (i.e. to only schedule exports after the end of the school day to avoid excessive load on the camera nodes during teaching activities)
3. Add a new page to the UI, along with corresponding operations in the API, to allow an admin enabled used to create, update and delete entities in t_exportTask.
4. Implement logic which will, based on timing defined export tasks, query the database and create corresponding records in t_export for entities matched, thus queueing them for export.

Note - CameraStatus has to be handled slightly differently as there’s no current search facility. The only mode that actually makes sense is to export all CameraStatus blocks that haven’t already been exported. This will end up exporting a CameraStatus with a null end time, as there will always be a ‘current’ status. Receiving parties need to handle this, particularly they must check for a prior ‘current’ block and handle update of the end points appropriately when a later status is provided. In general CameraStatus should be handled implicitly as part of another export type, there may be uses (i.e. a ‘what cameras exist’ aggregator) where we only care about the CameraStatus information itself.

Export processing

Export processing should ideally not require any ‘session-like’ state in either the exporting or importing party. Therefore, the algorithm I propose is to do the following:

1. While there are record in t_export with non-future timestamps, and which are live, pick the earliest such record.
2. Retrieve the referenced entity (a FileRecord or Event), and POST it to the endpoint using the same JSON form as elsewhere.
3. The importer must then respond with either:
   1. An indication that it has completed the import and that no further information is required, at which point the exporter must mark the record in t_export as non-live, whether the importer has actually done anything with the data is not the exporter’s problem, it must not send the same request again.
   2. An indication that more information is needed. This is currently only going to be the binary content of files, so the response will contain a file ID. The exporter must respond to this by immediately sending the binary contents of the file.
4. If the importer does not respond, or responds with an error of any kind, the timestamps of all pending entries in t_export should be updated to effectively push all such jobs into the future, effectively ignoring that particular target for a while (they could also simply be deleted, relying on a re-run of the entity selection process to populate them).

This approach will result in multiple calls in the case where we’re using it to actually replicate entities to other nodes (i.e. the specified ‘central server’). The calls for a FileRecord in such a case would be:

1. Exporter: Here’s a FileRecord (including the serialised form of the FileRecord but nothing else)
2. Importer: Send me the binary data
3. Exporter: Here’s the binary data for the FileRecord with ID xxx
4. Importer: Got it.
5. Exporter: Here’s a FileRecord (exactly the same message as in 1)
6. Importer: Got it, that’s all I need.

So, we have a degree of redundancy, but gain the simplicity that we don’t have to track complex state on the import or export parties. The process for an Event is similar, but there will be multiple requests, one for each associated FileRecord. For cases where we don’t need the actual files there’s only going to be a single transaction required, this would be the case for e.g. a service which pushed tweets out about observed event every evening (it might opt to grab an image or two to include in the tweet, but it’s up to the importing party to decide whether it needs this, and up to the exporting party to offer the entity in the first place).

A note on FileRecords associated with Events: These would be offered as part of the serialised Event, so queries that are intended explicitly to push stand-alone files out will always internally have the ‘exclude events’ flag set.

As an aggregating node must also track CameraStatus blocks in order to correctly interpret the FileRecords and Events, we need to augment the existing serialisation formats for FileRecord and Event to include the ID of the corresponding CameraStatus, and include the option for a receiving party to request this if it doesn’t already have it. CameraStatus blocks have UUIDs as of the current code, but it’s not surfaced anywhere in the Event or FileRecord serialisation.

Import of Events and FileRecords

A party wishing to receive Events and FileRecords from another node must create a user and assign the ‘import’ role to it, providing this username and password to the exporting party (typically this would be a manual process, and allow aggregating servers to individually enable and disable different exporting parties). We will include the import functionality along with the conversational protocol for full file replication in the server code, as well as providing a Python module which can provide the basic protocol handling for third party developers. The implementation we provide will make use of the same user table as used elsewhere to handle e.g. admin users.

Partial imports of FileRecords

At the point we first receive a FileRecord we don’t have the corresponding file. To indicate this, a receiving instance of the server must set the file length to zero. When the actual binary file is received this can be updated to the length of the received file. Clients, including the web interface, need to be coded to interpret a file size of zero to mean ‘data not yet available’. To simplify exports, exports should only run if the FileRecord, or all FileRecords in an Event, have non-zero file sizes set. That is, we don’t initiate the export of an entity until we have all the supporting data on the exporting node itself. To handle this we need to augment the Event and FileRecord queries to allow us to exclude those entities associated with zero size files.

Keeping track of imported vs locally created entities

We will introduce a new table t_import used to track the origin of a given Event or FileRecord (including the FileRecord instances referenced by an Event). This table will track the source address, and timestamp of import of each such entity. We’re not actually that interested in this information, but we do care whether an entity has participated in an export or an import - entities which have been locally created and not yet exported can be mutated, particularly by additions to their metadata blocks, but to simplify replication and ensure data consistency we disallow any form of mutability once data have been exported. By adding the ability to include this in the query specification for Events and FileRecords we can exclude non-local entities from export, and better tailor UIs intended to allow editing so that only applicable entities are shown.

Use cases

Replication

The most obvious use case is to push Events and FileRecords from a Camera node into a more accessible data warehouse, potentially aggregating the data from multiple Camera nodes into a single store. The database and code is written to handle this, any given node can contain data from an arbitrary number of Cameras (including the Camera nodes, although we’d typically only expect these to include their own data). This particular use case should be supported by code we write as part of the core project.

Notification / Social media integration

The export mechanism is also in effect a notification system, with the additional facility to allow the receiving party to specify whether they wish to receive substantial actual data or solely the (much lighter) metadata. We could envisage a social media gateway which received such data and used it to update blogs, tweet, post to Instagram and similar etc.