GTFS API proposal

[clezag]

Regarding the issue #32, I've thought a bit about how the GTFS API could be done.

GTFS is specified as a zip file containing csv / txt files, so by providing the data as json or similar formats is counter to the GTFS specifcation.

If we want to transfer GTFS into JSON, there are already existing projects, but nothing that sees too much active development.

Nevertheless, having something a bit more fancy than an S3 bucket could be useful, if only to have some metadata, and have the whole thing in one place instead of a bunch of S3 links.

I don't know how common it is to request single files outside of the zip, so if we don't need that, the service could be even simpler:

API host: gtfs.api.opendatahub.com/v1

/dataset
    Accept header json or nothing
    -> JSON list of datasets with metadata. See config for rough structure
            + last update time (if we have that data)

/dataset/<dataset ID>
    Accept header json or nothing
    -> JSON same as /dataset, but only for this one.
            Maybe have list of files inside zip

/dataset/<dataset ID>/raw
    Accept header zip or nothing
    -> <dataset ID>.zip

/dataset/<dataset ID>/raw/<filename>
<filename> ::= Agency | Calendar_dates | Calendar
    Accept header txt / csv or nothing
    -> <filename>.txt content

/invalidate-cache/<dataset ID>
    -> Invalidates the cache for this dataset

The datasets themselves could be configured in a *.yml file following this rough outline:

config file:
    datasets:
            <dataset ID> :
                    source <file URL (e.g. S3 URL) to *.zip>
                    description
                    origin
                    metadata: {}
                    update schedule / validity / caching time?

I think this way we could provide useful additional information / metadata, and aid discovery by providing a single access point.
Behind this service, we can continue using S3 buckets as the actual storage

If we want to extend this in the future, we can add JSON format by doing /json instead of /raw, or by respecting the Accept header MIME type (e.g. if someone requests JSON, they get JSON, otherwise zip/txt)

Let me know what you think

[rcavaliere]

@clezag I think this is OK. Typically GTFS Data Consumer do want directly the ZIP, so I would go for an MVP in which you can simply download to a proper API call the ZIP file. Question: do we want to store all GTFS ZIP files generated and provide them as well?

[clezag]

A MVP of this is now up in testing:

https://gtfs.api.opendatahub.testingmachine.eu/v1/dataset
https://gtfs.api.opendatahub.testingmachine.eu/v1/dataset/skyalps-flight-data
https://gtfs.api.opendatahub.testingmachine.eu/v1/dataset/skyalps-flight-data/raw

Configuration of the datasets is done via datasets.yml in the repo

[clezag]

Regarding storing the history etc:

Currently this is a very minimal system:
A public http server (in our case S3 bucket - but it could be STA or someone else) hosts the zip file.
The API shows some static metadata from a config file, and calls the static URL of the zip, passing the data along to the caller.

If we want to have history, lots of files, changing metadata etc. then we have to connect this to a database, extend the schema to support it, decide how we want to store the binary files (we currently don't, I think)

So it would become a much more serious project than it is now.
Are there actual use cases for historical timetable data?
How many datasets of this type do we will/plan to have?