Provide total statistics metrics

[jan-di]

It would be great, if the exporter also provides metrics that represent the statistics from the UI:

• Total count of documents/documents that are marked as new/characters
• Statistics about file extensions

[hansmi]

I had a quick look at what's necessary/possible. Statistics aren't part of the documented REST API.

• "Documents in inbox" needs the "inbox" flag from tags to be exported, then it can be reproduced in PromQL (sum(paperless_tag_document_count) filtered by inbox flag).

• "Total documents" is possible via the API. Either by iterating (see below) or requesting /api/documents/ which has the total count (currently not implemented by the client code.

• "Character count" would require iterating over all documents. The underlying implementation for the statistics is a single SQL query summing the character count from all documents. Depending on the scrape interval and document count the metric would be rather expensive. Would have to be behind a flag.

• The break-down by document type is also a single SQL query grouping by MIME type. Would require document iteration as well. Not sure how useful the break-down is in metrics.

Implementation:

• Web UI
• Internal, undocumented API

[hansmi]

The "inbox" tag flag is reported as a metric with df7648d.

"Total documents" will/would require changes to the underlying Paperless-ngx client library. @jan-di, do you have a usecase other than "it would be great"?

[jan-di]

No, actually not. Its just for measuring growth as I thought its just an API that can be piped through to metrics.
If its that much work, its not worth it.

I'm instead thinking of providing a docs MR to paperless, maybe the statistics API is already official but just not documented. If its assured that it is stable, this would make this issue much easier.

[hansmi]

@jan-di, inspired by #41 I've added a document count in 071024d. It will be part of the next release.

In summary that makes the implementation of what's done easily without additional documented APIs complete (c.f. #39 (comment)).