README.txt

eprinttools

eprinttools is a collection of command line tools written in Go, a Go
package and set of command line utilities for working with EPrints 3.x
EPrint XML and REST API written in Python 3. Eventually it is planned to
have this project become a pure Python project.

This project also hosts demonstration code to replicate a public facing
version of an EPrints repository outside of EPrints. Think of it as the
public views and landing pages.

Go base code

The command line programs

-   eputil is a command line utility for interacting (e.g. harvesting)
    JSON and XML from EPrints’ REST API
    -   uses minimal configuration because it does less!
    -   it superceded the ep command
-   epfmt is a command line utility to pretty print EPrints XML and
    convert to/from JSON
    -   in the process of pretty printing it also validates the EPrints
        XML against the eprinttools Go package definitions
-   doi2eprintxml is a command line program for turning metadata
    harvested from CrossRef and DataCite into an EPrint XML document
    based on one or more supplied DOI
-   eprintxml2json is a command line program for taking EPrint XML and
    turning it into JSON

The first two utilities can be configured from the environment or
command line options. The environment settings are overridden by command
line options. For details running either command envoke the tool name
with the ‘-help’ option.

Python base code

Python Modules

eprints3x

This python module wraps the eputil Go comand in Python. It makes it
trivial to implement harvesting an EPrints repository using the EPrints
REST API.

eprintviews

This python module uses py_dataset and the harvested content to generate
a htdocs directory similar to the URL layout of EPrints. It features
classes for working with Views, Users (needed to attribute names to
userid fields EPrint XML harvested from the REST API), Subjects (a way
to load the subjects text file from an EPrints archive and generate the
path to label mapping used when rendering views into an htdocs
directory) and Aggregator (this does the heavy lifting of processing a
dataset collection of harvested EPrint XML and generating the views as
JSON documents in the htdocs directory).

command line tools

harvester_full.py, harvester_recent.py

These two Python programs use eprints3x module to implement harvesters
of EPrint XML and any related digitl objects (e.g. PDFs, images) into a
dataset collection

genviews.py

This Python program processes a dataset collection and renders an htdocs
tree populating it with JSON documents and key lists. This skeleton of
metadata and directory structure can then be processed into a rendered
website mirroring the content from an EPrints repository. This module
relies on eprintviews.

indexer.py

This Python program indexes the contents of our replicated EPrints site
by creating scheme.json files along side the index.json files that
represent the landing pages for the replicated repository. These can
then be easily ingested into Lunr.js or Elasticsearch. Currently the
proof of concept targets Lunr.js. This module relies on eprintviews.

mk_website.py

This Python program creates the HTML pages from Markdown documents in
the static folder (e.g. home page and major landing pages) as well as
the individual views and abstracts from the JSON documents created by
genviews.py. The final result is a static website ready to serve out to
the public. This module relies on eprintviews.

publisher.py

This Python program copys (syncs) the content with an AWS S3 bucket via
the AWS command line tools.

Related GitHub projects

py_dataset

This Python module provides access to dataset collections which we use
as intermediate storage for JSON documents and related attachments.

AMES

The eprintools command line programs have been made available to Python
via the AMES project. This include support for both read and write to
EPrints repository systems.