From Boox -> Dropbox -> Raspberry Pi -> Mega
This README acts as a journal of all the progress I've made on this project and how I continue to develop it.
Attempting to transfer my notes from my Boox notebook to my MEGA cloud storage. The working iteration of the project involves orchestrating CLI commands to download note files from Dropbox to a Raspberry Pi, then upload those files to MEGA. Along the transfer process, we're also adding more PDF features like watermarks with my logo, PDF passwords (more to show that I can add a password rather than it being effective), adding a PDF page outline and making the lines of the PDF highlightable (by adding a bunch of invisible characters to the PDF XD).
- Getting a BAD_REQUEST error when creating the
NoteStoreClient- Decided to stop trying to find the root cause and focus on a Dropbox solution
- Access tokens have a lifetime of 4 hours
- Currently, don't know how to get a refresh token
- I think I'll have to manually hit some endpoint, but I don't know what endpoint I have to look for, documentation was unclear
- Manually upload the files whenever I feel like it XD
- Pretty cringe
- Is there even a Java API??
- Not that I could find as of writing this
Going through several ideas and iterations of the project to make it function without 'human intervention'
Attempt 1 used this library to render the webpage
Citation: GitHub Link
- The site would throw an exception when Javascript was enabled; however, Javascript is necessary for the page to render correctly for future use
Attempt 2 used the HBrowser library to try to do the same thing as HTML Unit
Citation: GitHub Link
- The site wouldn't render completely
- Not sure how much time I'd want to spend figuring out if it's plausible to get a completely rendered page
For attempt 3, there is an unofficial Dropbox Command Line Interface written in Go and has compiled versions for
Windows and Debian Linux via linux-arm. Perfect!!
Now we can move forward with an idea.
Citation: GitHub Link
Design a Java wrapper around the CLI similar to Eliux MEGA cmd4J library
- This fixes the access lifetime issue
- You log into the CLI once and that's it
- The CLI compatibility with Windows and Linux made for easy testing before deploying to the Raspberry Pi
- Premise: The program is dysfunctional when running the JAR file on a Raspberry Pi Cron Job
- Way to look into the problem: Changed the internal logging library to
ch.qos.logback:logback-classic(GitHub Link)- This contains a dependency to
SLF4J-apiso that swapping libraries is no problem - With the library change, I can output the log events to a file rather than relying on the console for everything
- Since console logging doesn't work for Cron Jobs anyway
- This contains a dependency to
- Logging event: The error that was occurring was
Cannot run program "dbxcli": error=2, No such file or directory- This tells me that the
dbxclicommand is not getting recognized by the system.- So, my thought process was 'Is the user different for the Cron Scheduler?'
- This tells me that the
- Solution: In Linux, we can prepend a command with
sudo -u [USERNAME]so that the command can be executed by a particular user with their set of permissions. In other words, I can substitute myself in as the user running thejarfile
- If the
Crash-Cloud-Pathargument is setTrue, then we wipe the cloud path where we store notes in MEGA before the transfer- Also deletes the
revision-list.txtif it exists so that we can conduct a full transfer from Dropbox to MEGA
- Also deletes the
- Fetch the full list of files from Dropbox
- The Dropbox CLI sometimes would cut the
lscommand prematurely, so we weren't able to get the list of files- The fix for this is to try again with a linear drop-off rate similar to how browsers employ Exponential Backoff for inaccessible websites to prevent heavy traffic
- The Dropbox CLI sometimes would cut the
- Sort files strictly by filename (Excluding the absolute path)
- Detect any file movements within the Dropbox system
- A file movement is defined as the same file that appears twice in Dropbox with a difference of one directory
- This indicates that the file was moved to a new directory and the older file should be deleted
- A file movement is defined as the same file that appears twice in Dropbox with a difference of one directory
- If any file movements are found...
- Determine the older file among the pair (Older files are determined by older ages and smaller file sizes)
- Delete the older file from Dropbox
- Remove the older file from the fetched file list
- If an older file cannot be determined, disregard the pair
- Determine the older file among the pair (Older files are determined by older ages and smaller file sizes)
- If the
revision-list.txtfile exists, filter out all Dropbox files that haven't changed since last transfer- If it filters out all files, the program stops here. (No new files were added to Dropbox)
- Rewrite the
revision-list.txtfile with all new files and hash codes - Prepare files for transfer
- If the amount of files reaches a certain threshold, conduct the transfers in parallel
- Download the PDFs from Dropbox (one at a time because some notes will have the same name, and we try to prevent local overwriting as much as possible)
- Process the files
- Make the lines of each page highlightable
- Add a simple table of contents that labels each page with the
Page No. - Encrypting PDFs and locking permissions for certain operations (more for fun than security. PDF security is garbage, see here)
- Adding a watermark to each page of the 'Jragon' logo
- Log and remove any erroneous files from the
revision-list.txtfile so that the program can try again at a later time