Skip to content

Latest commit

 

History

History
235 lines (205 loc) · 6.32 KB

README.v1.md

File metadata and controls

235 lines (205 loc) · 6.32 KB

file_io

Deterministic File Lib to make working with Files across Object Storage easier

Quickstart

!pip install --upgrade git+https://github.com/trisongz/file-io.git
!pip install --upgrade file-io


from fileio import File

pathlike = File('gs://path/to/item.txt')
pathlike = File('s3://path/to/item.txt')

Changelogs

May 21, 2022 v0.3.1

  • Complete Overhaul and refactor.

Aug 31, 2021 v0.3.0alpha

  • Major refactor to remove tensorflow as primary dependency
  • Started secondary support of gs using google-cloud-storage
  • Started primary support of s3 using tensorflow
  • Working on secondary support of s3 using aioaws
  • Planning to integrate async support
  • Planning to add deeper integration with smart_open
  • Planning to add support for supabase storage
  • Started adding auto-auth support: s3, gs, supabase
  • Added compat module for previous File API to prevent breakage
    • All previous File APIs are still usable.
    • Does not check for tensorflow dependency. So using without tensorflow will break

Aug 3, 2021 - v0.1.16

  • A lot. But its pretty lazily done.

July 7, 2021 - v0.1.15

  • Modified behavior of open and direct __call__
  • Remove Explicit need for Tensorflow in setup, but still require it at the moment.
    • This may help with macos Tensorflow installations using tensorflow-macos

July 2, 2021 - v0.1.13

  • Change .textread to return string rather than list
    • .textreadlines replaces original function
  • Update .textlist to support option for stripping newlines and have replacements
    • strip_newlines = True, will strip all newlines prior to return
    • replacements: [ list | dict | str ] = None, will iterate through and replace
  • Update .base(filename, with_ext=True) to allow return without File Extension
  • Add .readfile method to return .read() API
  • Add .mod_fname(filename, new_name=None, prefix=None, suffix=None, ext=None, directory=None, create_dirs=True, filename_only=False, space_replace='_')
    • src = 'gs://mybucket/path/file.txt'
    • res = File.mod_fname(src, newname='newfile', ext='json', directory='/newdir', prefix='test_', suffix='_001')
    • >> res = /newdir/test_newfile_001.json

June 30, 2021 - v0.1.11

  • Added Dill as default pickler if installed
  • Ability to set any pickle method that supports .dumps/.loads call with File.set_pickler(name='pickler') or File.set_pickler(function=cloudpickle)
  • Hotfix to change method to dumps/loads
  • Hotfix for .gsutil method which did not initialize properly.

June 11, 2021 - v0.1.8

  • Hotfix for methods .split_file/.split_files

June 9, 2021 - v0.1.7

  • Hotfix for Method .get_local
  • Hotfix for method .jlgs

May 28, 2021 - v0.1.6

  • Added Method to get User Dir
    • File.userdir

May 21, 2021 - v0.1.5

  • Added TSV/CSV Write Methods
    • File.csvwrite
    • File.tsvwrite

May 20, 2021 - v0.1.4

  • Hotfix for file.split_file(s) method to also return resulting filenames with output_files key

May 20, 2021 - v0.1.3

  • Py Version Requirement Fix

May 19, 2021 - v0.1.2

  • Minor Fixes
  • Added Methods for Splitting Files/Items
    • File.calc_splits
    • File.split_items
    • File.split_file
    • File.split_files

May 12, 2021 - v0.1.1

  • Minor Fixes
  • Added Method
    • File.fmv

May 12, 2021 - v0.1.0

  • Refactored Library
  • Organized Methods
  • Added MultiThreaded Wrapper
    • from fileio import MultiThreadPipeline
  • Added gsutil wrapper method
    • File.gsutil
  • Added Methods for Yaml
    • File.yload
    • File.yloads
    • File.ydump
    • File.ydumps
    • File.yparse
  • Updated Methods for Json
    • File.jsonload
    • File.jsonloads
    • File.jsondump
    • File.jsondumps
    • File.jp
    • File.jwrite
    • File.jg
    • File.jgs
  • Updated Methods for Jsonlines
    • File.jll
    • File.jlp
    • File.jldumps
    • File.jlwrite
    • File.jlwrites
    • File.jlg
    • File.jlgs
    • File.jlload
    • File.jlw
    • File.jlsample
  • Updated Methods for Text
    • File.textload
    • File.textwrite
    • File.textread
    • File.textlist
  • Added Methods for Requests
    • File.rget
    • File.rpost
    • File.reqsess
  • Added Methods for URL Encoding/Decoding
    • File.urlencode
    • File.urldecode
  • Added Methods for Hashing
    • File.hash
    • File.checkhash
  • Added Methods to Disable/Enable TQDM
    • File.enable_progress
    • File.disable_progress
  • Added Utility Methods
    • File.cat
    • File.backup
    • File.findir
    • File.append_ext
    • File.copydir
    • File.dirglob
    • File.absdir
    • File.get_local
    • File.finalize
    • File.print
    • File.set_printer
  • Fixed/Updated Methods
    • File.isfile
    • File.download
    • File.batch_download
    • File.pexists
    • File.whichpath
    • File.copy
    • File.bcopy
  • Added TFDSIODataset

Previous Version

from fileio import File

'''
Recognized File Extensions

.json               - json
.jsonl/.jsonlines   - jsonlines
.csv                - csv
.tsv                - tsv with "\t" seperator
.txt                - txtlines
.pkl                - pickle
.pt                 - pytorch
.tfrecords          - tensorflow
'''

# Main auto classes
File.open(filename, mode='r', auto=True, device=None) # device is specific to pytorch. Set auto=False to get a barebones Posix via Gfile
File.save(data, filename, overwrite=False) # if not overwrite, will attempt to append for newline files
File.load(filenames, device=None) # yields generators per file, meaning you can have different file types
File.download(url, dirpath=None, filename=None, overwrite=False) # Downloads a single url
File.gdown(url, extract=True, verbose=False) # uses gdown lib to grab a google drive drive

# Main i/o classes (Not Binary)
File.read(filename) # 'r'
File.write(filename) # 'w'
File.append(filename) # 'a'

# Binary
File.wb(filename) # 'wb'
File.rb(filename) # 'rb'


# Batch downloaders
File.batch_download(urls, directory=None, overwrite=False) # downloads all urls into a directory, skipping if overwrite = True and exists
File.batch_gdown(urls, directory=None, extract=True, verbose=False) # downloads all gdrive urls to a directory

# Extension Specific 

# .json
File.jsonload(filename)
File.jsondump(dict, filename)

# .jsonl/.jsonlines (Single File)
File.jlg(filename)
File.jlw(data, filename, mode='auto', verbose=True)

# Multifile Readers

# .jsonl/.jsonlines
File.jgs(filenames)