[WIP] Jupyter-Fsspec Roadmap

[RRosio]

🚀 Jupyter-Fsspec Roadmap [WIP]

📌 Project Overview

• Description: This project aims to integrate fsspec into JupyterLab, enabling seamless file system interaction across various backends as supported by fsspec (local, cloud, and memory). By providing both a server-side API and a frontend UI, users will be able to browse, open, and manage files efficiently within JupyterLab.

• Scope:

  • Backend Integration: Expose CRUD file operations via a JupyterLab server extension using fsspec.
  • Frontend Integration: Provide a custom file browsing UI for both JupyterLab and Notebook, that interacts with the server extension and integrates with JupyterLab’s existing file viewers (not intended to replace or extend the default JupyterLab file browser).
  • File Rendering: Ensure that files are displayed using JupyterLab’s existing rendering mechanisms.
  • Data Handling: Support structured data previews and potential transformation operations.
  • Extensibility: Design with flexibility in mind to support additional backends and new file operations in the future.
  • Testing: Verify configuration, filesystem instances and frontend state. As well as testing the server API functions with various file systems.

🎯 Goals

• File system interaction in Jupyter using fsspec
  • UI and server-side API for browsing, opening, and managing files across multiple backends.
  • Efficient handling of large directories and diverse file types.
  • Support core file operations (copy, move, delete).
• Facilitate structured data handling and transformation
  • Enable users to preview and interact with structured data formats.

🛠 Development Phases & Timeline

Phase	Milestones	Estimated Completion
Phase 1	Core feature implementation	YYYY-MM-DD
Phase 2	UI improvements & optimizations	YYYY-MM-DD
Phase 3	Additional integrations & testing	YYYY-MM-DD

Phase 1: Core feature implementation

📌 Feature List & Tasks

📂 Backend

• File-based config
  • Improve error handling for config Config error handling updates #74
• Support any fsspec backend
• Listing operations
• Get/put bytes operations
• Sync filesystems [WIP] Add sync handler base #73

🖥 Frontend

• List files, icons, file info (size)
• Lazy loading
• URL/code block context menu
• Upload/download with UI and local filesystem picker
  • Upload from browser file picker (local, remote)
  • Upload bytes from kernel
  • Upload from Jupyter file browser (local, remote)

📄 Data

• Code block context for access from kernel
• Integration with bytes in the kernel (helper: get bytes)
• Integrate with JupyterLab’s file viewer to render files according to their type Integrate with JupyterLab’s file viewer to render files according to their type #77

🔧 Testing

• Test remote write caching

🔧 Maintenance

• TBD: Refactoring, error handling etc.

🔮 Future

• Fully async/non-blocking
• Download to current notebook directory (or Jupyter root)
• Download via helper
• Click-to-preview functionality
• Intake catalog specification/editing
• Make server independent as a bytes proxy
• fsspec/pyodide integration
• Explicit link to projects: filesystem config and/or anaconda: remote
• Drag & Drop

Aspirational Data Goals 🏆

• Auto data source setup and configure UI
• Catalog sharing
• Artefact viewer
• Data config as part of a project