Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for reading from a WebStreams #635

Merged
merged 13 commits into from
Jul 7, 2024
1 change: 1 addition & 0 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ jobs:
fail-fast: false
matrix:
node-version:
- 22
- 20
- 18
steps:
Expand Down
29 changes: 0 additions & 29 deletions browser.d.ts

This file was deleted.

15 changes: 0 additions & 15 deletions browser.js

This file was deleted.

19 changes: 12 additions & 7 deletions core.d.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
import type {Readable as ReadableStream} from 'node:stream';
/**
* Typings for primary entry point, Node.js specific typings can be found in index.d.ts
*/

import type {ReadableStream as WebReadableStream} from 'node:stream/web';
Copy link
Owner

@sindresorhus sindresorhus Jul 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This type should be available globally, so I don't think we need to import it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The global type in incompatible with Node.js type. If I change it here, the problem will appear elsewhere,

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what that means, but this is core.d.ts, so it shouldn't import types only available for Node.js.

Copy link
Collaborator Author

@Borewit Borewit Jul 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Node.js Readable types are incompatible with the global lib.dom.d.ts types, one things the challenges mentioned in #588 (comment), related to this issue DefinitelyTyped/DefinitelyTyped#60377, which unfortunately got closed with PR aiming to resolve that.

I used the Node.js, as Node.js has been the primary supported platform. Using the lib.dom.d.ts I need hack in types mapping cast somewhere.

Maybe good accept both, an do an ugly type cast, just for the convenience for our users.

Off-topic: The other mind f*ck, was the BYOB Readable Stream, The "Bring-Your-Own-Buffer" reader:

defines a reader for a ReadableStream that supports zero-copy reading from an underlying byte source

Well the first things the zero-copy method does is hijacking the buffer you bring, and essentially turns into junk (there is formal property for this state, forgot the name), meaning it can no longer be used. Which essentially forces you to create a new Buffer (as it becomes totally useless after providing it), and then copying the data to the Buffer you wanted to have it written in the first place. So the only feature BYOB has, versus the ReadableStreamDefaultReader is that you can control the chunk length to be written. How confusing, what a disappointment.

import type {Readable as NodeReadableStream} from 'node:stream';
import type {ITokenizer} from 'strtok3';

export type FileExtension =
Expand Down Expand Up @@ -318,7 +323,7 @@ export type FileTypeResult = {
readonly mime: MimeType;
};

export type ReadableStreamWithFileType = ReadableStream & {
export type ReadableStreamWithFileType = WebReadableStream & {
readonly fileType?: FileTypeResult;
};

Expand All @@ -339,10 +344,10 @@ Detect the file type of a Node.js [readable stream](https://nodejs.org/api/strea

The file type is detected by checking the [magic number](https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files) of the buffer.

@param stream - A readable stream representing file data.
@param stream - A Node.js Readable stream or Web API Readable Stream representing file data. The Web Readable stream **must be a byte stream**.
@returns The detected file type, or `undefined` when there is no match.
*/
export function fileTypeFromStream(stream: ReadableStream): Promise<FileTypeResult | undefined>;
export function fileTypeFromStream(stream: WebReadableStream): Promise<FileTypeResult | undefined>;

/**
Detect the file type from an [`ITokenizer`](https://github.com/Borewit/strtok3#tokenizer) source.
Expand Down Expand Up @@ -420,7 +425,7 @@ if (stream2.fileType?.mime === 'image/jpeg') {
}
```
*/
export function fileTypeStream(readableStream: ReadableStream, options?: StreamOptions): Promise<ReadableStreamWithFileType>;
export function fileTypeStream(readableStream: WebReadableStream<Uint8Array>, options?: StreamOptions): Promise<ReadableStreamWithFileType>;

/**
Detect the file type of a [`Blob`](https://nodejs.org/api/buffer.html#class-blob) or [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File).
Expand Down Expand Up @@ -511,7 +516,7 @@ export declare class FileTypeParser {
/**
Works the same way as {@link fileTypeFromStream}, additionally taking into account custom detectors (if any were provided to the constructor).
*/
fromStream(stream: ReadableStream): Promise<FileTypeResult | undefined>;
fromStream(stream: NodeReadableStream): Promise<FileTypeResult | undefined>;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this and toDetectionStream support web streams too?

Copy link
Collaborator Author

@Borewit Borewit Jul 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, should probably be done as well. (see #636)


/**
Works the same way as {@link fileTypeFromTokenizer}, additionally taking into account custom detectors (if any were provided to the constructor).
Expand All @@ -526,5 +531,5 @@ export declare class FileTypeParser {
/**
Works the same way as {@link fileTypeStream}, additionally taking into account custom detectors (if any were provided to the constructor).
*/
toDetectionStream(readableStream: ReadableStream, options?: StreamOptions): Promise<FileTypeResult | undefined>;
toDetectionStream(readableStream: NodeReadableStream, options?: StreamOptions): Promise<FileTypeResult | undefined>;
}
9 changes: 6 additions & 3 deletions core.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
/**
* Primary entry point, Node.js specific entry point is index.js
*/

import * as Token from 'token-types';
import * as strtok3 from 'strtok3/core';
import {includes, indexOf, getUintBE} from 'uint8array-extras';
Expand Down Expand Up @@ -88,12 +92,11 @@ export class FileTypeParser {
}

async fromBlob(blob) {
const buffer = await blob.arrayBuffer();
return this.fromBuffer(new Uint8Array(buffer));
return this.fromStream(blob.stream());
}

async fromStream(stream) {
const tokenizer = await strtok3.fromStream(stream);
const tokenizer = await strtok3.fromWebStream(stream);
try {
return await this.fromTokenizer(tokenizer);
} finally {
Expand Down
31 changes: 24 additions & 7 deletions index.d.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,30 @@
import type {FileTypeResult} from './core.js';

/**
Detect the file type of a file path.
* Typings for Node.js specific entry point
*/

import type {Readable as NodeReadableStream} from 'node:stream';
import type {ReadableStream as WebReadableStream} from 'node:stream/web';
import type {FileTypeResult} from './core.js';
import {FileTypeParser} from './core.js';

The file type is detected by checking the [magic number](https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files) of the buffer.
export declare class NodeFileTypeParser extends FileTypeParser {
/**
*
* @param stream Node.js Stream readable or Web API StreamReadable
*/
fromStream(stream: WebReadableStream<Uint8Array> | NodeReadableStream): Promise<FileTypeResult | undefined>;
}

@param path - The file path to parse.
@returns The detected file type and MIME type or `undefined` when there is no match.
*/
/**
* Detect the file type of a file path.
*
* The file type is detected by checking the [magic number](https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files) of the buffer.
*
* @param path
* @returns The detected file type and MIME type or `undefined` when there is no match.
*/
export function fileTypeFromFile(path: string): Promise<FileTypeResult | undefined>;

export function fileTypeFromStream(stream: WebReadableStream<Uint8Array> | NodeReadableStream): Promise<FileTypeResult | undefined>;

export * from './core.js';
22 changes: 21 additions & 1 deletion index.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
/**
* Node.js specific entry point
*/

import {ReadableStream as WebReadableStream} from 'node:stream/web';
import * as strtok3 from 'strtok3';
import {FileTypeParser} from './core.js';

export class NodeFileTypeParser extends FileTypeParser {
async fromStream(stream) {
const tokenizer = await (stream instanceof WebReadableStream ? strtok3.fromWebStream(stream) : strtok3.fromStream(stream));
try {
return super.fromTokenizer(tokenizer);
} finally {
await tokenizer.close();
}
}
}

export async function fileTypeFromFile(path, fileTypeOptions) {
const tokenizer = await strtok3.fromFile(path);
try {
Expand All @@ -11,4 +27,8 @@ export async function fileTypeFromFile(path, fileTypeOptions) {
}
}

export * from './core.js';
export async function fileTypeFromStream(stream, fileTypeOptions) {
return (new NodeFileTypeParser(fileTypeOptions)).fromStream(stream);
}

export {fileTypeFromBuffer, fileTypeFromBlob, fileTypeStream, FileTypeParser, supportedMimeTypes, supportedExtensions} from './core.js';
9 changes: 5 additions & 4 deletions index.test-d.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import {createReadStream} from 'node:fs';
import {ReadableStream as WebReadableStream} from 'node:stream/web';
import {expectType} from 'tsd';
import {
type FileTypeResult as FileTypeResultBrowser,
} from './browser.js';
} from './core.js';
import {
fileTypeFromBlob,
fileTypeFromBuffer,
Expand Down Expand Up @@ -54,13 +55,13 @@ expectType<ReadonlySet<FileExtension>>(supportedExtensions);

expectType<ReadonlySet<MimeType>>(supportedMimeTypes);

const readableStream = createReadStream('file.png');
const streamWithFileType = fileTypeStream(readableStream);
const webStream = new WebReadableStream<Uint8Array>();
const streamWithFileType = fileTypeStream(webStream);
expectType<Promise<ReadableStreamWithFileType>>(streamWithFileType);
(async () => {
const {fileType} = await streamWithFileType;
expectType<FileTypeResult | undefined>(fileType);
})();

// Browser
expectType<Promise<FileTypeResultBrowser | undefined>>(fileTypeFromBlob(new Blob()));
expectType<Promise<FileTypeResultBrowser | undefined>>(fileTypeFromBlob(new Blob([])));
5 changes: 1 addition & 4 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"exports": {
".": {
"node": "./index.js",
"default": "./browser.js"
"default": "./core.js"
},
"./core": "./core.js"
},
Expand All @@ -28,8 +28,6 @@
"files": [
"index.js",
"index.d.ts",
"browser.js",
"browser.d.ts",
"core.js",
"core.d.ts",
"supported.js",
Expand Down Expand Up @@ -210,7 +208,6 @@
"fbx"
],
"dependencies": {
"readable-web-to-node-stream": "^3.0.2",
"strtok3": "^7.1.0",
"token-types": "^6.0.0",
"uint8array-extras": "^1.3.0"
Expand Down
4 changes: 3 additions & 1 deletion readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ The file path to parse.

### fileTypeFromStream(stream)

Detect the file type of a Node.js [readable stream](https://nodejs.org/api/stream.html#stream_class_stream_readable).
Detect the file type of a [Node.js readable stream](https://nodejs.org/api/stream.html#stream_class_stream_readable) or a [Web API ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream).

The file type is detected by checking the [magic number](https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files) of the buffer.

Expand All @@ -168,6 +168,8 @@ A readable stream representing file data.

Detect the file type of a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob).

It will **stream** the underlying Blob, and required a [ReadableStreamBYOBReader](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamBYOBReader) which **require Node.js ≥ 20**.

The file type is detected by checking the [magic number](https://en.wikipedia.org/wiki/Magic_number_(programming)#Magic_numbers_in_files) of the buffer.

Returns a `Promise` for an object with the detected file type:
Expand Down
Loading