#

subword-tokenization

Here are 6 public repositories matching this topic...

DolbyUUU / byte_pair_encoding_BPE_subword_tokenization_implementation_python

Byte-Pair Encoding (BPE) (subword-based tokenization) algorithm implementaions from scratch with python

python nlp natural-language-processing tokenizer data-preprocessing data-cleaning bpe byte-pair-encoding subword-tokenization

Updated Jan 30, 2023
Python

SD7Campeon / Comment-Toxicity-Detection-and-Classification

LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.

nlp sklearn transformer discourse-analysis multi-label-classification affective-computing keras-tensorflow text-vectorization bilstm nlp-pipeline deep-sequential-model toxicity-analysis toxicity-prediction toxicity-detection toxicity-classification llm subword-tokenization real-time-inference contextual-nlp

Updated May 15, 2025
Jupyter Notebook

kennethsible / segmented-permutation

An Artiﬁcial Subword Translation Task

machine-learning seq2seq subword-tokenization

Updated May 16, 2025
Jupyter Notebook

Simparaisco / Comment-Toxicity-Detection-and-Classification

LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.

nlp transformer discourse-analysis multi-label-classification affective-computing keras-tensorflow bilstm toxic-comments nlp-pipeline deep-sequential-model toxicity-analysis toxicity-prediction toxicity-detection toxicity-classification llm subword-tokenization real-time-inference contextual-nlp

Updated May 19, 2025
Jupyter Notebook

shivendrra / shredword

Fast & efficient BPE tokenizer written in C & python for LLM tranining

natural-language-processing tokenizer word-segmentation tokenization subword-segmentation subword-tokenization tiktoken

Updated May 18, 2025
C++

Tripp01 / Comment-Toxicity-Detection-and-Classification

LLM-inspired BiLSTM pipeline for real-time, multi-label toxicity inference across adversarial discourse modalities.

nlp transformer discourse-analysis multi-label-classification affective-computing keras-tensorflow bilstm toxic-comments nlp-pipeline deep-sequential-model toxicity-analysis toxicity-prediction toxicity-detection toxicity-classification llm subword-tokenization real-time-inference contextual-nlp

Updated May 21, 2025
Jupyter Notebook

Improve this page

Add a description, image, and links to the subword-tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the subword-tokenization topic, visit your repo's landing page and select "manage topics."