Does Floret support Fasttext trained models?

[alanault]

The Floret documentation seems to suggest that it's compatible with Fasttext - i.e. train with -mode fasttext

However, if I try to load a model trained with Fasttext, Floret hangs and spins the CPU to 100%. Does this mean that although Floret will return a fasttext-esque model, it's only compatible with Floret?

I ask as we have lots of existing Fasttext trained models in production and would love to adopt Floret, but need to be able to load existing models, rather than re-training everything. Obviously we can run Fasttext and Floret in parallel, but it's not ideal.

We're testing using the command line version to avoid any Python issues.

[adrianeboyd]

floret can't currently load .bin files produced by fasttext (and vice versa).

The actual differences are very small, just two added settings in the arguments, and it might be possible to make it backwards-compatible without a lot of effort? The direct changes are in Args::save/load.

https://github.com/explosion/floret/blob/b3078fe77db75d13d81d61b4fc64623d433f4767/src/args.cc#L361-L395

But I think it might be hard to auto-detect the format.

It might be easier to write a short program that loads a model as fasttext, adds the default values for these two fields, and writes it back out for floret?

[adrianeboyd]

Ah, looking at this again, floret probably should have used a new FASTTEXT_VERSION or similar.

[alanault]

Interestingly, we typically use quantized versions of Fasttext models in production so *.ftz. Would these also work? I noticed that the documentation suggests you can quantize Floret-based models?

[adrianeboyd]

Interesting, I haven't experimented with this at all.

I don't think it makes sense to use floret mode for the pretrained vectors because you're only providing the .vec table. There's a slight chance you might see very slight improvements if you use more than one hash and also a large bucket size. But I think the default ngram bucket size is already so large that you probably wouldn't see much of a difference. And plain fasttext vectors will be slightly faster to train.

I haven't tried using floret mode with supervised models, but my guess is that the quantization would probably just work because the internal representations should be similar enough to fasttext. I'd be interested to hear if it works?

[alanault]

Ah - one of the key drivers for us to use Floret was so that we can get a smaller vector table that also included the sub-word information. As well as being smaller we hoped to pass these instead of the *.vec file to train supervised mode and benefit from sub-word information.

Is it possible to pass the Floret table directly using the pre_tarined_vectors option to utilise sub-words?

[adrianeboyd]

I took a look and the current "supervised" implementation currently only supports reading in the .vec table, not the .bin format with both tables. It might be possible, but I'm not sure how difficult it would be to implement.

[alanault]

Ah interesting; we were hoping the .floret file would allow us to have a smaller vector space. I'm a bit confused as to what the Floret file contains? if the .vec is the word vectors, and the .bin is the model + sub_words, what is in .floret? I thought it was the hashed version of the .bin (it's roughly half the size), but I can't actually load it in floret for nn, or analogies etc? Instead, I have to load the .bin... in which case how and where does the .floret get used without the .bin? (we'd assumed we'd get a tiny .bin file, rather than having to use that + .floret?)

[adrianeboyd]

In order not to change much about how the internals work, the .bin file is basically the same as for fasttext. It contains the vector table for the known words in the vocab (the data in .vec) plus the hash table (the .floret data). In floret mode, it is true that you could derive the entire large .vec table from the hash table, but in fasttext mode you need both tables to derive the vectors.

If you try it out with all settings the same except fasttext vs. floret mode (with a smaller bucket size), the floret .bin files are smaller because you have ~100K rows in the hash table rather than 2M, but the vector table part stays the same size.

The spacy pipelines only import the .floret tables (with spacy init vectors), so they stay small. We were more interested in using the vectors as features for the statistical models like the tagger or NER models and less in using them for similarity comparisons.

It is a problem that there's no easy way to do nearest neighbors in a spacy pipeline with floret vectors, which is because there is no word list and no stored vectors for individual words to compare against. To use spacy's built-in similarity methods, you actually have to export the vectors for some external word list into a separate (larger) spacy pipeline with default vectors, and then you can run most_similar.

Basically we only cared about the .floret table, which is why we didn't spend a lot of time on the details related to the .bin file format. I've made a note to take another look at the file headers to see if we can address some of the compatibility and loading problems, but I can't make any promises about when I'll have time to work on it: explosion/floret#19