Working with informal online textual data

[SupaHam]

Hi everyone,

I'm trying out spacy and am impressed with the provided functionality and usability. One problem that comes up in my use-case is my dataset of chatter tends to be rushed messages, and not properly written out. "Someone" might be spelt "Some one" or "some 1", "looooooooool" might be better replaced with a simple "lol". I'm not very well-versed in NLP technology and spacy in general, so any guidance or tips here would be very much appreciated

Thank you

[rmitsch]

Hi @SupaHam, sounds like you're looking for spelling correction. spaCy doesn't do that natively. There is the contextualSpellCheck extension though - I recommend you give that one a shot. There are also plenty of other libraries out there worth trying, e.g. https://github.com/wolfgarbe/SymSpell.

[SupaHam]

Hi rmitsch, thanks for your quick response. I've had a look at contextualSpellCheck and I prefer SymSpell over the two. contextualSpellCheck seems to have quite intense overhead in my brief tests, and SymSpell has a few ports, but it seems the symspellpy works best and most up to date. Thanks again for your help!