Best practices for entity linker pipeline

[lbelpaire]

Hi everyone

Just had some general questions on how to tackle a project. The goal is to build a system that processes news articles in multiple languages, extracts relevant company/business updates, and links them with a database containing company descriptions. Not all articles necessarily contain a relevant update.

Concrete questions:

• Should I build a classifier that decides whether an article contains a relevant update or not. If so, is a binary one okay, or should it be a multi-class classifier that also decides on the theme of the article? E.g. investment, product launch, …
• Is a classifier unnecessary if I just train a span classifier that is able to classify a sentence as the relevant update?
• How do I best tackle the entity linking training wise: I have a prodigy license and found the video about entity linking (https://spacy.io/universe/project/video-entity-linking) but I was wondering how to to best tackle this for a huge knowledge base (100k+), as we won’t have training articles/updates to link with all companies in our knowledge base (most of them will probably never be linked with an article). Our knowledge base will also keep expanding. If I just labeled X examples would that be enough to help the entity linker generalise for unseen companies in the knowledge base? (how big should X be?).

Any feedback or suggestions are very welcome :)

[rmitsch]

Hi @lbelpaire,

The goal is to build a system that processes news articles in multiple languages, extracts relevant company/business updates, and links them with a database containing company descriptions.

Can you provide an example of such an "company/business update"?

Should I build a classifier that decides whether an article contains a relevant update or not. If so, is a binary one okay, or should it be a multi-class classifier that also decides on the theme of the article? E.g. investment, product launch, …

If I understand your use case correctly, you'll do span classification and/or entity linking down the line anyway, so I don't think this would be beneficial (you'll be able to determine whether an article contains update by checking whether any relevant spans/entities have been detected in this case).

...as we won’t have training articles/updates to link with all companies in our knowledge base...

It's a bit unclear to me what kind of information exactly you want to detect (your business updates) and which entities you want to link to (only companies?). Can you elaborate on that?

[lbelpaire]

Hi @rmitsch, thanks for the quick response!

Can you provide an example of such an "company/business update"?

Example in the following article:

Edinburgh's Fanbase has closed a £1 million round led by payment service and software provider SaltPay.
Fanbase's sports franchise digitisation software enables mobile ticketing for spectator sports, allowing fans to pass through the turnstiles using a barcode on their smartphone. It also collects personalised audience data and lets sports teams design additional products aligned to their brand.

With the bold section as the update I would want to extract using span classification. After we can extract the two companies, Fanbase and SaltPay using (pre-trained should suffice?) NER. We have a database/knowledge base containing names and descriptions of companies. Using the entity linker we would want to link this update with the correct companies.

[rmitsch]

Right, so for the company linking part NER + NEL should be a good fit. As for how many examples you'd need to label, that's difficult to say in advance. It depends on the difficulty of the task and which kind of error rates and scenarios are acceptable to you. I'd recommend starting with maybe 500 labeled examples - establish a baseline by evaluating the results, then label a few hundred more. Rinse and repeat until your evaluation metrics stop increasing.

I'm more skeptical about the span classification. Do you have a concise definition of what constitutes an business update?

[lbelpaire]

Alright thanks a lot!

I assume that it is important to also include examples of companies the entity linker has not seen when evaluating performance, to get a good idea on how well it would perform in production.

As for the concise definition of business updates I would have several classes:

• Investment: X invests in Y (as in the article mentioned previously), company X raises Y amount
• Product launch: Company X launches a product
• Hiring: company X appoints Y as new CEO

Is this a use case for span classification? Or would you recommend other techniques? (e.g. extract key sentence from article then classify which kind of update it is?)

[rmitsch]

In general you should always follow a train/dev/test set approach. You can certainly do that by separating by company - I don't know if and to which degree it would make your model more robust though.

I think this is a reasonable use case for spancat, yes. Since you have a very narrowly defined use case, you could also start with a naive rule-based sentence/span classification to establish a baseline. E. g. if a sentence contains "invest*", classify it as investment; "launch*" -> product launch; "hiring" -> hiring. Ideally more elaborate and with more terms, but you get the idea. You'll have to come up with a way to resolve conflicts, but that should be doable.

This is cheap and easy to implement and will help you evaluate the quality of your trained model later.

[lbelpaire]

Thanks a lot!