Challenges of sentence tokenization

At first glance, you would ask, what's the big deal about finding out the sentence boundary from the given raw text?

Sentence tokenization varies from language to language.

Things get complicated when you have the following scenarios to handle. We are using examples to explain the cases:

• If there is small letter after a dot, then the sentence should not split after the dot. The following is an example:
  • Sentence: He has completed his Ph.D. degree. He is happy.
  • In the preceding example, the sentence tokenizer should split the sentence after degree, not after Ph.D.
• If there is a small letter after the dot, then the sentence should be split after the dot. This is a common mistake. Let's take an example:
  • Sentence: This is an apple.an apple is good for health.
  • In the preceding example, the sentence tokenizer should split the sentence after apple.
• If there is an initial name in the sentence, then the sentence should not split after the initials:
  • Sentence: Harry Potter was written by J.K. Rowling. It is an entertaining one.
  • In the preceding example, the sentence should not split after J. It should ideally split after Rowling.

• Grammarly Inc., the grammar correction software, customized a rule for the identification of sentences and achieves high accuracy for sentence boundary detection. See the blog link:
  https://tech.grammarly.com/blog/posts/How-to-Split-Sentences.html.

To overcome the previous challenges, you can take the following approaches, but the accuracy of each approach depends on the implementation. The approaches are as follows:

• You can develop a rule-based system to increase the performance of the sentence tokenizer:
  • For the previous approach, you can use name entity recognition (NER) tools, POS taggers, or parsers, and then analyze the output of the described tools, as well as the sentence tokenizer output and rectify where the sentence tokenizer went wrong. With the help of NER tools, POS taggers, and parsers, can you fix the wrong output of the sentence tokenizer. In this case, write a rule, then code it, and check whether the output is as expected.
  • Test your code! You need to check for exceptional cases. Does your code perform well? If yes, great! And, if not, change it a bit:
    • You can improve the sentence tokenizer by using machine learning (ML) or deep learning techniques:
      • If you have enough data that is annotated by a human, then you can train the model using an annotated dataset. Based on that trained model, we can generate a new prediction from where the sentence boundary should end.
      • In this method, you need to check how the model will perform.