What is tokenization in the context of Test Mining?

 Posted by Niladri.Biswas on 5/6/2013 | Category: Others Interview questions | Views: 972 | Points: 40
Answer:

Tokenization is the process of breaking piece of text into smaller pieces like words, phrases, symbols and other elements which are called tokens. Even a whole sentence can be considered as a token. During the tokenization process some characters like punctuation marks can be removed. The tokens then become an input for other
processes in text mining like parsing.


Asked In: Many Interviews | Alert Moderator 

Comments or Responses

Login to post response