What is tokenization in the context of Test Mining?

 Posted by Niladri.Biswas on 5/6/2013 | Category: Others Interview questions | Views: 899 | Points: 40

Tokenization is the process of breaking piece of text into smaller pieces like words, phrases, symbols and other elements which are called tokens. Even a whole sentence can be considered as a token. During the tokenization process some characters like punctuation marks can be removed. The tokens then become an input for other
processes in text mining like parsing.

Asked In: Many Interviews | Alert Moderator 

Comments or Responses

Login to post response