Types of Scoring Techniques for Extractive Text Summarization

 Scoring Techniques:

  • Proper noun
  • Sentence length character
  • Sentence length words
  • Sentence position
  • Word frequency
  • Numerical value
  • Named entity
  • Iterative query score
  • Cue words 



Word frequency:

As the name suggests the more the frequency of the word in the sentence the higher will be it's score. In other words the sentence containing the most frequent words of the document has high chances of getting selected for the Final summary. This is based on the assumption that the higher the frequency of the word in the text , it is more likely to be related to the subject of the document.


S(L) = N(w) / N(d)


Where,

N(w) =  Sum of the frequency of the words of the sentence

N(d) = Sum of the frequency of the words of the document.



Proper noun:

Proper noun refers to an individual,place or organisation. It is considered to be carrying greater information from the rest of the words.

The Sentence containing a higher number of proper nouns is more  likely to be selected for the final summary. It is a specialisation of the upper case method. 


The score is calculated as the number of proper nouns of the sentence to the number of proper noun of the document.


S(P) = N(P) / N(D)


Where, 

N(P) = Number of the proper nouns of the sentence

N(D) = Number of the proper nouns of the document.


Sentence length character:

This method is used for calculating the sentence score lengthwise. The average sentence length is calculated by following function given below:


Avg(L)= max(L)+min(L)/2


Where,

max(L) = maximum length of the sentence (character-wise)

min(L) = minimum length of the sentence (character-wise)


Score= ( | Sl -Avg(L) | ) / Msl


Where,

Sl= Sentence length

Avg(L) = The average sentence length of the document

Msl = The maximum sentence length


Sentence length words:

This method is used for calculating the sentence score lengthwise. The average sentence length is calculated by following function given below:


Avg(L)= max(L)+min(L)/2


Where,

max(L) = maximum length of the sentence (word-wise)

min(L) = minimum length of the sentence (word-wise)


Score= ( | Sl -Avg(L) | ) / Msl


Where,

Sl= Sentence length

Avg(L) = The average sentence length of the document

Msl = The maximum sentence length


Sentence position:

It is assumed to be the most important feature for sentence scoring. Normally a few sentences at the beginning and at the end are considered to be more important than the other sentences. They are more likely to be selected for the final summary.


Numerical value:

Numerical values represent important figures. According to this method the sentence containing numerical values are considered to be more important than other sentences.


S(L) = Nnv / T


Where,

Nnv = Number of numerical values in the sentence

T = Total number of numerical values in the document.


Named entity:

The sentence containing more number of named entities will be considered to be having more weight than other sentences.


S(L)= Net / T


Where,

Net =  Number or named entity of the sentence

T =  Number of named entities of the document.




Iterative query score:

In this technique first we calculate the frequency of the words contained in the document and then the words are sorted based on their frequency. Then the top frequent words are chosen from the sorted list which is known as the initial keyword set. Then the sentence containing the higher number of the initial keywords are considered to be more important than other sentences and they have high chances to get selected for the final summary.


Score= Nit / T


Where,

Nit = Number of the initial keywords of the sentence

T = Total number of initial keywords of the document.


Cue words :

In this technique the sentences are scored based on the number of cue words it contains. For using this technique a set of cue words like ‘In brief’ ,’Summing up’, ‘thus’, ‘to conclude’ has to prepared.


Score = Ncw / T


Where,

Ncw = Number of cue words in the sentence

T = Number of cue words of the document

Comments

Post a Comment

Popular posts from this blog

Cuckoo Search Algorithm

Image
What is Cuckoo Search Algorithm? Cuckoo search is an optimization algorithm used in Natural Language Processing. It was inspired by the obligate brood parasitism of some cuckoo species by laying their eggs in the nests of host birds of other species. Some host birds can engage direct conflict with the intruding cuckoos.  For example, if a host bird discovers the eggs are not their own, it will either throw these alien eggs away or simply abandon its nest and build a new nest elsewhere. Cuckoo search idealized such breeding behavior, and thus can be applied for various optimization problems. It has been shown that cuckoo search is a special case of the well-known (μ + λ)-evolution strategy. Cuckoo search (CS) uses the following representations: Each egg in a nest represents a solution, and a cuckoo egg represents a new solution. The aim is to use the new and potentially better solutions (cuckoos) to replace a not-so-good solution in the nests. In the simplest form, each nest has one...
Read more

Different Types of Text Summarization

Image
  Broad Classification: •            Single Document Summarization Ø    Given a single document to generate a summary   •            Multiple Document Summarization Ø    Given a group of documents to generate a summary out of it   •            Generic Summarization Ø    Summarize the content of a document   •            Domain Specific Ø    Text summarization aims at extracting the essential information from a text to produce a shorter version,  such as generating headlines for news and subject lines for emails.   •            Query-focused Summarization Ø    Summarize a document with respect to an information need expressed in a user qu...
Read more

Introduction to Text Summerization

Image
What is Text Summerization ? Text Summerization is the process of summarizing long text into short piece of text. It's main focus is to only add necessary and important sentences and text into the final summary. The goal of Text Summerization is to provide a shorter version of the source text containing only important aspects of the source text. It can be used in if we want to get only important information from a very long source text.  Why Text Summerization is used ? Text Summerization provides us a shorter version of long text containing only the important and necessary points and aspects of the source file. It helps in ignoring irrelevant information from the source text. It saves the reader's time by reducing the text size. It is mainly used to make precise documents of  long publications. How Text Summerization summerize the text ? Firstly it separates the document sentence wise. Then it uses various word scoring and sentence scoring techniques to decide score's of i...
Read more