I contributed to an article titled “Understanding Text Summarization with AI” on the Pure AI web site. See https://pureai.com/Articles/2024/07/01/text-summarization.aspx.

One type of natural language problem where AI is generally regarded as being at least as good as humans, and in many cases superior to humans, is text summarization. Examples include summarizing the contents of a long email message down to a couple of sentences, or summarizing the contents of a research paper to a paragraph.

I explained that there ate two different approaches for text summarization. One approach is to use an AI service such as Microsoft Azure or Google Cloud Services or Amazon SageMaker. This is a high-level approach and has the advantage of being relatively easy to use.

The five main disadvantages of using an AI service are:

* Doing so locks you into the service company’s ecosystem to a great extent
* It’s difficult to customize the text summarization service for specialized scenarios
* Security can be an issue if sensitive information is being summarized
* AI services are mostly black-box systems where it’s often difficult or impossible to know what the system is doing
* A summarization service can be pricey if large numbers of documents must be summarized

A second approach for doing text summarization is to write in-house computer programs to do so. This approach gives maximum flexibility but requires in-house programming expertise or the use of a vendor with programming expertise.

I contributed an example that summarizes the first four paragraphs of the Wikipedia article on World War I. The summary result is:

[{'summary_text': ' World War I or the First World War (28
 July 1914 - 11 November 1918) was a global conflict between
 the Allies (or Entente) and the Central Powers . Fighting
 took place mainly in Europe and the Middle East, as well as
 parts of Africa and the Asia-Pacific . The causes of the
 conflict included the rise of Germany and decline of the
 Ottoman Empire, which disturbed the balance of power in
 place for most of the 19th century .'}]

In the article, I toss out a couple of opinions:

Dr. McCaffrey offered an opinion, “I think for most organizations and most text summarization scenarios, using an existing large language model that has been trained using general purpose data, such as news articles, is relatively simple and effective. The most difficult part of deploying a text summarization system will likely be integrating the system with other systems in the organization.”

McCaffrey added, “On the other hand, fine-tuning a pretrained model with domain-specific data is much more difficult than might be expected. Most of the time and effort will be directed at preparing the summary part of the training data, which must be done manually in most cases.”

Two of the books I love the most are compilations of short stories. “The Lawrenceville Stories” by Owen Johnson (1878-1952) has five stories about life in a boys prep school circa 1900. “The Original Illustrated Sherlock Holmes” by Arthur Conan Doyle (1859-1930) has 37 short stories plus one short novel and needs no explanation. It’s not an exaggeration to say that these two books changed my life for the better when I was a teenager.