Qualitative Analysis, Part II

Data Literacy

Coding/Tagging Data at Scale: Natural Language Processing

  • Natural Language Processing (NLP): a subfield of artificial intelligence that focuses on the interaction between computers and humans through natural language.
  • Text Analysis: a set of techniques used to extract useful information from text data.

Machine Readable Text & Optical Character Recognition (OCR)

  • Text a machine can read is called machine-readable text.
  • Optical Character Recognition (OCR): converts images of text into machine-readable text.

Text Analysis

  • Catch all for working with text
  • Text-clearning
  • Text-searching

Cleaning Text Data

  • stop words
  • punctuation
  • case
  • stemming: reducing words to their root form

Cleaning Text: Example

The qu1ck br0wn f0x jump5overthe iazy dog.

Sentiment Analysis: Supervised

  • Evaluates the sentiment of text data: positive, negative, neutral.
  • Usually based on a pre-defined dictionary of words/training.
  • Sentiment: the attitude or opinion expressed in text.

Named Entity Recognition (NER): Supervised

  • Trains a model to find pre-defined entities within text
  • Entities: People, places, organizations, dates, etc.
  • Must use pre-trained models
  • Or train model yourself.

NER Example: Headlines

  • Entities: ORG, GPE, MONEY
  • Text: Apple is looking at buying U.K. startup for $1 billion.

NER Example: Headlines training (cont.)


TRAIN_DATA = [
    ("Apple is looking at buying U.K. startup for $1 billion.", 
     {"entities": [(0, 5, "ORG"), (24, 26, "GPE"), (39, 48, "MONEY")]}),
    ("San Francisco considers banning sidewalk delivery robots.",
     {"entities": [(0, 13, "GPE")]})
]
  • Tag subset of data
  • Train on 80% of data
  • Test on 20% of data

NER Example: Headlines training (cont.)

Output:

  • Processed Text: Apple is looking at buying U.K. startup for $1 billion.
  • Text: Apple, Start: 0, End: 5, Label: ORG
  • Text: U.K., Start: 24, End: 26, Label: GPE
  • Text: $1 billion, Start: 39, End: 48, Label: MONEY

Topic Modeling: Unsupervised

  • Discovers abstract topics within text data.
  • LDA (Latent Dirichlet Allocation): most common topic modeling technique.
  • Topics: a collection of words that represent a theme.
  • Documents: a collection of text data.
  • Corpus: a collection of documents.
  • Finds “topics” across a corpus of documents

LDA and Clustering

  • Based on creating a bag of words
  • Models distribution of words in a document.
  • Documents = mixture of topics
  • Topics = mixture of words

LDA Example

  • Doc1: “The quick brown fox”
  • Doc2: “The lazy dog went to town”
  • BoW: [“The”, “quick”, “brown”, “fox”, “lazy”, “dog”, “went”, “to”, “town”]
  • Vector1 = [1, 1, 1, 1, 0, 0, 0, 0, 0]
  • Vector2 = [1, 0, 0, 0, 1, 1, 1, 1, 1]

Topic Modeling Output

  • Topics and their key words:
  • Topic 0: 0.200“quick” + 0.200”brown” + 0.200“fox” + 0.200”very” + 0.200*“dog”
  • Topic 1: 0.250“lazy” + 0.250”dog” + 0.125“quick” + 0.125”brown” + 0.125“fox” + 0.125”town”
  • Document-Topic Distributions:
  • Document 1: [(0, 0.5), (1, 0.5)]
  • Document 2: [(1, 1.0)]
  • Document 3: [(0, 1.0)]

Tools

  • Python: nltk, spaCy, gensim
  • R: tm, topicmodels, text2vec
  • MALLETT: