path: root/demo/Stanza_Beginners_Guide.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Stanza-Beginners-Guide.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "56LiYCkPM7V_",
        "colab_type": "text"
      },
      "source": [
        "# Welcome to Stanza!\n",
        "\n",
        "![Latest Version](https://img.shields.io/pypi/v/stanza.svg?colorB=bc4545)\n",
        "![Python Versions](https://img.shields.io/pypi/pyversions/stanza.svg?colorB=bc4545)\n",
        "\n",
        "Stanza is a Python NLP toolkit that supports 60+ human languages. It is built with highly accurate neural network components that enable efficient training and evaluation with your own annotated data, and offers pretrained models on 100 treebanks. Additionally, Stanza provides a stable, officially maintained Python interface to Java Stanford CoreNLP Toolkit.\n",
        "\n",
        "In this tutorial, we will demonstrate how to set up Stanza and annotate text with its native neural network NLP models. For the use of the Python CoreNLP interface, please see other tutorials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yQff4Di5Nnq0",
        "colab_type": "text"
      },
      "source": [
        "## 1. Installing Stanza\n",
        "\n",
        "Note that Stanza only supports Python 3.6 and above. Installing and importing Stanza are as simple as running the following commands:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "owSj1UtdEvSU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Install; note that the prefix \"!\" is not needed if you are running in a terminal\n",
        "!pip install stanza\n",
        "\n",
        "# Import the package\n",
        "import stanza"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ixllwEKeCJg",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For common troubleshooting, please visit our [troubleshooting page](https://stanfordnlp.github.io/stanfordnlp/installation_usage.html#troubleshooting)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aeyPs5ARO79d",
        "colab_type": "text"
      },
      "source": [
        "## 2. Downloading Models\n",
        "\n",
        "You can download models with the `stanza.download` command. The language can be specified with either a full language name (e.g., \"english\"), or a short code (e.g., \"en\"). \n",
        "\n",
        "By default, models will be saved to your `~/stanza_resources` directory. If you want to specify your own path to save the model files, you can pass a `dir=your_path` argument.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HDwRm-KXGcYo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Download an English model into the default directory\n",
        "print(\"Downloading English model...\")\n",
        "stanza.download('en')\n",
        "\n",
        "# Similarly, download a (simplified) Chinese model\n",
        "# Note that you can use verbose=False to turn off all printed messages\n",
        "print(\"Downloading Chinese model...\")\n",
        "stanza.download('zh', verbose=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7HCfQ0SfdmsU",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "Pretrained models are provided for 60+ different languages. For all languages, available models and the corresponding short language codes, please check out the [models page](https://stanfordnlp.github.io/stanza/models.html).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3-WZJrzWD2o",
        "colab_type": "text"
      },
      "source": [
        "## 3. Processing Text\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrnKl2m3fq2f",
        "colab_type": "text"
      },
      "source": [
        "### Constructing Pipeline\n",
        "\n",
        "To process a piece of text, you'll need to first construct a `Pipeline` with different `Processor` units. The pipeline is language-specific, so again you'll need to first specify the language (see examples).\n",
        "\n",
        "- By default, the pipeline will include all processors, including tokenization, multi-word token expansion, part-of-speech tagging, lemmatization, dependency parsing and named entity recognition (for supported languages). However, you can always specify what processors you want to include with the `processors` argument.\n",
        "\n",
        "- Stanza's pipeline is CUDA-aware, meaning that a CUDA-device will be used whenever it is available, otherwise CPUs will be used when a GPU is not found. You can force the pipeline to use CPU regardless by setting `use_gpu=False`.\n",
        "\n",
        "- Again, you can suppress all printed messages by setting `verbose=False`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HbiTSBDPG53o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Build an English pipeline, with all processors by default\n",
        "print(\"Building an English pipeline...\")\n",
        "en_nlp = stanza.Pipeline('en')\n",
        "\n",
        "# Build a Chinese pipeline, with customized processor list and no logging, and force it to use CPU\n",
        "print(\"Building a Chinese pipeline...\")\n",
        "zh_nlp = stanza.Pipeline('zh', processors='tokenize,lemma,pos,depparse', verbose=False, use_gpu=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Go123Bx8e1wt",
        "colab_type": "text"
      },
      "source": [
        "### Annotating Text\n",
        "\n",
        "After a pipeline is successfully constructed, you can get annotations of a piece of text simply by passing the string into the pipeline object. The pipeline will return a `Document` object, which can be used to access detailed annotations from. For example:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k_p0h1UTHDMm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Processing English text\n",
        "en_doc = en_nlp(\"Barack Obama was born in Hawaii.  He was elected president in 2008.\")\n",
        "print(type(en_doc))\n",
        "\n",
        "# Processing Chinese text\n",
        "zh_doc = zh_nlp(\"达沃斯世界经济论坛是每年全球政商界领袖聚在一起的年度盛事。\")\n",
        "print(type(zh_doc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DavwCP9egzNZ",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For more information on how to construct a pipeline and information on different processors, please visit our [pipeline page](https://stanfordnlp.github.io/stanfordnlp/pipeline.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_PYLEGziQWR",
        "colab_type": "text"
      },
      "source": [
        "## 4. Accessing Annotations\n",
        "\n",
        "Annotations can be accessed from the returned `Document` object. \n",
        "\n",
        "A `Document` contains a list of `Sentence`s, and a `Sentence` contains a list of `Token`s and `Word`s. For the most part `Token`s and `Word`s overlap, but some tokens can be divided into mutiple words, for instance the French token `aux` is divided into the words `à` and `les`, while in English a word and a token are equivalent. Note that dependency parses are derived over `Word`s.\n",
        "\n",
        "Additionally, a `Span` object is used to represent annotations that are part of a document, such as named entity mentions.\n",
        "\n",
        "\n",
        "The following example iterate over all English sentences and words, and print the word information one by one:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B5691SpFHFZ6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i, sent in enumerate(en_doc.sentences):\n",
        "    print(\"[Sentence {}]\".format(i+1))\n",
        "    for word in sent.words:\n",
        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
        "              word.text, word.lemma, word.pos, word.head, word.deprel))\n",
        "    print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-AUkCkNIrusq",
        "colab_type": "text"
      },
      "source": [
        "The following example iterate over all extracted named entity mentions and print out their character spans and types."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Uu0-WmvsnlK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Mention text\\tType\\tStart-End\")\n",
        "for ent in en_doc.ents:\n",
        "    print(\"{}\\t{}\\t{}-{}\".format(ent.text, ent.type, ent.start_char, ent.end_char))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ql1SZlZOnMLo",
        "colab_type": "text"
      },
      "source": [
        "And similarly for the Chinese text:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XsVcEO9tHKPG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for i, sent in enumerate(zh_doc.sentences):\n",
        "    print(\"[Sentence {}]\".format(i+1))\n",
        "    for word in sent.words:\n",
        "        print(\"{:12s}\\t{:12s}\\t{:6s}\\t{:d}\\t{:12s}\".format(\\\n",
        "              word.text, word.lemma, word.pos, word.head, word.deprel))\n",
        "    print(\"\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUhWAs8pnnHT",
        "colab_type": "text"
      },
      "source": [
        "Alternatively, you can directly print a `Word` object to view all its annotations as a Python dict:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_UafNb7HHIg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "word = en_doc.sentences[0].words[0]\n",
        "print(word)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TAQlOsuRoq2V",
        "colab_type": "text"
      },
      "source": [
        "### More Information\n",
        "\n",
        "For all information on different data objects, please visit our [data objects page](https://stanfordnlp.github.io/stanza/data_objects.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hiiWHxYPpmhd",
        "colab_type": "text"
      },
      "source": [
        "## 5. Resources\n",
        "\n",
        "Apart from this interactive tutorial, we also provide tutorials on our website that cover a variety of use cases such as how to use different model \"packages\" for a language, how to use spaCy as a tokenizer, how to process pretokenized text without running the tokenizer, etc. For these tutorials please visit [our Tutorials page](https://stanfordnlp.github.io/stanza/tutorials.html).\n",
        "\n",
        "Other resources that you may find helpful include:\n",
        "\n",
        "- [Stanza Homepage](https://stanfordnlp.github.io/stanza/index.html)\n",
        "- [FAQs](https://stanfordnlp.github.io/stanza/faq.html)\n",
        "- [GitHub Repo](https://github.com/stanfordnlp/stanza)\n",
        "- [Reporting Issues](https://github.com/stanfordnlp/stanza/issues)\n",
        "- [Stanza System Description Paper](http://arxiv.org/abs/2003.07082)\n"
      ]
    }
  ]
}