Package overview | Character and character-shingle tokenizers | Word and word-stem tokenizers | N-gram and skip n-gram tokenizers | Sentence and paragraph tokenizers | Text chunking | Counting words, characters, sentences
Package: tokenizers 0.3.1
Thomas Charlon
tokenizers: Fast, Consistent Tokenization of Natural Language Text
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.
Authors:
tokenizers_0.3.1.tar.gz
tokenizers_0.3.1.zip(r-4.7-x86_64)tokenizers_0.3.1.zip(r-4.7-arm64)tokenizers_0.3.1.zip(r-4.6-x86_64)tokenizers_0.3.1.zip(r-4.6-arm64)tokenizers_0.3.1.zip(r-4.5-x86_64)
tokenizers_0.3.1.tgz(r-4.6-x86_64)tokenizers_0.3.1.tgz(r-4.6-arm64)tokenizers_0.3.1.tgz(r-4.5-x86_64)tokenizers_0.3.1.tgz(r-4.5-arm64)
tokenizers_0.3.1.tar.gz(r-4.7-arm64)tokenizers_0.3.1.tar.gz(r-4.7-x86_64)tokenizers_0.3.1.tar.gz(r-4.6-arm64)tokenizers_0.3.1.tar.gz(r-4.6-x86_64)
tokenizers_0.3.1.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
tokenizers/json (API)
| # Install 'tokenizers' in R: |
| install.packages('tokenizers', repos = c('https://ropensci.r-universe.dev', 'https://cloud.r-project.org')) |
Reviews:rOpenSci Software Review #33
Bug tracker:https://github.com/ropensci/tokenizers/issues
Pkgdown/docs site:https://docs.ropensci.org
Datasets:
- mobydick - The text of Moby Dick
nlppeer-reviewedtext-miningtokenizercpp
Last updated from:b80863d088. Checks:4 NOTE, 12 OK. Indexed: yes.
| Target | Result | Time | Files | Syslog |
|---|---|---|---|---|
| linux-devel-arm64 | NOTE | 100 | ||
| linux-devel-x86_64 | NOTE | 113 | ||
| pkgdown docs | OK | 166 | ||
| source / vignettes | OK | 182 | ||
| linux-release-arm64 | OK | 101 | ||
| linux-release-x86_64 | OK | 127 | ||
| macos-release-arm64 | OK | 113 | ||
| macos-release-x86_64 | OK | 270 | ||
| macos-oldrel-arm64 | OK | 71 | ||
| macos-oldrel-x86_64 | OK | 201 | ||
| windows-devel-arm64 | NOTE | 226 | ||
| windows-devel-x86_64 | NOTE | 145 | ||
| windows-release-arm64 | OK | 222 | ||
| windows-release-x86_64 | OK | 105 | ||
| windows-oldrel-x86_64 | OK | 115 | ||
| wasm-release | OK | 106 |
Exports:chunk_textcount_characterscount_sentencescount_wordstokenize_character_shinglestokenize_characterstokenize_linestokenize_ngramstokenize_paragraphstokenize_ptbtokenize_regextokenize_sentencestokenize_skip_ngramstokenize_word_stemstokenize_words
Introduction to the tokenizers Package
Last update: 2022-12-19
Started: 2016-08-11
The Text Interchange Formats and the tokenizers Package
Last update: 2022-09-23
Started: 2018-03-14
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| Basic tokenizers | basic-tokenizers tokenize_characters tokenize_lines tokenize_paragraphs tokenize_regex tokenize_sentences tokenize_words |
| Chunk text into smaller segments | chunk_text |
| Count words, sentences, characters | count_characters count_sentences count_words |
| The text of Moby Dick | mobydick |
| N-gram tokenizers | ngram-tokenizers tokenize_ngrams tokenize_skip_ngrams |
| Character shingle tokenizers | tokenize_character_shingles |
| Penn Treebank Tokenizer | tokenize_ptb |
| Word stem tokenizer | tokenize_word_stems |
| Tokenizers | tokenizers-package tokenizers |