Answer by Pradeep Singh for Efficiently count word frequencies in python
Combining every ones else's views and some of my own :)Here is what I have for youfrom collections import Counterfrom nltk.tokenize import RegexpTokenizerfrom nltk.corpus import stopwordstext='''Note...
View ArticleAnswer by Murtadha Alrahbi for Efficiently count word frequencies in python
you can try with sklearnfrom sklearn.feature_extraction.text import CountVectorizer vectorizer = CountVectorizer() data=['i am student','the student suffers a lot'] transformed_data...
View ArticleAnswer by nat gillin for Efficiently count word frequencies in python
Here's some benchmark. It'll look strange but the crudest code wins.[code]:from collections import Counter, defaultdictimport io, timeimport numpy as npfrom sklearn.feature_extraction.text import...
View ArticleAnswer by Nizam Mohamed for Efficiently count word frequencies in python
Instead of decoding the whole bytes read from the url, I process the binary data. Because bytes.translate expects its second argument to be a byte string, I utf-8 encode punctuation. After removing...
View ArticleAnswer by alvas for Efficiently count word frequencies in python
A memory efficient and accurate way is to make use of CountVectorizer in scikit (for ngram extraction)NLTK for word_tokenizenumpy matrix sum to collect the countscollections.Counter for collecting the...
View ArticleAnswer by ShadowRanger for Efficiently count word frequencies in python
The most succinct approach is to use the tools Python gives you.from future_builtins import map # Only on Python 2from collections import Counterfrom itertools import chaindef countInFile(filename):...
View ArticleAnswer by Goodies for Efficiently count word frequencies in python
This should suffice.def countinfile(filename): d = {} with open(filename, "r") as fin: for line in fin: words = line.strip().split() for word in words: try: d[word] += 1 except KeyError: d[word] = 1...
View ArticleAnswer by Stephen Grimes for Efficiently count word frequencies in python
Skip CountVectorizer and scikit-learn.The file may be too large to load into memory but I doubt the python dictionary gets too large. The easiest option for you may be to split the large file into...
View ArticleEfficiently count word frequencies in python
I'd like to count frequencies of all words in a text file.>>> countInFile('test.txt')should return {'aaa':1, 'bbb': 2, 'ccc':1} if the target text file is like:# test.txtaaa bbb cccbbbI've...
View Article
