Sigmoid Social

Karsten SchmidtRecently I've combined various functions which I've been using in other projects (e.g. my personal PKM toolchain) and published them as new library <a href="https://thi.ng/text-analysis" rel="nofollow noopener" translate="no" target="_blank">https://thi.ng/text-analysis</a> for better re-use:- customizable, composable & extensible tokenization (transducer based) - ngram generation - Porter-stemming & stopword removal - vocabulary (bi-directional index) creation - dense & sparse multi-hot vector encoding/decoding - histograms (incl. sorted versions) - tf-idf (term frequency & inverse document frequency), multiple strategies - k-means clustering (with k-means++ initialization & customizable distance metrics) - similarity/distance functions (dense & sparse versions) - central terms extractionThe attached code example (also in the project readme) uses this package to creeate a clustering of all ~210 <a href="https://mastodon.thi.ng/tags/ThingUmbrella" class="mention hashtag" rel="nofollow noopener" target="_blank">#ThingUmbrella</a> packages, based on their assigned tags/keywords...The library is not intended to be a full-blown NLP solution, but I keep on finding myself running into these functions/concepts quite often, and maybe you'll find them useful too...<a href="https://mastodon.thi.ng/tags/Text" class="mention hashtag" rel="nofollow noopener" target="_blank">#Text</a> <a href="https://mastodon.thi.ng/tags/Analysis" class="mention hashtag" rel="nofollow noopener" target="_blank">#Analysis</a> <a href="https://mastodon.thi.ng/tags/Cluster" class="mention hashtag" rel="nofollow noopener" target="_blank">#Cluster</a> <a href="https://mastodon.thi.ng/tags/KMeans" class="mention hashtag" rel="nofollow noopener" target="_blank">#KMeans</a> <a href="https://mastodon.thi.ng/tags/TFIDF" class="mention hashtag" rel="nofollow noopener" target="_blank">#TFIDF</a> <a href="https://mastodon.thi.ng/tags/Ngram" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ngram</a> <a href="https://mastodon.thi.ng/tags/Vector" class="mention hashtag" rel="nofollow noopener" target="_blank">#Vector</a> <a href="https://mastodon.thi.ng/tags/TypeScript" class="mention hashtag" rel="nofollow noopener" target="_blank">#TypeScript</a> <a href="https://mastodon.thi.ng/tags/JavaScript" class="mention hashtag" rel="nofollow noopener" target="_blank">#JavaScript</a>

petersuberFellow finicky writers: Do you prefer "advance notice" or "advanced notice"?Both are attested. But FYI, <a href="https://fediscience.org/tags/ngram" class="mention hashtag" rel="nofollow noopener" target="_blank">#ngram</a> says that "advance notice" is much more common, even if it's in decline. <a href="https://books.google.com/ngrams/graph?content=advance+notice%2C+advanced+notice&year_start=1800&year_end=2022&corpus=en&smoothing=3" rel="nofollow noopener" translate="no" target="_blank">https://books.google.com/ngrams/graph?content=advance+notice%2C+advanced+notice&year_start=1800&year_end=2022&corpus=en&smoothing=3</a>

François Renaville 🇺🇦🇪🇺<a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#Google</a> Books Is Indexing <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a>-Generated Books👉 <a href="https://mastodon.social/tags/GoogleBooks" class="mention hashtag" rel="nofollow noopener" target="_blank">#GoogleBooks</a> is indexing low quality, AI-generated books that will turn up in search results, and could possibly impact Google <a href="https://mastodon.social/tags/Ngram" class="mention hashtag" rel="nofollow noopener" target="_blank">#Ngram</a> viewer, an important tool used by researchers to track <a href="https://mastodon.social/tags/language" class="mention hashtag" rel="nofollow noopener" target="_blank">#language</a> use throughout history. <a href="https://timesofindia.indiatimes.com/technology/tech-news/google-books-important-source-for-academics-may-have-a-bot-problem/articleshow/109089043.cms" rel="nofollow noopener" translate="no" target="_blank">https://timesofindia.indiatimes.com/technology/tech-news/google-books-important-source-for-academics-may-have-a-bot-problem/articleshow/109089043.cms</a> <a href="https://mastodon.social/tags/GoogleNgram" class="mention hashtag" rel="nofollow noopener" target="_blank">#GoogleNgram</a> <a href="https://mastodon.social/tags/NgramViewer" class="mention hashtag" rel="nofollow noopener" target="_blank">#NgramViewer</a> <a href="https://mastodon.social/tags/linguistics" class="mention hashtag" rel="nofollow noopener" target="_blank">#linguistics</a> <a href="https://mastodon.social/tags/diachrony" class="mention hashtag" rel="nofollow noopener" target="_blank">#diachrony</a> <a href="https://mastodon.social/tags/diachroniclinguistics" class="mention hashtag" rel="nofollow noopener" target="_blank">#diachroniclinguistics</a> <a href="https://mastodon.social/tags/research" class="mention hashtag" rel="nofollow noopener" target="_blank">#research</a> <a href="https://mastodon.social/tags/languages" class="mention hashtag" rel="nofollow noopener" target="_blank">#languages</a> <a href="https://mastodon.social/tags/aigeneratedcontent" class="mention hashtag" rel="nofollow noopener" target="_blank">#aigeneratedcontent</a> <a href="https://mastodon.social/tags/AIgeneratedBooks" class="mention hashtag" rel="nofollow noopener" target="_blank">#AIgeneratedBooks</a>

Harald SackOne of the basic questions we tackle when working towards statistical language models is "Can we predict a word?" This was also one of the intro questions to the students last Wednesday in our <a href="https://sigmoid.social/tags/ise2023" class="mention hashtag" rel="tag">#ise2023</a> lecture no.4, when we were introducing simple n-gram language models.<a href="https://sigmoid.social/tags/nlp" class="mention hashtag" rel="tag">#nlp</a> <a href="https://sigmoid.social/tags/lecture" class="mention hashtag" rel="tag">#lecture</a> <a href="https://sigmoid.social/tags/ngram" class="mention hashtag" rel="tag">#ngram</a> <a href="https://sigmoid.social/tags/languagemodels" class="mention hashtag" rel="tag">#languagemodels</a> <a href="https://sigmoid.social/tags/language" class="mention hashtag" rel="tag">#language</a> <a href="https://sigmoid.social/tags/aiart" class="mention hashtag" rel="tag">#aiart</a> <a href="https://sigmoid.social/tags/stablediffusion" class="mention hashtag" rel="tag">#stablediffusion</a> <a href="https://sigmoid.social/tags/creativeAI" class="mention hashtag" rel="tag">#creativeAI</a> <a href="https://sigmoid.social/@fizise" class="u-url mention">@fizise</a> <a href="https://mastodon.social/@KIT_Karlsruhe" class="u-url mention">@KIT_Karlsruhe</a> <a href="https://troet.cafe/@nfdi4ds" class="u-url mention">@nfdi4ds</a> <a href="https://nfdi.social/@nfdi4culture" class="u-url mention">@nfdi4culture</a>

Recent searches

Search options

Administered by:

Server stats:

#ngram