Techniques to Encode Categorical Features with many Levels/Categories
Feature Engineering is one of the important elements of the data science model development pipeline. A data scientist spends most of their time doing data processing and feature engineering, in order to train a robust model. The dataset constitutes various types of features including categorical, numerical, text, DateTime, etc.
Since most of the machine learning models understand numerical vectors, so all kinds of features need to be engineered to numerical format. There are various encoding techniques to transform text data, into a numerical format, including Bag of Words, Tf-Idf vectorization, and many more. …
Decision Tree model Interpretation
Random Forest or Random Decision Forest is a supervised ensemble machine learning technique, for training classification and regression models. The training algorithm of random forest uses bagging or bootstrap aggregation technique, using decision tree as base learners. Random Forest uses randomization to reduce the bias in decision tree models.
What is a Decision Tree?
A Decision Tree is a supervised machine learning technique, where the data is split according to a certain parameter and forms a tree-like structure. The tree can be explained using decision nodes and leaves.
How important is Model Interpretability?
After training a robust model and evaluating the performance of the model, the question arises, what is…
Spelling and Grammar Correction tool in Python
Usage of good grammar and correctly spelled words helps you to write and communicate clearly and get what you want. Whether you are working on an article, essay, or email, presenting your ideas with clear and correct language makes a good impression on your readers. Often while typing emails, essays, articles, etc one makes a lot of grammatical and spelling mistakes.
Grammarly is an American-based technology company that provides an AI and NLP-based digital writing assessment tool. It comes up with a lot of free and paid tools including grammar checkers, spell checkers, writing assistance, etc. …
Annoyed writing multiple import statements? Let PyForest do the work for you
Python is a primary language used by data scientists for data science projects, because of the presence of thousands of open-source libraries that can ease and perform a data scientist task. Over 235,000 Python packages can be imported through PyPl.
Multiple libraries and frameworks need to be imported to perform tasks in a data science case study. Every time a data scientist or analyst starts a new jupyter notebook or any other IDE, they need to import all the libraries as per their requirements. Sometimes writing multiple lines of the same import statement over and over again can be frustrating…
Ever wish to work on an NLP project, this article is for you
Natural Language Processing (NLP) is a subfield of Artificial Intelligence involving the interactions between a computer and human language, in particular how to program or train an AI model to understand and implement the natural human language.
There are various topics or challenges in NLP tasks such as Tokenization, Lemmitazation, Text Encoding techniques, LSTM, RNN, etc. Combinations of these concepts can be used for NLP scenarios such as Question Answering, Text Classification and Summarization, Sentiment Analysis, Language Recognition and Translation, OCR, and many more.
Did you ever wish you could make an NLP project of your own? …
Deep dive understanding of Dask data frame, and how it works under the hood
Python has tons of open-source libraries that ease data science project development. Some of the famous Python libraries such as Pandas, Numpy, Scikit-Learn provides high level usable and flexible API along with high-performance implementation. These libraries focus on providing a vast list of API’s but largely ignores performance and scalability.
In other words, these libraries fail to load large-sized data or out-of-memory datasets and perform explorations and visualization. Dask library comes into the rescue, which has a similar API that of Pandas and Numpy and accelerates the workflow by parallelizing across all CPUs.
In this article, you can get a…
Predict the Sentiment of your speech in real-time
Sentiment Analysis is a Natural Language Processing technique to predict the sentiment or opinion of a given text. It involves the use of NLP, text analysis, computer linguistic to identify or extract subjective information. Sentiment Analysis is widely used to predict the sentiment of reviews, comments, survey responses, social media, etc.
A sentiment analyzer model can predict whether a given text refers to positive, negative, or neutral sentiment. In this article, we will focus on developing a real-time sentiment analyzer, that can predict the sentiment of speech in real-time using open-sourced Python libraries such as NLTK and TextBlob.
Idea:
In this…
Explore different ways to develop a model to predict the sentiment of a given text
Sentiment Analysis is a Natural Language Processing technique to determine the sentiment or opinion of a given text. A sentiment analysis model can predict whether a given text data is positive, negative, or neutral by extracting meaning from the natural language and assigning it to a numerical score.
There are various ways to develop or train a sentiment analysis model, in this article we will discuss 5 different ways:
- Custom Trained Supervised Model
- TextBlob
- Word-Dictionary based model
- Bert
- Named Entity based Sentiment Analyzer
Sentiment analysis is used by various organizations to understand the sentiment of their customers, using reviews, social…
Why every company is dependent on the synthetic dataset, and how is it generated
Finding the right dataset for any data science project is a challenging task. A machine learning model is dependent on the quality and quantity of the dataset. And, training a robust AI model, requires a vast quantity of data.
What is Synthetic Dataset?
As the name suggests, the synthetic dataset is similar to a real-world dataset but is generated programmatically. Unlike real-world datasets it is not collected by any real-life means, surveys or experiments. …
Tired of reading long text documents, summarize them using open-sourced python libraries
Text summarization is the process of creating a short, accurate, and fluent summary of a long text document. It is the process of distilling the most important information for a text document. The intention of text summarization is to create a summary of a large corpus having important points describing the entire corpus.
The need for Text Summarization?
A vast quantity of text data is generated on the internet, be it social media articles, news articles, etc. Manual creation of summaries is time-consuming, and therefore a need for automatic text summaries has arisen. …
