Tips & Recommendations for Project 1 (NLP Automated Customer Reviews)

Dataset

You can use one, two, or more datasets.
The "primary" dataset is more than sufficient. You can use more than one dataset if you want, but keep in mind that this will increase the time required for cleaning and preprocessing.
It's a large dataset, with many features (you don't need all the features).
Also, don't forget to split the data (e.g., Train/Test or Train/Validation/Test).

GitHub has a 100 MB hard limit per file, and dataset files can easily exceed this limit.
For datasets:
- To avoid hitting that limit, keep all your dataset files in a datasets/ directory and add this directory to your .gitignore file. This prevents Git from tracking the dataset files.
- If you're still not familiar with .gitignore, you can find more info here
Fine-Tuned Models:
- Fine-tuned model files can also be larger than GitHub's 100 MB per-file limit. If your model is too large to commit to GitHub, you won't be able to include it directly in your repository.
- A common solution is to upload the model to the Hugging Face Hub and download it at runtime using the transformers library. This keeps large model files out of your GitHub repository and allows your deployed application to access the model when needed.

The dataset includes many columns, including columns containing long text
On VS Code, you'll see that pandas doesn't display all columns and it will also crop the text
To read them, you can actually configure pandas to display all the columns and avoid truncating long text values (do some research)

You'll be working with realistic, real-world datasets
For example, in one of the datasets, the column "id" is not very reliable (in some cases, different products are assigned to the same ID)
Spend some time exploring and understanding the dataset. You may need to fix data quality issues, discard irrelevant features, handle missing values, and make other preprocessing decisions before training your model.

You can find an example of NLP + clustering in this notebook (section 3.5 All together: News clustering example): https://github.com/ironhack-rmt-AI-student-materials/ai_eng_lesson_templates/blob/main/H_NLP/NLP_4.ipynb
After creating the clusters, it can be helpful to display a few examples from each cluster, and see if the clusters make sense. For example:
- If you display some samples from a cluster and most of them focus on a similar topic, the cluster may be meaningful.
- If the samples within a cluster cover very different topics, you may need to review the process (e.g. review the process for feature extraction and/or pick a different value for K)

Task 3 (Summarization) depends on Task 2 (Clustering).
In other words, you must complete the clustering first before you can perform the summarization.

Focus on an MVP
- MVP (Minimum Viable Product) = a version that may not be perfect but meets all the minimum requirements.
Aim to have an MVP as soon as possible (3 baseline models + a simple app).
Once you have an MVP, continue iterating (e.g., try different approaches to improve the metrics, experiment with different models, implement additional features, etc.).