nsfw_data_scraper automates the collection of categorized NSFW and SFW images to build datasets for training machine learning classifiers. It provides shell scripts that download images across five classes alongside a pre-collected directory of source URLs.
Project overview
It combines pre-collected, categorized source URLs with scripts for collecting and preparing image data for NSFW classifier training.
Project type
Model Development · Data Processing · Infrastructure
Deployment
Refer to project documentation
License
MIT
Best for
Data teams and AI engineers looking to aggregate categorized image data for training custom machine learning models.
Key capabilities
Shell scripts that use the ripme application to download URLs of images from websites like Reddit across five categories: porn, hentai, sexy, neutral, and drawings.
Scripts that create train and test directories, copy valid image files, remove corrupted images, and split data (N=2000 per class) for validation.
A directory of source URLs and raw data outputs already generated by the author for immediate use.
Limitations and risks
The data collection script might run for several hours and is recommended to be left overnight.
As expected, drawings and hentai categories are frequently confused with each other during classification.
Getting started
The setup difficulty is medium because it requires Docker to build and run the scraping environment, and the full run takes several hours. To start, build the image with `docker build . -t docker_nsfw_data_scraper`, run it using `docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh`, and then check the generated `data/train` and `data/test` directories.
Evidence and sources
README: This is a set of scripts that allows for an automatic collection of _tens of thousands_ of images for the following (loosely defined) categories to be later used for training an i…
README: I was able to train a CNN classifier to 91% accuracy with the following confusion matrix:  As expected, `drawings` and `hentai` are confused with…
README: $ # Next command might run for several hours. It is recommended to leave it overnight $ docker run -v $(pwd):/root/nsfw_data_scraper docker_nsfw_data_scraper scripts/runall.sh
README: - `5_create_train_.sh` - creates `data/train` directory and copy all `*.jpg` and `*.jpeg` files into it from `raw_data`. Also removes corrupted images. - `6_create_test_.sh` - cre…
README: - Install [fastai](https://github.com/fastai/fastai): `conda install -c pytorch -c fastai fastai` - Run `train_model.ipynb` top to bottom