yokaidb

日本語のREADMEはこちらです: README.ja.md

A database of Japanese yōkai (supernatural creatures) scraped from Yokai.com and provided as a CSV file.

Dataset

• yokaidb.csv - The scraped data, licensed under CC BY-NC-SA 4.0 by Yokai.com.

Features

• Two-Step Scraping: First gathers all yōkai URLs, then scrapes the content for each, allowing for incremental updates.
• Structured Data: Parses HTML to extract fields like names (English/Japanese), habitat, diet, appearance, legends, and more.
• Resilient: Avoids re-downloading existing entries and caches fetched pages locally.
• Pagination: Automatically navigates through all pages on the source website.

How to Build

This project requires Deno.

The database is built in two stages:

# 1. Scrape the index of all yōkai URLs into yokaidb_idx.csv
deno run -A scrape.js

# 2. Scrape the content for each URL and append it to yokaidb.csv
deno run -A scrapeContent.js

CSV Schema

The output file yokaidb.csv contains the following columns:

• url
• name
• name_ja
• name_ja_kana
• translation
• alternate_names
• habitat
• diet
• appearance
• behavior
• interactions
• description
• origin
• legends
• other_forms
• critical_weakness