notebook/yohane.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lACVssChmYNf"
   },
   "source": [
    "# yohane\n",
    "\n",
    "GitHub project: https://github.com/Japan7/yohane\n",
    "\n",
    "---\n",
    "\n",
    "Click the badge below to open the latest version of the notebook:\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/Japan7/yohane/blob/main/notebook/yohane.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "---\n",
    "\n",
    "**Before proceeding, change your runtime type to T4 GPU (Toolbar > Runtime > Change runtime type > T4 GPU).**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xwJTVAXcu-Ln"
   },
   "source": [
    "## Install\n",
    "\n",
    "Execute the next cells to install yohane.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "WreI26vRmKmO"
   },
   "outputs": [],
   "source": [
    "!python3 --version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "DUuXDO8enIK2"
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "REPO_URL=https://github.com/Japan7/yohane.git\n",
    "LATEST_TAG=$(git ls-remote --tags --sort -v:refname $REPO_URL | head -n1 | cut --delimiter='/' --fields=3)\n",
    "\n",
    "pip3 install uv\n",
    "uv pip install --system git+$REPO_URL@$LATEST_TAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l8b80QOHWjHQ"
   },
   "outputs": [],
   "source": [
    "!pip3 show yohane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uHDmLc9R7x8J"
   },
   "source": [
    "## Parameters\n",
    "\n",
    "The next cells will set the parameters for the yohane pipeline.\n",
    "\n",
    "You can either upload your own song file (_Song Upload_ cell), or use yt-dlp below (_Song Download_ cell).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PGHogUA-7qcz"
   },
   "outputs": [],
   "source": [
    "# @title Song (upload)\n",
    "# @markdown Run this cell to **upload your song** using the form below.\n",
    "#\n",
    "# @markdown Accepted formats: audio or video files.\n",
    "#\n",
    "# @markdown **Note**: If the upload fails, try using a different browser or upload the file manually in the Files section on the left sidebar.\n",
    "\n",
    "from google.colab import files\n",
    "\n",
    "upload = files.upload()\n",
    "for file in upload:\n",
    "    song_filename = file\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2Kg99ykCWjHR"
   },
   "outputs": [],
   "source": [
    "# @markdown **If you uploaded the file manually, enter the song filename here.**\n",
    "\n",
    "song_filename_override = \"\" # @param {type:\"string\"}\n",
    "song_filename = song_filename_override or song_filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sp0sHZPCWjHS"
   },
   "outputs": [],
   "source": [
    "# @title Song (yt-dlp)\n",
    "# @markdown **Enter** the remote URL below, **then execute** this cell to download the song.\n",
    "\n",
    "!uv pip install -q --system yt-dlp[default]\n",
    "\n",
    "from yt_dlp import YoutubeDL\n",
    "\n",
    "song_url = \"\" # @param {type:\"string\"}\n",
    "\n",
    "with YoutubeDL({\"format_sort\": [\"res:1080\", \"vcodec:h264\", \"acodec:aac\"]}) as ydl:\n",
    "    info = ydl.extract_info(song_url)\n",
    "    song_filename = ydl.prepare_filename(info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "luFhCjhb-Dfm"
   },
   "outputs": [],
   "source": [
    "# @title Lyrics { display-mode: \"form\", run: \"auto\" }\n",
    "# @markdown Run this cell, then **paste your lyrics** in the box below.\n",
    "\n",
    "from IPython.display import display\n",
    "from ipywidgets import Layout, Textarea\n",
    "\n",
    "lyrics_area = Textarea(layout=Layout(width=\"100%\", height=\"200px\"))\n",
    "display(lyrics_area)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U0i4FT5V-NSG"
   },
   "outputs": [],
   "source": [
    "# @title Source Separator { display-mode: \"form\", run: \"auto\" }\n",
    "# @markdown Run this cell and select a **Source Separator**:\n",
    "# @markdown - **VocalRemoverSeparator**: Based on the [`vocal-remover`](https://github.com/tsurumeso/vocal-remover) library. Choose this if you're unsure.\n",
    "# @markdown - **HybridDemucsSeparator**: Uses `torchaudio`'s [Hybrid Demucs model](https://pytorch.org/audio/2.1.0/tutorials/hybrid_demucs_tutorial.html), which is faster but less aggressive.\n",
    "# @markdown - **None**: Skips the vocal extraction step if it's not needed.\n",
    "\n",
    "from yohane.audio import VocalRemoverSeparator, HybridDemucsSeparator\n",
    "\n",
    "separator_class = VocalRemoverSeparator # @param [\"VocalRemoverSeparator\", \"HybridDemucsSeparator\", \"None\"] {type:\"raw\"}\n",
    "separator = separator_class() if separator_class is not None else None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lW-jK163FcBW"
   },
   "source": [
    "## Run\n",
    "\n",
    "When ready, execute the next cells to run the pipeline.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nGnkPuSzFZzk"
   },
   "outputs": [],
   "source": [
    "# @title Generate\n",
    "\n",
    "import logging\n",
    "from pathlib import Path\n",
    "from yohane import Yohane\n",
    "\n",
    "logging.basicConfig(level=\"INFO\", force=True)\n",
    "\n",
    "song_path = Path(song_filename)\n",
    "\n",
    "yohane = Yohane(separator)\n",
    "yohane.load_song(song_path)\n",
    "yohane.load_lyrics(lyrics_area.value)\n",
    "yohane.extract_vocals()\n",
    "yohane.force_align()\n",
    "subs = yohane.make_subs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bgfd-NS6INkN"
   },
   "outputs": [],
   "source": [
    "# @title Save and download karaoke file\n",
    "\n",
    "from google.colab import files\n",
    "\n",
    "subs_path = song_path.with_suffix('.ass')\n",
    "subs.save(subs_path)\n",
    "files.download(subs_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Qy3iC4WFYuc9"
   },
   "outputs": [],
   "source": [
    "# @title Download song file\n",
    "\n",
    "files.download(song_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YVB0lS8ENJvo"
   },
   "source": [
    "The karaoke file should have downloaded. If not, open Files in the left sidebar and look for `*.ass`.\n",
    "\n",
    "**Next recommended steps in Aegisub:**\n",
    "\n",
    "1. Load the .ass and the video\n",
    "2. Replace the _Default_ style with your own\n",
    "3. Due to the normalization during the process, lines are lowercased and special characters have been removed: use the original lines in comments to fix the timed lines\n",
    "4. Subtitle > Select Lines… > check _Comments_ and _Set selection_ > OK and delete the selected lines\n",
    "5. Listen to each line and fix their End time\n",
    "6. Add a 1s karaoke lead-in to every line\n",
    "7. Iterate over each line in karaoke mode and merge/fix syllable timings\n",
    "\n",
    "**Happy editing!**\n",
    "\n",
    "<img src=\"https://github.com/user-attachments/assets/614cd8ca-d471-447c-8596-4ac800d690cf\" width=\"25%\" >\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}