Sync Upstream (0f65165)

.gitignore

@@ -18,4 +18,4 @@ cookies.json
 docker-compose.yml
 
 # ide
-.vscode
+.vscode

api/README.md

@@ -0,0 +1,106 @@
+# cobalt api
+this directory includes the source code for cobalt api. it's made with [express.js](https://www.npmjs.com/package/express) and love!
+
+## running your own instance
+if you want to run your own instance for whatever purpose, [follow this guide](/docs/run-an-instance.md).
+we recommend to use docker compose unless you intend to run cobalt for developing/debugging purposes.
+
+## accessing the api
+there is currently no publicly available pre-hosted api.
+we recommend [deploying your own instance](/docs/run-an-instance.md) if you wish to use the cobalt api.
+
+you can read [the api documentation here](/docs/api.md).
+
+> [!WARNING]
+> the v7 public api (/api/json) will be shut down on **november 11th, 2024**.
+> you can access documentation for it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md).
+
+## supported services
+this list is not final and keeps expanding over time. if support for a service you want is missing, create an issue (or a pull request 👀).
+
+| service           | video + audio | only audio | only video | metadata | rich file names |
+| :--------         | :-----------: | :--------: | :--------: | :------: | :-------------: |
+| bilibili          | ✅            | ✅         | ✅         | ➖         | ➖              |
+| bluesky           | ✅            | ✅         | ✅         | ➖         | ➖              |
+| dailymotion       | ✅            | ✅         | ✅         | ✅         | ✅              |
+| instagram         | ✅            | ✅         | ✅         | ➖         | ➖              |
+| facebook          | ✅            | ❌         | ✅         | ➖         | ➖              |
+| loom              | ✅            | ❌         | ✅         | ✅         | ➖              |
+| ok.ru             | ✅            | ❌         | ✅         | ✅         | ✅              |
+| pinterest         | ✅            | ✅         | ✅         | ➖         | ➖              |
+| reddit            | ✅            | ✅         | ✅         | ❌         | ❌              |
+| rutube            | ✅            | ✅         | ✅         | ✅         | ✅              |
+| snapchat          | ✅            | ✅         | ✅         | ➖         | ➖              |
+| soundcloud        | ➖            | ✅         | ➖         | ✅         | ✅              |
+| streamable        | ✅            | ✅         | ✅         | ➖         | ➖              |
+| tiktok            | ✅            | ✅         | ✅         | ❌         | ❌              |
+| tumblr            | ✅            | ✅         | ✅         | ➖         | ➖              |
+| twitch clips      | ✅            | ✅         | ✅         | ✅         | ✅              |
+| twitter/x         | ✅            | ✅         | ✅         | ➖         | ➖              |
+| vimeo             | ✅            | ✅         | ✅         | ✅         | ✅              |
+| vk videos & clips | ✅            | ❌         | ✅         | ✅         | ✅              |
+| youtube           | ✅            | ✅         | ✅         | ✅         | ✅              |
+
+| emoji   | meaning                 |
+| :-----: | :---------------------- |
+| ✅      | supported               |
+| ➖      | impossible/unreasonable |
+| ❌      | not supported           |
+
+### additional notes or features (per service)
+| service    | notes or features                                                                                                    |
+| :--------  | :-----                                                                                                               |
+| instagram  | supports reels, photos, and videos. lets you pick what to save from multi-media posts.                               |
+| facebook   | supports public accessible videos content only.                                                                      |
+| pinterest  | supports photos, gifs, videos and stories.                                                                           |
+| reddit     | supports gifs and videos.                                                                                            |
+| snapchat   | supports spotlights and stories. lets you pick what to save from stories.                                            |
+| rutube     | supports yappy & private links.                                                                                      |
+| soundcloud | supports private links.                                                                                              |
+| tiktok     | supports videos with or without watermark, images from slideshow without watermark, and full (original) audios.      |
+| twitter/x  | lets you pick what to save from multi-media posts. may not be 100% reliable due to current management.               |
+| vimeo      | audio downloads are only available for dash.                                                                         |
+| youtube    | supports videos, music, and shorts. 8K, 4K, HDR, VR, and high FPS videos. rich metadata & dubs. h264/av1/vp9 codecs. |
+
+## license
+cobalt api code is licensed under [AGPL-3.0](LICENSE).
+
+this license allows you to modify, distribute and use the code for any purpose
+as long as you:
+- give appropriate credit to the original repo when using or modifying any parts of the code,
+- provide a link to the license and indicate if changes to the code were made, and
+- release the code under the **same license**
+
+## acknowledgements
+### ffmpeg
+cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should.
+
+you can [support ffmpeg here](https://ffmpeg.org/donations.html)!
+
+#### ffmpeg-static
+we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform.
+
+you can support the developer via various methods listed on their github page! (linked above)
+
+### youtube.js
+cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible without it.
+
+you can support the developer via various methods listed on their github page! (linked above)
+
+### many others
+cobalt also depends on:
+
+- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers.
+- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs.
+- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file.
+- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers.
+- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints.
+- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services.
+- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting).
+- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream.
+- [psl](https://www.npmjs.com/package/psl) as the domain name parser.
+- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services.
+- [undici](https://www.npmjs.com/package/undici) for making http requests.
+- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns.
+
+...and many other packages that these packages rely on.

api/package.json

@@ -1,45 +1,50 @@
 {
-    "name": "teamhydra-videodl-api",
-    "description": "Team Hydra's video downloader (API)",
-    "version": "10.0.0",
-    "author": "Team Hydra",
+    "name": "@imput/cobalt-api",
+    "description": "save what you love",
+    "version": "10.5.2",
+    "author": "imput",
     "exports": "./src/cobalt.js",
     "type": "module",
     "engines": {
         "node": ">=18"
     },
     "scripts": {
-        "start": "node src/thvideodl",
-        "setup": "node src/util/setup",
+        "start": "node src/cobalt",
         "test": "node src/util/test",
-        "token:youtube": "node src/util/generate-youtube-tokens"
+        "token:youtube": "node src/util/generate-youtube-tokens",
+        "token:jwt": "node src/util/generate-jwt-secret"
     },
     "repository": {
         "type": "git",
-        "url": "git+https://teamhydra.io/Team-Hydra/Video-Downloader.git"
+        "url": "git+https://github.com/imputnet/cobalt.git"
     },
     "license": "AGPL-3.0",
+    "bugs": {
+        "url": "https://github.com/imputnet/cobalt/issues"
+    },
+    "homepage": "https://github.com/imputnet/cobalt#readme",
     "dependencies": {
+        "@datastructures-js/priority-queue": "^6.3.1",
+        "@imput/psl": "^2.0.4",
         "@imput/version-info": "workspace:^",
         "content-disposition-header": "0.6.0",
         "cors": "^2.8.5",
         "dotenv": "^16.0.1",
-        "esbuild": "^0.14.51",
-        "express": "^4.18.1",
-        "express-rate-limit": "^6.3.0",
+        "express": "^4.21.2",
+        "express-rate-limit": "^7.4.1",
         "ffmpeg-static": "^5.1.0",
         "hls-parser": "^0.10.7",
-        "ipaddr.js": "2.1.0",
-        "nanoid": "^4.0.2",
-        "node-cache": "^5.1.2",
-        "psl": "1.9.0",
+        "ipaddr.js": "2.2.0",
+        "nanoid": "^5.0.9",
         "set-cookie-parser": "2.6.0",
         "undici": "^5.19.1",
         "url-pattern": "1.0.3",
-        "youtubei.js": "^10.3.0",
+        "youtubei.js": "^12.2.0",
         "zod": "^3.23.8"
     },
     "optionalDependencies": {
-        "freebind": "^0.2.2"
+        "freebind": "^0.2.2",
+        "rate-limit-redis": "^4.2.0",
+        "redis": "^4.7.0"
     }
 }

api/src/cobalt.js

@@ -0,0 +1,32 @@
+import "dotenv/config";
+
+import express from "express";
+import cluster from "node:cluster";
+
+import path from "path";
+import { fileURLToPath } from "url";
+
+import { env, isCluster } from "./config.js"
+import { Red } from "./misc/console-text.js";
+import { initCluster } from "./misc/cluster.js";
+
+const app = express();
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename).slice(0, -4);
+
+app.disable("x-powered-by");
+
+if (env.apiURL) {
+    const { runAPI } = await import("./core/api.js");
+
+    if (isCluster) {
+       await initCluster();
+    }
+
+    runAPI(express, app, __dirname, cluster.isPrimary);
+} else {
+    console.log(
+        Red("API_URL env variable is missing, cobalt api can't start.")
+    )
+}

api/src/config.js

@@ -1,20 +1,20 @@
-import { getVersion } from '@imput/version-info';
-import { services } from './processing/service-config.js';
+import { getVersion } from "@imput/version-info";
+import { services } from "./processing/service-config.js";
+import { supportsReusePort } from "./misc/cluster.js";
 
 const version = await getVersion();
 
 const disabledServices = process.env.DISABLED_SERVICES?.split(',') || [];
-const enabledServices = new Set(
-    Object.keys(services).filter((e) => {
-        if (!disabledServices.includes(e)) {
-            return e;
-        }
-    }),
-);
+const enabledServices = new Set(Object.keys(services).filter(e => {
+    if (!disabledServices.includes(e)) {
+        return e;
+    }
+}));
 
 const env = {
     apiURL: process.env.API_URL || '',
     apiPort: process.env.API_PORT || 9000,
+    tunnelPort: process.env.API_PORT || 9000,
 
     listenAddress: process.env.API_LISTEN_ADDRESS,
     freebindCIDR: process.platform === 'linux' && process.env.FREEBIND_CIDR,
@@ -24,39 +24,58 @@ const env = {
 
     cookiePath: process.env.COOKIE_PATH,
 
-    rateLimitWindow:
-        (process.env.RATELIMIT_WINDOW &&
-            parseInt(process.env.RATELIMIT_WINDOW)) ||
-        60,
-    rateLimitMax:
-        (process.env.RATELIMIT_MAX && parseInt(process.env.RATELIMIT_MAX)) ||
-        20,
+    rateLimitWindow: (process.env.RATELIMIT_WINDOW && parseInt(process.env.RATELIMIT_WINDOW)) || 60,
+    rateLimitMax: (process.env.RATELIMIT_MAX && parseInt(process.env.RATELIMIT_MAX)) || 20,
 
-    durationLimit:
-        (process.env.DURATION_LIMIT && parseInt(process.env.DURATION_LIMIT)) ||
-        10800,
-    streamLifespan: 90,
+    durationLimit: (process.env.DURATION_LIMIT && parseInt(process.env.DURATION_LIMIT)) || 10800,
+    streamLifespan: (process.env.TUNNEL_LIFESPAN && parseInt(process.env.TUNNEL_LIFESPAN)) || 90,
 
-    processingPriority:
-        process.platform !== 'win32' &&
-        process.env.PROCESSING_PRIORITY &&
-        parseInt(process.env.PROCESSING_PRIORITY),
+    processingPriority: process.platform !== 'win32'
+        && process.env.PROCESSING_PRIORITY
+        && parseInt(process.env.PROCESSING_PRIORITY),
 
     externalProxy: process.env.API_EXTERNAL_PROXY,
 
+    turnstileSitekey: process.env.TURNSTILE_SITEKEY,
     turnstileSecret: process.env.TURNSTILE_SECRET,
     jwtSecret: process.env.JWT_SECRET,
     jwtLifetime: process.env.JWT_EXPIRY || 120,
 
+    sessionEnabled: process.env.TURNSTILE_SITEKEY
+                        && process.env.TURNSTILE_SECRET
+                        && process.env.JWT_SECRET,
+
+    apiKeyURL: process.env.API_KEY_URL && new URL(process.env.API_KEY_URL),
+    authRequired: process.env.API_AUTH_REQUIRED === '1',
+    redisURL: process.env.API_REDIS_URL,
+    instanceCount: (process.env.API_INSTANCE_COUNT && parseInt(process.env.API_INSTANCE_COUNT)) || 1,
+    keyReloadInterval: 900,
+
     enabledServices,
-};
+}
+
+const genericUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";
+const cobaltUserAgent = `cobalt/${version} (+https://github.com/imputnet/cobalt)`;
+
+export const setTunnelPort = (port) => env.tunnelPort = port;
+export const isCluster = env.instanceCount > 1;
+
+if (env.sessionEnabled && env.jwtSecret.length < 16) {
+    throw new Error("JWT_SECRET env is too short (must be at least 16 characters long)");
+}
 
-const genericUserAgent =
-    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36';
-const teamHydraUserAgent = `teamhydravideodl/${version} (support@hep.gg)`;
+if (env.instanceCount > 1 && !env.redisURL) {
+    throw new Error("API_REDIS_URL is required when API_INSTANCE_COUNT is >= 2");
+} else if (env.instanceCount > 1 && !await supportsReusePort()) {
+    console.error('API_INSTANCE_COUNT is not supported in your environment. to use this env, your node.js');
+    console.error('version must be >= 23.1.0, and you must be running a recent enough version of linux');
+    console.error('(or other OS that supports it). for more info, see `reusePort` option on');
+    console.error('https://nodejs.org/api/net.html#serverlistenoptions-callback');
+    throw new Error('SO_REUSEPORT is not supported');
+}
 
 export {
     env,
     genericUserAgent,
-    teamHydraUserAgent as cobaltUserAgent, // lol
-};
+    cobaltUserAgent,
+}