jirayu_sri/YouTube.js
1
0
Fork 0
mirror of https://github.com/LuanRT/YouTube.js.git synced 2026-06-26 08:08:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(docs): overhaul parser documentation

Browse Source 
[skip ci]
This commit is contained in:
LuanRT
2023-03-14 05:57:37 -03:00
parent 3df3261488
commit ad1d3dbf91
1 changed files with 113 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

232
src/parser/README.md
View File

@@ -1,38 +1,43 @@
# Parser
Sanitizes and standardizes InnerTube responses while maintaining the integrity of the data.
Structure:
* [`/classes`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/classes) - InnerTube nodes.
* [`/types`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/types) - General response types.
* [`/youtube`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/youtube) - Contains the logic for parsing YouTube responses.
* [`/ytmusic`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/ytmusic) - Contains the logic for parsing YouTube Music responses.
* [`/ytkids`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/ytkids) - Contains the logic for parsing YouTube Kids responses.
* [`helpers.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/helpers.ts) - Helper functions/classes for the parser.
* [`parser.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/parser.ts) - The core of the parser.
* [`map.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/map.ts) - A list of all InnerTube nodes, it is used to determine which node to use for a given renderer. Note that this file is auto-generated and should not be edited manually.
The parser is responsible for sanitizing and standardizing InnerTube responses while preserving the integrity of the data.
## Table of Contents
<ol>
<li>
<a href="#api">API</a>
</li>
<li>
<a href="#usage">Usage</a>
<ul>
<li><a href="#observedarray">ObservedArray</a></li>
<li><a href="#superparsedresponse">SuperParsedResponse</a></li>
<li><a href="#ytnode">YTNode</a></li>
<li><a href="#memo">Memo</a></li>
</ul>
</li>
<li><a href="#adding-new-nodes">Adding new nodes</a></li>
<li><a href="#how-it-works">How it works</a></li>
</ol>
- [Parser](#parser)
- [Table of Contents](#table-of-contents)
- [Structure](#structure)
- [Core](#core)
- [Clients](#clients)
- [API](#api)
- [`parse(data?: RawData, requireArray?: boolean, validTypes?: YTNodeConstructor<T> | YTNodeConstructor<T>[])`](#parsedata-rawdata-requirearray-boolean-validtypes-ytnodeconstructort--ytnodeconstructort)
- [`parseResponse(data: IRawResponse): T`](#parseresponsedata-irawresponse-t)
- [Usage](#usage)
- [ObservedArray](#observedarray)
- [SuperParsedResponse](#superparsedresponse)
- [YTNode](#ytnode)
- [Type Casting](#type-casting)
- [Accessing properties without casting](#accessing-properties-without-casting)
- [Memo](#memo)
- [Adding new nodes](#adding-new-nodes)
- [How it works](#how-it-works)
___
## Structure
### Core
* [`/types`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/types) - General response types.
* [`/classes`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/classes) - InnerTube nodes.
* [`helpers.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/helpers.ts) - Helper functions/classes for the parser.
* [`parser.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/parser.ts) - The core of the parser.
* [`map.ts`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/map.ts) - Contains a list of all the InnerTube nodes, which is used to determine the appropriate node for a given renderer. It's important to note that this file is automatically generated and should not be edited manually.
### Clients
The parser itself is not tied to any specific client. Therefore, we have a separate folder for each client that the library supports. These folders are responsible for arranging the parsed data into a format that can be easily consumed and understood. Additionally, the underlying data is also exposed for those who wish to access it.
* [`/youtube`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/youtube)
* [`/ytmusic`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/ytmusic)
* [`/ytkids`](https://github.com/LuanRT/YouTube.js/blob/main/src/parser/ytkids)
## API
@@ -44,86 +49,82 @@ ___
<a name="parse"></a>
#### parse(data, requireArray, validTypes)
### `parse(data?: RawData, requireArray?: boolean, validTypes?: YTNodeConstructor<T> | YTNodeConstructor<T>[])`
Responsible for parsing individual nodes.
| Param | Type | Description |
| --- | --- | --- |
| data | `any` | The data |
| data | `RawData` | The data to parse |
| requireArray | `?boolean` | Whether the response should be an array |
| validTypes | `YTNodeConstructor<T> \| YTNodeConstructor<T>[] \| undefined` | Types of `YTNode` allowed |
When `requireArray` is `true`, the response will be an `ObservedArray<YTNodes>`.
- If `requireArray` is `true`, the response will be an `ObservedArray<YTNodes>`.
- If `validTypes` is `undefined`, the response will be an array of YTNodes.
- If `validTypes` is an array, the response will be an array of YTNodes that are of the types specified in the array.
- If `validTypes` is a single type, the response will be an array of YTNodes that are of the type specified.
When `validTypes` is `undefined`, the response will be an array of YTNodes.
When `validTypes` is an array, the response will be an array of YTNodes that are of the types specified in the array.
When `validTypes` is a single type, the response will be an array of YTNodes that are of the type specified.
If you do not specify `requireArray`, the return type of the function will not be known at runtime, and therefore we return the response wrapped in a helper, `SuperParsedResponse`, to gain access to the response.
If you do not specify `requireArray`, the return type of the function will not be known at runtime. Therefore, to gain access to the response, we return it wrapped in a helper, `SuperParsedResponse`.
You may use the `Parser#parseArray` and `Parser#parseItem` methods to parse the response in a deterministic way.
<a name="parseresponse"></a>
#### parseResponse(data)
### `parseResponse(data: IRawResponse): T`
Unlike `parse`, this can be used to parse the entire response object.
| Param | Type | Description |
| --- | --- | --- |
| data | `object` | Raw InnerTube response |
| data | `IRawResponse` | Raw InnerTube response |
## Usage
## ObservedArray
You may use `ObservedArray<T extends YTNode>` as a normal array, but it provides additional methods for typesafe access and casting.
### ObservedArray
You can utilize an `ObservedArray<T extends YTNode>` as a regular array, but it also offers further methods for accessing and casting values in a type-safe manner.
```ts
// For example, we have a feed, and want all the videos:
const feed = new ObservedArray<YTNode>([...feed.contents]);
const videos = feed.filterType(GridVideo);
// This is now a GridVideo[]
// Or we want only the first video:
// Here, we use the filterType method to retrieve only GridVideo items from the feed.
const videos = feed.filterType(GridVideo);
// `videos` is now a GridVideo[] array.
// Alternatively, we can use firstOfType to retrieve the first GridVideo item from the feed.
const firstVideo = feed.firstOfType(GridVideo);
// We may cast the whole array to a GridVideo[] and throw if we have any non-GridVideo elements:
// If we want to make sure that all elements in the `feed` array are of the `GridVideo` type, we can use the `as` method to cast the entire array to a `GridVideo[]` type. If the cast fails because of non-GridVideo items, an exception is thrown.
const allVideos = feed.as(GridVideo);
// There are some extra methods for ObservedArray<T extends YTNode>
// which we use internally but not documented here (yet).
// see the source code for more details.
// Note that ObservedArray provides additional methods beyond what's shown here, which we use internally. For more information, see the source code or documentation.
```
## SuperParsedResponse
Represents a parsed response in an unknown state. Either a `YTNode` or an `ObservedArray<YTNode>` or `null`.
You will need to assert the type and unwrap the response to get the actual value.
### SuperParsedResponse
Represents a parsed response in an unknown state. Either a `YTNode`, an `ObservedArray<YTNode>`, or `null`. To extract the actual value, you must first assert the type and unwrap the response.
```ts
// We can assert we have a YTNode:
// First, parse the data and store it in `response`.
const response = Parser.parse(data);
// Check whether `response` is a YTNode.
if (response.is_item) {
// If so, we can assert that it is a YTNode and retrieve it.
const node = response.item();
}
// We can assert we have an ObservedArray<YTNode>:
const response = Parser.parse(data);
// Check whether `response` is an ObservedArray<YTNode>.
if (response.is_array) {
// If so, we can assert that it is an ObservedArray<YTNode> and retrieve its contents as an array of YTNode objects.
const nodes = response.array();
}
// Or lastly a null response:
const response = Parser.parse(data);
// Finally, to check if `response` is a null value, use the `is_null` getter.
const is_null = response.is_null;
```
## YTNode
All renderers returned by InnerTube are converted to this generic class and then extended for the specific renderers.
This class is what allows us a typesafe way to use data returned by the InnerTube API.
### YTNode
All renderers returned by InnerTube are converted to this generic class and then extended for the specific renderers. This class is what allows us a type-safe way to use data returned by the InnerTube API.
Here's how to use this class to access returned data:
@@ -131,10 +132,10 @@ Here's how to use this class to access returned data:
```ts
// We can cast a YTNode to a child class of YTNode
const results = node.as(TwoColumnSearchResults);
// This will throw if the node is not a TwoColumnSearchResults
// We thus may want to check for the type of the node before casting
// This will throw an error if the node is not a TwoColumnSearchResults.
// Therefore, we may want to check for the type of the node before casting.
if (node.is(TwoColumnSearchResults)) {
// We do not need to recast the node, it is already a TwoColumnSearchResults after calling is() and using it in the branch where is() returns true
// We do not need to recast the node; it is already a TwoColumnSearchResults after calling is() and using it in the branch where is() returns true.
const results = node;
}
@@ -144,21 +145,20 @@ const results = node.as(TwoColumnSearchResults, VideoList);
// Similarly, we can check if the node is of a certain type.
if (node.is(TwoColumnSearchResults, VideoList)) {
// Again no casting is needed, the node is already of the correct type.
// // Again, no casting is needed; the node is already of the correct type.
const results = node;
}
```
### Accessing properties without casting
Sometimes multiple nodes have the same properties and we don't want to check the type of the node before accessing the property, for example, the property "contents" is used by many node types, and we may add more in the future, as such we want to only assert the property instead of casting to a specific type.
Sometimes multiple nodes have the same properties, and we don't want to check the type of the node before accessing the property. For example, the property 'contents' is used by many node types, and we may add more in the future. As such, we want to only assert the property instead of casting to a specific type.
```ts
// Accessing a property on a node which you aren't sure if it exists.
// Accessing a property on a node when you aren't sure if it exists.
const prop = node.key("contents");
// This returns the value wrapped into a `Maybe` type
// which you can use to find the type of the value
// note, however, this throws an error if the key doesn't exist
// we may want to check for the key before accessing it.
// This returns the value wrapped into a `Maybe` type, which you can use to determine the type of the value.
// However, this throws an error if the key doesn't exist, so we may want to check for the key before accessing it.
if (node.hasKey("contents")) {
const prop = node.key("contents");
}
@@ -169,19 +169,18 @@ if (prop.isString()) {
const value = prop.string();
}
// We can do more complex assertions too,
// like checking for instanceof.
// We can do more complex assertions, like checking for instanceof.
const prop = node.key("contents");
if (prop.isInstanceof(Text)) {
const text = prop.instanceof(Text);
// and then use the value as the given type
if (prop.isInstanceOf(Text)) {
const text = prop.instanceOf(Text);
// Then use the value as the given type.
text.runs.forEach(run => {
console.log(run.text);
});
}
// There are some special methods for using with the parser —
// such as getting the value as a YTNode.
// There are special methods for use with the parser, such as getting the value as a YTNode.
const prop = node.key("contents");
if (prop.isNode()) {
const node = prop.node();
@@ -200,13 +199,12 @@ if (prop.isNodeOfType([TwoColumnSearchResults, VideoList])) {
}
// Sometimes an ObservedArray is returned when working with parsed data.
// We've got a helper for that too;
// We also have a helper for this.
const prop = node.key("contents");
if (prop.isObserved()) {
const array = prop.observed();
// Now we may use all the ObservedArray methods as normal,
// like finding nodes of a certain type for example.
// Now we can use all the ObservedArray methods as normal, such as finding nodes of a certain type.
const results = array.filterType(GridVideo);
}
@@ -215,8 +213,8 @@ const prop = node.key("contents");
if (prop.isParsed()) {
const result = prop.parsed();
// SuperParsedResult is another helper for typesafe access to the parsed data,
// it is explained above with the `Parser#parse` method.
// SuperParsedResult is another helper for type-safe access to the parsed data.
// It is explained above with the `Parser#parse` method.
const results = results.array();
const videos = results.filterType(Video);
}
@@ -226,51 +224,47 @@ if (prop.isParsed()) {
const prop = node.key("contents");
const value = prop.any();
// Arrays are also a special case as every element may be of a different type,
// the `arrayOfMaybe` method will return an array of `Maybe`s.
// Arrays are a special case, as every element may be of a different type.
// The `arrayOfMaybe` method will return an array of `Maybe`s.
const prop = node.key("contents");
if (prop.isArray()) {
const array = prop.arrayOfMaybe();
// This will return Maybe[]
// This will return `Maybe[]`.
}
// Or if you want zero type safety you can use the `array` method.
// Or, if you don't need type safety, you can use the `array` method.
const prop = node.key("contents");
if (prop.isArray()) {
const array = prop.array();
// This will return any[]
// This will return any[].
}
```
## Memo
The `Memo` class is a helper class for memoizing values in the `Parser#parseResponse` method. It is useful for finding nodes after parsing the response.
### Memo
The `Memo` class is a helper class for memoizing values in the `Parser#parseResponse` method. It can be used to conveniently access nodes after parsing the response.
Say we want all of the videos in a search result. We can use the `Memo` to find all of them quickly without recursing through the response.
For example, if we'd like to obtain all of the videos from a search result, we can use the `Memo#getType` method to find them quickly without needing to traverse the entire response.
```ts
const response = Parser.parseResponse(data);
const videos = response.contents_memo.getType(Video);
// This returns the nodes as a ObservedArray<Video>.
// This returns the nodes as an `ObservedArray<Video>`.
```
`Memo` extends `Map<string, YTNode[]>` and can be used as a normal `Map` too if you want.
`Memo` extends `Map<string, YTNode[]>` and can be used as a regular `Map` if desired.
## Adding new nodes
Instructions can be found [here](https://github.com/LuanRT/YouTube.js/blob/main/docs/updating-the-parser.md).
## How it works
If you decompile a YouTube client and analyze it for a while you will notice that it has classes named `protos/youtube/api/innertube/MusicItemRenderer`, `protos/youtube/api/innertube/SectionListRenderer`, etc.
If you decompile a YouTube client and analyze it, it becomes apparent that it uses classes such as `../youtube/api/innertube/MusicItemRenderer` and `../youtube/api/innertube/SectionListRenderer` to parse objects from the response, map them into models, and generate the UI. The website operates similarly, but instead uses plain JSON. You can think of renderers as components in a web framework.
These classes are used to parse objects from the response, map them into models and generate the UI. The website works similarly, the difference is that it uses plain JSON (likely converted from protobuf server-side, hence the weird structure of the response).
Our approach is similar to YouTube's: our parser goes through all the renderers and parses their inner element(s). The final result is a nicely structured JSON, it even parses navigation endpoints, which allow us to make an API call with all required parameters in one line and emulate client actions, such as clicking a button.
Here we're taking a similar approach, the parser goes through all the renderers and parses their inner element(s). The final result is a nicely structured JSON, and on top of that, it also parses navigation endpoints which allow us to make an API call with all required parameters in one line and even emulate client actions (eg; clicking a button).
Here is your average, arguably ugly InnerTube response:
<details>
<summary>Click to see</summary>
<p>
To illustrate the transformation we make, let's take an unstructured InnerTube response and parse it into a cleaner format:
Raw InnerTube Response:
```js
{
sidebar: {
@@ -309,14 +303,7 @@ Here is your average, arguably ugly InnerTube response:
}
```
</p>
</details>
And what we get after parsing it:
<details>
<summary>Click to see</summary>
<p>
Clean Parsed Response:
```js
{
sidebar: {
@@ -324,21 +311,28 @@ And what we get after parsing it:
contents: [
{
type: 'PlaylistSidebarPrimaryInfo',
title: { text: '..' },
description: { text: '..' },
title: { text: '..', runs: [ { text: '..' } ] },
description: { text: '..', runs: [ { text: '..' } ] },
stats: [
{
text: '..'
text: '..',
runs: [
{
text: '..'
}
]
},
{
text: '..'
text: '..',
runs: [
{
text: '..'
}
]
}
]
}
]
}
}
```
</p>
</details>
```