Peek 2.1.0 Crash: Large Stash Libraries & String Limit

Encountering crashes with Peek 2.1.0 when building a cache for large Stash libraries? You're not alone. This article dives into the issue of Peek crashing due to string limits and the absence of pagination when dealing with substantial video collections in Stash. We'll explore the root cause, symptoms, and potential solutions to get Peek working smoothly with your extensive library.

Understanding the Problem: Peek and Large Stash Libraries

When using Peek 2.1.0 with a sizable Stash library, such as one containing around 104,000 videos (approximately 74 TB), you might face crashes during the cache-building process. The culprit often lies in how Peek fetches scene data from Stash. It appears that Peek attempts to retrieve all scenes in a single GraphQL call, resulting in an extremely large JSON response. This massive response can exceed Node.js's maximum string length limit, leading to the crash.

Specifically, the crash occurs during the findScenesCompact request. This request, intended to gather scene information for the cache, ends up overwhelming the system due to the sheer volume of data. While Stash itself might function normally, Peek's cache initialization process falters repeatedly, making it impossible to use Peek with your Stash library.

To put it simply, the string limit of the system is exceeded when Peek tries to process the data from such a large library all at once. This is further compounded by the lack of pagination, which would allow Peek to fetch data in smaller, more manageable chunks.

Symptoms of the Crash

Several telltale signs indicate that you're experiencing this issue:

• Peek crashes during cache initialization: The most obvious symptom is that Peek fails to build its cache and crashes shortly after starting.
• Repeated retries: Peek might attempt to rebuild the cache multiple times (e.g., 5 times) before giving up, leading to repeated crashes.
• Stash instability: In some cases, Stash itself might crash alongside Peek, especially when Peek is restarted and attempts to rebuild the cache.
• Resource exhaustion: You might notice that Peek consumes a significant amount of system resources, such as CPU and memory, before crashing.
• Error messages: The error logs will likely contain messages like:
  • Error: Scene fetch failed: Cannot create a string longer than 0x1fffffe8 characters
  • Error JSON: ERR_STRING_TOO_LONG

These errors clearly point to the issue of exceeding the maximum string length limit during the scene fetching process.

Replicating the Issue

To replicate this crash, you'll need:

1. Stash v0.29.3: The issue has been observed specifically with this version of Stash.
2. A large Stash library: A library containing a substantial number of scenes (e.g., ~104,000) is crucial for triggering the crash.
3. Peek 2.1.0: This is the version of Peek known to have this issue.

Here are the steps to reproduce the problem:

1. Run Stash v0.29.3 with your large video library.
2. Start Peek 2.1.0.
3. Observe the cache initialization process. After approximately 15–20 seconds, Peek will likely crash.
4. Peek might retry the cache initialization process multiple times, crashing repeatedly.

By following these steps, you can confirm whether you're encountering the same issue caused by the large library size and the string limit.

Diving Deeper: The Root Cause

The core issue stems from Peek's approach to fetching scene data. Instead of utilizing pagination or other techniques to break down the data retrieval into smaller chunks, Peek attempts to fetch all scenes in a single request. This all-or-nothing approach works well for smaller libraries but becomes problematic with large collections.

When the findScenesCompact request is made, Stash responds with a massive JSON payload containing information about all the scenes in your library. This JSON data is then loaded into a string within the Node.js environment. However, Node.js has a limit on the maximum string length, which is approximately 2GB (2^31 - 1 characters) on 64-bit systems. If the JSON string exceeds this limit, the ERR_STRING_TOO_LONG error is thrown, and Peek crashes.

The lack of pagination is the primary contributing factor here. Pagination would allow Peek to request scenes in batches, process them individually, and avoid exceeding the string length limit. Without pagination, the entire dataset must be loaded into memory at once, leading to the crash.

Potential Solutions and Workarounds

While a permanent fix ideally involves implementing pagination in Peek's data fetching mechanism, several workarounds can help mitigate the issue in the meantime:

1. Reduce library size: If possible, consider reducing the number of videos and scenes in your Stash library. This might involve archiving older content or splitting your library into smaller, more manageable parts. However, this is often not a practical solution for users with large collections.

2. Increase Node.js memory limit: You can try increasing the memory allocated to the Node.js process running Peek. This might provide more space for the large JSON string, but it's not a guaranteed fix and could lead to other performance issues. The method for increasing memory limits varies depending on your environment (e.g., using the --max-old-space-size flag in Node.js).

3. Optimize Stash database: Ensure that your Stash database is properly optimized. A fragmented or poorly indexed database can slow down data retrieval and increase the size of the JSON response. Regularly performing database maintenance tasks can help improve performance.

4. Use a different browser: While the issue is not specific to a particular browser, some browsers might handle large JSON responses more efficiently than others. Experimenting with different browsers (e.g., Chrome, Safari) could potentially provide some relief.

5. Implement a rate limiter: Implement a rate limiter on the GraphQL requests sent to Stash. This can prevent Peek from overwhelming Stash with requests and potentially causing crashes. However, this workaround might slow down the cache building process.

6. Wait for a Peek update: The most promising solution is to wait for an update to Peek that addresses this issue directly. Hopefully, future versions of Peek will incorporate pagination or other techniques to handle large libraries more efficiently.

Practical Steps Taken and Observations

One user reported experiencing this issue and shared their troubleshooting steps. They observed that after deleting all metadata from Stash and gradually rebuilding the library, Peek would crash Stash intermittently. Initially, Peek would work with a smaller library, but as the library grew beyond approximately 50,000 videos, crashes became more frequent. This observation further supports the theory that the library size is the primary factor contributing to the issue.

The user also noted that Peek's crashes would sometimes resolve themselves after multiple attempts, suggesting that the issue might be related to timing or resource availability. However, this is not a reliable solution, and the crashes would consistently reappear as the library grew.

Error Message Breakdown

The error messages provide valuable clues about the nature of the crash. The Error: Scene fetch failed: Cannot create a string longer than 0x1fffffe8 characters message clearly indicates that the program is attempting to create a string that exceeds the maximum allowable length. The 0x1fffffe8 value represents the maximum string length limit in bytes (approximately 2GB).

The Error JSON: ERR_STRING_TOO_LONG message further confirms that the string limit is the cause of the crash. This error code is specific to Node.js and is thrown when an attempt is made to create a string that exceeds the maximum length.

By understanding the error messages, you can confidently diagnose the issue and focus on solutions that address the string limit problem.

Conclusion

The Peek 2.1.0 crash during cache build with large Stash libraries is a frustrating issue, but understanding the root cause – the string limit and lack of pagination – is the first step towards finding a solution. While workarounds can provide temporary relief, the ultimate fix lies in implementing pagination or other efficient data fetching mechanisms in Peek. Keep an eye out for future updates to Peek that address this issue.

In the meantime, consider exploring alternative stash management tools or contributing to the Peek project to advocate for improved handling of large libraries.

For more in-depth information on GraphQL and pagination techniques, consider exploring resources like the Official GraphQL Documentation.