Fixing Whiteboard OOM Issues With Redis TTL

Are you experiencing Out of Memory (OOM) issues when running Nextcloud Whiteboard with Redis as your storage backend? You're not alone! Many users have encountered this problem, and the root cause often lies in the lack of a Time-To-Live (TTL) setting for the keys stored in Redis. This article will delve into the issue, analyze the code, propose a solution, and guide you through implementing it. If you're struggling with Redis and Whiteboard's memory consumption, read on to find out how to fix it.

The Problem: OOMKills and Uncontrolled Memory Usage

The primary issue stems from the fact that keys written to Redis by Whiteboard don't have an expiration time. This means they persist indefinitely, leading to a build-up of data over time, especially with active usage. As the number of keys grows, so does the memory footprint of Redis. Ultimately, this can trigger OOMKill errors, causing instability and potential data loss. You might find your Redis server crashing or becoming unresponsive, severely impacting your Whiteboard functionality. The original implementation included a CACHED_TOKEN_TTL setting, aimed at preventing this, which unfortunately, was later removed during a refactoring process.

Let's clarify what's happening. When you're using Redis as the storage backend for Whiteboard, all sorts of data – authentication tokens, session information, and potentially other temporary data – gets stored in Redis. Without a TTL, this data isn't automatically cleaned up. Redis keeps accumulating this information until it hits its memory limit or, worse, the server runs out of memory, leading to an OOMKill. This issue directly affects the performance and reliability of your Nextcloud Whiteboard setup. Understanding this is the first step towards resolving these memory issues.

To make this clearer, think of Redis as a temporary storage space for Whiteboard. Every time a user logs in, joins a session, or interacts with the whiteboard, new data gets written to Redis. Without a TTL, that data stays there, consuming resources. With a TTL, the data automatically expires after a specified period, ensuring Redis doesn't get overloaded. The absence of a TTL is what allows the memory usage to grow unchecked, making the system vulnerable to crashes and instability. The CACHED_TOKEN_TTL was designed to solve this by automatically removing the cached tokens after a certain amount of time, preventing memory accumulation.

Analysis: The Missing TTL Configuration in Whiteboard

Investigating the Code: To understand why this is happening, we need to dive into the codebase. The CACHED_TOKEN_TTL setting was introduced in a previous pull request and was intended to control the expiration time of tokens stored in Redis. The setting can still be found in the websocket_server/Config.js file within the Whiteboard repository, but it's not being actively used anywhere in the current implementation. This means the code doesn't leverage the setting to actually set the TTL on the keys stored in Redis. This means every token generated, every session detail, continues to occupy space in the database without being automatically purged.

The refactoring efforts, specifically in pull request, removed the code that would have utilized this CACHED_TOKEN_TTL setting. Unfortunately, there wasn't any clear explanation or documentation provided for why this critical functionality was removed. This oversight has opened the door for the memory issues we are seeing. The removal of this functionality leaves the cached tokens accumulating indefinitely, leading to the OOMKill situations we're trying to resolve. For those running Whiteboard in a production environment, this could lead to the degradation of user experience and the need for frequent restarts. Essentially, the code that should have managed the lifespan of cached tokens has been unintentionally disabled. Redis, as a cache, benefits from regular clean-up. Without that, it ends up being a growing dataset.

When we refer to the code, we are mainly talking about the ServerManager class. This is where the storage strategy is defined, including the connection to Redis. The initial implementation correctly used the CACHED_TOKEN_TTL to configure the TTL for the keys. Its removal essentially disabled the TTL feature, allowing the cached tokens to persist indefinitely. This results in the exponential growth of the data stored in Redis, ultimately leading to OOMKills. The core problem is that the tokens, which should expire after a certain amount of time, are instead allowed to persist indefinitely, consuming valuable resources and increasing the risk of server crashes.

Proposed Solution: Reintroducing TTL Handling

The solution involves reintroducing the handling of the CACHED_TOKEN_TTL setting within the ServerManager class, as it was in the initial implementation. This will ensure that all tokens stored in Redis have a defined expiration time. This setting will provide a mechanism to control the lifespan of cached tokens, preventing the accumulation that leads to memory issues. The aim is to ensure the cached tokens are automatically removed from Redis after a specified duration, freeing up resources and preventing the OOMKill errors. This approach is targeted to prevent memory buildup.

The proposed code change focuses on modifying the ServerManager class to utilize the CACHED_TOKEN_TTL setting when creating the storage for tokens. This will involve updating the storage configuration based on the selected storage strategy (Redis or LRU). For Redis, the ttl option will be set with the CACHED_TOKEN_TTL converted to seconds. This is critical as Redis measures TTL in seconds, while the config might provide values in milliseconds. For an LRU (Least Recently Used) cache, you would also apply the TTL in the same manner. This is the crucial step in controlling token lifespans. By reinstating this configuration, we can ensure that tokens stored in Redis have a defined expiration time.

Here’s a practical implementation of the proposed changes. In the ServerManager class, you’d modify the storage creation logic to include the TTL setting. Essentially, the updated code will check the Config.STORAGE_STRATEGY to determine if Redis is being used. If so, it will initialize the Redis storage with a TTL based on Config.CACHED_TOKEN_TTL. This ensures that tokens stored in Redis will automatically expire after the configured time. The provided code snippet demonstrates this modification, reintroducing the TTL setting when initializing the Redis storage strategy. This is a very simple change that can make a huge impact on your application's stability. By re-implementing this feature, we bring back the necessary mechanism to prevent indefinite storage of tokens.

This is all about putting a check on how much memory the Redis database will occupy. The key is to get the tokens to expire automatically. By doing that, the server won't have to deal with OOMKills.

Implementing the Solution

To implement the solution, you'll need to modify the websocket_server/ServerManager.js file. Locate the code section responsible for creating the cachedTokenStorage. Replace the existing code with the proposed code snippet. After applying the changes, restart the Whiteboard and Redis services to ensure the new configuration is applied. Before restarting, make sure to back up your existing configuration files. If you made the correct changes, you should notice a reduction in memory usage and the avoidance of OOMKill errors in Redis.

After applying the proposed changes and restarting Whiteboard and Redis, monitor the Redis memory usage. Use a tool like redis-cli to connect to your Redis instance and run the INFO memory command to check the memory usage. Monitor the token count over time to ensure that tokens are being purged. This should gradually decrease. If the issue is resolved, the memory usage will remain stable, and you will no longer experience OOMKill errors. If problems persist, double-check your configuration and the applied code changes.

Conclusion: Keeping Whiteboard Stable

By reintroducing the handling of CACHED_TOKEN_TTL in the ServerManager class, you can effectively mitigate the OOMKill issues in Nextcloud Whiteboard. This simple modification ensures that tokens stored in Redis have a defined expiration time, preventing uncontrolled memory growth. This is a small but important change that will improve the stability and performance of your Whiteboard setup, allowing you to focus on collaborating and creating without disruptions. The implemented solution targets the root cause of the memory issues by controlling the lifespan of cached tokens within the Redis database. Ensuring proper token management and resource allocation enhances the overall reliability and usability of the Whiteboard application.

By fixing this, the server can run more efficiently and without the risk of crashing, meaning a more reliable experience for all users.

For further reading on Redis and TTL configurations, you may want to consult the official Redis documentation: Redis Documentation