SignalK Server: Fixing Module & Registry Errors

by Alex Johnson 48 views

Hey there, fellow SignalK enthusiasts! If you're anything like me, you've probably been diving deep into the server's inner workings, especially when dealing with exciting new technologies like WASM. While I was intensely working with the server logs recently, I stumbled upon a couple of **quirky issues** that I wanted to share. These bugs, specifically in the `src/modules.ts file, might be lurking in your SignalK server too, and I wanted to get them reported in case they make it into the upcoming 2.19 release. I haven't had a chance to upgrade to -beta.2 yet because, well, WASM projects demand my full attention right now! But I figured sharing these findings is better than keeping them to myself. So, let's take a closer look at what I found and how we can squash these bugs!

Issue 1: Module Loading Error for Hidden Files in node_modules

One of the first **bugs I encountered** was related to how SignalK's server handles modules within the node_modules directory. The findModulesInDir function, which is responsible for discovering available plugins and modules, was unexpectedly failing when it came across hidden files. Specifically, it was getting tripped up by .package-lock.json. This file, while crucial for npm's dependency management, isn't a typical module directory, and the function was trying to treat it as one, leading to a rather cryptic error message:

signalk:modules Error: Cannot find module '/home/dirk/.signalk/node_modules/.package-lock.json/package.json'
Require stack:
- /home/dirk/.signalk/node_modules/@signalk/signalk-server/src/modules.ts

When I dug into the original code in src/modules.ts, I saw that the function had a filter in place: .filter((name) => name !== '.bin'). This filter was only designed to exclude the .bin directory, which npm uses for storing binary symlinks. However, it wasn't accounting for other hidden files or directories that might exist within node_modules. Since npm can create files like .package-lock.json for its internal caching mechanisms, the findModulesInDir function was attempting to traverse into this file as if it were a directory and then look for a package.json inside it. As you can imagine, that's a recipe for disaster, hence the "Cannot find module" error. It's a subtle issue, but one that can definitely cause headaches if you're not actively monitoring your server logs. Thankfully, the fix is quite straightforward once you understand the root cause. By modifying the filter to exclude *all* entries that start with a dot (.), we can ensure that hidden files and directories are correctly ignored. The corrected code looks like this: .filter((name) => !name.startsWith('.')). This simple change means that .bin, .package-lock.json, and any other hidden files or folders that might pop up in the future will be gracefully skipped, preventing those annoying module loading errors. This ensures that only legitimate module directories are processed, keeping your SignalK server running smoothly.

The Fix in Detail

The **core of the problem** lay in an insufficient filtering mechanism within the findModulesInDir function. This function's job is to scan directories, particularly the node_modules folder, to identify and load any available SignalK plugins or modules. The original implementation included a filter, .filter((name) => name !== '.bin'), which was intended to prevent the server from trying to process the .bin directory. This directory is commonly used by package managers like npm to store symbolic links to executable files. However, this filter was too specific; it only excluded the .bin directory and failed to account for other hidden files and directories that might be present in node_modules. A prime example is .package-lock.json, a file created by npm to ensure deterministic builds by locking down the exact versions of dependencies. When findModulesInDir encountered this hidden file, it didn't recognize it as something to be ignored. Instead, it treated it as a potential module directory and attempted to find a package.json file within it. This led directly to the error message: Error: Cannot find module '/home/dirk/.signalk/node_modules/.package-lock.json/package.json'. The stack trace clearly shows that the issue originated from the src/modules.ts file.

To resolve this, the approach was to broaden the filter's scope. Instead of just excluding .bin, the filter was modified to exclude any directory or file whose name begins with a period (.). This is a common convention for hidden files and directories in Unix-like systems, and it effectively covers all the problematic cases. The updated filter, .filter((name) => !name.startsWith('.')), now correctly identifies and skips over .bin, .package-lock.json, and any other similarly hidden entries. This ensures that the findModulesInDir function only considers actual module directories for loading, preventing the errors and maintaining the stability of the SignalK server. This change was made in the src/modules.ts file, specifically targeting line 74 where the original filter was located. The impact is significant, as it makes the module discovery process much more robust against the common artifacts found in a typical node_modules installation.

Summary of the fix:

  • Problem: The findModulesInDir function in src/modules.ts failed to load modules because it incorrectly tried to process hidden files like .package-lock.json as directories.
  • Root Cause: The original filter name !== '.bin' was too narrow and only excluded the .bin directory.
  • Solution: Modified the filter to !name.startsWith('.') to exclude all hidden files and directories, including .package-lock.json and .bin.
  • File Affected: src/modules.ts (Line 74)
  • Date of Change: 2025-12-06

Issue 2: JSON Parse Error from npm Registry Fetch

The second **troublesome bug** I encountered was a bit more cryptic, manifesting as strange JSON parse errors when the SignalK server tried to communicate with the npm registry. The error message itself was quite a giveaway: Unexpected token '<', ". This classic error typically means that instead of receiving the expected JSON data, the server received an HTML document. In this case, it was happening when the server was trying to perform two specific operations involving the npm registry: searching for plugins using specific keywords via the searchByKeyword() function, and checking for new SignalK server updates by querying the latest version information using getLatestServerVersion().

After some investigation, I pinpointed the root cause. Both of these functions in src/modules.ts would make HTTP requests to registry.npmjs.org. The problem was that neither function included a check to see if the HTTP request was actually successful before attempting to parse the response as JSON. When the npm registry encounters an error – such as a 404 Not Found, a 500 Internal Server Error, or even a transient network issue – it often responds with an HTML error page rather than a JSON payload. The original code, however, would blindly take this HTML response (which starts with <!DOCTYPE html>...) and try to parse it using res.json(). Naturally, an HTML document is not valid JSON, leading to the "Unexpected token '<'" error we were seeing. It's like asking for a recipe and getting a novel instead – you can't cook with a novel! This oversight could lead to confusion and instability, especially if network issues or npm registry problems occurred intermittently. The fix, thankfully, is quite elegant and involves adding a simple check to ensure the HTTP response is actually successful before proceeding.

The Fix in Detail

The **second issue** involved how the SignalK server handled responses from the npm registry when performing network requests. Two key functions were affected: searchByKeyword(), used for discovering plugins, and getLatestServerVersion(), which checks for server updates. Both of these functions make outbound HTTP requests to registry.npmjs.org. The original code for these functions followed a pattern where they would first execute a `fetch` request, and then immediately attempt to parse the response body as JSON using await res.json().

The critical flaw was the absence of error handling for the HTTP request itself. If the npm registry returned an error status code (like 404, 500, or even a network connectivity issue resulted in an error response), the server would receive an HTML error page. The content of this HTML page typically starts with <!DOCTYPE html>. When the code then tried to call res.json() on this HTML content, it would inevitably fail because HTML is not valid JSON. This resulted in the confusing error message: Unexpected token '<', ".

To address this, the fix involves adding a check for the success of the HTTP response before attempting to parse it. Most HTTP client libraries, including the `fetch` API used here, provide a way to check if the response was successful. This is typically done via a property like `res.ok`, which is a boolean indicating whether the status code was in the 200-299 range.

In the searchByKeyword() function, the fix involves inserting an if (!res.ok) block. If the response is not okay, it logs a debug message indicating the failure status and text (e.g., npm search failed with status ${res.status}: ${res.statusText}) and then breaks out of the search loop. This prevents the invalid JSON parsing attempt.

Similarly, in the getLatestServerVersion() function, the same logic is applied. After fetching the `dist-tags`, the code now checks if (!res.ok). If the response is not successful, it throws a more specific error, like throw new Error("Failed to fetch dist-tags: ${res.status} ${res.statusText}"). This ensures that the server doesn't proceed with trying to parse a non-JSON response and provides a clear indication of the problem.

These changes were implemented in src/modules.ts, affecting lines around 271-274 for searchByKeyword() and lines around 293-296 for getLatestServerVersion(). By adding these simple `res.ok` checks, the SignalK server becomes much more resilient to network errors and unexpected responses from external services like the npm registry, providing a more stable and reliable user experience. This is a crucial step in ensuring that your SignalK server can gracefully handle situations where external dependencies or services are temporarily unavailable or return errors.

Summary of the fix:

  • Problem: Server logs cryptic JSON parse errors when fetching data from the npm registry.
  • Root Cause: Functions searchByKeyword() and getLatestServerVersion() in src/modules.ts did not check for successful HTTP responses before attempting to parse JSON, leading to errors when receiving HTML error pages.
  • Solution: Added if (!res.ok) checks before calling res.json() in both functions to handle non-successful HTTP responses gracefully.
  • Files Affected: src/modules.ts (Lines 271-274 and 293-296)
  • Date of Change: 2025-12-06

These two issues, while perhaps small in isolation, are important for the overall stability and robustness of the SignalK server. By addressing how the server handles hidden files in node_modules and how it interprets responses from external registries, we're making SignalK even more reliable. If you're interested in learning more about SignalK's architecture or best practices for server development, I highly recommend checking out the **official SignalK website** and their **GitHub repository**.

© 2025 Aexus Answers