Write logs to the persistent file system

Oct 13, 2025 by pynkocean.com 18 views

Introduction

Log retention is a crucial aspect of any system, allowing developers to analyze and troubleshoot issues that may arise. However, the current implementation of log retention in the launcher has several limitations. In this article, we will explore a proposal to improve log retention by writing logs to the persistent file system on the instance.

Current Limitations

The current implementation of log retention in the launcher has several limitations. Firstly, the duration of retained logs depends on how noisy Node-RED is. This means that if Node-RED is generating a high volume of logs, the retained logs may be truncated or lost. Secondly, the logs are only available if the launcher is running, and any history is lost if the launcher itself is restarted (e.g., suspend/resume). These limitations make it difficult for developers to analyze and troubleshoot issues that may arise.

Proposal

To address the current limitations, we propose writing logs to the persistent file system on the instance. This will provide a more robust and reliable log retention mechanism. Here are the key features of the proposal:

Log File Rotation

The file should be rotated daily to prevent the log file from growing indefinitely.
This will ensure that the log file remains manageable and easy to analyze.

Log Retention

Only retain the last N days of logs (14/30 days... could be configurable in the future).
This will ensure that the log retention mechanism is scalable and efficient.

API for Log Retrieval

Provide an API in the launcher to retrieve logs for a given time period.
This will allow developers to analyze and troubleshoot issues that may arise.
The API will be a separate task, as mentioned in the proposal.

Log File Location

Log files will show up under the 'assets' view of the instance.
This will provide a convenient location for developers to access and analyze log files.

Benefits

The proposed log retention mechanism will provide several benefits, including:

Improved log retention: The proposed mechanism will ensure that logs are retained for a longer period, making it easier for developers to analyze and troubleshoot issues.
Increased reliability: The proposed mechanism will ensure that logs are retained even if the launcher is restarted or suspended.
Scalability: The proposed mechanism will ensure that the log retention mechanism is scalable and efficient.

Implementation

To implement the proposed log retention mechanism, we will need to make the following changes:

Write logs to the persistent file system: We will need to modify the launcher to write logs to the persistent file system on the instance.
Implement log file rotation: We will need to implement a log file rotation mechanism to prevent the log file from growing indefinitely.
Implement log retention: We will need to implement a log retention mechanism to retain the last N days of logs.
Implement API for log retrieval: We will need to implement an API in the launcher to retrieve logs for a given time period.

Conclusion

In conclusion, the proposed log retention mechanism will provide several benefits, including improved log retention, increased reliability, and scalability. We believe that this proposal will improve the overall quality and reliability of the system.

Future Work

There are several future work items that we will need to address, including:

Configurability: We will need to make the log retention mechanism configurable, allowing developers to customize the log retention period.
API for log retrieval: We will need to implement an API in the launcher to retrieve logs for a given time period.
Log file location: We will need to ensure that log files are easily accessible and manageable.

References

FlowFuse/flowfuse/issues/5301

Appendix

The following is a list of technical details that are relevant to the proposed log retention mechanism:

Log file format: We will use a standard log file format, such as JSON or CSV.
Log file compression: We will use log file compression to reduce the size of the log file.
Log file encryption: We will use log file encryption to ensure that log files are secure and tamper-proof.
Improve Log Retention: Q&A ==========================

Introduction

In our previous article, we proposed a new log retention mechanism that writes logs to the persistent file system on the instance. This article provides a Q&A section to address any questions or concerns that you may have about the proposed mechanism.

Q: What are the benefits of writing logs to the persistent file system?

A: Writing logs to the persistent file system provides several benefits, including:

Improved log retention: The proposed mechanism will ensure that logs are retained for a longer period, making it easier for developers to analyze and troubleshoot issues.
Increased reliability: The proposed mechanism will ensure that logs are retained even if the launcher is restarted or suspended.
Scalability: The proposed mechanism will ensure that the log retention mechanism is scalable and efficient.

Q: How will log file rotation work?

A: Log file rotation will work as follows:

The log file will be rotated daily to prevent the log file from growing indefinitely.
The rotated log files will be stored in a separate directory, allowing developers to easily access and analyze them.

Q: How will log retention work?

A: Log retention will work as follows:

Only the last N days of logs will be retained (14/30 days... could be configurable in the future).
The retained logs will be stored in a separate directory, allowing developers to easily access and analyze them.

Q: How will the API for log retrieval work?

A: The API for log retrieval will work as follows:

The API will allow developers to retrieve logs for a given time period.
The API will be a separate task, as mentioned in the proposal.

Q: Where will log files be stored?

A: Log files will be stored in the 'assets' view of the instance, under a well-defined directory.

Q: How will log file compression and encryption work?

A: Log file compression and encryption will work as follows:

Log file compression will be used to reduce the size of the log file.
Log file encryption will be used to ensure that log files are secure and tamper-proof.

Q: What are the technical details of the proposed log retention mechanism?

A: The technical details of the proposed log retention mechanism are as follows:

Log file format: We will use a standard log file format, such as JSON or CSV.
Log file compression: We will use log file compression to reduce the size of the log file.
Log file encryption: We will use log file encryption to ensure that log files are secure and tamper-proof.

Q: What are the future work items for the proposed log retention mechanism?

A: The future work items for the proposed log retention mechanism are as follows:

Configurability: We will need to make the log retention mechanism configurable, allowing developers to customize the log retention period.
API for log retrieval: We will need to implement an API in the launcher to retrieve logs for a given time period.
Log file location: We will need to ensure that log files are easily accessible and manageable.

Conclusion

References

FlowFuse/flowfuse/issues/5301

Appendix

The following is a list of frequently asked questions and answers that are relevant to the proposed log retention mechanism:

Q: What is the purpose of the proposed log retention mechanism? A: The purpose of the proposed log retention mechanism is to improve log retention, increase reliability, and scalability.
Q: How will the proposed log retention mechanism be implemented? A: The proposed log retention mechanism will be implemented by writing logs to the persistent file system on the instance.
Q: What are the benefits of the proposed log retention mechanism? A: The benefits of the proposed log retention mechanism include improved log retention, increased reliability, and scalability.