A few weeks ago we had so many logs that apparently the actual physical storage was full.
The bug here is not necessarily that this happened, but that it caused all ‘save storage actions’ to lead to a ‘null’ in that storage entity. Probably because the save action meant that the actual file couldn’t be written to the drive.
The issue here is that in production, storages are often used for critical processes. This means that just an overflow of logs, can lead this way to a full failure of the environment.
This should be mitigated - if possible with auto-scaling of such storage, or by giving storages and logs different partitions to work with.
PS: the problem is mitigated on our side, I was asked by Roland to submit this bugreport.