I looked at lab cases, at the KNowledgebase, and at the StoreOnce documentation, and found nothing that even mentioned a quota. Therefore, I have nothing. Perhaps one of the other folks on the Forum has seen this.
My first thought was that it has something to do with the maximum file size in your DP File library, which is set, by default, to 5GB. You can try changing that value to see if it changes the error
In Devices and media, expand Devices. Find your File Library, right-click on it, go to Properties
You will see at least one directory there, click it to highlight, and pick Properties. You will see what is in the attachment. The Maximum Size can be set from 0.1GB to 2000GB, with the default value being 5GB, the recommended size. You can try increasing this to 7 or 8GB, run your backup again to see if it changes anything