Hi Vincent,
I never got any feedback to this question either in the ITRC forum or off-list. I also didn't actively search for a workaround, most of the Samba boxes slowly went away (specifically when backup admins discovered how nicely VSS on a Windows server allows for user self service recoveries and reduces their workload), those that are still running them have accepted it's a Windows thing (locking has been an issue there for ages, it's just no longer true due to VSS again).
The situation is weird and I never really understood why it behaves that way even though locking is clearly not mandatory. I wasn't even aware O_NONBLOCK open(2) would lead to EAGAIN on a non-mand-mounted file system. To me it appears this is a well meant feature specifically implemented to cooperate with Samba locking, but it badly needs a switch to disable it - the locking behavior is not god-given like on Windows, it's under admin control, and I don't like software that tries to take this control away from me.
As to your issue with 8.x consolidation that turns this locking problem from just a nuisance into a data loss nightmare, this sounds really bad. It sometimes feels that certain feature sets that have been implemented some day (and were pushed as the latest and greatest by the sales folks) later fall out of fashion, start to bitrot and nobody really likes the admins like us who come out of the woodwork, declare that they actually used this stuff and are not amused about it breaking. Note I phrased this in a generic way, it's by far not just DP which shows this developmental problem. I would go as far as seeing the foreshadows of a new coming software crisis hidden in there, as the story is repeating all over the place with a lot of different products, FLOSS or proprietary alike. In short, I was fed up with all the issues around the feature set of consolidation, DFMF file libraries, enhanced incrementals and virtual synthetic fulls in an incremental forever chain and eliminated that completely wherever I had used it before. The file libraries are now StoreOnce software stores, the backups are back to classic staggered full/incremental/differential cadences, and the issues are - ahem - different ones. But there are less of them, at least with 7.03 (I have so far just toyed with 8.x, so I hold my breath here - I just read more about issues than ever before).
As for a reproducer, the best idea would probably be a small C program (or even a perl script) that does the minimum file opening and locking which would trigger the behavior as seen from Samba, but without all the complexity. Of course the developers should know why DP does the open(2) dance in this specific way and why it was implemented with these precautions (I assume in true mandatory locking setups, without these precautions, the DA itself would block, which of course is a no-go and has to be prevented). Another way to close in onto the issue would be hacking the Linux kernel and preventing it from returning EAGAIN on O_NONBLOCK opens of regular files when they aren't actually on a mandatory locking file system. This could even be considered a bug in Linux, given an open on such file system would not block (it can only block with mandatory locking engaged) and as such, it should not return EAGAIN in the O_NONBLOCK case either. Given mandatory locking is a traditional unloved step child in the Unix world, a corner case bug like that may have delevoped without anyone ever noticing or (given we did notice it) correctly attributing it to a kernel misbehavior.
HTH,
Andre.