locking: deal with clock skew

docs/faq.rst

@@ -29,6 +29,17 @@ Can I back up from multiple servers into a single repository?
 
 Yes, you can! Even simultaneously.
 
+The clocks of machines sharing a repository should be roughly synchronized
+(e.g. via NTP): repository locks and archive/manifest timestamps are based on
+the clients' clocks, so big clock differences between clients can cause
+trouble. Where the storage backend provides object timestamps (file, sftp, s3
+and current rest servers - but not rclone), borg cross-checks lock staleness
+against the storage's clock (so a client with a wrong clock can not break
+another client's healthy lock) and logs a warning when it detects that the
+clocks of concurrently active clients differ by more than a few minutes.
+The storage's own clock does not need to be correct - it is only used as a
+common reference between the clients.
+
 Can I back up to multiple swapped backup targets?
 --------------------------------------------------
 

docs/internals/data-structures.rst

@@ -1170,17 +1170,33 @@ To implement locking based on ``borgstore``, borg stores objects below locks/.
 
 The objects contain:
 
-- a timestamp when lock was created (or refreshed)
+- a timestamp when lock was created (or refreshed), stamped by the clock of
+  the machine writing the lock
 - host / process / thread information about lock owner
 - lock type: exclusive or shared
 
+Where the storage backend provides object timestamps (file, sftp, s3 and
+current rest servers - but not rclone), borg additionally uses the lock
+object's store-side mtime, which is stamped by the storage's clock.
+
 Using that information, borg implements:
 
-- lock auto-expiry: if a lock is old and has not been refreshed in time,
-  it will be automatically ignored and deleted. the primary purpose of this
-  is to get rid of stale locks by borg processes on other machines.
 - lock auto-removal if the owner process is dead. the primary purpose of this
   is to quickly get rid of stale locks by borg processes on the same machine.
+- lock auto-expiry: if a lock is old and has not been refreshed in time,
+  it will be automatically ignored and deleted. the primary purpose of this
+  is to get rid of stale locks by borg processes on other machines. to never
+  kill a healthy lock just because its writer's clock is skewed against ours
+  (see :issue:`9870`), a lock is only expired by age if it looks stale both by
+  the clients' clocks (content timestamp) and by the storage's clock
+  (store-side mtime); store-side timestamps can veto an expiry, but never
+  cause one.
+- a warning if the clocks of concurrently active clients differ by more than
+  a few minutes.
+
+See the module docstring of ``src/borg/storelocking.py`` for the details
+(clock domains, how store "now" is derived, what happens without store-side
+mtimes).
 
 Breaking the locks
 ------------------

pyproject.toml

@@ -31,7 +31,7 @@ license = "BSD-3-Clause"
 license-files = ["LICENSE", "AUTHORS"]
 dependencies = [
   "borghash ~= 0.2.0",
-  "borgstore[rest,blake3] ~= 0.6.0",
+  "borgstore[rest,blake3] ~= 0.6.1",
   "msgpack >=1.0.3, <=1.2.1",
   "packaging",
   "platformdirs >=3.0.0, <5.0.0; sys_platform == 'darwin'",  # for macOS: breaking changes in 3.0.0.
@@ -57,9 +57,9 @@ mfusepy = ["mfusepy >= 3.1.0, <4.0.0"]  # fuse 2+3, high-level
 # a pypi release of borgbackup can't contain a dependency on github!
 # mfusepym = ["mfusepy @ git+https://github.com/mxmlnkn/mfusepy.git@master"]
 nofuse = []
-s3 = ["borgstore[rest,blake3,s3] ~= 0.6.0"]
-sftp = ["borgstore[rest,blake3,sftp] ~= 0.6.0"]
-rclone = ["borgstore[rest,blake3,rclone] ~= 0.6.0"]
+s3 = ["borgstore[rest,blake3,s3] ~= 0.6.1"]
+sftp = ["borgstore[rest,blake3,sftp] ~= 0.6.1"]
+rclone = ["borgstore[rest,blake3,rclone] ~= 0.6.1"]
 cockpit = ["textual>=6.8.0"]  # might also work with older versions, untested
 
 [project.urls]
@@ -155,6 +155,7 @@ dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
 "src/borg/testsuite/archiver/return_codes_test.py" = ["F811"]
 "src/borg/testsuite/benchmark_test.py" = ["F811"]
 "src/borg/testsuite/platform/platform_test.py" = ["F811"]
+"src/borg/testsuite/storelocking_test.py" = ["F811"]
 
 [tool.pytest.ini_options]
 markers = []

src/borg/constants.py

@@ -93,6 +93,12 @@
 # this, the pack is re-verified.
 MAX_CLOCK_SKEW = 7200  # [s]
 
+# Maximum tolerated clock skew between the clocks of borg clients concurrently using the same
+# repository before a warning is emitted (see storelocking). Must be well below the lock stale
+# timeout (30 min) / refresh interval (15 min) so users get warned long before skew could
+# interfere with lock staleness judgment or manifest timestamps.
+MAX_MUTUAL_CLOCK_SKEW = 300  # [s]
+
 # How many segment files Borg puts into a single directory by default.
 DEFAULT_SEGMENTS_PER_DIR = 1000