borg2 b19/b21: stale lock issue on `repo-create`

[nicojx]

I'm trying to convert a borg1 backup to borg2. On setting up the new store, I get a stale lock error, even though the lock is new. It could be because I'm sshing into a server in a different time zone, or because that server is not in GMT.

root@server:~# export BORG_DISPLAY_PASSPHRASE=no
root@server:~# export BORG_NEW_PASSPHRASE='new_passphrase'
root@server:~# export BORG_OTHER_PASSPHRASE='old_passphrase'
root@server:~# borg2 --repo=/mnt/intenso8/borgbackup2 repo-create --encryption=repokey-aes-ocb --other-repo=/mnt/intenso8/borgbackup --from-borg1 --debug
using builtin fallback logging configuration
17 self tests completed in 0.21 seconds 
borgstore: store('config/readme') -> 73B in 1.3ms 
borgstore: store('config/version') -> 1B in 1.1ms 
borgstore: store('config/id') -> 64B in 2.3ms 
caching 0 chunks (incremental=True). 
borgstore: list_start('cache', deleted=False) 
borgstore: list_end('cache', deleted=False) -> 0 
cached chunk indexes: [] 
caching chunks index as cache/chunks.a283591f50db914c in repository... 
borgstore: store('cache/chunks.a283591f50db914c') -> 297B in 1.2ms 
borgstore: load('config/readme', offset=0, size=None, deleted=False) -> 73B in 0.3ms 
borgstore: load('config/version', offset=0, size=None, deleted=False) -> 1B in 0.3ms 
borgstore: load('config/id', offset=0, size=None, deleted=False) -> 64B in 0.2ms 
LOCK-INIT: initialising. store: <Store(url='file:///mnt/intenso8/borgbackup2', levels=[('archives/', [0]), ('config/', [0]), ('cache/', [0]), ('locks/', [0]), ('data/', [2]), ('keys/', [0])])>, stale: 1800s, refresh: 900s. 
LOCK-ACQUIRE: trying to acquire a lock. exclusive: True. 
borgstore: list_start('locks', deleted=False) 
borgstore: list_end('locks', deleted=False) -> 0
LOCK-CREATE: creating lock in store. key: 0661aa3684531fb2, lock: {'exclusive': True, 'hostid': 'server@237747344171252', 'processid': 244588, 'threadid': 0, 'time': '2026-08-22T22:40:50.045+00:00'}. 
borgstore: store('locks/0661aa3684531fb2') -> 131B in 1.1ms 
borgstore: list_start('locks', deleted=False) 
borgstore: list_end('locks', deleted=False) -> 1 
borgstore: load('locks/0661aa3684531fb2', offset=0, size=None, deleted=False) -> 131B in 0.4ms 
LOCK-ACQUIRE: we are the only exclusive lock! 
borgstore: list_start('locks', deleted=False) 
borgstore: list_end('locks', deleted=False) -> 1 
borgstore: load('locks/0661aa3684531fb2', offset=0, size=None, deleted=False) -> 131B in 0.3ms 
LOCK-ACQUIRE: success! no non-exclusive locks are left! 
bVerified integrity of /mnt/intenso8/borgbackup/index.11374 
security: read previous location '/mnt/intenso8/borgbackup' 
security: read manifest timestamp '2026-07-08T04:51:52.287928' 
security: determined newest manifest timestamp as 2026-07-08T04:51:52.287928 
security: repository checks ok, allowing access 
Initializing repository at "/mnt/intenso8/borgbackup2" 
borgstore: store('keys/repokey') -> 486B in 1.6ms 
Key in "<Repository proto='file', user=None, host=None, port=None, path='/mnt/intenso8/borgbackup2'>" created. 
Keep this key safe. Your data will be inaccessible without it. 
borgstore: list_start('locks', deleted=False) 
borgstore: list_end('locks', deleted=False) -> 1 
borgstore: load('locks/0661aa3684531fb2', offset=0, size=None, deleted=False) -> 131B in 0.4ms 
LOCK-STALE: lock is too old, it was not refreshed. lock: {'exclusive': True, 'hostid': 'server@237747344171252', 'processid': 244588, 'threadid': 0, 'time': '2026-08-22T22:40:50.045+00:00', 'key': '0661aa3684531fb2', 'dt': datetime.datetime(2026, 8, 22, 22, 40, 50, 45000, tzinfo=datetime.timezone.utc)}. 
LOCK-DELETE: deleting lock from store. key: 0661aa3684531fb2. 
borgstore: delete('locks/0661aa3684531fb2', deleted=False) -> 0B in 0.5ms 
LOCK-REFRESH: our lock was killed, there is no safe way to continue. 
borgstore: list_start('locks', deleted=False) 
borgstore: list_end('locks', deleted=False) -> 0 
Failed to release the lock <Store(url='file:///mnt/intenso8/borgbackup2', levels=[('archives/', [0]), ('config/', [0]), ('cache/', [0]), ('locks/', [0]), ('data/', [2]), ('keys/', [0])])> (was not locked). 

Error:

NotLocked: Failed to release the lock <Store(url='file:///mnt/intenso8/borgbackup2', levels=[('archives/', [0]), ('config/', [0]), ('cache/', [0]), ('locks/', [0]), ('datSun23 [844/1284]
s/', [0])])> (was not locked).

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/usr/lib/borgbackup2/borg/archiver/_common.py", line 143, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/borgbackup2/borg/archiver/_common.py", line 211, in wrapper
    return method(self, args, **kwargs)
  File "/usr/lib/borgbackup2/borg/archiver/repo_create_cmd.py", line 33, in do_repo_create
    manifest.write()
    ~~~~~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/manifest.py", line 555, in write
    assert self.archives.count() <= MAX_ARCHIVES
           ~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/manifest.py", line 204, in count
    return len(list(self.ids()))
               ~~~~^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/manifest.py", line 108, in ids
    infos = list(self.repository.store_list("archives", deleted=deleted))
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/repository.py", line 547, in store_list
    self._lock_refresh()
    ~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/repository.py", line 233, in _lock_refresh
    self.lock.refresh()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/storelocking.py", line 263, in refresh
    raise LockTimeout(str(self.store))
borg.storelocking.LockTimeout: Failed to create/acquire the lock <Store(url='file:///mnt/intenso8/borgbackup2', levels=[('archives/', [0]), ('config/', [0]), ('cache/', [0]), ('locks/', [0]), ('data/', [2]), ('keys/', [0])])> (timeout). 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/borgbackup2/borg/archiver/__init__.py", line 647, in main
    exit_code = archiver.run(args)
  File "/usr/lib/borgbackup2/borg/archiver/__init__.py", line 541, in run
    rc = func(args)
  File "/usr/lib/borgbackup2/borg/archiver/_common.py", line 118, in wrapper
    with repository:
         ^^^^^^^^^^
  File "/usr/lib/borgbackup2/borg/repository.py", line 195, in __exit__
    self.close()
    ~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/repository.py", line 277, in close
    self.lock.release()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/borgbackup2/borg/storelocking.py", line 218, in release
    raise NotLocked(str(self.store))
borg.storelocking.NotLocked: Failed to release the lock <Store(url='file:///mnt/intenso8/borgbackup2', levels=[('archives/', [0]), ('config/', [0]), ('cache/', [0]), ('locks/', [0]), ('data/', [2]), ('keys/', [0])])> (was not locked).

Platform: Linux server 6.18.39+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.18.39-1+rpt1 (2026-07-29) aarch64
Linux: Unknown Linux  
Borg: 2.0.0b19  Python: CPython 3.13.5 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
PID: 244588  CWD: /root
sys.argv: ['/usr/bin/borg2', '--repo=/mnt/intenso8/borgbackup2', 'repo-create', '--encryption=repokey-aes-ocb', '--other-repo=/mnt/intenso8/borgbackup', '--from-borg1', '--debug']
SSH_ORIGINAL_COMMAND: None

I also compiled the latest beta (b21), but the result was the same.

[ThomasWaldmann]

borg uses utc time, no matter what local time is configured to be on the machines.

but the machines should roughly agree about their utc time. doesn't necessarily need to be ntp synced, but at least roughly.

the latest beta is b23.

[nicojx]

Mmh, let me try again. Apart from the time difference, they do agree, so not sure what caused this error. Any other ideas?

[ThomasWaldmann]

b23 has this: #10077

With that, clock skew should be less problematic.

If you try b23, read that: #10187

[ThomasWaldmann]

Closing this, because it was encountered when using quite an old beta.

If there are issues with b23+, please file a new ticket.