- formerly "macOS Sequoia + HomeBrew + cron = pain!"
Each year as Apple's World Wide Developer Conference (WWDC) approaches, I start to think about upgrading macOS to the version that was released at WWDC the previous year. I reckon that's a good way of minimising your pain and suffering, not to mention giving developers a chance to patch things that got broken when the API goal-posts moved.
The end of May 2025 was when I decided to upgrade both a 2019-era Intel iMac and an Apple M2 MacBook from Sonoma 14.7.1 to Sequoia 15.5.
Aside from the seven fractions of eternity that macOS updates seem to demand these days, it all seemed to go swimmingly. The only explicit grizzle was from Carbon Copy Cloner, which wanted a later version (on my to-do list).
I'm an old guy. I've used a bunch of source code utilities in my life, including CVS, Subversion and Git (plus tools most people have never heard of like UPDATE and sccs).
Right now I have a mixture of Subversion and Git. The Subversion stuff is in a mental bucket marked "should be migrated sometime."
Now, without boring anyone with all the gory details, I have a crontab
entry on each Mac which fires every couple of hours to keep some subversion trees in sync:
20 */2 * * * subversion_autosync
Basically, if I commit something on one machine, it'll be on the other machine within an hour or so without me needing to remember to do anything. Sort of like a poor man's Dropbox that doesn't involve the cloud.
This scheme has been working away, silently, for almost 20 years. I kid you not. The initial log entry on the subversion_autosync
script is 05 Jul 2005.
Long story short, I noticed some oddities on the iMac which led me to suspect that the synchronisation process had stopped working.
I'm a firm believer in logging. The only log that is useful is the one that's already captured the problem. Whenever I set up scripts that are intended to run in the background, I instrument them like this:
# set up logging if launched by cron or as a launch agent
if [ "$(tty)" = "not a tty" ] ; then
LOGFILE="$HOME/Library/Logs/$(basename "$0").log"
touch "$LOGFILE"
exec >> "$LOGFILE"
exec 2>> "$LOGFILE"
fi
That way, any logging messages will get preserved in ~/Library/Logs
.
The log was full of this:
svn: E170013: Unable to connect to a repository at URL 'svn://svnserver.your.domain.com/trunk/Documents/Synchronised'
svn: E000065: Can't connect to host 'svnserver.your.domain.com': No route to host
Meanwhile, over on the M2 laptop, the script was working as usual.
Triggering the command by hand worked as expected on both machines so it seemed to be the combination of:
- launched by
cron
; and - running on
x86_64
(Intel).
I wondered if this was Apple's way of telling me that I should've stopped using cron
in favour of a Launch Agent so I gave that a whirl:
{
"Debug" => 1
"Label" => "arpa.home.your.svn.autosync"
"ProgramArguments" => [
0 => "/Users/moi/.local/bin/subversion_autosync"
]
"StandardErrorPath" => "/Users/moi/Library/Logs/subversion_autosync.log"
"StandardOutPath" => "/Users/moi/Library/Logs/subversion_autosync.log"
"StartInterval" => 30
}
The start interval of 30 seconds was chosen so I could keep trying things in conjunction with a tail -f
watching the log.
Same problem. So, not cron
but a problem of background tasks more generally, but only on Intel.
I also have another (unrelated) crontab
entry:
*/5 * * * * publish_mac_temperature
This only runs on the Intel iMac because the utility it relies on to fetch the CPU temperature doesn't work on Apple Silicon. The publish_mac_temperature
script packages-up an MQTT message and transmits it using:
mosquitto_pub -h mosquitto.your.domain.com
That script was working.
So, we have the situation where:
-
cron
and/orlaunchd
is initiating two scripts:subversion_autosync
publish_mac_temperature
-
Both of those scripts invoke a utility which involves network access:
script utility domain name subversion_autosync
svn
svnserver.your.domain.com publish_mac_temperature
mosquitto_pub
mosquitto.your.domain.com -
In terms of the data communications:
- The domain names referenced by the utilities are local DNS CNAMEs which resolve to two different Raspberry Pis.
- The DNS server is the same host as
svnserver
. - All devices (Macs, Raspberry Pis) are on the same /24 subnet.
-
In terms of the utilities:
- both were installed using HomeBrew.
- One utility is failing while the other isn't.
It looks like svn
has resolved the alias to the IP address and that connectivity has failed after that. Because it's working, it's kinda self-evident that mosquitto_pub
has resolved its alias so that implies that, despite Jeff Geerling's T-shirts, it ain't a DNS issue. All other things being equal, whatever is stopping svn
from working should also be stopping mosquito_pub
. Why are they different?
Googling the topic finds things like:
-
Reddit:
-
GitHub:
-
Apple discussions:
In the past, not granting background-process tools full disk access has caused all manner of grief so, while I can't see how that's related, I make sure none of that got lost on the Sequoia upgrade. There's also advice about turning off the Firewall. I've never turned it on but I double-check anyway.
While I'm there, I wade through System Preferences » Privacy and Security in case anything else jumps out. That's when I notice a brand new category: "Local Network".
This was hinted at in DBeaver, 765513 and 778457 but the penny didn't really drop.
On the Intel Mac, "Local Network" has the Arduino IDE. On the M2 laptop, it's got the Arduino IDE and svn
.
W.T.A.F.?
778457 also mentions code-signing.
The vital clue!
Compare/contrast:
-
Intel Mac:
$ codesign -d -v /usr/local/Cellar/subversion/1.14.5_2/bin/svn /usr/local/Cellar/subversion/1.14.5_2/bin/svn: code object is not signed at all
-
M2 Laptop:
$ codesign -d -v /opt/homebrew/Cellar/subversion/1.14.5_2/bin/svn 2>&1 | grep -e "^Signature" Signature=adhoc
why
codesign
writes its output tostderr
is a mystery.
Back to the Intel Mac where the Launch Agent is still triggering the script every 30 seconds and the watch on the log is showing constant failure:
$ codesign --sign «MyIdentityHere» /usr/local/Cellar/subversion/1.14.5_2/bin/svn
The next time the Launch Agent fires, I get a prompt to allow svn
which takes up residence in "Local Network", so now the Intel and M2 Macs are the same.
More importantly, the subversion_autosync
script starts working again.
Cookin' with photons!
A slightly deeper dive into this code-signing malarky.
First, the Intel Mac:
$ cd /usr/local/Cellar/subversion/1.14.5_2/bin
$ ls -1 | xargs file
svn: Mach-O 64-bit executable x86_64
svnadmin: Mach-O 64-bit executable x86_64
svnbench: Mach-O 64-bit executable x86_64
svndumpfilter: Mach-O 64-bit executable x86_64
svnfsfs: Mach-O 64-bit executable x86_64
svnlook: Mach-O 64-bit executable x86_64
svnmucc: Mach-O 64-bit executable x86_64
svnrdump: Mach-O 64-bit executable x86_64
svnserve: Mach-O 64-bit executable x86_64
svnsync: Mach-O 64-bit executable x86_64
svnversion: Mach-O 64-bit executable x86_64
$ ls -1 | xargs -I % codesign -d -vv %
svn: code object is not signed at all
svnadmin: code object is not signed at all
svnbench: code object is not signed at all
svndumpfilter: code object is not signed at all
svnfsfs: code object is not signed at all
svnlook: code object is not signed at all
svnmucc: code object is not signed at all
svnrdump: code object is not signed at all
svnserve: code object is not signed at all
svnsync: code object is not signed at all
svnversion: code object is not signed at all
$ cd /usr/local/Cellar/mosquitto/2.0.21/bin
$ ls -1 | xargs file
mosquitto_ctrl: Mach-O 64-bit executable x86_64
mosquitto_passwd: Mach-O 64-bit executable x86_64
mosquitto_pub: Mach-O 64-bit executable x86_64
mosquitto_rr: Mach-O 64-bit executable x86_64
mosquitto_sub: Mach-O 64-bit executable x86_64
$ ls -1 | xargs -I % codesign -d -vv %
mosquitto_ctrl: code object is not signed at all
mosquitto_passwd: code object is not signed at all
mosquitto_pub: code object is not signed at all
mosquitto_rr: code object is not signed at all
mosquitto_sub: code object is not signed at all
Interpretation:
- All the Subversion and Mosquitto binaries are pure
x86_64
(not "universal"). - None is code-signed.
- The lack of code-signing for
mosquitto_pub
means this doesn't explain why thepublish_mac_temperature
script kept working when thesubversion_autosync
script went nuts.
A slightly wider search of the HomeBrew Cellar is yet to discover any x86_64
binary that is code-signed. I'm not saying HomeBrew packages deployed on Intel Macs are never signed. I just couldn't find any examples.
Now let's take a look at the M2 laptop (I'll cut down the lines a bit but you can take it as read there are no exceptions buried in the ...
):
$ cd /opt/homebrew/Cellar/subversion/1.14.5_2/bin
$ ls -1 | xargs file
svn: Mach-O 64-bit executable arm64
svnadmin: Mach-O 64-bit executable arm64
...
$ ls -1 | xargs -I % codesign -d -vv % 2>&1 | grep -e "^Signature" -e "^Executable"
Executable=/opt/homebrew/Cellar/subversion/1.14.5_2/bin/svn
Signature=adhoc
Executable=/opt/homebrew/Cellar/subversion/1.14.5_2/bin/svnadmin
Signature=adhoc
...
$ cd /opt/homebrew/Cellar/mosquitto/2.0.21/bin
$ ls -1 | xargs file
mosquitto_pub: Mach-O 64-bit executable arm64
mosquitto_sub: Mach-O 64-bit executable arm64
...
$ ls -1 | xargs -I % codesign -d -vv % 2>&1 | grep -e "^Signature" -e "^Executable"
Executable=/opt/homebrew/Cellar/mosquitto/2.0.21/bin/mosquitto_pub
Signature=adhoc
Executable=/opt/homebrew/Cellar/mosquitto/2.0.21/bin/mosquitto_sub
Signature=adhoc
...
Interpretation:
- All the Subversion and Mosquitto binaries are pure
arm64
(not "universal"). - All are code-signed.
This time a slightly wider search of the Cellar didn't chuck up any examples of arm64
binaries that were not code-signed.
All of which leads to the key question:
Why does HomeBrew appear to be signing
arm64
binaries but not signingx86_64
binaries?
Our journey begins with the HomeBrew v4.3.0 release notes 14 May 2024 which includes:
Homebrew/homebrew-cask requires code signing of all casks. Expect removal of casks that are not code signed from Homebrew/homebrew-cask in future. This is because code signing is required on Apple Silicon which is used by a growing majority of all Homebrew users.
Subversion isn't a cask (at least not as I understand it) so this begs the question of when code-signing arm64
binaries expanded to non-casks. I couldn't find anything to explain that history so we'll put that to one side.
The above quote includes a link to HomeBrew Pull Request 17002 which might have been the end of the journey but for Pull Request 17009 which seemed to revert it.
In short, stuffed if I know!
Why does svn
have this problem when mosquitto_pub
does not?
My guess is TN3151 Choosing the right networking API might have something to do with it.
If you've reached this point in the discussion, well done you! No doubt you're asking how to sign any binaries of your own because the «MyIdentityHere»
token used above doesn't actually help anyone.
The simplest is to use what is called a "ad-hoc signing":
$ codesign --sign "-" path/to/binary
You can also use other code-signing certificates, if you have them. To find out, run:
$ security find-identity -v -p codesigning
If you have any identities, they are displayed as a list in a «sequence») «hexID» "«name»"
format, like this:
1) 0123456789ABCDEF0123456789ABCDEF01234567 "certificate 1 name"
2) 76543210FEDCBA9876543210FEDCBA9876543210 "certificate 2 name"
...
You can use any of those fields to indicate which identity to use. In the following example, all three commands refer to the same certificate:
$ codesign --sign "1" path/to/binary
$ codesign --sign "0123456789ABCDEF0123456789ABCDEF01234567" path/to/binary
$ codesign --sign "certificate 1 name" path/to/binary
The issue with subversion is really one instance of a class of problem which can loosely be grouped under "No route to host", which is why I've changed the title of this gist.
One of the "hits" I found when searching was DBeaver issue 35705. I didn't have DBeaver installed but, having solved the subversion problem, I went back and installed DBeaver on the Intel iMac.
I told it to connect to a local MariaDB instance and, bang, hit the "No route to host" problem. DBeaver turned up in:
System Settings » Privacy & Security » Local Network
All I had to do was turn on the blue switch, retry the connection request, and the problem went away.
At that point, my "Local Network" list contained:
- Microsoft Excel
- Syncthing
- Arduino IDE
- svn (subversion)
- DBeaver
That list was interesting. I had codesigned svn
myself but all the others were already signed by their developers. It is also true to say that svn
was the only standalone binary; all the others were true "apps" with a bundle.
Using this formula:
$ APP="DBeaver.app"
$ codesign -d --entitlements - --xml "/Applications/$APP" | plutil -p -
I was able to determine that Microsoft Excel, Arduino IDE and DBeaver had entitlements lists, while Syncthing did not. Of the three apps with entitlements, only Excel had any specific network-related entitlement:
com.apple.security.network.client
If I hypothesise that the lack of this entitlement explains why Arduino IDE, DBeaver and Syncthing appear in the "Local Network" list and need explicit user authorisation (ie turning on the blue switch), then Excel represents confounding evidence.
Meanwhile, over on an M2-chip MacBook Pro, Excel did not appear in the "Local Network" list. Also of note is that neither machine had any other Microsoft applications in the list.
I wondered if Excel being listed on the Intel iMac might have been the result of my usage patterns? I'm a heavy Excel user (multiple times a day) whereas Word is more like once a month, PowerPoint once a year, if I'm unlucky. Other Office apps are not even installed. Outlook? Not in a pink fit! It would not, therefore, be beyond the bounds of possibility for there to have been a timeline like this:
- At some point, Excel did not have any entitlements;
- I used Excel which caused it to appear in "Local Network";
- All Office apps were subsequently updated to include entitlements; so
- Excel's continued presence in "Local Network" is redundant.
How to test that theory? I started to wonder how to remove entries from the "Local Network" list. There's no — button and right-clicking on entries doesn't produce a contextual menu with "Delete". Googling led me to 758008 which talks about disabling SIP, blowing away a couple of plists, then re-enabling SIP again.
As someone somewhat mixing their metaphors commented: that's a sledgehammer approach to a problem which requires a tweezers. It is not something you'd want to be doing on a daily basis and I certainly don't recommend that course of action. Screwing with that low-level stuff, especially on Apple Silicon Macs is a good way to have a DFU in your future. You have been warned!
But I found a solution:
-
Make sure the target app is not running.
-
Launch "System Preferences" and navigate to "Privacy & Security" then "Local Network".
-
In "Finder":
- Go to the
/Applications
folder. - If you don't actually want to remove the target app completely, right-click on the target app and choose
Compress
to make a.zip
backup. - Delete the target app and empty the trash. This hides the app from macOS.
- Go to the
-
In "System Preferences":
- The target app will still be in the "Local Network" list with its switch turned on.
- Turn the switch off. The target app will disappear from the list.
- Quit "System Preferences". This step is important. Please don't skip it.
-
If you did not make a
.zip
backup in step 3, your work is done and you can stop here. -
If you did make a
.zip
backup, then in "Finder":- Go to the
/Applications
folder - Double-click the
.zip
to unpack the target app. - Launch the target app.
- Take some action which causes local network activity and triggers the "No route to host" error. Dismiss the error dialog.
- Go to the
-
Launch "System Preferences" and navigate to "Privacy & Security" then "Local Network":
- The target app should be back in the list with its switch turned off.
- Turn the switch on.
-
Go back to the target app and retry the action which causes local network activity. It should succeed.
Having tried all that with Excel, it popped back into the list.
Questions:
- What local network activity is Excel causing? 🤷
- Why doesn't the
com.apple.security.network.client
entitlement deal with it? 🤷 - Why on the Intel iMac but not the M2 MacBook? 🤷
- Is Excel perhaps doing something that would warrant the
com.apple.security.network.server
entitlement? Runninglsof
doesn't reveal Excel to be listening on any ports so, presumably not. Unless it's a transient listener ... 🤷 - What happens if I turn off the switch for Excel? Absolutely no change in behaviour that I can discern. See "What local network activity is Excel causing?" above. 🤷
- Why the heck does Apple even care when two devices on the same subnet wish to communicate? 🤷🤷
To paraphrase Winston Churchill:
Gatekeeper is a riddle wrapped in a mystery inside an enigma,
to which I'll add:
and an example of galloping scope-creep, coated in some slimy corporate interference goo, apparently designed to be the very antithesis of Apple's "it just works".