In my opinion, any program that is supposed to copy files, but which is not able to make perfect copies, i.e. copies that do not lose any bit of data or metadata that was present in the original file, is just unusable garbage.
Unfortunately, most copying programs available in UNIX-like operating systems (and also many archiving programs) do not make perfect file copies with their default options and many of them are never able to make perfect copies, regardless what options are used.
I have not looked recently at the scp command of ssh, but at least until a few years ago it was not possible to make perfect file copies with scp, especially when the copies were done between different operating systems and file systems. That is why I never use scp, but only rsync over ssh.
Rsync is the only program that I have seen, which is able (with the right options) to make perfect file copies even between different operating systems and file systems (for instance between FreeBSD with UFS and Linux with XFS), preserving also metadata like extended file attributes, access control lists and high-precision file timestamps (some copying programs and archiving programs truncate high-precision timestamps).
The current documentation of openrsync does not make any guarantee that it can make complete file copies, so by default I assume that it cannot, so for now it is a program that I consider useless.
Beside rsync for copying, one of the few Linux archiving programs that can archive perfect file copies is bsdtar (when using the pax file format; the ancient tar and cpio file formats cannot store all modern file metadata).
(FYI: I always alias rsync to '/usr/bin/rsync --archive --xattrs --acls --hard-links --progress --rsh="ssh -p XXX -l YYYYYYY"')
(With the right CLI options, "cp" from coreutils can make perfect file copies, but only if it has been compiled with appropriate options; some Linux distributions compile coreutils with wrong options, e.g. without extended file attributes support, in which case "cp" makes only partial file copies, without giving any warnings or errors.)
(We restore everything super carefully but sometimes I feel like we're the only ones who care)
Also, a lot of application-specific data formats already don't care about the "extra" attributes available in various filesystems because those aren't universally supported and implement them themselves in the file format they operate on. For example, DICOMs, or password-protected PDFs or Zip archives etc.
We would only care if the software our customers were running did. Big enterprise software suits were defined to run in hostile environments, in such they mostly rely on their data formats and don’t care about attributes from the filesystem other than so they have access.
Except sometimes it's a 3rd party's app whose data you have to restore, and you don't have control over their engineers.
$ /usr/bin/rsync –version
reminds me of that yogi berra quote "nobody goes there anymore, it's too crowded"
For example, many people don't even understand target disk mode on apple hardware, but it has saved me countless hours over the years and made administering apple systems a breeze. Ask people who've used target display mode if they can imagine going without it.
on another subject - it's worth mentioning that time machine is based on rsync.
Is it possible this is just a documentation style-tone mismatch? My default assumption would be that openrsync is simply a less restrictively licensed rsync, and I wouldn’t assume it works any differently. Have you verified your strong hypothesis? Or are you just expressing skepticism. It’s hard to tell exactly.
Edit: I read the openrsync readme. It says it’s compatible with rsync and points the reader to rsync’s docs. Unless extended file attributes, ACLs, and high resolution timestamps are optional at the protocol level, it must support everything modern rsync supports to be considered compatible, right? Or are you suggesting it lies and accepts the full protocol but just e.g. drops ACLs on the floor?
> The openrsync command line tool is compatible with rsync, but as noted in the documentation openrsync accepts only a subset of rsync’s command line arguments.
I'm still using the multi volume support in tar for example. Which was something that stems from the time when tar was used for tape archives (hence the name tar) on actual tapes. Without that I'd be really screwed because I use a box full of harddrives as backup "tapes" (which works surprisingly well I must say, I needed a small restore only a week or two ago and it really saved my bacon). But I bet 99.9% of tar users have no idea it can even do that.
Rsync is another one of those swiss army knives that people use for a lot more stuff than you might expect. Especially the remote capabilities are amazing.
The problem is that when you clone something but don't provide full compatibility, you're putting your users through much headscratching and frustration. It would be better to not name it after the original then so it's clear it's something different.
The software packages I really value the most are the ones where a situation causes something really weird I need to do, and I read in the documentation to try to find some workaround, and then I discover that it already has exactly that feature that I need hidden in there somewhere. It's like the developer read my mind :) There's been very few packages that I truly cherished (and very few in this day and age, software in the early PC days was often more powerful IMO).
One of them was SP (SK Packet Radio), where this happened several times. That was truly amazing software, there was so much it could do and it all worked on an 8088 together with a TSR-based softmodem (connected to a radio not a phone line). Wow. Even the insanest stuff that popped into my head I could make happen with just some settings.
It's a world of difference to Gnome devs that will just shut down everything.
PS I do think dolphin is the weakest link in the KDE experience though. But they have made some really good improvements with KDE 6. Like the typable crumb trail.
It's a bit of a contrast though with macOS where the finder is one of the least opinionated parts of the OS (and thus for me one of the best). I think that's more of a historical thing though, Apple's vision seems to be more centered around moving file management into the domain of different apps like on iOS. Another thing i don't like but I think ios has loosened that somewhat as they had to contend that it was necessary to make the iPad more of a productivity device (it still really isn't one though)
> We are still working on it... so please wait.
openrsync is very likely good code, but that doesn't mean it replicates every feature of the rsync utility.
I consider this a very serious problem, because most naive users will assume automatically that when they give a file copy command they obtain a perfect duplicate of the original file.
It is surprising for them to discover that this is frequently not true.
Among permissions you use in your alias: --xattrs, --acls, --hard-links are all missing.
[0]https://github.com/kristapsdz/openrsync/blob/a257c0f495af2b5...
Valid or not in anyone else's opinion, it doesn't really matter, the risk that someone will attempt to use a court to enforce one of these tends to mean companies don't want to even go near it.
Working in a Bank we won't touch anything GPL3, even to build our software/services or mobile app, because we don't want to even open that Pandora's box.
We don't have to find out if a court would try to force us to release our signing keys if we don't use or ship any code that contains language that could in some ways be phrased to do that.
We had very keen developer upgrade all the libraries in our codebase as a "reducing technical debt" task that they decided to undertake themselves.
They couldn't get something working and posted a stack-trace to ask for help..... Some enterprising sales person in iText saw it and emailed them offering to help and asked a question about what they were running and the developer effectively told them they were running version 5 which they didn't even check (or possibly understand) is relicensed under AGPL or commercial license.
The legal threats from iText and the resulting fallout means we now do not allow developers access to the internet from their machines, even via a proxy, they have a separate RDP machine for that.
And they can only pull in libraries that are scanned via jFrog xRay and ensure the license of said library is "acceptable".
On the plus side, means we're doing something about supply-chain vulnerabilities.
By default, a file copy should include everything that is contained in the original file. Sometimes the destination file system cannot store all the original metadata, but in such cases a file copying utility must give a warning that some file metadata has been lost, e.g. like when copying to a FAT file system or to a tmpfs file system as implemented by older Linux kernels. (Many file copy or archiving utilities fail to warn the user when metadata cannot be preserved.)
Some times you may no longer need some of the file metadata, but the user should be the one who chooses to loose some information, it should not be the default behavior, especially when this unexpected behavior is not advertised anywhere in the documentation.
The origin of the problem is that the old UNIX file systems did not support many kinds of modern file metadata, i.e. they did not have access control lists or extended file attributes and the file timestamps had a very low resolution.
When the file systems were modernized (XFS was the first Linux file system supporting such features, then slowly also the other file systems were modernized), most UNIX utilities have not been updated until many years later, and even then the additional features remained disabled by default.
Copying like rsync, between different computers, creates additional problems, because even if e.g. both Windows and Linux have extended file attributes, access control lists and high-resolution file timestamps, the APIs used for accessing file metadata differ between operating systems, so a utility like rsync must contain code able to handle all such APIs, otherwise it will not be able to preserve all file metadata.
Take the file owner for example if I take a copy of a file then by default I should be the owner of that file as it's my copy of the file, and not the original file owner's copy.
An alternative way of looking at it is if I have created a file on my local machine that's owned by root and has the setuid bit set on it's file permissions then there's no way that I should be able to copy that file up to a server with my normal user account and have those atttibutes still set on the copy.
Yes. Sometimes you need that additional information too. And if you do, then rsync is your tool. If you only need the data stored in the file, then drag & drop suffices.
When a copy a file you will be the owner because the new copy is your copy. Other attributes however like modification date for example will remain the same. It's not as if you wrote the contents of the file anew, especially not for copy-on-write architectures like Apple's APFS.
My expectations are more specific than "not all of them", so please don't misrepresent them.
Now say Alice and Bob have the same copy of file F, Bob modifies it first which gets stored at timestamp T, then Alice modifies her copy at time T+1.
Bob syncs his files on a filer, its timestamp gets reset to now, which is say T+2. Then Alice does the same, but her file does not get copied, since the remote timestamp T+2 is newer than her local timestamp T+1.
I honestly don't have a strong preference either way on this. I don't use date created except for misbehaving media downloaders that think the file modified date is a good place to put the video publication date. I'm sure there's a flag somewhere that I don't care enough to find.
When installing these things from archives or moving/distributing relevant files to large fleets, I expect the file contents and all metadata incl. datestamps to be carried the way I want, because all of that data is useful for me and the application which uses the file.
If the user doing the copying has no right to copy the file exactly, I either expect a loud warning or an error depending on the situation.
For example, copying a file from a user's home directory (perhaps user_home_t) into /var/www/html/ usually requires it to get the httpd_sys_content_t context (or similar) to be served by the webserver correctly and securely. Blindly copying the original user_home_t context would likely prevent the webserver from accessing the file.
Doesn't this suggest that some metadata, specifically the SELinux context, often shouldn't be copied verbatim from the source but rather be determined by the destination and the system's security policy?
SELinux contexts shall be sticky, and needs to be manually (re)set after copying.
This is the default behavior, BTW. SELinux contexts are not (re)set during copy operations in most cases, from my experience. You need to change/fix the context manually.
When foo copy a file from user bar, and put it on his homedir, the last thing h want is for it to be owned by the foo user.
Your expectations are irrealistics.
Yes, and that's OK.
> When foo copy a file from user bar, and put it on his homedir, the last thing h want is for it to be owned by the foo user.
It depends.
> Your expectations are irrealistics (sic).
No, rsync can do this (try -avSHAX) and tar does this by default, and we're talking about rsync here.
That is exactly what I am saying, rsync do not do this by default either, you have to tell him to via optionnal parameters.
Strip all the execute attributes out of your *nix system and see what happens.
Rsync only cp the metadata when you specifically ask it to anyway. I haven't had a look at openrsync man page but I would assume it is the same in the case of the later.
Openrsync lacks the options of rsync for making exact copies.
Moreover, the OpenBSD file systems are unable to store all metadata that can accompany files in Linux filesystems or Windows filesystems, so that is the likely reason for removing the rsync options.
I also doubt that the developers of an utility for OpenBSD are also interested in taking care to preserve file metadata when copying to/from Windows, because the metadata access API is not portable, so a complete "rsync" utility must include specific code paths at least for Windows, for Linux and for FreeBSD. I do not know if the API of MacOS is also specific to it, or it is compatible with anything else.
I expect file attributes of the target to be what I say they should be, not copied over from wherever the content happened to live before.
I have switched to bsdtar many years ago precisely because I have discovered that at least at that time it was the only Linux utility that could make exact archives for the backup of my filesystems.
I make extensive use of extended file attributes. For instance any file on my filesystems has a hash stored in an extended attribute for detecting errors/modifications in its content and for deduplication (the hash is updated whenever the file is modified intentionally).
When I make backups, I always store at least two copies on different media and it is essential that the file hashes in extended file attributes are preserved by the archiving/backup program, so that I will be able to detect corrupted files if I try to restore them. If some file is corrupted, I can retrieve it from the other backup copy. This has saved me several times with archives stored for many years, because neither HDDs nor any other available archival media are currently reliable enough to trust them for long-term storage without errors.
Like I have said, for modern backups one must use the pax file format. GNU tar, and also other "tar" programs have made some custom non-standard extensions to the standard "tar" file format, in order to be able to store some things not allowed in standard tar files, but those workarounds are inferior to what can be done in the more recent "pax" file format.
I personally dislike the GPL because I think my modifications should belong to me. I spent the effort on them, and I don't think any license which requires me to forfeit my effort is worth spending time with. Corporations agree and this is why the code licensed with GPLv2 is sparingly accepted and code licensed with GPLv3 is outright rejected by most large companies.
For example, consider the case of a software distributed under a permissive license like MIT or BSD. If you modify and re-distribute it, anyone else can further modify the software that you patched and improved but they are not obliged to release the new source code to you. In such a scenario, you are willing "forfeiting" your effort. With GPL, at least your right to access future source codes (of publicly distributed GPL software) cannot be denied to not just you, but any user of the software.
A classic example, at least from my experience, is `unzip`. On two of my servers (one running Debian and the other an older Ubuntu), neither of their bundled `unzip` versions can handle AES-256 encrypted ZIP files. But apparently, according to some Stack Overflow posts, some distributions have updated theirs to support it.
So here is what I ran into:
1. I couldn't easily find an "updated" version of `unzip`, even though I assume it exists and is open source.
2. To make things more confusing, they all claim to be "version 6.00", even though they obviously behave differently.
3. Even if I did find the right version, I'm not sure if replacing the system-bundled one is safe or a good idea.
So the end result is that some developer out there (probably volunteering their time) added a great feature to a widely used utility, and yet I still can’t use it. So in a sense, being a core system utility makes `unzip` harder to update than if it were just a third-party tool.
I get that it's probably just as bad if not worse on Windows or macOS when it comes to system utilities. But I honestly expected Linux to handle this kind of thing better.
(Please feel free to correct me if I’ve misunderstood anything or if there’s a better way to approach this.)
Related to that, the Unix philosophy of simple tools that do one job and do it well, also applies here a bit. More typical workflow would be a utility to tarball something, then another utility to gzip it, then finally another to encrypt it. Leading to file extensions like .tar.gz.pgp, all from piping commands together.
As for versioning, I'm not entirely sure why your Debian and Ubuntu installs both claim version 6.00, but that's not typical. If this is for a personal machine, I might recommend switching to a rolling release distro like Arch or Manjaro, which at least give upto date packages on a consistent basis, tracking the upstream version. However, this does come with it's own set of maintenance issues and increased expectation of managing it all yourself.
My usual bugbear complaint about Linux (or rather OSS) versioning is that people are far too reluctant to declare v1.00 of their library. Leading to major useful libraries and programs being embedded in the ecosystem, but only reaching something like v0.2 or v0.68 and staying that way for years on end, which can be confusing for people just starting out in the Linux world. They are usually very stable and almost feature complete, but because they aren't finished to perfection according to the original design, people hold off on that final v1 declaration.
Arithmetic is built into POSIX shell, and it's universally implemented. The following works in basically every shell, and starts 0 new processes, not 2:
$ bash -c '[ $((1 + 1)) = 2 ]; echo $?'
0
$ zsh -c '[ $((1 + 1)) = 2 ]; echo $?'
0
$ busybox ash -c '[ $((1 + 1)) = 2 ]; echo $?'
0
$ ysh -c 'if (1 + 1 === 2) { echo hi }'
hi
$ echo '{"foo": 42}' > test.json
$ ysh
ysh-0.28$ json read < test.json
ysh-0.28$ echo "next = $[_reply.foo + 1]"
next = 43
$ echo "q = $[_reply.foo / 5]"
q = 8.4
Care to explain to me how all the problems of X-Windows have been solved so it's no longer valid to criticize the fallout from its legacy vestigial historic baggage we still suffer from even today? How many decades ago did they first promise the Year of the Linux Desktop?
The X-Windows Disaster: This is Chapter 7 of the UNIX-HATERS Handbook. The X-Windows Disaster chapter was written by Don Hopkins.
https://medium.com/@donhopkins/the-x-windows-disaster-128d39...
Why it took THREE processes and a shitload of context switches and punctuation that we are still stuck with to simply test if 1 + 1 = 2 in classic Unix [TM]:
[ "$(expr 1 + 1)" -eq 2 ]
expr 1 + 1
$(...) (Command substitution)
[ ... ]
It took THREE separate processes because:
Unix lacked built-in arithmetic.
The shell couldn't do math.
Even conditionals ([) were external.
Everything was glued together with fragile text and subprocesses.
All of this just to evaluate a single arithmetic expression by ping-ponging in and out of user and kernel space so many times -- despite the CPU being able to do it in a single cycle.
That’s exactly the kind of historical inefficiency the "Unix Philosophy" retroactively romanticizes.
This gave me a big laugh, I love the UNIX-haters Handbook despite loving UNIXy systems. Thank you for decades of enjoyment and learning, especially in my late-90s impressionable youth.
It's fine to hate Unix, but you should update your examples :)
And even that, only bash, because I wouldn't consider the state of X Windows and Wayland fixed in any way, shape, or form.
For all the weirdos smashing that downvote button: How about you name me some UNIX distros you have ran in the past year? Other than Linux, OpenBSD (~0.1% market share btw) and ostensibly MacOS (which we all know has dropped any pretense of caring to be UNIX-like many years ago), that is.
https://www.youtube.com/watch?v=JOeY07qKU9c
macOS is absolutely Unix, and a lot more like mainstream Unix than many of the other vastly different Unix systems of the past and present, so exactly when did the definition of Unix suddenly tighten up so much that it somehow excludes macOS? And how does your arbitrary gatekeeping and delusional denial of the ubiquity and popularity of macOS, and ignorance of the Unix 03 certification, the embedded, real time, and automotive space, and many other Unix operating systems you've never heard of or used, suddenly change the actual definition of Unix that the rest of the world uses?
Have you ever even attended or presented at a Usenix conference? Or worked for a company like UniPress who ports cross platform software to many extremely different Unix systems? Maybe then you'd be more qualified to singlehandedly change the definition of the word, and erase Unix 03 certification from existence, and shut down all the computers and devices running it, but you're not. Who do you think you are, one of Musk's DOGE script kiddies? Because you sound as overconfident and factually incorrect as one.
https://en.wikipedia.org/wiki/No_true_Scotsman
>The "no true Scotsman" fallacy is committed when the arguer satisfies the following conditions:
>1) not publicly retreating from the initial, falsified a posteriori assertion: CHECK
>2) offering a modified assertion that definitionally excludes a targeted unwanted counterexample: DOUBLE CHECK
>3) using rhetoric to signal the modification: TRIPLE CHECK
macOS, AIX, HP-UX, Solaris (still technically certified), Inspur K-UX, EulerOS, etc.
POSIX-compliant and Unix-alike OSes (e.g., FreeBSD, QNX, etc.) are very active in many common domains (networking, firewalls, embedded, industrial).
Mission-critical infrastructure, telco, financial systems, military/spacecraft, automotive, and embedded still widely use non-Linux Unix or Unix-like systems.
QNX in cars, AIX in banks, Illumos in storage, RTEMS in space systems.
Orbit OS on the PlayStation.
On embedded space, with various levels of POSIX compliancy without being Linux distributions, QNX, vxWorks, INTEGRITY, NuttX, FreeRTOS, Zephyr, RTEMS.
> $ bash -c '[ $((1 + 1)) = 2 ]; echo $?'
Not even Perl uses that much arcane punctuation to test if 1 + 1 = 2. As if [] isn't enough, you've got to throw in two more levels of (()), plus enough grawlix profanity for a Popeye comic strip. And people complain Lisp has too many parens. Sheez.
https://en.wikipedia.org/wiki/Grawlix
There's even an emoji for Unix shell syntax:
https://gunkies.org/wiki/ITS_DDT_Guide
..PROMPT
holds the instruction which DDT uses to type out a "*".
You can replace it with any other instruction.
To use "%" instead of "*", deposit $1#%$> in that location
($> to avoid clobbering the opcode without having
to know what it is)
https://en.wikipedia.org/wiki/Incompatible_Timesharing_Syste...
https://en.wikipedia.org/wiki/Dynamic_debugging_technique
https://web.archive.org/web/20061011004003/http://www.sigfs....
https://its.victor.se/wiki/luser
https://www.hactrn.net/sra/alice/alices.pdp10
https://gunkies.org/wiki/Incompatible_Timesharing_System
(I recognize that photo I took of MIT-MC at the MIT-AI Lab!)
https://gunkies.org/wiki/File:Mit-mc.jpg
Even the Apple ][ Integer Basic monitor ROMs had a build-in mini assembler and disassembler.
I mean it makes write-only languages like Perl look like beautiful prose but it’s hard to argue about efficiently setting the 20 environment variables used by my terraform jobs with a mere 20 clock cycles. It may seem silly but every clock cycle truly matters.
One day, one remote day, mankind will attain Reason and will award you a Nobel Prize for speaking out loud against retrograde computing.
I salute you, hero, veteran.
ESR's claptrap book The Art of Unix Programming turns Unix into philosophy-as-dogma, where flaws are reframed as virtues. His book romanticizes history and ignores inconvenient truths. He's a self-appointed and self-aggrandizing PR spokesperson, not a designer, and definitely not a hacker, and he overstates and over-idealizes the Unix way, as well as and his own skills and contributions. Plus he's an insufferable unrepentant racist bigot.
Don't let historical accident become sacred design. Don’t confuse an ancient workaround with elegant philosophy. We can, and should, do better.
Philosophies need scrutiny, not reverence.
Tools should evolve, not stagnate.
And sometimes, yelling at clouds stirs the winds of change.
https://en.wikipedia.org/wiki/Unix_philosophy#Criticism
>In a 1981 article entitled "The truth about Unix: The user interface is horrid" published in Datamation, Don Norman criticized the design philosophy of Unix for its lack of concern for the user interface. Writing from his background in cognitive science and from the perspective of the then-current philosophy of cognitive engineering, he focused on how end-users comprehend and form a personal cognitive model of systems—or, in the case of Unix, fail to understand, with the result that disastrous mistakes (such as losing an hour's worth of work) are all too easy.
Donald A. Norman: The truth about Unix: The user interface is horrid:
http://www.ceri.memphis.edu/people/smalley/ESCI7205_misc_fil...
>In the podcast On the Metal, game developer Jonathan Blow criticised UNIX philosophy as being outdated. He argued that tying together modular tools results in very inefficient programs. He says that UNIX philosophy suffers from similar problems to microservices: without overall supervision, big architectures end up ineffective and inefficient.
On the Metal: Jonathan Blow:
https://archive.org/details/on-the-metal-jonathan-blow
>Well, the Unix philosophy for example it has been inherited by Windows to some degree even though it's a different operating system, right? The Unix philosophy of you have all these small programs that you put together in two like Waves, I think is wrong. It's wrong for today and it was also picked up by Plan Nine as well and so -
>It's micro services, micro services are an expression of Unix philosophy, so the Unix philosophy, I've got a complicated relationship with Unix philosophy. Jess, I imagine you do too, where it's like, I love it, I love a pipeline, I love it when I want to do something that is ad hoc, that is not designed to be permanent because it allows me- and you were getting inside this earlier about Rust for video games and why maybe it's not a fit in terms of that ability to prototype quickly, Unix philosophy great for ad hoc prototyping.
>[...] All this Unix stuff, it's the sort of the same thing, except instead of libraries or crates, you just have programs, and then you have like your other program that calls out to the other programs and pipes them around, which is, as far from strongly typed as you can get. It’s like your data coming in a stream on a pipe. Other things about Unix that seemed cool, well, in the last point there is just to say- we've got two levels of redundancy that are doing the same thing. Why? Get rid of that. Do that do the one that works and then if you want a looser version of that, maybe you can have a version of a language that just doesn't type check and use that for your crappy spell. There it is.
>[...] It went too far. That's levels of redundancy that where one of the levels is not very sound, but adds a great deal of complexity. Maybe we should put those together. Another thing about Unix that like- this is maybe getting more picky but one of the cool philosophical things was like, file descriptors, hey, this thing could be a file on disk or I could be talking over the network, isn't it so totally badass, that those are both the same thing? In a nerd kind of way, like, sure, that's great but actually, when I'm writing software, I need to know whether I'm talking over the network or to a file. I'm going to do very different things in both of those cases. I would actually like them to be different things, because I want to know what things that I could do to one that I'm not allowed to do to another, and so forth.
>Yes, and I am of such mixed mind. Because it's like, it is a powerful abstraction when it works and when it breaks, it breaks badly.
Shame that it is dogmatically followed by a very loud but vocal minority.
> "The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." George Bernard Shaw.
Man, I'm with you, but I'll put my efforts elsewhere :)
> He inspired Richard Stallman, who described him as a "very imaginative fellow", to use the term copyleft. ... He ported the SimCity computer game to several versions of Unix and developed a multi player version of SimCity for X11, did much of the core programming of The Sims, ... He is also known for having written a chapter "The X-Windows Disaster" on X Window System in the book The UNIX-HATERS Handbook.
I hope this experience helps you realize that jumping immediately to contempt can easily backfire.
> TWO heavyweight processes
> `eval` being heavy
> inexplicable flurry of punctuation
In this case:
- `[` and `]` delimit a test expression
- `"..."` ensure that the result of an expression is always treated as a single-token string rather than splitting a token into multiple based on spaces, which is the default behaviour (and an artifact of sh and bash's basic type system)
- `$(...)` denotes that the expression between the parens gets run in a subshell
- `-eq` is used for numerical comparison since POSIX shells default to string comparison using the normal `=` equals sign (which is, again, a limitation of the type system and a practical compromise)
> even though the processor has single cycle instructions to add two numbers and test for equality
If I wanted to test if one plus one equals two at a multi-terabit-per-second bandwidth I'd write a C program for it that forces AVX512 use via inline assembly, but at that point I think I'd have lost the plot a bit.
> they usually don't have to be fast since they're usually just used to delegate operations off to other code for performance
Even now you're bending over backwards to make ridiculous rationalizations for the bankrupt "Unix Philosophy". And you're just making my point for me. Does the Unix Philosophy say that the shell should be designed to be slow and inefficient and syntactically byzantine on purpose, or are you just making excuses? Maybe you don't think YOUR shell scripts have to be fast, or easy to write, read, and maintain, or perform simple arithmetic, or not have arsenals of pre-loaded foot guns, but speak for yourself.
Even if Unix is bad, I like it.
I think it's a perfectly reasonable assessment to think of shell scripts as a glue layer between more complex software. It does a few things well, including abstracting away stuff like pipelining software, navigating file systems, dispatching batch jobs, and exposing the same interface to scripts as you'd use to navigate a command line as a human, interactively.
> Maybe you don't think YOUR shell scripts have to be fast, or easy to write, read, and maintain, or perform simple arithmetic, or not have arsenals of pre-loaded foot guns, but speak for yourself.
I also reject the idea that faster or more full-featured alternatives lack footguns, pre-loaded or otherwise.
- C has a relatively limited type system by modern standards, no memory safety, no bounds checking, a slew of non-reentrant stdlib functions, UB, and relies on the user to account for all of that to benefit from its speed.
- C++ offers some improvements, but, being a near superset of C, it still has the footguns of its predecessor, to say nothing of the STL and the bloat issues caused by it.
- Rust improves upon C++ by miles, but the borrow checker can bite you in nontrivial ways, the type system can be obtuse under some circumstances, cargo can introduce issues in the form of competing dependency versions, and build times can be very slow. Mutable global state is also, by design, difficult to work with.
- Python offers ergonomic and speed improvements over POSIX shells in some cases, and a better type system than anything in POSIX shells, but it can't compete with most serious compiled languages for speed. It's also starting to have a serious feature bloat issue.
Pick your poison. The reality is that all tools will suck if you use them wrong enough, and most tools are designed to serve a specific domain well. Even general-purpose programming languages like the ones I mentioned have specializations -- you can use C to build an MVC website, yes, but there are better tools out there for most real-world applications in that domain. You can write an optimizing compiler in Ruby, but if you do that, you should reevaluate what life choices led you to do that.
Bash and co. are fine as shell languages. Their syntax is obtuse but it's everywhere, which means that it's worth learning, cause a bash script that works on one host should, within reason, work on almost any other *nix host (plus or minus things like relying on a specific host's directory structure or some such). I'd argue the biggest hurdle when learning are the difference between pure POSIX shell scripting idioms and bashisms, which are themselves very widely available, but that's a separate topic.
It got lucky ridding the UNIX adoptiong wave, an OS that got adopted over the others, thanks to having its source available almost at a symbol price of a tape copy, and a book commenting its source code, had it been available as commercial AT&T product at VMS, MVS, et al price points, no one would be talking about UNIX philosophy.
That is a feature, not a bug. Add your own bound checks if you want it, or use Ada or other languages that add a lot of fluff (Ada has options to disable the addition of bound checks, FWIW).
I am fine with Bash too (and I use shellcheck all the time), but I try to aim to be POSIX-compliant by default. Additionally, sometimes I just end up using Perl or Lua (LuaJIT).
My argument was that C puts the onus on the user to work within those limitations. Implementing your own bounds checks, doing shared memory management, all that stuff, is extra work that you either have to do yourself or know and trust a library enough to use it, and in either case carry around the weight of having to know that nonstandard stuff.
The warnings of GCC and Clang can help you spot a lot of possible bugs, too, FWIW.
DDT is a hell of a lot older than Bourne shell, is not interpreted, does have full efficient access to the machine instructions and operation system, and it even features a built-in PDP-10 assembler and disassembler, and lets you use inline assembly in your login file to customize it, like I described here:
https://news.ycombinator.com/item?id=43609418
And even the lowly Windows PowerShell is much more recent, and blows Bourne shell out of the water along so many dimensions, by being VASTLY more interoperable, powerful, usable, learnable, maintainable, efficient, and flexible, with a much better syntax, as I described here:
https://news.ycombinator.com/item?id=43609571
>When even lowly Windows PowerShell blows your Unix shell out of the water along so many dimensions of power, usability, learnability, maintainability, efficiency, and flexibility, you know for sure your that your Unix shell and the philosophy it rode in on totally sucks, and self imposed ignorance and delusional denial is your only defense against realizing how bankrupt the Unix Philosophy really is.
>It's such a LOW BAR to lose spectacularly to, and then still try to carry the water and make excuses for the bankrupt "Unix Philosophy" cargo cult. Do better.
This is like bashing Windows for the look of its buttons.
Then, a decade or two later, I realized the Grand Unified $thing$ was itself composed, but not by me so I had no control over it. Then I thought to myself, how great would it be if we decompose this Grand Unified $thing$ into many reusable $thing$s? That way we can be optimally productive by not being dependent on the idiosyncrasies of Grand Unified $thing$.
And so it was written and so it was done. We built many a $thing$ and the world was good, excellent even. But then one of the Ancients realized we could increase our productivity dramatically if we would compose our many $thing$s into one Grand Unified $thing$ so we wouldn't have to learn to use all these different $thing$s.
And so it was written and so it was done. Thus goes the story of the Ancients and their initiation of the most holy of cycles.
It's a general observation of how we are infatuated with composibility, then tire of it and unify and then learn to love it again because the unifications grow stale and weird of which Perl is an excellent example.
When you have a look at Plan 9 (now 9fron) with rc as a shell, awk and the power of rio/acme scripting and namespaces among aux/listen... Perl feels bloated and with the same terse syntax as SH derived shells.
The idea of small composable tools doing one thing and doing it well may have been mostly an ideal (and now pretty niche), but I don't think it was purely invented after the fact. Just crippled by the "worse is better".
Even Risc OS seems to do better even if it doesn't have memory protection too (I think it hasn't, I didn't try it for more than a few days).
But Windows it's too heavyweight, from 8 it has been a disaster. And the NT kernel+explorer can be really slim (look at ReactOS, or XP, or a debloated W7).
The problem it's that Apple and MS (and RedHat) are just selling shiny turds wasting tons of cycles to do trivial tasks.
Worse, you can't slim down your install so it behaves like a sane system for 1GB of RAM.
I can watch 720p@30FPS videos under a n270 netbook with MPV. Something even native players for WXP can't do with low level direct draw calls well enough.
The Windows > XP philosophy among RedHat and Apple it's: let bloat and crap out our OSes with unnecesary services and XML crap (and interpreted languages such as JS and C#) for the desktop until hardware vendors idolize US so the average user has to buy new crap to do the same task ever and ever.
Security? Why the fuck does Gnome 3 need JS at first? Where's Vala, where it could shine here and Mutter could get a big boost and memory leaks could be a thing of the past?
Vala it's a toy because Miguel de Icaza went full MS with C# since Ximian. If Vala had more support from Red Hat, Gnome 4 could support Vala as the main language for it. JS? Lua and Luajit wouldb be a better choice for Mutter scripting. If you have a look on how Luakit and Vimb behave, the difference it's almost nil.
PowerShell even has a hosting api that lets you embed it inside other applications -- try doing that with bash. At least you can do that with python!
When even lowly Windows PowerShell blows your Unix shell out of the water along so many dimensions of power, usability, learnability, maintainability, efficiency, and flexibility, you know for sure your that your Unix shell and the philosophy it rode in on totally sucks, and self imposed ignorance and delusional denial is your only defense against realizing how bankrupt the Unix Philosophy really is.
It's such a LOW BAR to lose spectacularly to, and then still try to carry the water and make excuses for the bankrupt "Unix Philosophy" cargo cult. Do better.
https://en.wikipedia.org/wiki/PowerShell
>Pipeline
>PowerShell implements the concept of a pipeline, which enables piping the output of one cmdlet to another cmdlet as input. As with Unix pipelines, PowerShell pipelines can construct complex commands, using the | operator to connect stages. However, the PowerShell pipeline differs from Unix pipelines in that stages execute within the PowerShell runtime rather than as a set of processes coordinated by the operating system. Additionally, structured .NET objects, rather than byte streams, are passed from one stage to the next. Using objects and executing stages within the PowerShell runtime eliminates the need to serialize data structures, or to extract them by explicitly parsing text output.[47] An object can also encapsulate certain functions that work on the contained data, which become available to the recipient command for use.[48][49] For the last cmdlet in a pipeline, PowerShell automatically pipes its output object to the Out-Default cmdlet, which transforms the objects into a stream of format objects and then renders those to the screen.[50][51]
>Because all PowerShell objects are .NET objects, they share a .ToString() method, which retrieves the text representation of the data in an object. In addition, PowerShell allows formatting definitions to be specified, so the text representation of objects can be customized by choosing which data elements to display, and in what manner. However, in order to maintain backward compatibility, if an external executable is used in a pipeline, it receives a text stream representing the object, instead of directly integrating with the PowerShell type system.[52][53][54]
> Hosting
>One can also use PowerShell embedded in a management application, which uses the PowerShell runtime to implement the management functionality. For this, PowerShell provides a managed hosting API. Via the APIs, the application can instantiate a runspace (one instantiation of the PowerShell runtime), which runs in the application's process and is exposed as a Runspace object.[12] The state of the runspace is encased in a SessionState object. When the runspace is created, the Windows PowerShell runtime initializes the instantiation, including initializing the providers and enumerating the cmdlets, and updates the SessionState object accordingly. The Runspace then must be opened for either synchronous processing or asynchronous processing. After that it can be used to execute commands. [...]
Congrats, PSH, you did what TCL did ~30 years ago, but worse. With TCL everything it's a string, even numbers. Yes, it sucks you need to [ eval ] math operations, but well, the advantages outnumber the quirks.
If you came from Lisp, you will be at home in the spot. Use the l* functions as you were doing the same with Lisp lists, but without juggling with car, cdr, caar, cddr and so on.
And there's Expect which is utterly underrated.
Yes, I hate upvar sometimes, but with namespaces you can almost avoid that issue.
On TCL done for serious stuff... if people have been using Excel with millions of rows for covid patients and census, TCL/Tk with SQLite would outperform these by a huge margin.
https://github.com/SimHacker/micropolis/blob/master/micropol...
With strings and lists the complexity goes away.
I played the original Zork on MIT-DM, and read the original source code written in MDL, which is essentially Lisp with angled brackets and data types, and it's neither object nor text oriented, so I have no idea what point you're trying to make about its descendent ZIL, because it makes no sense and has no bearing on this discussion.
https://news.ycombinator.com/item?id=31846457
https://news.ycombinator.com/item?id=42771971
You're arguing with a well vetted factually correct evidence based wikipedia page, so if you disagree, go try to edit it, and see how long your hallucinations and vandalisms last without citations to reality or coherent sentences.
At least my code doesn't shit its pants when you pass it a filename with a space in it.
On Powershell/C#, TCL/Tk might not be as powerful but it works under Windows XP with IronTCL unlike MS' own and newest C# implementations ( >= 4.5). Double irony there. TCL can help to write some useful software such as a Gopher /Gemini client with embedded TLS support. And the resource usage will still be far lower.
On embedding, TCL wins here, hands down. It's everywhere.
And JimTCL can run under a potato.
Golang it's almost the byproduct of Limbo, and we owe tons of legacy or 9front's C compilers into Golang too.
I do this for my own files, but half of the time I zip something, it’s to send it to a Windows user, in which case zip is king.
So for Windows use I then started to recommend a fork called NanaZip [1] that enabled some Windows security features (CFG, CET, Package Integrity Check...) and added support for additional formats that other forks already had [2] [3].
---
0: https://en.wikipedia.org/wiki/7-Zip#Security
1: https://github.com/M2Team/NanaZip
2: https://github.com/mcmilk/7-Zip-zstd
3: https://github.com/myfreeer/7z-build-nsisDebian tends to have long release cycles, but is very stable. Everything will work perfectly together on stable (in fact, testing tends to be almost as good at stability vs other OSes).
Ubuntu is basically Debian with "but what if we released more frequently?".
If you want the latest tools, then you'll have to settle for a less stable OS (sort of). Nix and Arch come to mind. Neither are super user friendly.
If you want stable and the latest tools, Gentoo is the way to go. However, it's even more intimidating than Arch.
If you want stability and simplicity, then the other way to go is sacrificing disk space. Docker/podman, flatpak, appcontainers, and snap are all contenders in this field.
Windows and Mac both have the same problem. Windows solved this by basically just shipping old versions of libraries and dynamically linking them in based on what app is running.
I’ve administered at least one each of: Ubuntu server (set up by another; the rest were by me), Ubuntu desktop at least ten years ago, Arch desktop, Arch server.
The Arch machines get very occasional breakages, generally either very obvious, or signposted well. I did have real trouble once, but that was connected with cutting corners while updating a laptop that had been switched off for two years. (I’ve updated by more than a year at least two other times, with no problems beyond having to update the keyring package manually before doing the rest. The specific corners I cut this one time led to the post-upgrade hooks not running, and I simply forgot to trigger them manually in order to redo the initcpio image, because I was in a hurry. Due to boot process changes, maybe it was zstd stuff, can’t remember, it wouldn’t boot until I fixed it via booting from a USB drive and chrooting into it and running the hooks.)
Now Ubuntu… within a distro release it’s no trouble, except that you’re more likely to need to add external package sources, which will cause trouble later. I feel like Ubuntu release upgrades have caused a lot more pain than Arch ever did. Partly that may be due to differences in the sorts of packages that are installed on the machines, and partly it may be due to having used third-party repositories and/or PPAs, but there were reasons why those things had to be added, whether because software or OS were too old or too new, and none of them would have been needed under Arch (maybe a few AUR packages, but ones where there would have been no trouble). You could say that I saw more trouble from Ubuntu because I was using it wrong, but… it wouldn’t have been suitable without so “using it wrong”.
The term "stable" is the most polluted term in Linux, it's not something to be proud of. Similar to how high uptime was a virtue, now it just means your system probably has been pwned at some point.
We've been doing this for a long time at my current workplace (for dev containers) and haven't run into any problems.
[0]: https://infozip.sourceforge.net/UnZip.html
[1]: here the build recipe from Arch, where you can see the number of patches that are applied: https://gitlab.archlinux.org/archlinux/packaging/packages/un...
So unzip is not really that special, its a mode general problem with waning interest.
But the way I learned that unzip is unmaintained was pretty horrible. I found an old zip file I created ages ago on Windows. Extracting it on Arch caused no problem. But on FreeBSD, filenames containing non-ASCII characters were not decoded correctly. Well, they probably use different projects for unzip, this happens. Wrong, they use the same upstream, but each decided to apply different patches to add features. And some of the patches address nasty bugs.
For something as basic as unzip, my experience as a user is that when it has so many issues, it either gets removed completely or it gets forked. The most reliable way I found to unzip a zip archive consists of a few lines of python.
As an example, look how Debian patches the Linux kernel: https://udd.debian.org/patches.cgi?src=linux&version=6.12.21... . And the kernel is a very active project.
Funnily, this makes recoding the version number for a SBOM pretty useless.
But for unzip the situation is particularly bad because it has no maintainer. Normally, you would raise feature requests for basic functionality upstream and once added, the maintainer would cut a new release. So software with the same version number generally, but not always, behaves similarly across distros.
But for unzip, because upstream is unmaintained, distro maintainers started to add features while keeping the version number. So in the end you end up with different behavior for what looks like the same release.
FRRouting is the fork.
That said, most distros have bsdtar in their repositories so you might want to use that instead. The package might be called libarchive depending on the distro. It can extract pretty much any format with a simple `bsdtar xf path/to/file`. AES is also supported for zips.
macOS includes it by default and Windows too IIRC, in case you're forced to become a paying Microsoft product^Wuser.
Everything I do gets a git repo and a flake.nix, and direnv activates the environment declared in the flake when I cd to that dir. If I write a script that uses grep, I add the script to the repo and I add pkgs.gnugrep to the flake.nix (also part of the repo).
This way, it's the declared version that gets used, not the system version. Later, when I hop from MacOS to Linux, or visa versa, or to WSL, the flake declares the same version of grep, so the script calls the same version of grep, again avoiding whatever the system has lying around.
It's a flow that I rather like, although many would describe nix as unfriendly to beginniners, so I'm reluctant to outright recommend it precisely. The important part is: declare your dependencies somehow and use only declared dependencies.
Nix is one way to do that, but there's also docker, or you could stick with a particular language ecosystem. python, nodejs, go, rust... they all have ways to bundle and invoke dependencies so you don't have to rely on the system being a certain way and be surprised when it isn't.
A nice side effect of doing this is that when you update your dependencies to newer versions, that ends up in a commit, so if everything breaks you can just check out the old commit and use that instead. And these repos, they don't have to be for software projects--they can just be for "all the tools I need when I'm doing XYZ". I have one for a patio I'm building.
Again, not a criticism of Debian, just a friction I noticed moving between a "bleeding edge" and more stable distro regularly.
You need to understand that you are now in Unix land which means you compose this pipeline using programs that perform each step of the processes. So when creating an encrypted backup you would use: `tar -c /home/foo | gzip | aescrypt >backup.tgz.aes` or something to that effect. This lets you use whatever compression program in the pipe.
Breaking this composability leads to the kind of problem you are complaining about. It also removes the ability of splitting this pipeline across machines allowing you to distribute the compute cost.
A symmetrically encrypted foo.zip.gpg or foo.tgz.gpg would work in a lot more places than a bleeding edge zip version. Also you get better tested and audited encryption code
But I assume, you should be able to update unzip without issues. And if no critical service depends on it, just update and see.
I've been in pager rotations for most of the last 20 years so I'm sympathetic to that. If some genius symlinked unzip to 7z with no testing in production and caused an incident I'd be real mad. But uh I don't think that's remotely what OP was suggesting here.
"if no critical service depends on it, just update and see"
It did not sound like OP was running a hospital infrastructure. And I never did either, nor intend to. I try to have a linux that does what I want on my computer. 7z was helpful to me, so I shared it, that's it.
But I get your point...
"I made plenty of good comments, but this one thread counts as flame war and he has to go by the worst, not by the best comments"
I thought about replying with some Don Hopkins comments, that were way worse than what was here and he is clearly not restricted. But I didn't, as I don't do Kindergarten, I just took a time off from HN.
But it definitely is not equal standards.
So I respect Don Hopkins for his knowledge and experience, but not his style of communication sometimes.
Try using atool (which wraps the various options for different archives and should hopefully fix your problem) or the tools provided by https://libzip.org/documentation/.
Practically, what you're hitting is the problem when upstream is dead, and there is no coordination between different distros to centrally take over maintenance.
Maybe I’ll write it.
Anyone using MacOS as a unix platform is installing updated tooling with brew or similar.
Half sarcastic with that one
Also, zsh is not installed by default on most distros.
Why they went with Bash 2 as the defualt shell is beyond me. I always switched to and used Zsh which had a more recent version. Now I'm also using it on Linux and FreeBSD, because I want a consistent shell.
The reason was the same as here: bash moved to GPL v3.
On the other hand, it's great that there are multiple independent implementations of rsync now. It means that it's actually being treated as a protocol, not just a piece of software.
fun fact: Dropbox internally used rsync binary diff to quickly upload small changes to large file. I assume they still do. But their public API endpoints don't offer this and a small change to a large file means the whole file must be updated.
It's a decent protocol, but it has shortcomings. I'd expect most future use cases for that kind of thing to reach for a content-defined chunking algorithm tuned towards their common file formats and sizes.
Now that's an idea I never considered. Nice.
Very handy if SSH+rsync is locked down in your corporate environment, but building services with gRPC isn’t :)
rsync has a lot of features, surely this will take a good amount of time.
I can see no reason why Apple wouldn't be fine with that.
If someone else wants to release their software under a more permissive license, that’s none of your damn business.
My understanding is that this is the whole reason for the existence of openrsync. The people doing work on the rpki standards wanted to use rsync for one type of transfer, the standards body (IETF?) balked with a concern that the rsync protocol had only one implementation, so the openbsd folk, specifically Kristaps Dzonson stepped up and wrote a second implementation. It does not do everything rsync does but it interoperates enough for the rpki project.
One counter-example to this is in desktop GUI environments. You want one single strong reference implementation there for stability/consistent expectations of what will run. Pretty much everything that will run on the eleventh X protocol will work X.orgs X11 everywhere. Whereas the core wayland protocol is not feature complete and the reference implementation weston is weak. So every wayland compositor implements what should be core wayland protocol features in their own choice of third party lib or custom code. Like libei vs libinput vs no support at all (weston) for normal keyboard/mouse features. Software that works on one wayland won't work on others.
My point here is that strong single reference implementations prevent fragmentation. And sometimes that's important. This is not one of those cases and I'm glad to see more rsync protocol implementations.
I don't know that I'd go that far. The benefit of having only one implementation of a protocol is that the protocol can evolve much faster. You don't have to have committee meetings to tweak how it works. And as a first pass, the more iterations you make of something, the better the result.
Rsync is mature enough to benefit from multiple implementations. But I'm glad it had some time to iterate on the protocol first.
Or you design the protocol to allow non-standard extensions, like with SSH, so you can have foo@example.com implemented by one product (and others can look for it if useful), and bar@example.org by another product. And if enough folks file the feature(s) useful they can be standardize with tweaks that fixed issues that were found though operational experience.
Lots of IETF standards have a "x-" prefix mentioned for private / proprietary extensions.
Bombich "Carbon Copy Cloner" is a GUI app that wraps it.
https://support.bombich.com/hc/en-us/articles/20686446501143...
I started following Mike Bombich from his posts on macOS Server sysadmin boards; see
https://web.archive.org/web/20140707182312/http://static.afp...
Nathaniel Gray created a testing tool to verify the fidelity of backups; files with multiple streams, extended attributes and ACLs, all the good stuff... Backup Bouncer:
https://github.com/n8gray/Backup-Bouncer
See also this SwiftUI app that wraps rsync, RsyncX.
https://github.com/rsyncOSX/RsyncOSX
We used to really care about this stuff, back when we were still running software from "Classic" macOS on top of our new UNIX systems.
https://web.archive.org/web/20161022012615/http://blog.plast...
IFileOperation (Windows) and FileManager (macOS) will do the most performant copy supported by the underlying FS.
Enabling CRC checks is a checkbox in SMB and ReFS - rsync's content matching step is redundant to a modern SMB share on a modern Windows Server. Windows to Windows, IFileOperation will be like 1.5-8x faster throughput with lower CPU usage than rsync, and maybe 1.2-3x faster than doing a file copy using vanilla golang.
And if you don't care about the operating systems that actually use all the complex filesystem metadata, if you only care about Linux, then you only need openrsync or simpler programs.
So for iOS, it’s pretty obvious why they don’t use gplv3… because it would violate the terms.
For macOS they could certainly get away with shipping gplv3 code, but they do a lot of code sharing between iOS and macOS (and watchOS/tvOS/visionOS/etc) and it doesn’t make much sense to build on a gplv3 foundation for just one of these operating systems and not the others. So it’s simpler to just not use it at all.
It also means they’re more free to lock down macOS from running your own code on it in the future, without worrying about having to rip out all the gpl3 code when it happens. Better to just not build on it in the first place.
It's not widely known but what TiVo actually did was something different than this, and both RMS and the SFC believe that both the GPLv2 and GPLv3 allow what TiVo actually did. Some discussion and further links via https://lwn.net/Articles/858905/
[1]: https://support.apple.com/guide/security/signed-system-volum...
> GPLv3 stops tivoization by requiring the distributor to provide you with whatever information or data is necessary to install modified software on the device
By my reading of this, there is not a requirement that the operating system is unlocked, but the device. Being able to install an alternate operating system should meet the requirement to "install modified software on the device."
> This may be as simple as a set of instructions, or it may include special data such as cryptographic keys or information about how to bypass an integrity check in the hardware.
As you've mentioned with disabling SSV, and as Asahi Linux has shown, Apple Silicon hardware can run 3rd party operating systems without any problems.
Edit: "without any problems" is definitely a stretch.
Can't install Asahi Linux on those!
[1] https://brew.sh
You must be able to modify and change the code, not merely append to the PATH:
> Tivoization: Some companies have created various different kinds of devices that run GPLed software, and then rigged the hardware so that they can change the software that's running, but you cannot.
1) The so-called anti-Tivoization clauses are scoped to "consumer products". Don't ask me why, but the language is very deliberately constructed to limit these terms to products "which are normally used for personal, family, or household purposes" - if you're building hardware for commercial or industrial use, none of this applies.
2) These clauses are also scoped to object code which is conveyed "as part of a transaction" in which the user purchases or rents a consumer product which the code is intended for use with. The intent was to limit this to software which was incorporated in the device; however, it accidentally ends up applying to any consumer transaction where the user purchases (e.g.) both a computer and a piece of software which includes GPLv3 code - regardless of who's selling them. So, in practice, this actually applies to any GPLv3 software, regardless of whether it's part of a device's firmware or not.
3) The end result of these clauses is to require that any software distributed under these conditions (which is to say, any GPLv3 software) be distributed with "Installation Information". It's somewhat ambiguous what precisely this encompasses, but it's quite possible that, if Apple distributed GPLv3 software, some of their internal software signing keys and/or build processes would be considered part of that Installation Information.
Sometimes I feel like I'm deluding myself with the small inconveniences I put myself through only using Linux, but finding out about stuff like this wipes that away.
Also, GPLv2 requires the ability to modify and reinstall, just like GPLv3.
https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t...
Neither GPLv2 nor GPLv3 prevent what TiVo actually did.
https://events19.linuxfoundation.org/wp-content/uploads/2017...
Apple using "openrsync" because they want to close the code more than the rsync license lets them.
how does locking down macOS have anything to do w/ GPL compliance? Apple is free to do whatever BS with the OS they ship in terms of terminal access, user permission level, etc regardless of GPL of any code on the device. I could ship a GPLv3 system tomorrow that disallows user root access and as long as I make the OS source freely available and redistributable, it's fine.
> I could ship a GPLv3 system tomorrow that disallows user root access and as long as I make the OS source freely available and redistributable, it's fine.
Is not true for gpl3. It’s called the “tivo-ization” loophole, and it’s one of the principal reasons the GPL3 was made in the first place. I think you’re just wrong.
(Note: I’m not claiming Apple is would be in violation for shipping e.g. a GPLv3 bash on macOS, today, only that they would be in violation for doing that on iOS today, or if in the future they locked down macOS in the same way that iOS was, then for macOS too.)
That's actually quite scary what you wrote there.
That's also even more scary to me, as I am really watchful for such restrictions which can IMO happen in current OSes any time now ...
Now, macOS would be fine in shipping a gpl3 bash. But not iOS. (Yes, iOS has bash. Or ar least they used to, they may be all on zsh now, I’m not sure.)
So, the question becomes to Apple, do we ship different bash versions for different devices, and treat macOS as being different, and have to worry about only using newer bash features on macOS? Or do we keep the same old version on all platforms, and just eschew the new bash everywhere? It’s a pretty simple decision IMO, especially because users can just use brew on macOS and put their own bash on there if they want.
Others are pointing out that gpl3 is less tested in court and that lawyers are just more uncertain/afraid of gpl3 than gpl2, especially with respect to patents… but I don’t think these are mutually exclusive. It’s clear that they can’t ship gpl3 on 4 out of their 5 operating systems. macOS is an outlier, and from an engineering standpoint it’s a lot simpler to just keep them all the same than it is to ship different scripts/etc for different platforms. It can be both reasons.
Even limiting that to “in the USA” I would never say certainly for a license for which so little jurisprudence exists.
Once you add in multiple countries, it doesn’t get clearer.
And yes, that applies to GPLv2, too, but that ship has sailed. I also don’t see them add much new GPLv2 licensed software.
For GPLv3, they also may be concerned about patents. If, to support some MacOS feature, they change a GPLv3 licensed program that uses one of their patents, GPLv3 gives others the rights to use those patents in versions of the tool that run on other platforms.
I suppose the way they prevent you from replacing system files could violate the GPLv3 clause, but still, it seems silly.
I don’t know Apple’s reasoning, but I know that choosing non-GPL licenses when available was one of the guiding principals given to us by corporate lawyers at another company.
On the engineering level, other liceneses likely get selected because it’s easy. You don’t need to consult the legal department to know how to comply with licenses like MIT, BSD, etc, so you just pull the thing in, make any required attributions, and continue on with your day. It’s a lot less friction, which is extremely attractive.
Yeah, there are some assumptions which can be made about licenses and their suitability for our purposes, but no serious organization is touching that code until there has been a full audit of those license terms and the origin of every commit to the repository.
And to prevent your scenario, usually CI/CD systems are gapped to internal repos, unless dependencies are validated and uploaded into those repos, the build is going to break.
Likewise, if you don't do any embedded, you don't need to worry about v3, it's functionally identical to v2 except the compliance story is slightly easier (you don't immediately lose your license if you fuck up a source release).
There's very few companies that have their fingers in both the embedded and desktop markets; those are the ones that need to worry about GPLv3 doomsday. AFAIK that's only Apple and Microsoft[1], both of which have very hostile attitudes towards v3 as a result.
[0] To be clear, when you hear "embedded development", think "TiVoization". The business model of embedded development is putting your proprietary software in a box to sell. GPLv3 wants to make it so that if you do that, you can't stop someone from modifying the OS around the software by making the software detect that and break. But that also makes it significantly harder to defend your business model. Remember: the embedded landscape is chock full of very evil DRM schemes, many of which would break trivially if the app had to support running on arbitrarily modified OSes or with arbitrarily modified libraries.
[1] Microsoft controls the signing keys for UEFI, and while they are willing to sign stuff to let Linux boot, they will not sign GRUB because that's GPLv3 and they worry signing any v3 software will obligate them to release their signing keys.
In the early 2000s all the miscellaneous small projects on sourceforge used GPLv2 (v3 was not out yet).
These days you'll be hard pressed to find any new projects using GPLv3, except the ones with close ties to the GNU or FSF.
The GPL is getting more irrelevant and more easy to avoid. That's why nobody is afraid of GPLv3 any more.
The web stack is such an example. Almost everything you use -- chrome, webpack, electron, babel, React etc all adopted the permissive license.
Not quite so for other areas, but I can count with one hand the number of GPLv3 licenses I have seen in new projects.
No big company wants to spend a million(s) dollars defending themselves from an NPE with an East Texas mailbox in a frivolous licensing suit. Worst case is a judge deciding the license infects their proprietary code because they're built on the same cluster.
The rest of the world has hardly moved on. I've heard of multiple companies with the same GPLv3 policy as Apple for largely the same reasons.
One large side of the industry is turning to managed services. They run free/libre software, but build lock-in on higher level and avoid giving direct contact.
On the other market, the desktop free/libre software won as with Android and free/libre parts of MacOS/iOS.
However they don't do that to benefit the free/libre software in any way, but for getting software cheap or even for free.
The amount by which this flows in one direction, there isn't a win.
1 - Sun decided to inovate by spliting UNIX into user and developer SKUs, thus making the until then irrelevant GCC, interesting to many organisations not willing to pay for UNIX development SDK.
2 - AT&T tried to get control back over UNIX's destiny, and made BSD's future uncertain
Sure, but in this case Apple has chosen, for 20 years, to not go with GPLv3 when there was no alternative.
Apple, Nokia, and many other large companies, employ lawyers that advice them to steer clear of things like GPLv3. The history of that particular license is that it tried to make a few things stricter relative to GPLv2 which unintentionally allowed for things like commercial Linux distributions mixing closed and open source. That's why Android exists and is Linux based, for example. That could not have happened without the loopholes in GPLv2. In a way that was a happy accident and definitely not what the authors of that license had in mind when they wrote the GPL.
It's this intention that is the problem. GPLv3 might fail to live up to its intentions in some respects because of untested (in court), ambiguous clauses, etc. like its predecessor. But the intention is clearly against the notion of mixing proprietary and OSS code. Which, like it or not, is what a lot of big companies do for a living. So, Apple is respecting licenses like this by keeping anything tainted by it at arms length and just not dealing with it.
At a big company I worked for, GPL licenses were strictly forbidden. But I got the vibe that was more about not wanting to wind up in a giant court case because of engineers not being careful in how they combined code.
I'd be super curious if there are explicit intentional acts that people generally think are okay under GPL but where lawyers feel the risk is too high.
Can you elaborate?
I never worked at any company that allows for GPLv3 dependencies, and even GPLv2 aren't welcomed, unless validated by legal team first.
This isn't like the normal "take someone else's work for free but don't give anything back" approach most companies follow when they decide to avoid GPL code.
Especially when a piece of software changes from GPLv2 to GPLv3, it's asking Apple to stop updating, and they do as asked.
Every company has its stupid superstitions.
recall the John Gilmore camp handing out "don't tread on me Apple" buttons 35 years ago.. it has been going on that long. Apple knows very well what they are doing.
It’s a shame that openrsync is not 100% compatible with rsync — I noticed that Apple was starting to switch to openrsync because my own tests broke on macOS 15.
Could anyone suggest why these organizations would want to fund this development?
https://github.com/kristapsdz/openrsync?tab=readme-ov-file#p...
https://news.ycombinator.com/item?id=43605846
Companies fund things because they're useful or necessary. My guess is that some of the companies listed might use BSD — and perhaps wanted/needed an implementation of rsync that was not GPL3 licensed.
And/or they simply have an interest in funding Open Source projects / development.
I meant 'company' in the sense of a legal entity, probably paying some kind of tax, probably having to register/file their accounts every year. Here in the UK, all of these various different types of 'companies' all have to register with Companies House, and file tax returns to HMRC. 'Company' is the overarching legal term here.
— But sure, my bad: the post I was replying to actually used a term that is arguably better, 'organisations'. And I should have used that.
But my point still stands, whether a private limited company, or a non-profit of some kind, or an organisation, or a foundation, or a charity, or whatever — they're all legal entities of some kind — and they're all able to fund anything they please, if they see value in it.
- NetNod is actually a private limited company according to Wikipedia [1]. Corporate identity number: 556534-0014.
- Swedish Internet Foundation, formerly IIS, have corporate identity number: 802405-0190 (on their website [2])
- Sunet is a department of the Swedish Research Council, and uses the Swedish Research Council’s corporate identity number 2021005208, according to their website [3]
So they are all registered with the Swedish Companies Registration Office. Which I assume is their equivalent of Companies House here in the UK.
Maybe if you still think that they're not 'companies' — of some kind — then perhaps take it up with the Swedish Companies Registration Office! ;)
[1] https://en.wikipedia.org/wiki/Netnod
One particular annoyance of openrsync is that it claimed to support the /./ magic path element for --relative. I sent a bug report to Apple for this about a month ago.
rsync_samba is gone as of Sequoia 15.4.
I've installed rsync from homebrew.
* in "shared source" form
In contrast, by 1984, Stallman had already formed a close working relationship with a competent lawyer (Eben Moglen) to devise a legal strategy to maximize the probability that everyone will continue to enjoy a list of freedoms (chosen by Stallman) around any software put under the GPL.
Is that the measure: a complete OS? When exactly did GNU ship a complete OS?
IMHO none of the above is relevant to the question which was first. IMHO both were not first. IBM, among others, were shipping source code with their product, until they didn't. OSS is and was a reaction to an only object model. And there were seeds at Berkeley and MIT.
And Stallman isn't strictly responsible for the MIT strain. As Keith Packard said in his "A Political History of X", the X11 project chose not use the GPL license, because Stallman was simply too annoying.
The fact that BSD was incomplete is relevant because it illustrates the fact that the only people who could run BSD were shops that had a source-code license for the proprietary AT&T Unix.
> the only people who could run BSD were shops that had a source-code license for the proprietary AT&T Unix.
So -- finally! -- that's the measure of OSS? It must run on non-proprietary systems? Not simply the source code? OSS that runs on Windows or MacOS or VMS is not actually OSS?
You figure that Linux is the first non-proprietary system in 1991? Not 4.3BSD released in 1989?
I think you can understand my and others reluctance to state definitively Stallman was first, when by a dozen different metrics he wasn't. I'm still trying to understand what he was supposedly first at? First to find a lawyer?
Linux is important. GNU is important. BSD is important. And they remain important. I don't think any of them are made more important by distinguishing only one and not the others. Like -- as much as it pains me to say it, because of how I loath Stallman and the FSF, GCC was more than important to the entire ecosystem for years. Until LLVM, it was required. Etc, etc.
I want to say around 2006 or 2007 was the first time a 'normal' *nix hacker could install and boot[0] a complete GNU OS[1] and get something resembling work done (ie edit and compile C code in vi or emacs). (yes I know the question was rhetorical)
[0] without having to to a bunch of bootstrapping steps and other hackery
[1] Technically 'shipped' by Debian rather than GNU/FSF
Was this necessary? What exactly are we arguing about again?
I mean if we're going to split hairs and play this game, SCO claimed ownership to alleged Unix code in Linux which wasn't initially resolved until 2008 or so (and further continued for another decade). That never stopped anybody.
So BSD has severe shortcomings as an answer to the question that started this thread, namely, "Can you show examples of impactful open software that predates fsf and stallman?"
So 5 years after he started with an empty repo and some political ramblings?
GNU did not have a working system until Linus released Linux in 1992. They had pieces and components which were worthless on their own.
There is a lot of deliberate ignorance of public domain code being posted on BBSes at the time. I'm not discounting anything Richard did but let's not rewrite history here.
Or around 4 years after the first public GNU Emacs release, 4 years after the first public GNU Bison release, 3 years after the first public GDB release, and 2 years after the first public GCC release.
> GNU did not have a working system until Linus released Linux in 1992. They had pieces and components which were worthless on their own.
Correct, just like how the initial public BSD release was just the networking stack (worthless on its own).
> There is a lot of deliberate ignorance of public domain code being posted on BBSes at the time.
Not sure where you got that from. Nobody claims that Stallman was the first one to come up with publicly releasing source code. I will say that a lot of the "public domain" software from back then lacks the uniformity you see from later movements like free software or open source. Some of it isn't even public domain, and has a license like "this is copyright me, any modified copies must have my copyright statement preserved, this software may not be used for commercial purposes".
People were installing GNU onto existing Unix systems because GNU was better than they were distributed with. Maybe they did that with components of BSD Net/1 - no one has ever told me they did but it probably happened - but that was definitively post GNU.
Anyway, I'm not sure if this matters so much to the debate. Stallman was reacting to a change. He rambled politically and wrote some code to back it up because he used to be able to do things, and now he could only do them if he would write some code and win some allies.
Whether or not GNU had an OS or would ever have an OS has nothing to do with anything, though. What are you trying to illustrate? Those "pieces and components" are some of the most used pieces of software in history.
Of course the answer to your question depends on the definition of 'open source' and 'impactful'.
My view is that having more than one choice is good. It is good for both people and companies that we have BSD and Linux. It is good we have both BSD and GPL.
Sometimes, having too many choices is bad because it leads to fragmentation, creates support and technical issues and leads to analysis paralysis and procrastination. But it's not the case here.
You see the same on embedded as well, all new kids on the block as embedded FOSS OSes or bare metal libraries, are either Apache or MIT licensed.
In practice though authors of GPLv3 see it as a clarification of GPLv2, i.e. they should have the same practical intent.
Worth a click just to see how we used to live!
This is due to the software patent terms that appeared in GPLv3.
"Early drafts of version 3 of the GNU General Public License (GPLv3) contained several patent retaliation clauses that varied in scope, some of which were later removed due to concerns about their efficacy. The final published version of GPLv3 contains a patent retaliation clause similar to those in the Apache License and Mozilla Public License, which terminates rights granted by the license in response to litigation alleging patent infringement in the software."
https://en.wikipedia.org/wiki/Software_patents_and_free_soft...
That would be a significant factor in avoiding GPLv3.
And relying on open source CLI tools in Macintosh for helping end users is not a good idea.
What this signals to me most of all is "oh we can't steal from GNU anymore so we'll steal from openbsd".
Because even if it is a neglible part of the appeal of Macintosh computers, they still make an effort to ship these tools with their OS and they make a lot of money doing it.
> openrsync accepts only a subset of rsync’s command line arguments.
I have not upgraded to MacOS Sequoia yet so I cannot verify but from the source (https://github.com/apple-oss-distributions/rsync/blob/rsync-...) it appears that there is a wrapper and they ship both samba rsync and openrsync and fall back to samba if you use an unsupported option?
On my Sequoia machines, there is no `/usr/libexec/rsync`, and the `rsync` binary at /usr/bin seems to just be the regular `openrsync` binary.
If you're used to Linux (I am) it feels there are lots of quality of life changes, but I realized it's because I'm used to Linux.
The OS itself is fine.
Finder is annoying as hell. The icons / layouts do not snap to resizing, proper navigation requires arcane keyboard shortcuts, it's difficult to open new instances in the expected way, tabs suck, navigation sucks. Finder is made for non-power users.
The preinstalled apps are annoying and can't be removed. Media apps constantly nag to be used or logged into, and they open with magic URLs despite your intention.
Window management and virtual desktops are a pain. Plugins like SizeUp, Amethyst, and BetterTouchTool are awful hacks and feel like it.
I do not want to "define" terms with the shitty built in dictionary tool, yet that option eats up context window space in every tool.
If the DOJ breaks them up, I hope it's into a hardware company and software company. I'll buy their hardware, but I want far away from their software.
Coming from someone who would "prefer to be in Gnome or KDE", this is a hard criticism to take seriously. KDE is obviously designed by committee and is therefore UI elements are all over the place, and Gnome is just horrible.
> Finder is annoying as hell. The icons / layouts do not snap to resizing, proper navigation requires arcane keyboard shortcuts, it's difficult to open new instances in the expected way, tabs suck, navigation sucks. Finder is made for non-power users.
Those "arcane keyboard shortcuts" have been around for 40 years. Or is it the Emacs keybinding that you don't like? Opening a new instance of a Finder window is a Cmd N away. New tab? Cmd T - I do not understand what is hard about that. Oh, and tabs are a feature of the system, and native apps (some non-native too) automatically implement them. It seems to me that someone hasn't RTFM'ed. Some "power user" you seem to be...
The rest is rant that macOS isn't $MyFavouriteDistro and that is the only way computers should work.
> I do not want to "define" terms with the shitty built in dictionary tool, yet that option eats up context window space in every tool.
Oh no! A whole line in a context menu! The horror.
Patience goes a long way for a harmonious relationship.
They were whiny shits.
This is an included package from a 3rd party that was kept at a previous version for licensing reasons.
If you want the latest version of rsync, you can just install it.
Are you upset that MacOs doesn’t include a copy of Libre Office, or every other bit of 3rd party software?
I'd be kind of unhappy if my OS shipped an old version of LO.
Much better than Windows.
They had a good run where they really were open, like when they created OpenCL and Darwin was kept up to date. However these days most of their APIs are closed and Darwin is always lagging behind. Not that anyone actually uses that anyway, but still..
https://www.opengroup.org/openbrand/register/
Perhaps what you mean is that the definition of what it means to be a UNIX has changed for you and you prefer to run UNIX-inspired or UNIX-compatible systems.
The short syntax used to be "rsync -Eva"
Since openrsync this is now "rsync -va ---extended-attributes"
I will need to go through a bunch of scripts now I'm afraid. Bye bye Eva.
The problem with Apple's ancient userspace is so many of the utilities are outdated and don't support things like files bigger than 4GB. So switching to a tool updated in the last 19 years may be an improvement. But then rsync is such a standard, is openrsync 100% compatible?
The need to install and maintain Homebrew was a big part of why I switched from MacOS to Windows. WSL is a very good Unix environment, being just Ubuntu or Debian.
citation needed
I asked a couple of AIs just now and they think most MacOS utilities now work with big files.
Mac OS userland is slowly rotting away because they're terrified of GPL. It's strange, as installing a modern version of rsync seems like it would be specifically allowed by GPL without "infecting" any other parts of the OS
Installing a package manager and a package from it apparently is harder than installing an OS and the installing a package from its package manager.
And lets be honest here. It’s not like homebrew is a set in stone necessity.
I use Nix, theres macports, you can build the package from source. All with less complexity than running what is in the end a whole os in a vm
Macports and nix and fink will build under a new user id and install as root as per any other Unix. Thus the build can be controlled to only use known versions of other libraries.
Homebrew installs as the current user. - Try using it when you do have multiple users on a mac (which is uncommon).
That is only true on Intel—on Apple Silicon it uses /opt/homebrew—so it will only become less true as time goes on.
The only reason it used /usr/local in the first place was that at the time many of the software packages it supported were broken if installed anywhere but /usr/local. In other words, Homebrew didn’t pick that location to annoy you or because they didn’t know better, but for a real practical reason.
That location was always an optional default, you’re free to change it. If you do, everything will be compiled from source rather than using compiled binaries and won’t be offered support. But it does work.
Anything using normal Unix build tools like autotools allow easy change of install directory.
So, thank you, Apple.
But please open source System 7.
https://github.com/apple-oss-distributions/Chess/tree/Chess-...
alias rsy="rsync -avP"
I do this with many unix utils that have insensible (imo) defaults
Why? Because it's right to. If you create good karma, the world will get better. If you do bad things, the world and your world (ie, your karma) will get worse. Paying for software you use extensively is good karma. Not doing so is bad karma that erodes the world (and your world), because it severs the exchange of value and erodes the justice that arises from that, which then reflects back on you inevitably.
For the participants in this archetypal case:
Apple - because it's not right to not pay the developers of software you use a lot, even if it was released under permissive licenses. Apple paying rsync producers for their software is just and right. Apple wants to be a good company, so they want to do this, too. Plus they could get a tailored custom license that works for them, and gives them standard good rsync.
The rsync developers - so they get the just reward for the value they produce, as is right and absolutely correct. They can choose to allocate that however they want, which is them expressing their good interest. What's good for them, is good for what they produce. Everything gets better. Happy cycle.
Everybody else - to participate in that just and right exchange of value, which nourishes the good of both the software, the developers, Apple, and everybody else, supporting the karma of the world, rather than participating in an exploitative abuse that erodes it.
More generally, using software extensively that is permissively licensed is not piracy, but it has the effect of piracy in that value consumed is severed from value rewarded to the producers. This is fundamentally exploitative and abusive, in the limit leads to poor software quality by eroding productive capacity.
One caveat is large well-organized ad-hocracies that maintain giant FOSS projects, like the FOSS or FOSS-like Linux distributions. These are sort of hybrid volunteer, corporate volunteer forces that are large enough to make such fossonomics work. But there's plenty of hyperuseful software built by tiny, single-person or single-company teams for whom those economies don't work as they don't have that scale nor fractional-corp-labor.
To conclude: normalize improving the world and spreading good karma by normalizing paying for the software you use. Even if a given developer team is yet to realize how to bank the value they created for you, as a savvy and responsible software consumer it's your responsibility to seek out and initiate opportunities to pay them, and not to seek out what you can take and exploit. If they make payment available, use it.
Basically, it's fairly simple. Don't be evil. And respond to and create opportunities to do good!
Though I totally get what you mean and license terms are part of the cost that’s why a commercial offering with terms that can be negotiated and a price set to compensate for that seems a great option enough slack to capture the value for both sides.
Just like I would find it weird for a project to be called openlinux or librelinux...
Still it's great to have multiple implementations, of course!
(I will admit: I had to check and openssh is actually "the OpenBSD Secure Shell" project, so I guess it makes sense :-) ).
I don't really get the point about libtls, though.
But I get it, OpenBSD has been using Open* as a prefix for many projects, I didn't know it :-).
Their intent with the naming is indeed to say that Linux isn't open-source enough.
open != "free"
Both rsync and openrsync are both free software and open source software.
There's quite a number in this list that meet OSI's definition (or are even explicitly approved by OSI), but are explicitly labelled non-free by FSF:
https://www.gnu.org/licenses/license-list.html#NonFreeSoftwa...
Two clear examples are:
Apple Public Source License 1.x
NASA Open Source Agreement
I'm glad I'm no longer using their stuff.
And yes I would need a complete implementation obviously.
Mac OS is getting a bit worse every release, clearly trending towards an iOS world where we have to ask apple for permission to run anything, even in a sandbox
And btw, it's included on Setapp subscriptions.