Comment by eru

I recreated a little tool to simultaneously mount all the commits in a git repository as directories at the same time (but re-use the same inodes for the same content).

The code is at https://github.com/matthiasgoergens/git-snap-fs

The original was in Python and actually had a decent excuse for existing for a very specific problem at work a few years ago. The new version is in Rust and exists just for fun.

This was also a small experiment in coding with OpenAI's codex. I wrote the Python original by hand---like a caveman. Codex was mostly ok at the actual code, especially once I told it to make `cargo clippy` happy, but it needed lots of help with the design. It kept insisting on extra complications and state.

But perhaps I'm a bit unfair here, because I only figured out the nice and simple design after reflecting on the connection between Linux's fuse and git's design for a while when writing the original. So it's only fair that the computer would also need some help to see how to match them up nicely.

We had a bunch of quants who were writing Python and Matlab code.

Previously they just saved it in a (Windows) Shared Folder and it automatically showed up in the test cluster. No version control, no nothing.

The test cluster had actually grown to a few thousand machines or so. Running Shared Folders over that was crazy, and no version control was crazy, too.

In addition, they expected to be able to write output files next to the source files, and expected them to show up to be used by the other machines.

We were trying to help them migrate to something saner. We could convince them to not intermix source code and output files, but as part of that bargain otherwise they wanted everything to look as similar as possible to before, but still support some git-goodness we have promised.

To make matters worse, they had checked in some rather large files into their repository. Like Gigabytes, and lots of them.

As before, we wanted to support running multiple processes at the same time, but this time on different versions. As a joke I suggested to 'just mount' the git repository directly (that we constantly pull to every computer in the cluster), but my boss thought it was a grand idea, thus the tool.

An additional nicety: under the hood 'git stash' consists of two phases, the first phase make something like a commit from what you have lying around in your repo, the second phase cleans up what you have lying around. If you use libgit2 (or a similar library) you can use just the first phase to get something like a commit, and send that to the server to execute, while changing nothing on the quant's machine, and not forcing them to explicitly make a commit nor polluting their git history.

One saner alternative would have been to just make a checkout for each run. But naively that would have taken more storage space than we had, thanks to those big files. Alternatively, we could have done some sharing for running the same version. But that would have involved some reference counting etc and cleaning up.

So my suggestion was to 'just let the kernel caches handle it'.

In the end, the prototype was useful to get the quants to get along with what we did. And luckily for our sanity, we could soon convince the quants to store those large files somewhere else, and not in the repository along with the code. That restored our sanity, and we could move to a more conventional scheme.

The working life of the tool was blessedly short, but it played an important role in getting the quants to move along in their journey to using version control. Though even though on paper it might have looked like an abortive and wasted effort, in terms of business value it was very successful.

I love the quants. They are very technical and very smart and effectively write software all day every day, but they don't see themselves as software engineers, and they aren't.

I recreated it just for fun, because I like the connection between git and how filesystems work. You can really tell that Linus Torvalds, the original author and designer of git is an operating systems guy.

hrimfaxi 2 days ago

Could you expand on what problem the tool was originally built to solve?

Reply View 1 reply

eru a day ago

We had a bunch of quants who were writing Python and Matlab code.
Previously they just saved it in a (Windows) Shared Folder and it automatically showed up in the test cluster. No version control, no nothing.
The test cluster had actually grown to a few thousand machines or so. Running Shared Folders over that was crazy, and no version control was crazy, too.
In addition, they expected to be able to write output files next to the source files, and expected them to show up to be used by the other machines.
We were trying to help them migrate to something saner. We could convince them to not intermix source code and output files, but as part of that bargain otherwise they wanted everything to look as similar as possible to before, but still support some git-goodness we have promised.
To make matters worse, they had checked in some rather large files into their repository. Like Gigabytes, and lots of them.
As before, we wanted to support running multiple processes at the same time, but this time on different versions. As a joke I suggested to 'just mount' the git repository directly (that we constantly pull to every computer in the cluster), but my boss thought it was a grand idea, thus the tool.
An additional nicety: under the hood 'git stash' consists of two phases, the first phase make something like a commit from what you have lying around in your repo, the second phase cleans up what you have lying around. If you use libgit2 (or a similar library) you can use just the first phase to get something like a commit, and send that to the server to execute, while changing nothing on the quant's machine, and not forcing them to explicitly make a commit nor polluting their git history.
One saner alternative would have been to just make a checkout for each run. But naively that would have taken more storage space than we had, thanks to those big files. Alternatively, we could have done some sharing for running the same version. But that would have involved some reference counting etc and cleaning up.
So my suggestion was to 'just let the kernel caches handle it'.
In the end, the prototype was useful to get the quants to get along with what we did. And luckily for our sanity, we could soon convince the quants to store those large files somewhere else, and not in the repository along with the code. That restored our sanity, and we could move to a more conventional scheme.
The working life of the tool was blessedly short, but it played an important role in getting the quants to move along in their journey to using version control. Though even though on paper it might have looked like an abortive and wasted effort, in terms of business value it was very successful.
I love the quants. They are very technical and very smart and effectively write software all day every day, but they don't see themselves as software engineers, and they aren't.
I recreated it just for fun, because I like the connection between git and how filesystems work. You can really tell that Linus Torvalds, the original author and designer of git is an operating systems guy.

Reply View | 0 replies