Ask HN: How do you automate file upload to GitHub?

6 points by sirspacey 9 hours ago

12 comments

View on Hacker News

I’ve been exploring Git workflows and curious how others would approach this challenge:

We need to upload a file to GitHub on demand (a table routing JSON file to minimize API calls) and want to automate it so non-technical users can push the file.

Our ideal workflow is to generate the JSON and push automatically via webhook or API call

How would you approach it? Things you’d be concerned about with this approach?

Some additional context:

- We have to solve this problem in this way due to limitations with another code base we rely on.

- this is a private repo so we aren’t concerned about merge issues, just overwriting the file is fine

dan_can_code 6 hours ago

I'm not sure if it's overkill but decapCMS [0] (formerly netlifyCMS) uses git for its publishing method to store static files and content. It's quite simple and open source, and might even just serve as good inspiration for your final solution.

[0] - https://github.com/decaporg/decap-cms

Reply View 0 replies
hotdogs 7 hours ago

If it's JSON it sounds more like config than code. Does it actually need to live in the GitHub repo? Could your application read it from an external location (where ever the rest of your configs live) and reload it on an API call or periodically?

Reply View 1 reply
  • sirspacey 6 hours ago

    We tried this but the performance hit was too high

    * edit: we're building an integration with a SaaS product that doesn't allow us to reference external files due to security, hence having to upload direct to git

    Reply View 0 replies
codingdave 8 hours ago

I mean, we're talking about git here - so you don't need to re-invent pushing a file to the repo. What you need to focus on is building a UX that abstracts it so the non-tech folk don't need to do it via command line.

So the question is less about pushing to git and more about what causes the file to update in the first place. If it is an automated update, just have whatever task updates it also run the git command to push it. If it is a manual update, what tool are the non-tech people using that makes it happen? Add a button to that tool which runs the command.

Reply View 4 replies
  • sirspacey 6 hours ago

    Agreed - I'm new to git so the question is what code that button is firing? Ideally in a cloud app not desktop app.

    Reply View 3 replies
    • solardev an hour ago

      You also don't have to use `git` directly for this (the command-line tool / protocol).

      Because you're on Github and wanting to edit it from another cloud app, it might be easier to just use the Github API (which is its own thing, not git-based) to edit the file directly: https://docs.github.com/en/rest/repos/contents?apiVersion=20...

      Then you can skip all the CI/CD stuff (it's overkill for something like this) and just send a plain fetch from your app's backend (NOT the frontend/client!! Don't expose your Github tokens to your users!). You can either use their JS SDK or just send the raw HTTP commands.

      Reply View 0 replies
    • alemanek 5 hours ago

      I wrote an ansible pipeline that pushes config backups, json and ndjson, to a GitHub repo. It just uses the “git” cli tool to clone the repo, commit any changes, and push those commits to “origin/main”.

      Only piece that you have to be a bit careful with the guarding credentials or token you will be using for the “git” commands. In my case we use Hashicorp Vault for secrets management so I can just checkout the token to use from there.

      Reply View 1 reply
      • sirspacey 2 hours ago

        Got it. CI/CD tools coming up often as a possible solve. Time to do some YT vids!

        Appreciate the guidance on credentials, fully agree on using a vault.

        Reply View 0 replies
mystified5016 6 hours ago

#!/bin/bash

commandToGenerateFile > file.json #or wget some.server.tld/autogen.json file.json git add file.json git commit -m "Automatic upload <date>" git push

You can do this in any number of ways. Powershell, batch script, bash, you could even build a little GUI if you want.

If you want to get fancier, throw the script on a server and give users an SSH script they can double-click, or hook it up to some intranet web interface.

This isn't a particularly complex problem, and git was designed to handle use cases just like this.

Of course you could absolutely overcomplicate and burn dozens of engineer hours building some kubernetes cluster or AWS swarm, but you can do this with a script so simple that I wrote it on my phone.

Reply View 1 reply
  • sirspacey 6 hours ago

    yeah running a bash script is one option

    the k8 approach is tempting though haha

    Reply View 0 replies