Show HN: Sol – A de-minifier for shell programs

(github.com)

142 points by noperator 4 days ago

24 comments

I've built a tool called sol (like "soul") that helps you inspect and format complex shell one-liners. Features:

- Choose which transformations you want (break on pipe, args, redirect, whatever)

- "Peeks" into stringified commands (think xargs, parallel) and formats those, too

- Auto-breaks at a given width (e.g., 80 characters)

- Shows you non-standard aliases, functions, files, etc. that you might not have in your shell environment

- Breaks up long jq lines with jqfmt because—let's be honest—they're getting out of hand

As a security researcher and tool developer, I often encounter (or create) long pipelined Bash commands. While quick and powerful, they can be a nightmare to read or debug. I created sol to make it easier to understand and share these commands with others.

kmarc 2 days ago

I usually do this by hand. Good to see a tool for it :-)

Feature request, which I would love to have in all my automation scripts:

Replace short flags with the long switches. Short flags are great when typing in a terminal but I don't want to figure out 2 years from now what the

    obscurecommand -v -f -n
does, and I have to assume that it's NOT --version --file --dry-run, but --verbose, --force, and --dont-ask-before-deleting-everything

I try to use long options in my script, therefore (especially in a team, where not everyone is familiar with every single command)

  • notpushkin 2 days ago

    It would be a great rule for shellcheck, by the way.

      Line 6:
        curl -fsSL "${url}"
             ^-- SC8537 (warning): use long options instead (`--fail`, `--silent`, `--show-error`, `--location`).
    • yjftsjthsd-h 2 days ago

      I would want it opt-in, because I use shellcheck on scripts that will be run on busybox or *BSD where there aren't long options

      • notpushkin a day ago

        Of course.

        • yjftsjthsd-h a day ago

          Oh, I didn't realize shellcheck already had optional checks (see `shellcheck --list-optional` for a list), so that was not obvious to me initially. Then yes, that'd be a good thing to have available.

  • jakub_g a day ago

    When I saw "deminifier for shell commands" in title I had exactly the same in mind.

snatchpiesinger 2 days ago

Cool! My personal preference is Knuth-style line-breaks on binary operators and pipes, which means breaking before the operator/pipe symbol.

  foo -a -b \
  | bar -c -d -e \
  | baz -e -f
instead of

  foo -a -b | \
  bar -c -d -e | \
  baz -e -f
This doesn't seem to be an option, but could be easy to implement.
  • js2 a day ago

    If you end with a pipe, you don't need the backslash before the newline. It's implicit.

    https://unix.stackexchange.com/questions/253518/where-are-ba...

    When writing bash, if I have a command with many switches, I use an array to avoid backslashes and make the intent clearer:

      curl_cmd=(
        curl
        --silent
        --fail
        --output "$output_file"
        "$url"
      )
      "${curl_cmd[@]}"
    
    I also insist on long-form options in shell scripts since they are much more self-documenting as compared to single-letter options.
  • basemi 2 days ago

    `shfmt` formats multi-line pipes into:

        foo -a -b |
            bar -c -d -e |
            baz -e -f
    
    which it's not that bad
  • TristanBall a day ago

    \ linebreaks are not something I love,and a while ago I started using chained blocks..

    These are usually a step between "overely complicated one liner" and structured script, and often get refactored to functions etc if the script evolves that far. But lots don't, and if I just want something readable, that also lends itself to comments etc, this works for me.

    { foo -a -b }|{ bar -c -d -e }|{ baz -e -f }

    But I suspect it's not everyone's cup of tea!

  • ramses0 2 days ago

    Next level:

       foo -a -b \
       | bar -c -d -e \
       | baz -e -f \
       && echo "DONE."   # && /bin/true
    
    ...means you can safely (arbitrarily) add more intermediate commands to the pipeline w/o having to worry about modifying the trailing slash (eg: `yyp` won't cause a syntax error).
    • yjftsjthsd-h 2 days ago

      A pattern I typically do

          foo && \
          bar && \
          baz && \
          :
      
      or so, which is less verbose but short and sweet. Obviously slightly different, but : (no-op) seems applicable to your situation.
      • js2 a day ago

        You don't need the backslashes in that case. As with lines ending in pipes and a few other places, the line continuation is implicit after the &&:

        https://unix.stackexchange.com/questions/253518/where-are-ba...

        • yjftsjthsd-h a day ago

          Huh, neat. So I picked that habit up from writing Dockerfiles, which does let you do

              RUN foo && \
                  bar && \
                  :
          
          but not

              RUN foo &&
                  bar &&
                  :
          
          (I just tested it), but more recently you can just write

              RUN <<EOF
              foo
              bar
              EOF
          
          so with the caveat of needing to `set -e` the whole thing might be a moot point now:)
      • ramses0 a day ago

        Clever! ...almost TOO clever... ;-)

        That's a great technique, but the `:` as no-op is tough to expect bash-normies to understand (and a tough "operator" to search for). Thanks for sharing, it'll definitely stay in my back pocket!

    • pxc 2 days ago

      I do it this way but I indent the rest of the pipeline (like one typically would in a functional language with a pipeline operator or thread macro, or in an OOP language with method chaining via `.`):

        foo --long-something \
          | bar --long-other
      
      and if the lines in subsequent commands in the pipeline start to get long, I also indent their arguments:

        foo --long-flag \
          | bar \
              --long-other-flag \
              --long-option a \
              --long-option b \
          | baz \
              --long-another-flag \
              --long-flag-again \
          > /path/to/some/file
      
      I really like to use && on its own line like that. One of my favorite things about Fish is how it turns && and || from special syntax into commands, which it calls combiners, so you could write:

        foo \
          | bar \
          | baz
        and echo done
      
      I use this often for conditions whose bodies would only be one line, avoiding the nesting and indentation:

        test -n "$SOMETHING"
        or set -x SOMETHING some-default
      
        command $SOMETHING
      
      In Bash, I usually use parameter substitution for this, but in other situations (other than setting default values for vars) I throw a backslash at the end of a line, indent and use && or ||, imitating the Fish style.

      One of my favorite patterns for readability is to use indented, long-form pipelines like this to set variables. They work fine inside subshells, but for uniformity and clarity I prefer to do

        shopt -s lastpipe
      
        foo \
          | bar \
          | baz \
          | read SOMEVAR
      
      I really enjoy 'maximizing' pipelines like this because it makes it possible to use long pipelines everywhere without making your program terse and mysterious, or creating unwieldy long lines.

      If you do this, you end up with a script is mostly 'flat' (having very little nested control flow, largely avoiding loops), has very few variable assignments, and predictably locates the variable assignments it does have at the ends of pipelines. Each pipeline is a singular train of thought requiring you to consider context and state only at the very beginning and very end, and you can typically likewise think of all the intermediate steps/commands in functional terms.

      I tend to write all of my shell scripts this way, including the ones I write interactively at my prompt. One really cool thing about shell languages is that unlike in 'real' programming languages, loops are actually composable! So you can freely mix ad-hoc/imperative and pipeline-centric styles like this (example is Fish):

        find -name whatever -exec basename '{}' \;
            | while read -l data
                set -l temp (some-series $data)
                set -l another (some-command $temp)
                blahdiblah --something $temp --other $another
            end \
            | bar \
            | baz \
            > some-list.txt
      
      (I typically use 2 spaces to indent when writing Bash scripts, but Fish defaults to 4 in the prompt, which it also automatically indents for you. I'm not sure if that's configurable but I haven't attempted to change it.)

      I tend to follow my guideline suggested earlier and do this only close to the very beginning or very end of a pipeline if that loop actually modifies the filesystem or non-local variables, but it's really nice to have that flexibility imo. (It's especially handy if you want to embed some testing or conditional logic into the pipeline to filter results in a complex way.)

      • stouset a day ago

        Shell script authors like yourself make me very happy. The pipe-to-read is a fun idea, I’ll use it.

        One stanza I have at the beginning of every script:

            [[ -n “${TRACE:-}” ]] && set -o xtrace
        
        This lets you trace any script just by setting the environment variable. And it’s nounset-safe.

        This was typed from memory on mobile so if the above is bugged, my bad :)

ComputerGuru a day ago

This is really cool; for a second I thought I could use it to stop manually maintaining both the minified and full-text versions of my “shell prefix” that makes it possible to run rust source code directly as if it were a shell script [0] where I’ve accidentally diverged between the two in the past, but then I revisited it and saw that the real value was in the comments and explanations more than just placing output in variables and breaking up command pipelines across shell lines.

But the opposite might work, does anyone have a good minifier they could recommend (preferably one that does more than just whitespace mangling, eg also renames variables, chains executions, etc) that doesn’t introduce bash-isms into the resulting script?

[0]: https://neosmart.net/blog/self-compiling-rust-code/

pxc a day ago

This looks really handy! I should add this to the environment for some of my shell-centric projects at work.