Comment by dagw

Comment by dagw 5 days ago

As someone in a similar situation, my number one piece of advice is to meticulously keep up your software engineering best practices even when doing your day to day data analysis job. All that bread-and-butter stuff like tests and easy deployment and well structured code and reproducible builds and writing reusable tools and packages tends to fall by the wayside when your 'just' hacking one off Jupyter notebooks.

As a software engineer you bring a perspective to the job that many of your colleagues may lack. Lean on that and use that background to help build better tools and ways of working. This not only makes your job (and the job of your colleagues) easier in the long run it will also let you keep your software engineering skills sharp.

aristofun 5 days ago

> even when doing your day to day data analysis job

This is a bad advice.

Usually coming from an over-engineering mentality (that many engineers suffer from, including myself).

Part of the engineering culture is to find the _right_ solution for the job. Not the most _engineered_ one.

There is no point to shoot at sparrows from a canon.

Reply View 3 replies

buffington 5 days ago

How is this bad advice?
> keep up your software engineering best practices
It's not a best practice to over-engineer solutions, and indeed, it is a best practice to find the right solution.

Reply View | 2 replies
- aristofun 4 days ago
  
  When your job is to cobble up disposable scripts - there is no place for “practices”, unit test for example. Thus there is nothing for a software engineer to pay special attention to. Therefore there is no value in suggesting that
  While some may take that as “turn up your engineering knob” that leads to over engineering
  
  Reply View | 1 reply
  
  mettamage 4 days ago
  
  While I am fairly good at overengineering and optimizing, I have always hated doing it. For the longest time I wasn't able to do it but then I learned about low level hacking, and throwing things into assembly. Then I had a friend enter the HFT field and he taught me a thing or two about optimization.
  godbolt.org is fun to check out :)
  But yea, I really like making quick scrappy prototypes that deliver business results quickly. I've always had that mindset rather than crafting good quality code. In software engineering teams I threw that mindset to the wayside because they'd get mad. Sometimes rightly so as tests have saved my life, but quite often I think SWEs focus too much on code and being technical rather than delivering impact. I think in part that's also because it seems that in most places I've worked at product owners are "supposed to" think about that. The dev is only supposed to think about the technical implications, maybe the UX implications (depending on the company) but the moment a dev thinks about business implications, I've gotten quite a few times to "stay in my lane".
  In most cases the care for code quality isn't needed, sometimes it is. I'm taking a page from Jonathan Blow here (the game dev that created The Witness and Braid). He'd develop everything unoptimized and only later when he'd run into performance issues would he optimize it.
  
  Reply View | 0 replies