I was the exact same until I started using relative line numbers. Then I can just look at a line and see that it's N lines down and jump there immediately. With j or k repeat I'll often over shoot and then have to go back which is kind of annoying.
Funny, I was thinking exactly the opposite. Stack traces, logs, test failures, warning, etc will always reference an absolute number. If you want to tell a coworker where to find a section of code, you use the absolute number.
Relative numbers are useful to save a few keystrokes. But in every other context, absolute numbers are what's relevant.
How is this easier than jumping straight to the line with absolute numbers?