Comment by RivieraKid 5 days ago
I've been using Beancount, really enjoyed learning it, writing tools for data import and having a hands-on experience with accounting. But it's been almost a year since I last imported data, I planned to do it monthly but it's a bit of a chore (it takes 30 to 60 mins despite a lot of it being automated).
I use beancount and I use ofxget for some accounts and then keep a CSV archive for the rest and wrote some scripts to sync them into beancount when a new file shows up. It's annoying to download everything and keep it organized, but it gives me better piece of mind than all the other SaSS accounting systems
I've found that so few of my accounts support ofxget that it's not worth the trouble of dealing with it. I just occasionally download year-to-date statements from all my accounts and then run scripts to convert them to ledger format and append only the things after a given cutoff date (things like the credit card script are pretty interactive so I don't want to overwrite the old data). To know which accounts are in most dire need of a check, I have a script that shows the last transaction on each statement-generating account.
This basically works for me and catching up doesn't really take long. There are some accounts that I update very infrequently, but I've gotten over my OCD about this and it doesn't really matter. My wife doesn't appreciate being hassled for the statement from her work HSA every month -- nor does it really matter how much is in there at any given moment, unless we have a big medical expense -- so we just occasionally sit down and catch everything up, maybe twice a year.
Overall I find the automation means it's vastly less work than when I used gnucash, and the flexibility in expense structures and ease of assigning things mean I have much better budget data than when I used mint.com.
Wish their was a standard data format that all banks would use.
The format exists - OFX https://en.wikipedia.org/wiki/Open_Financial_Exchange
But even banks that use it make massive mistakes and break the basic assumptions. Unique ID repeating in the same day for example. Or using the minutes in place of the month...
Banks are much less technological than the common stereotype about them. In fact, integrating or automating a bank is one of the worst things you can do. The list of easy integration goes like this:
High-tech internet services
Stock exchanges
Websites with no API
…
Telcos
…
…
Banks
Unreachable rock bottom of insanityYou should see what the major players (especially MS) do to iCalendar. The standard has been around since at least 2009, with the last update in 2016, but none of the major players implements the standard correctly, and most utterly fail on VTODO, I suspect on purpose because to-do lists are actually useful.
I can't decide whether the failure to implement iCalendar and OFX standards properly are examples of incompetence or malice, but I suspect the latter with an eye to vendor or bank lock-in.
Exactly my experience but with GnuCash. I used to do it weekly, if you miss one or even two or three weeks it's not too bad, but beyond that the effort required to catch up just snowballs. It's also been about a year since I've imported. I've been meaning to get around to reviving it, and the only way I can imagine doing it is to break it up into smaller chunks, like uploading one month at a time over a few days.
Or just start over on the day (or month) you decide to import.
As someone who wrote a budgeting app, historical data > 1 year is a lot less useful than most folks realize. I would say at least 80% of folks never look back further than 6 months. It's not even worthwhile for comparing YoY as your lifestyle, the economy, etc change so much.
I ended up adding per-transaction opt-out roll up functionality for this reason, effectively squashing transactions into monthly and yearly budget summary transaction. Saved a ton of space with little loss of granularity.
Years ago when I helped people with random computer stuff (circa 2002), I had a client who’d run into trouble with Quicken because they had about 20 years of historical data.
It was so weird to me - they were paying me hundreds of dollars so that they could reference their water bill in 1987. I have my own irrational tendencies like this, of course, but still seemed weird.
I agree that data much older than a year is less useful, but to me, historical data going back about a year is quite useful. The main time that my records are practically useful is when I need to file a tax return, which obviously requires a year's worth of data. Beyond that, I find it quite useful to track changes in spending, earning and investments over time.
This was one of YNAB's big "innovations" and wasn't even technical, just social: normalising "starting over" and not fetishing historical data.
Add an "error" account, put transactions there so that your transaction accounts have the correct value, and focus on inputting current data.
If for some reason you care about your historical data enough to import it, you can add new transactions into "error" to subtract whatever part of it they are responsible for. If the account reaches zero, you can even cheat and delete all transactions.
What is the bottleneck in your experience? I have around 100 transactions a month spread across various banks and providers. My experience is this:
I spent about a week defining a procedure to export all statements into CSV, converting the CSV into a Gnucash-friendly format where needed with some GPT3-authored PHP scripts. Once a month I do an import into Gnucash and the process takes at most 2 hours.
There are various frictions. I use multiple different banks, credit cards and investment platforms (partially due to my life being split between two countries). Each of them has their own clunky manual process for downloading statements (including MFA). The statements are in different formats. On the most recent version of GnuCash that I used I experienced weird bugs with the CSV importer which meant it would sometimes crash when I tried to use preset rules.
Classifying transactions is probably the biggest time sink. GnuCash's auto-categorise feature is somewhat helpful but requires a lot of manual intervention and honestly I spent a lot of time just trying to figure out what a particular expense related to (often the note on my bank statement is not helpful).
Ultimately, 2 hours per month is probably about what it took me as well. But x12 that's 24 solid hours of importing. And once you go back a few months it probably takes longer because it is more difficult to recall what particular expenses were for.
If you're willing to hold your nose a little bit, Plaid[0] might be worth looking into. They have (had?) a testing tier which is more than enough for personal uses (100 linked accounts).
[0]: https://plaid.com/
According to this Reddit thread from 10 months ago, some major US banks like Chase won't work even in development mode without production approval, and some people mention obstacles getting approved for OAuth without being a company or otherwise having that level of security posture. (Other people seem to have gotten successfully approved for OAuth.)
I agree this niche needs a better solution than it currently has, although the commercial and regulatory incentives might not be well-aligned to force that. In my family we have important accounts in the US and in the EU (as well as less important accounts in Canada and Mexico), and I don't know of any personal use-friendly solution that lets me pull my data automatically from all of these.
I'd honestly like some nice integration between Plaid and Excel or GSheets to let me track capital gains transactions for my non-euro accounts in the weird way that German tax law requires (over a surprisingly broad range of accounts starting next year at the latest). But nothing turnkey seems to exist right now.
Uh, somehow I forgot to link the Reddit thread I cited: https://www.reddit.com/r/mintuit/comments/17iq9gx/plaid_api_...
Also in June of this year Plaid decommissioned its development platform but added a (quite) limited free tier of the production platform: https://www.reddit.com/r/fintech/comments/1c8xxet/plaid_prod... This doesn't really invalidate the main point of my comment, but it does affect a few details.
teller.io doesn't work in any country except America.
I wish banks weren't actively sabotaging data export. I wrote some scripts for this in the past, reverse engineered the login with OCR, cleaned up the invalid entries, but... It's just too much effort to maintain this. Once the mobile confirmation started being required, I just gave up.