128 points by _1tan a day ago
I work mainly in energy market communications and systems that facilitate energy trading, balancing and such. Currently most parties there take minutes to process messages and I think there could be a lot to learn from financial systems engineering. Any good resources you can recommend?
I would recommend the Jane Street youtube channel, the ocamal focused ones in my opinion you can skip. But otherwise their engineering videos will give you a good sense of it. When it comes to low latency systems this is a very private ecosystem. There are some open source projects for learning some of the mechanics of trading, market data and the order to settlement lifecycle. I am not aware of any being built for low latency.
Market data ingest, analysis and resulting order execution is chewing through way more data way faster than any banking transaction system. I have worked on both of them.
If you want more in the weeds but still high level I gave a talk on the main concepts and systems you need to know to code low latency for markets. https://docs.google.com/presentation/d/1HIPJb0XX3JDHEYSrZC8v...
# Books
## *Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris*
- comprehensive overview written in an accessible way
## *The Microstructure of Financial Markets by Frank de Jong and Barbara Rindi*
- 1st 1/4 of the book is generally useful. Then the math starts. This math is not needed to get a basic overview.
These are two books I wish someone gave to me when I started my first capital markets software engineering job. I recommend them to all the people I place in financial system engineering roles.
This is my passion. Message me if you want to talk more about this — see HN profile for contact info.
I work on low(ish) latency trading systems in FX. FIX is the standard communication protocol and familiarity with it is essential for me. Here you can look up the standard message types and tag values: https://fiximate.fixtrading.org/
They also have docs for the standard message flows you can expect during trading. I use it regularly.
Also work in the FX space and could never find a good FIX specification resource. Always had to rely on whatever the broker shared with us, but it always felt incomplete. This is a super useful resource. Thanks for sharing!
Like someone else said your question is kind of broad. But I d recommend reading TigerBeetle's documentation to understand how financial transaction processing may differ from what they call general purpose databases like PostgreSQL.
There might be some gems for you in this old thread: https://news.ycombinator.com/item?id=22573204
Book recommendations for learning financial systems.
I recently read Central Banking 101¹ and learned a lot!
There a free online accredited master's degree in Financial Engineering (https://www.wqu.edu/mscfe), probably not for OP but maybe some find it useful for them
> Currently most parties there take minutes to process messages and I think there could be a lot to learn from financial systems engineering
Depends on what you mean by "financial systems". There are plenty of financial systems that takes days or weeks to fully process transactions.
You're probably referring to stock trading systems (which by their nature have to be extremely low latency). The term to search for is "HFT" (high-frequency trading).
Here is a recent paper on the topic (focused on C++, though most things you find will tend to be): https://arxiv.org/abs/2309.04259
I am mainly working on B2B message exchange, e.g. via AS4/EDIFACT/ebix/CIM files and most parties seem to still work with a lot of batch processing. With the ongoing digitization of e.g. metering hardware and faster settlement times in the energy markets (EU-centric) I think this might not cut it anymore in the future.
Currently most parties there take minutes to process messages
I suspect that the energy market is a bit different from markets where being ruthlessly fast might be a good long term behavior. So there might be good reasons for that way of doing business due to the specific nature of the energy market.
For example a limited number of counter parties might make stable B2B relationships more valuable. Or regulatory compliance might require extra steps. Or the messaging system might be noisy.
Or not. I wonder how executives at the trading companies would describe the delay.
One other thing that comes to mind - the players in the financial markets make so much money off that message-processing speed that it opens up a whole lot of options that may not be available to someone without those kinds of deep pockets. We're all still fighting Einstein, but the ability to just throw resources at the problem can surface some solutions that are out of reach when you manufacture something other than money.
"The Disruptor is the result of our efforts to build the world’s highest performance financial exchange at LMAX."
I recently learned of Kdb+ and the associated ecosystem of financial tools.
May be worth getting a head start on these, I understand these are the tools that the HFTs and the quants use.
Check out The Payments Engineer Playbook[0]. I don't work on financial systems but I'm subscribed and like the occasional distributed systems insights.
One of the posts turned up on HN front page a year ago[1]. Thats how I discovered it
If you just want to get up to speed on the mathematics, Luenberger's Investment Science is a classic.
https://www.amazon.com/Investment-Science-David-G-Luenberger...
A lot of settlement in financial markets is still pretty slow. That’s a big reason why there was so much fintech interest in blockchain.
You may be thinking of high frequency trading. In that case, traders interact directly with an exchange - e.g. via direct market access[1] - so it’s a pre-established two-party interaction. There’s no particular technical difficulty with making that fast. Usually, slow transaction times are a consequence of the structure of the market, not a technical issue particularly.
[1] https://corporatefinanceinstitute.com/resources/career-map/s...
In the broad picture of engineering, I would consider this to be technically difficult. Many pieces of the puzzle need to interact correctly to remove latency, from physical location, network gear, decision about where software runs, removing unnecessary layers of everything, to algos and data structures, and doing razor-tight tradeoff analysis favoring low-latency at every step along the way. It’s also expensive. So it’s hard to agree that this is not hard. On the flip side, if you find this easy and want a job, msg me.
You might want to checkout the books by Timothy Masters - http://www.timothymasters.info/
I had come across his books when searching for ML books in C++. He has applied ML/Statistical/Deep Learning/etc. techniques to Market Trading Systems and other domains. Generally all his books contain implementations in C/C++.
The reviews of his books on Amazon gives you confidence that he really knows his stuff from a practical/industry pov.
I wouldn't look further than asking what happens currently in the processing of these messages. It might turn out for example that the minutes delay is deliberate: high frequency trading can become unstable more easily than low frequency trading. Perhaps there's a human in the loop?
Question is a bit broad, but out of all the concepts under the financial engineering umbrella you're bound to explore the concept of ledgering eventually.
I've written a bit about it on my own co's product blog in an attempt to demystify some core concepts [1], [2], [3].
Still on ledgering and expanding into less mathematical and more applied concepts, I can also recommend a book called "The Accounting Game: basic accounting fresh from the lemonade stand" [4].
[1]: https://www.formance.com/blog/engineering/how-not-to-build-a... [2]: https://www.formance.com/blog/engineering/debits-and-credits... [3]: https://www.formance.com/blog/engineering/ledgering-all-the-... [4]: https://books.google.com/books/about/The_Accounting_Game.htm...