Comment by ttoinou 2 days ago
Make thunderbird supports a local database with 100k emails with proper search ! Make us pay for that optimization if needed. Email is a big tool of communication for all businesses, Pros who make money daily through emails need to handle tons of emails, we’re ready to pay for that
I have ~100k mails in Thunderbird. The GUI always feels a bit sluggish:
Sometimes spinners don't spin, reactions to clicks take ~500 ms, when I switch from Inbox to Calendar for the first time, I can see how the buttons in the top row render one after the other in ~100ms. (I don't think a human should _ever_ see buttons render!)
Sliding around the size of panels renders at 10 FPS, not so cool.
Opening "Account settings" first produces a full white-flash, then a grey-flash (dark mode), and then renders the UI element.
Startup takes ~5 seconds till the GUI fully shows. Then it hangs at "Opening folder INBOX..." for 60 seconds. Not sure why that sync takes so long when there are no new emails.
So it works acceptably but doesn't feel great.
Searching for e.g. "horse" in the Ctrl+K global search and hitting ender takes 5 seconds for full-text search to produce results. I think that part is OK. I mostly use the "Quick Filter" == "Filter messages" == Shift+Ctrl+K instead to search only subjects and correspondents.
I have ~100 IMAP folders (from +suffix emails). Unfortunately Thunderbird doesn't notice when a new folder gets spawned by a new +suffix email, I have to restart it Thunderbird to ever get to see that email.
RAM usage is 900 MB RES on Linux. (I could not check if that's glibc's fault as so often, because Thunderbird crashes when jemalloc is preloaded.)
When I move the mouse cursor around anywhere in the GUI, that causes 90% CPU usage. For comparison, in Sublime Text, moving the mouse cursor around causes 10% CPU usage over the text buffer and 25% over tabs.
Woah, how? My global search indexer seems to poop out when the sqlite db gets too big or something.
https://www.reddit.com/r/Thunderbird/comments/1jjzb6b/global...
On another note, Thunderbird feels quite snappy for me. Fast and responsive, especially global search.
Even so I think I prefer Recoll now that I've got it working. That thing is amazing.
I have a (non-published) plugin that I'm using that is capable of using elasticsearch for indexing & search from within Thunderbird. I never bothered publishing it, since I never really wanted to maintain it/build a business out of it. Would this be something you are interested in, potentially for a small fee?
Elasticsearch is a pretty bad choice for what really needs to be an embedded database. There are other FTS engines out there of varying quality that would be better suited to this particular case. Off the top of my head, Meillisearch and sqlite3's FTS5 would both be highly embed-able but have other tradeoffs (such as storage overhead).
I do agree that it is a bad choice, though the reason for this is historical; after quite some testing and comparisons, the best JS-based index I could find at the time was elasticlunr [0]. It may very well be the case that there are better wasm alternatives available currently.
Over time I then added a non-JS, external index. Since I already had an ES cluster running elsewhere anyway and the querying of elasticlunr and ElasticSearch is forwards compatible, I decided to just opt for re-using my ES cluster.
In short, the decisionmaking was a mix of historical and compatibility/ease-of-maintenance reasons.
Yeah, if you already have elasticsearch running it's a good choice. Most people are not going to have that or want to run and maintain elasticsearch though. Reasonable decisions for your situation though!
I'll see what I can cobble together somewhere in the course of the weekend/next week ;) Probably need to file off some rough edges and hardcoded stuff.
I mentioned the same problem in one other subthread as well. Current hardware is certainly performant enough not to become this sluggish at just 100 000 or so emails. There's actually no reason it shouldn't work well with say a million emails in one inbox.
> Outlook with multi-gigabye .pst files
What has been your experience? Mine in trying to use and support it is that Outlook is an Exchange client; PSTs are hacks to meet demand, though they work well enough in limited circumstances. Especially PSTs over a LAN connection are a disaster.
The Exchange server hardware was so underpowered (or the software so ill-designed for large mailboxes) that Exchange powered searches would fail, but ones run on the local pst would complete successfully (if slowly). This was on an HDD. SSD would be much faster.
I'm primarily a Linux user, but Mail.app is probably the best graphical email client I've ever had the pleasure of using (you can pry mutt from my cold, dead hands).
Yes, I know it's also an GNUStep application
Tell that to apple mail. Makes no sense how an app seemingly unchanged since the tiger days when I started using it could still be as performant as it always was on far better hardware. In fact I frequently find it to be the culprit when I wonder what the hell could be spinning my fans on this m3 pro just churning over the database.
Iphone version is arguably worse because it also has performance issues but doesn’t support inbox rules. Then again those inbox rules often fail to filter emails anyhow.
I just checked mine and I have about 250k emails sitting in my personal laptop's mailbox, no issues there. It might be dependent on the provider - I know at work where we use Exchange I get occasional slowdowns but I'm not sure whether that's due to Mail.app, due to Exchange, due to our dear endpoint security software taking its sweet time checking whatever I'm doing, or any combination of these. I probably have a couple million emails in my work laptop's mailbox though. Often the "Rebuild Mailbox" function fixes any problems for me, at least for a while.
The iPhone one regularly just doesn't search properly for me though. I'll search for the exact subject or contents of a message and just won't be able to find it, then when I go to my laptop and type in the exact same terms it finds it instantly.
I'd love it - email could use serious tools and refinement - but so many questions: Is it local or hosted? What is the story with privacy? Do you use an existing application (like a Thunderbird fork) or something you created?
Can you / will you integrate other messaging such as SMS, even WhatsApp, etc.? RSS?
Great questions.
1. It is _both_ local and hosted. The client itself is fully offline-capable, including proper full-text search (single digit ms), writing drafts – anything you would expect an email client to do. The "hosted" bit is to ensure rapid synchronisation across multiple clients (ie your desktop and mobile).
2. Some metadata is hosted in pg to facilitate cross-platform synchronisation, as mentioned. This is encrypted at rest on a provider with SOC 2 Type I certification. Further symmetric encryption (AES-256) of sensitive columns is also done. We're well aware that security is the most important aspect of this product and is our primary focus.
3. We've not forked Thunderbird. Marco has been built from the ground up, both on the FE and BE, and has been a monumental task.
4. We have no immediate plans to add SMS/WhatsApp/RSS. If those interest you, you might have a look at Missive.
We understand that storing email metadata is potentially a turn-off to some, but is actually the key driver to an entirely new email experience. It means that a Marco client itself is virtually stateless (save for some lightweight metadata) and syncs instantly across N number of clients – it runs on web/OSX/Windows/Android/etc, and changes propagate between them instantly. New client setup happens via Marco in a proprietary way on the order of seconds and doesn't take hours to sync via IMAP.
We're building this for ourselves. Thunderbird is "alright". Apple Mail is "alright". Superhuman is decent, but ridiculously expensive and Google/Microsoft only. Missive is fairly decent (and also stores metadata), but is built for team collaboration, not individual use.
Genuinely had to look that up. Not familiar with Microsoft speak.
From a quick look, it indeed looks similar. Although I imagine Microsoft's implementation is filled with cruft.
We use Replicache + Orama. _All_ data is fully offline on the client and synced to the BE when network is available. Orama handles indexing and full text search, filtering, sorting, etc, all in single-digit ms.
Let me know if you have any follow-up questions!
Apple Mail was actually the straw that broke the camel's back for me.
I wrote a blog post about our reasoning here:
Good luck with that, but as a first reaction I must say, what I see on that side is not that impressive. It's just the same feature-set & interface all over again. It's not selling me any reason why I should be more interested in this, than in all the other clients already available.
Granted there is very little on that side, but I hope if you really start from scratch, you will also look more outside the box of the established mail clients. Think about how RSS Feed readers are working and the interfaces they offer, think about task¬e-managment-tools are working and what they offer. For example, why is there no mail client with a kanban-board-view, allowing to organize mails by status or tags. Why is there no client with a social media feed-interface or even a tweetdeck-like view, allowing to observe multiple mail sources in parallel. This is the kind of innovation I'd like to see in a new mail client. Not just a bit better performance and new colors.
Yes, we've started from scratch. A detailed explanation for our reasoning can be found here:
https://marcoapp.io/blog/marco-an-introduction
TLDR: There are _no_ IMAP-primitive truly cross-platform email clients in existence, except for Missive, which is built for team collaboration. We are building something net new.
The content on the website is indeed a minimal representation and the actual alpha product has matured quite a bit beyond what you see there.
The kanban suggestion is brilliant, I have made a note of that.
Why tackle problems like search when you can redesign the UI/UX half a dozen times instead?
I don't blame the developers; they do whatever they're paid to do.
Mozilla has terrible leadership and no vision. It's the worst aspects of directionless, corporate software masquerading as an open source project.
> Make thunderbird supports a local database with 100k emails with proper search
Currently working with a Thunderbird database which contains over 300,000 messages and search works quite reliably (once in a blue moon have to switch from "Search Messages..." to "Global Search"), though the emails are stored in Maildir format rather than the default mbox: https://tinyapps.org/blog/202207100700_thunderbird_mbox_to_m... .