Beware: from what I understood, this isn't a self-sufficient MUA in traditional sense, it's a thin client that offloads the interaction with the actual mail server to a remote system. That said, the credentials to access the mailbox are entrusted to a third party.
> See the README for full instructions on setting up the sync engine, API, and N1 on your local machine.
Sadly, there is no instructions in the linked document yet.
I wonder, can the sync engine and whatever else be bundled with a client, so it'd be a self-sufficient standalone piece of software?
At least half of the blame should be not on them. IMAP doesn't know a thing about what happened to email in last 10 years (few minor exceptions apply).
They're merely implementing their own means to access email in convenient and modern way. Say, IMAP has no labels[1] or contacts[2] (and many other things) - there are proprietary extensions, MS Exchange being the most famous one, but they're... proprietary. And given that proprietary implementations differ, it makes sense to create an unifying wrapper.
And JS apps and web services are just an approach that's trendy today. Personally, I don't like it either, but can't blame for going with the flow. Especially since it's a business (and going against the flow has costs, both in development and adoption), not a private project where one can do things in a way that just feels right, ignoring anything else.
I believe it would absolutely make sense to bundle the "sync engine" within the mail client, though, to make it standalone. But I understand it's done this way just because there was only the engine (as a service, not a library) at the beginning.
IMAP actually does have labels. They are called flags. Servers tend to not implement them well at all (sometimes allowing only a handful to exist, and almost never keeping an index for that metadata, meaning that searches by flag are either slow or often not even allowed :/), but that is not the fault of IMAP.
And, FWIW, search defined virtual folder is a client concept and would be trivially implemented over IMAP's incredibly powerful (and I believe I remember being even extensible) search commands, if only IMAP servers tended to provide the right kinds of index to make such queries fast. (I know this, as I was working on a highly scalable and very efficient IMAP server a few years ago that I had to force myself to stop working on as it was too much of a distraction from the things in life that are actually capable of putting food on the table.)
Mark Crispin was actually pretty-damned prescient when it came to IMAP: he believed that client operations should be offloaded to a server that would have much more disk space and CPU than the client, which is exactly the model everyone reaches for today, and thereby designed a thin client protocol that had asynchronous updates baked into its core. A few years ago, he passed away: here is what I wrote at the time about his mission for IMAP and history on the project.
Really, the only thing in IMAP to actively dislike is how the thin client view requires the server to have a "dense" (here I mean the mathematics term, not from the technical spec) numbering of messages the client is modeling in its view (this is the reason opening a large mailbox via Zimbra takes forever, and why many IMAP servers have dumb global locks), but I have come up with some techniques that should allow me to handle that with minimal overhead. Otherwise, with some of the modern extensions in place (QRESYNC, for example), IMAP is pretty damned awesome.
IIRC, IMAP flags/keywords were limited compared to folders (and a message can't belong to multiple folders). But maybe I got a wrong idea when I read the RFCs...
I would need a more concrete complaint to tell you why you are probably wrong here. The only thing I know of that tends to cause people to balk at flags is the frustrating idea that the name of the flag for some reason has to be the same as the name shown to the user, so the inability to rename flags or even use special characters causes people to give up on them... but you should treat them as opaque database tokens, not user-visible metadata, and map them to names. There was then an extension being worked on to store high-level names in the IMAP metadata store, but I don't know if it got completed (I havent paid much attention since Mark Crispin passed away: I felt like he was one of the few people keeping IMAP from losing its way, so I am highly concerned that I'm going to read through the mailing list and find people frantically pushing through all the stuff he had vetoed). But like, adding your own "documented but non-standard" blob to IMAP's metadata would still be worlds better than throwing away the entire protocol and starting over again from scratch :/. Honestly, I think the core issue with people working on IMAP is that people these days have very little data structure and algorithm training, particularly when it comes to implementations of parsers, and so when they stare into IMAP they miss the forest for the trees and barely scratch the surface of what it can accomplish before giving up and screaming for their JSON and HTTP. (That, and the dense numbering for message views, which is a truly-hard thing to get right, even with a lot of academic background.)
Everything you ever wanted to know about Exchange [1] (but were afraid to ask). I personally use SOGo [2] as a Exchange ActiveSync provider but I know OpenChange is a project to implement the entire Exchange stack. So I would not consider it proprietary.
Bundling the sync engine with the client works if you're willing to download your entire mail archive onto every device that connects to your mail. An implementation of the sync engine that allowed only syncing partial mailboxes would be very difficult to implement on top of (all of) IMAP, Exchange, and Gmail. By abstracting that away into a new cloud service, clients end up being both simpler and more flexible.
I completely agree - things needs to be simpler not more complex. I believe a quote from Albert Einstein fits here:
> "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction."
Though I think I get why this email client and Android's Outlook [1] will hoover your emails to some other server - the server may have more processing power than your device. And/or use an RDBMS to index the emails which is better suited for a server than a smartphone. And that's fine - but the email client should be able to do this on it's own without the need for a server (and explicitly show a warning/message allowing you to opt-in to using the third party service). It will be slower indexing emails on your phone but that is the price you will pay.
And to be fair - I've tried almost every email client I can get my hands on: Evolution, Foxmail, Pegasus, Thunderbird, Outlook, ElementaryOS. To date no one has gotten calendar, contacts, and email better integrated than Outlook. Trust me I REALLY REALLY REALLY don't want to use Outlook - but I've yet to use an open source alternative where in 5 clicks or less I am syncing all those items previously mentioned. Thunderbird comes in second place - but I have to install 2 different extensions for contact and calendar syncing. The contact syncing has been hit/miss and the calendar syncing was stable for the most part. I should note that I don't use Exchange but I use iredadmin with SOGo.
I think SyncML was supposed to solve the whole syncing problem - but never really took off.
Most mail clients have a "sync engine" of some kind, except traditionally they're not written in Python and don't communicate with the rest of the app using REST APIs.
But why is that a problem? It's open source, someone should just package everything up into one nice downloadable desktop application.
Because it's a mail client, and it needs a server that will talk to my SMTP server and the mail client. Which means that I either need to give all my most private and important data (my emails) to this service, or I need to set it up locally. For a mail client.
> Which means that I either need to give all my most private and important data (my emails) to this service
You are already doing this unless you run your own mailserver, or if any of your inbound/outbound emails aren't GPG encrypted (or encrypted during connections with other mail servers using TLS).
Something has to receive email for you when your computer is not running, unless you simply rely on a local mailserver, updating your MX and A records whenever your IP changes, and let other mail servers queue your email until your local desktop is back online to accept them.
I am not already giving my emails to this service, I'm giving them to my email provider. I don't see what value these guys provide that's worth giving them all my mail.
That's a fair argument. In that case, they should provide an MTA you can plop down on a virtual machine that'll store your emails for you and they interface with using JMAP [1].
"JMAP is a transport-agnostic, stateless JSON-based API for synchronising a mail client with a mail server. It is intended as a replacement for IMAP. The specification is based on the API currently used by FastMail’s web app. It aims to be compatible with the IMAP data model, so that it can be easily implemented on a server that currently supports IMAP, but also allows for reduced data usage and more efficient synchronisation, bundling of requests for latency mitigation and is generally much easier to work with than IMAP."
Are you suggesting this is a complete waste of time based on your own needs? Well, fine. But I'm sure there are plenty of valid use cases for it and at the very least the architecture and libraries involved are HN-worthy.
Seems like they've addressed at least one of your concerns, and it might actually turn out to be very simple to set up locally, so here's the privacy trade-off for not using it in your case: privacy > value || privacy > work_of_local_install
Not sure I see the point you are making. Why the concern it's running a server if it is running locally? You already give your email data to another server, and even if you run your own mail server, then what's the issue with running another server yourself?
I don't want to have to keep my computer on and connected to the internet 24/7 for something my mail client does anyway. I don't see why someone would architect a mail client like this.
I can't see why you would need to run your computer all the time. Why do you think this is necessary? You just turn on your computer and the sync service restarts where it left off. Just like Thunderbird, or any other MUA.
But I can see plenty of reasons the design was chosen. If you are bringing in lots of mail accounts, having it running as a daemon isn't necessarily a bad way to go. A sync program doesn't need to be associated with a tty, is better off run in the background in its own process, is arguably more extensible in design terms, doesn't necessarily take up any more memory than a regular executable, allows for multiple UIs (e.g. terminal program, Gtk+ GUI, WxWindows GUI), seperates processing from the UI, lots and lots of reasons.
Maybe the developers can step in here with a response?
You can make a UI to turn off the service or use the command prompt, if that's the concern.
If you can give me a valid technical reason running the sync part of the program as a daemon is bad, I'm definitely all ears though.
Actually, I kind of like this. It means I can run a dockerized backend that will keep a copy of all my email from various servers and present it uniformly to any N1 client I happen to be using.
Existing email protocols aren't nearly that flexible.
Remember, though: Do not run the open sync-engine on a public-facing server!
It uses no effective authentication and everyone can just login to your accounts (due to /accounts and /n, the endpoints displaying the auth codes, being public)
I've used the Nylas API for 9 months for my side project iOS email client. It works very well and the team is super responsive. Documentation is clear and complete. It makes it super easy to build any email client.
Eventually though, I decided to stop using it because I'm not confortable with having all my messages stored by "one more" entity in the cloud. Back to good old IMAP with all the pain it brings.
I'm often wondering if such an API could work in a full end to end encrypted mode, i.e. messages would be encrypted as they arrive before they are stored on the server, and only the client would have the key to decrypt them.
Engineer from Nylas here: End-to-end encryption is definitely on the roadmap. In the meantime the Sync Engine service that we're currently hosting for the sake of convenience is fully open source. You can run everything yourself on your own machines and still have the modern RESTful API that Nylas exposes. No more IMAP!!
It really is. It's ancient, stateful, no one is compliant (everyone has their own extensions), doesn't provide easy syncing, and the guarantees for id consistency, well, aren't there. You can't simply say 'this was the last delta I saw, give me all the updates for my entire user'. That's why email clients spend so long 'syncing' across all the mailboxes.
I've recently started working on an application using email and was considering using Nylas but have hesitated for the same reasons as you. But IMAP is such a pain that I may go ahead anyway. Have you considered running their sync engine yourself?
Any chance you could tell us what the pain is that you're experiencing with IMAP? I don't think I've ever had any problems with it, aside from offline (but for that there's POP) so I'm interested to hear your experience.
There's two things primarily that burned me a few years ago when I was working on an MUA:
1. Message Identifiers/Primary Keys - if I recall correctly, IMAP doesn't have a way to definitively map a message and a unique identifier to the message in a way that is guaranteed to persist long-term.
2. Not exactly IMAP, but malformed messages. When I was working on this, I was using my GMail corpus, which goes back to about 2006 or so. Somewhere around 30-40,000 messages. Turns out it was a ridiculously good test case for parser robustness. Messages that claim to be ASCII-7bit, but have extended characters. Messages that claim to be UTF-8 but have invalid codepoints. Messages with malformed MIME headers. Messages with a text/plain section with garbage in it, while having a nicely formatted text/html section. Messages with embedded CSS that tries its hardest to blow yup on-screen display.
I think that through the duration of the project, I only had to submit 2 PRs to the upstream net::imap ruby library, but damn were there a lot of heuristics in place to try to successfully grab an email from IMAP and display it on screen.
My app does a lot of folder/label management on the user's behalf, and some of the issue I've run into are:
* UID MOVE not always being supported (meaning you need to copy and expunge), and weirdness around gmail's IMAP extensions. This could be abstracted away with a good high level library, but I haven't found any for Ruby that covers all the cases properly.
* Gmail offers OAuth, but for regular IMAP, needing to store user's email and password and the security risks that come with that.
* Needing to know when new messages arrive, so either polling regularly, or keeping a bunch of connections open and using IDLE.
Nothing unsurmountable, but using a service like Nylas means I can spend more time developing features instead.
What I mean is, Nylas takes care of asking for and storing the user's credentials to their email account(s), all you need to store is the token that accesses the Nylas API.
Hi HN-- Michael from Nylas here. We're super excited about this launch, and happy to answer questions about N1, including plugin architecture, design, features, etc.
A few Nylanauts will likely hang out in this thread today. Would love to hear what folks think! :)
How "natively" does this deal with GMail? My experience with most non-google mail clients is that the GMail support is almost always broken or ugly in some way. Does it archive correctly? Label correct? Does it make use of GMail's IMAP extensions for doing the right things? If not, how does it work? What level of integration into the GMail workflow is there? I have for a long time desired a good desktop client for GMail, but all of them rely on an unacceptably bad mapping between IMAP and GMail idioms that I still have not found an acceptable one. How does N1 differ?
Yep, it does all the right things. All the "heavy lifting" here is done by the Nylas sync engine which works natively with Gmail labels, actions, search, etc. and those same semantics are exposed in N1.
We've spent almost 2 years building the sync technology, so it's very stable and flexible. You can check out the IMAP code here: https://github.com/nylas/sync-engine
Wow awesome. If this works entirely locally, this is something I've been looking for for a long time. I might wind up making command line utilities out of it. Or even apply for a Nylas job. Email is something I care about an awful lot.
If, on the other hand, this makes use of some Nylas-hosted server, I'm a bit bummed out, since this would mean all my data is transferred through Nylas servers.
Which is it? Or should I just bite the bullet and read the source?
It's both! Most users are fine with hosted infrastructure, so they don't need to run+maintain it themselves. But you totally can if you want. The code for the sync engine and API is here on GitHub: https://github.com/nylas/sync-engine
We're working on putting together a better doc that outlines how to set up the entire stack locally. Ideally this could even be a Docker container.
And N1 has some easy way of integrating with a locally hosted sync engine, spawning it at N1 startup, and shutting it down at N1 tear down? Or is it the sort of thing were I'll need to host my own infra running this sync engine, and then hand tweak some config file to connect to my infra instead of yours?
I don't think GMail exposes the "primary/social/forums/promotional" classification (which they call categories). However, there's a workaround: in the GMail web interface, type "category:promotions" (for example) in the top search bar, and an option will appear to create a filter that automatically applies a label to all emails matching this condition. (in this case we can make a "promotions" label.)
Then your label will appear in your desktop client. Kind of a hack but I did this for thunderbird and it works.
As for making a plugin that handles this automatically - I'm not sure it's even possible, but I'd love to be proven wrong!
On second thought, the initial setup doesn't really need a plugin so much as the way categories are displayed. Especially cool would be to have them as tabs like in the web interface: once you get used to them, you find them sorely missing in desktop clients. I always find it weird to have my personal, social, and "promotions" emails brutally mixed together.
Unfortunately Gmail doesn't expose updates/forms/promotional via IMAP, so we can't show those. But they probably wouldn't be that hard to add in a plugin... ;)
The three features that my ideal mail client has are:
1. ability to snooze emails
2. rich handling of things like calendar invites, flight reminders, attachments, etc.
3. Vim keybindings for message composition
From other comments, it sounds like you're working on (2). How about (1) and (3)? I took a look at some of the documentation for plugins. Do these sound like features that could be easily added through plugins?
For example, Atom has a package for Vim keybindings [1]. How far from has N1 diverged from "dropping Atom packages in"?
Hey! I'm an engineer on N1 and I can talk about (1) and (3):
1. Snoozing emails would be great. You could totally do this client-side in a plugin using our current developer APIs, by moving snoozed threads to a label/folder, and then moving them back after a prescribed time. Might be a fun project! Feel free to ping us on our community slack channel if you want to try.
3. N1 uses the exact same package structure as Atom, but we're using the CommandRegistry less, and introducing our own UI layer concepts like the ComponentRegistry (for dynamically injecting React elements.) There are probably a handful of Atom packages you could load into N1, but probably only well-contained ones with very minimal UI. Our composer unfortunately doesn't share any code with Atom's text view - we needed to do rich HTML editing, so we wrote our own composer based on `contenteditable`.
Vim keybindings for the composer are also a great idea. There's actually an API for creating `DraftStoreExtensions` (https://nylas.com/N1/docs/DraftStoreExtensions.html), and if you implement `onInput` in your extension you can probably re-create vim keyboard handling. Unfortunately, that's a bit low-level: right now, the composer is not as easy to extend as Atom's editor. We've put a lot more effort into extensibility of the thread / message panels and views.
Enjoy! Feel free to reach out on the community slack channel if you have any other questions.
The FAQ sent me to the README for a completely-local installation instructions that I would like to try out, but they're not there. Where can I find them?
Also, want to jump in our community Slack channel and chat with us? Getting a full environment set up can sometimes be a bit tricky and we can walk you through it. :) http://slack-invite.nylas.com/
Good question.
* This mail client is hackable which can lead to many plugins created on this.
* It is open source and if they embed the sync part in the app will ensure your data are safe.
* And its design belongs to 2015.
I don't quite understand how the business model of the Nylas platform that this client depends on works.
Since it provides up to 10 accounts for free, it looks like that end-users (for whom 10 accounts are more then enough) are not supposed to be the paying customers, but rather other companies that want to integrate with Gmail, etc.
But... does this mean that these companies are supposed to encourage the user to connect their e-mail accounts to the company's Nylas platform account, thus giving Nylas all the user's e-mails and weakening e-mail privacy even more?
Nylas CEO here-- we build infrastructure that powers apps and developers pay us to run it (like AWS). Check out a few here: https://www.nylas.com/use_cases
N1 is a new platform, and fits within our free usage for developer accounts. Paid features (potentially via paid plugins) are coming later. For now, we're focused on ironing out the UI/UX bugs, scaling onboarding system, and opening the codebase to developers.
We're a product company, not an advertising company. We have no plans to sell user data or build the kind of targeting/tracking systems used by Facebook and Google. Our goal is to build rich new experiences and tools on top of email that empower developers and end-users.
Privacy and security are obviously hugely important to us, and we know that it's our job to earn your trust every day. It's one of the reasons we've open sourced nearly all of our codebase. We make money to continue developing this product.
Does that make sense? Happy to dive in more if you'd like, or you can ping me directly: mg@nylas.com
Did anyone else get an email from Nylas that they never signed up for? I don't recall ever signing up for newsletters, so I'm curious as to how I heard about this in my inbox instead of on HN. Kind of annoying.
We used to be called Inbox (www.inboxapp.com) but we changed the name last year (after Google shipped a product with the same name). You may have signed up then! Lots and lots of folks from HN certainly did. :)
I got the email this morning and was very confused as well, this makes sense. Maybe in the next email should write "Nylas (formerly inboxapp.com)", otherwise I'd imagine your unsubscribe rate is going to be very high.
Google has all my data, and that allows them to build incredible features for me, like Google Now and Inbox's automatic parsing of travel and event emails.
I'd rather have all my data myself so I could pick great features from anyone who makes them. Nylas sounds like a part of the puzzle to make that happen. I wish you guys the best of luck.
If you want to have your data yourself, self-hosted mail system is a way to go.
In terms of privacy, the experience is good. I must warn, though, it's quite unpleasant in terms of dancing pigs and bunnies[1]. You won't get anything fancy. The age of protocol development had passed long ago and mail protocols (IMAP) aren't evolving anymore. Almost all the modern stuff is proprietary to services and their APIs and clients.
This limitation doesn't apply to mail filtering and automated parsing, though. Mail filters and processing for self-hosted email is way more powerful one can normally get with third-party services.
If this is not powerful enough, you can make sieve plugin invoke external program and pass it MIME-encoded email message for further processing. This way your server can, for example, add events to your calendar software or do the accounting based on receipt emails (and I forward my SMS to my IMAP server too)
Before Sieve I've used tools like procmail (http://tldp.org/LDP/LG/issue14/procmail.html), although I'm currently only using Dovecot/Sieve setup for my own mail. Some servers I've configured, for example a DIY SMS gateway at the company I've worked for, use just postfix+procmail (+ custom bash and Python scripts).
As for the spam filtering, I've had a good experience with rspamd (https://rspamd.com/), which is light on resources, integrates with Postfix quite nicely, usable out-of-box and is scriptable with Lua.
Thanks a lot for recommending Sieve, I really mean this. There isn't even a single mention of it on the myriad of the half-baked 'how to setup isp-mail for personal use' blogspam I keep stumbling into when googling issues.
It looks like a great tool. Now I can probably have 'n' junk mailboxes with varying personal filters (always wanted this).
Engineer from Nylas: You can start by using our open source Sync Engine (https://github.com/nylas/sync-engine) to begin extracting your own mail and put it behind a modern RESTful API.
Our ultimate goal is to finally give you and others like you an outlet and set of tools to build the next generation of awesome features that currently only Google and a few others can do.
Their dominance is being eroded by Mailpile, RainLoop, and Nylas.
Around the edges, we have Peps[1], Mailr[2] & Kite[3] which may all someday take off.
Zimbra and the well known groupware vendors (OnlyOffice, Horde, Citadel, Kolab) are competing in more-or-less the same space.
Probably dozens of other projects that are on the same level that I have never even heard of.
I would love some sort of comparison, even a biased one written by the respective teams.
Anyone have useful links?
Yeah, it's a tough design decision (assuming you mean nested threads HN-style).
We may add something like that soon. I agree that it's superior in some situations. But most folks are familiar with Gmail, so we opted for linear threading for the first release.
Would love to see a plugin (or several) that explore new thread UIs! Here are a few you can look at to get started: https://nylas.com/n1/examples
I'm super excited about this. I'd love to see a serious collection of apps/extensions for the platform. I love the ecosystem around gmail, but I despise using their web-ui. I'd love to have the flexibility baked into a modern app. I've tried every email app and none of them cut it. They start off promising, but quickly degrade into feature-bloat (Airmail) or development is discontinued (Sparrow, Mailbox?).
Thanks for checking out N1! I work on the client. Unfortunately `keytar` is a bit of a nasty module. Try using `nvm` to run script/bootstrap with Node 10.x. I think this might be the issue you're seeing: https://github.com/atom/apm/issues/195.
If you run into anything else, feel free to ping us on the Nylas Community slack channel (http://slack-invite.nylas.com/)
Honestly, the main reason is that I previously worked at Dropbox and knew we could reliably throw ~petabytes at MySQL. Or at the very least, when we started scaling I could call my friends from Dropbox and Facebook to help out. :)
Most of our complexity is in the sync later, so we spend more time tuning our Python sync server. For example, we wrote a custom statistical profiler: https://nylas.com/blog/performance
How hard would it be to create a backend for PostgreSQL?
I'm currently in doubt to move to either Mailpile or your platform on my personal server. However, PostgreSQL would be a big plus for your platform, since I don't have any experience with MySQL in recent years, and I rather do not run it on my VPS just for mail.
It's not something we're likely to support officially in the near future since we don't run PostgreSQL and don't have the ability to test it well with production loads, but it'd be neat to see a working community fork.
Finally! Always wanted something like this! It looks beautiful! Sorry if this sounds to flimsy but how an app looks is probably a huge factor for me to consider using it. This always pushes me away from open source because they aren't designed (visually) as properly as their proprietory counterparts. Love this project!
Hey folks-- sorry about the invite system. I know we sent out download links, but it turns out way more folks have signed up than we originally planned.
Relatedly, if any experienced devops/sre folks are looking for a new job, please ping me. ;) mg@nylas.com
I thought this looked nice until I saw that it's not an email client at all, it's an interface to a backend service.
Sure, you can get the backend and run it yourself but who wants to do that. Pity, it looked nice.
I love this and want to try it, but I find it strange that they emailed me to tell me about it being available, when I can't actually use it without an invite code.
You can, but remember the sync engine doesn't provide any authentication layer by default, since people use it in so many different ways. You'll want to run it proxied behind some sort of authentication service. (With AWS, you can do this with different subnets via VPC.)
I don't expect that we'll have time to put this together officially very soon. If you'd like to help though, the best way would be to figure out how to get N1 building on arch from the open source repo!
Yep, the only reason we didn't have one for launch is a lack of time. :) You want to help? First step is making sure `script/grunt mkrpm` works from https://github.com/nylas/N1!
> See the README for full instructions on setting up the sync engine, API, and N1 on your local machine.
Sadly, there is no instructions in the linked document yet.
I wonder, can the sync engine and whatever else be bundled with a client, so it'd be a self-sufficient standalone piece of software?