I think this will be great for extensions. Currently the only solid choice is to use the dead simple storage.local which only allows to retrieve things by ID.
There's one problem though, this new API is for the web, so the nature of this storage is temporary - obviously the user must be able to clear it when clearing site data and this what makes it currently an unviable solution for a persistent extensions data storage. https://bugs.chromium.org/p/chromium/issues/detail?id=138321...
Agreed, people want permission based file system access and so I don't see why a site can't prompt a user to store a file in a user-specified location. If they clear their site settings then giving permission to the same file should retain their data. I don't want to have to worry about losing something important or having my browser act as a file system.
> I don't see why a site can't prompt a user to store a file in a user-specified location.
This is already possible with the file system access API. Clearing site data has no impact on these files. But you do have to re-grant permission every time you reload the page.
Therein lies the problem, because it should only re-prompt for permission only when you clear the site settings. Additionally, as far as I'm aware or at least last time I checked this doesn't enable directory based permissions, so a web IDE wouldn't be able to browser a directory or sub-directories after given permission.
> Additionally, as far as I'm aware or at least last time I checked this doesn't enable directory based permissions,
I haven't tried it yet, but I thought it did.. the first 3 paragraphs of the comment you're responding to all mention directories
> This API allows interaction with files on a user's local device, or on a user-accessible network file system. Core functionality of this API includes reading files, writing or saving files, and access to directory structure.
> Most of the interaction with files and directories is accomplished through handles. A parent FileSystemHandle class helps define two child classes: FileSystemFileHandle and FileSystemDirectoryHandle, for files and directories respectively.
> The handles represent a file or directory on the user's system. You can first gain access to them by showing the user a file or directory picker using methods such as window.showOpenFilePicker() and window.showDirectoryPicker(). Once these are called, the file picker presents itself and the user selects either a file or directory. Once this happens successfully, a handle is returned.
I’m getting Flash flashbacks. Flash apps had filesystem access in a designated area, with some amount of user control, that is not easily wiped from the browser; Flash games used that for saves and data. Years after the demise of Flash, the web platform is still catching up.
It's different though, extensions are like local apps and deserve persistent storage, while you are talking about how Flash was being used by remote sources. Also my proposed approach is that this data would be wiped when the extension is removed from the browser - which is what happens for storage.local
Some way to indicate which "WASM contexts" (?) have utilized how much disk space would be great for open source multiple implementations not-Flash, too.
Both APIs are part of the same standard. If you just want to store things for your site/webapp you want the origin private filesystem and you can just go for it. If you want to do something with one of the user's files it requires interaction from the user to select the folder/file since it'd be a security nightmare otherwise. You can also do both e.g. store a bunch of backend stuff for your photo editing webapp in the origin private filesystem but prompt the user to select their images folder so they can start editing them.
> the user must be able to clear it when clearing site data and this what makes it currently an unviable solution
What are you suggesting here? That extensions should have some special storage, that can't be (easily) deleted? Noooooooo thank you. Firefox has "forget about this site" and similar tools, which I use near daily. Don't fuck with my ability to do housekeeping.
yeah because full access to the filesystem is only solution.
asking for permission to a longer storage.local exemption for the user? noooo let's not even think about giving user control of things. let's the European union pass post-fact legislation after file system access is abused by adnetworks. then they get the blame of being annoying, not the browser vendor who sell ads.
let's stop being naive. there are a million ways to fix that bug with better UX and respect for user control, but this is a scape goat to have that feature.
Does anyone have hands-on experience with how well different browsers support this? This blog post and caniuse both say Chrome support is currently somewhat limited (though being improved); caniuse also mentions that Safari (both desktop and iOS) only support the Origin Private File System part of the File System API, and that Firefox doesn't support it all. I'm curious how all that actually shakes out in practice.
The Origin Private File System is coming to all browsers, Chrome, Safari, and Firefox. It's a universally agreed-on Web standard: https://fs.spec.whatwg.org/.
Just because the standard exists doesn't mean every browser has already implemented 100% of it, hence why I was asking if anyone's had any experience actually trying to use it.
EDIT: I saw from your other comments on this post that Firefox has implemented at least the origin private file system part in nightly builds; that's useful info that I was looking for.
However the sqlite demo page says OPFS is unavailable for me in Safari https://sqlite-wasm-opfs.glitch.me/ ...unsure whether it only supports a subset of the APIs that the official sqlite build uses?
What happens is that one browser (usually Chrome) implements a useful new feature. Then applications are released that use that feature. Then the other browser vendors start hearing from their users that they suck because they don't support the nice new feature and hence can't run the nice new applications. And eventually they add support.
> What happens is that one browser (usually Chrome) implements a useful new feature.
What happens is Chrome releases a Chrome-only non-standard (at a rate of 400 APIs per year). It cobbles together a barely legible spec [1], and then uses its multiple propaganda channels like web.dev to present them as actual completed standards that other browsers are just slow to implement.
Even if other browsers only learn about this a few days before Chrome ships, or have multiple objections.
That's a nice sentiment at the link. The reality is that Chrome ships hundreds of APIs with little to no input and only barely notifies other browser vendors that "yeah, we're shipping it".
We're literally in the discussion about File System API that is:
- not on any standards track
- considered harmful by other browser vendors
- shipped by default in Chrome
Edit: I can give several examples from Web Transport to HID to Constructible Stylesheets that were either shipped with no input from other vendors, or shipped against the multiple objections from other browser vendors. And that's just the ones I know about.
Constructible Stylesheets: the spec contained a trivially reproducible race condition, the API was badly specified. Google shipped against any objections and refused to bring it back under the flag. Full discussion here: https://github.com/WICG/construct-stylesheets/issues/45. Shipped in Chrome https://github.com/WICG/construct-stylesheets/issues/45#issu... (may be hidden on mobile) despite multiple unresolved issues. Two years later Chrome did add a better API that people originally requested, other issues potentially remain.
You may believe in benevolent Google working hand in hand with others for the betterment of the web. The reality is radically different.
It is literally being drafted in conjunction by all the major browsers.
> - shipped by default in Chrome
So what? I for one am thankful that Chromium enables features earlier than other browsers. If you don't want the Chromium implementation then don't use it.
> As others have pointed out the standard is here:
As others also pointed out, it's a different standard.
File System: Mozilla isn't opposed
File System Access API: considered harmful
There are parts (of whichever spec) that neither Mozilla nor Safari are opposed to like Private Origin File System. That one ships across all browsers.
> So what?
Ah yes. So what if a dominant browser ships features without any consensus, against objections, with badly written specs, or with glaring issues.
There was also Storage Foundation API to which the reaction was "I don't think it's an acceptable outcome for the web platform to have that many ways to work with files" :) https://github.com/mozilla/standards-positions/issues/481 This one never saw the light of day.
It's a good idea, or at least originally it was a good idea. In reality it devolved to, well, see the links I provided.
Edit:
Note: those three are the ones I remember because I was reading about them at the time, or participating in some form in the discussion. How many are there that just slip through? Well, you can kinda see for yourself (second hand guessing, by proxy etc.):
1. MDN lists various web APIs, including experimental ones: https://developer.mozilla.org/en-US/docs/Web/API Almost every single one that's marked "experimental" is shipped by default in Chrome
2. Mozilla lists its position on various standards. Almost all of those marked as "considered harmful/negative" are shipped by default in Chrome https://mozilla.github.io/standards-positions/ (exceptions are a few deprecated and removed ones like HTML Imports)
3. Chrome currently ships 1100 more APIs more than Firefox: https://web-confluence.appspot.com/#!/confluence (note: a single standard like "File System Access API" may expose dozens of these APIs) Is it because Firefox is lagging, or is it because Chrome just powers through, all consensus be damned?
There isn't an "official" NPM package or ES6 model yet, but I believe they are working on it. It's also very much designed for the browser currently, but again I believe they are intending to support server WASM environments too eventually.
> There isn't an "official" NPM package or ES6 model yet
(The sqlite team's JS/WASM Guy here...)
An ES6 module build was added shortly after the 3.40 release.
NPM/node.js are nowhere on our radar. Our build is structured such that people who want to plug it in the resulting JS file to whatever their favorite toolchain is are welcomed to do so, but we have neither the ambition nor the bandwidth to support every build/bundling platform out there, especially ones none of us otherwise use.
> It's also very much designed for the browser currently, but again I believe they are intending to support server WASM environments too eventually.
The JS code is targeted solely at browsers and there are no plans on changing that in the foreseeable future.
The WASM build itself, we are working to provide server-side support for. We have a branch which builds under wasi-sdk, but we cannot create a JS binding for that build until/unless we find some substitute for Emscripten's transparent translation of POSIX I/O APIs.
> but we have neither the ambition nor the bandwidth to support every build/bundling platform out there, especially ones none of us otherwise use.
For what it's worth (though it seems like the decisions is done), there's no need to support every build/bundling platform - the NPM registry is rather 'tooling agnostic' and publishing there would make it available to a lot more people with a lot less effort.
> the NPM registry is rather 'tooling agnostic' and publishing there would make it available to a lot more people ...
The sqlite project is populated entirely by C coders (my primary language), none of whom have either experience with, nor a need for, NPM, so we're not eager to claim any level of support for it. Trying to support a tool one neither uses nor understands is like trying to wave away darkness with one's hand.
> ... with a lot less effort.
A lot less effort on their own parts and a lot more on ours ;).
Future NPM support has not been outright ruled out, but it is admittedly nowhere even close to currently being on our radar.
I do think it would be worth SQLite having an "official" npm package - even though setting it up in the first place would require quite a bit of effort.
The JavaScript world is culturally so dependent on npm now that adoption of SQLite in that world would massively increase given the availability of a good "official" package.
And without an official one I imagine there will quickly emerge dozens of unofficial ones, many of which will end up poorly maintained in the future.
> And without an official one I imagine there will quickly emerge dozens of unofficial ones, many of which will end up poorly maintained in the future.
Seeing as none of us in the sqlite project use NPM or node in any capacity whatsoever, nor do we have any interest in doing so, i'll opine that "unofficial" ones would be of much higher quality than any we would put out.
To repeat what i just said in another comment on this topic: Trying to support a tool one neither uses nor understands is like trying to wave away darkness with one's hand.
> It seems strange that they would make an official version and then not add it to the official JS package manager.
None of us in the sqlite use npm in any way, shape, or form, so there's nothing at all unusual about us not publishing anything there.
> It seems like Chrome is paying them for the “deliverable” of a JS version, but the spec did not include “npm installable”.
Like all deliverables released via the sqlite project, this one was conceived and created as a deliverable which can be distributed through the existing project-level infrastructure. What other people do with it, through whatever infrastructure they like, is entirely up to them. Getting involved with such tooling is way out of scope for us.
Hey, I am the maintainer of sql.js. I was in contact with them, and I do not think they are working on a npm module, but I offered them to work on the existing "official" sql.js module.
For example, it should be possible to build websites for downloading YouTube videos that does not use server side code execution or bandwidth. These were popular at some point but disappeared after Java applets stopped being supported by most browsers.
And make every website part of a botnet? I sure hope not.
WebSockets are already a problem because many web devs don't know that other origins can connect to them and potentially fetch data they're not supposed to (even ignoring the fact you can construct these without a browser), unleashing raw sockets to the web platform would be hell.
WebRTC and WebSockets provide more than enough already. If you need even more, your browser just isn't the place for this type of code IMO.
I my experience, normal users don’t read the permission prompts. They immediate click Allow or Accept and then wonder later why they’re getting popup notifications in Windows pretending to be an anti-virus or think their entire computer has been hacked because “something is wrong.”
Or... you can use a websocket gateway if you need this type of functionality... a lot of people already are for things like VNC or Terminal interfaces.
mmap would make WebAssembly a lot better, unironically. Even if it was only supported for read-only pages. A lot of cpu time and IO are wasted doing reads into temporary buffers that then get copied into the wasm heap.
The custom header requirement will make this version of sqlite unusable for many usecases. Can SQLite fallback to using slower persistant APIs if neither Cross-Origin-Opener-Policy or Origin-Embedder-Policy are set? If not, does the SQLite project intend to provide a version that would?
Devs who don't need the high performance but want wider compatibility will probably continue to use the "unofficial" versions.
Also, last I checked, you could definitely load web assembly in a page that has a File:/// origin.
My use case is an entirely offline webapp that I'm building as a side project, partly just to see what is possible. I've tried indexeddb, pouchdb, absurd-sql, and various others, and have ended up back at localstorage, which is plain but low on edge cases.
The other big pain point I am hacking around is cross device sync; I'm leaning towards simple chrome and firefox extensions that'll slurp data into your profile in order to sync, which'll work for cross computer sync for my needs. However, chrome on android doesn't support extensions, and there's no other good way that I've found to sync between a user's chrome profile and android, so that is still a hole in the story. Some kind of webrtc based thing is imaginable, but it's clunky.
This is definitely better, yeah. The blog post[1] on absurd-sql notes that it's a hacky solution that would be improved with file system access; it references the old Storage Foundation proposal [2], which has since evolved into the current File System Access API proposal(s).
This is honestly very cool. VS Code uses a similar approach to their local file system provider, albeit with a wrapper around IndexDB instead of SQLite. There’s some interesting trade offs too, since IndexDB can store the browser’s native file handlers in a flat map — so there’s no need for a schema.
IMO, the Chrome team is being a bit deceptive with their phrasing on synchronous file handles. The problem is that the entire API is wrapped up in an asynchronous ceremony. `createSyncAccessHandle` is only available in a worker context. So you can only communicate with the worker using an asynchronous postMesssage event dispatcher. And even when you’re in the worker, file handles can only be accessed through methods that return a promise.
I understand the need for such boundaries when working with a single threaded language, but limiting the synchronous APIs to just workers seems like one too many layers of indirection. I recently attempted to write a POSIX-style BusyBox library and this sort of thing was a total show stopper.
The file handler is already tucked inside an asynchronous Promise based API. I think it’s reasonable that a mere attempt to get a handler synchronously is made possible. Whether to support synchronous read/write access is another matter.
I would be delighted if handlers were more akin to byte arrays that could be written and read synchronously, albeit with an asynchronous function to persist changes to the disk.
I have yet to see whether implementations allow users direct access via the filesystem to these contents. The idea of each origin having some files sounds fine. But if I genuinely want to help & empower my users, give them the most user agency, I'd want them to be able to access the sqlite sb themselves directly, with whatever hazards they thereby assume.
From my reading previously, no the currently OPFS implementations do not expose the files as files in directories the user would access.
The File System Access API does give the webpage the ability access files on the users file system, with permission. However not at the block level, that is only enabled for the OPFS. In order to provide ACID compliance SQLite needs to be able to hold a lock to a file and write at the block level, which is only possible within the OPFS sandbox.
My use case is wanting a git implementation. I don't care about ACID, but I do care a lot about speed, and it absolutely is there 100% for the user's sake. So I feel shit-out-of-luck. Also, I'd rather use JS than wasm, and whatwg-fs/OPFS focused entirely on sync apis, to appease wasm folk who have cumbersome shims they didn't want to deal with.
It's frustrating that the web community first build File System Access, but made a number of design choices compromising performance. Then AccessHandles/whatwg-fs, which unlocks performance, but severely limits it's usability to things users can't really interact with. And only release a sync implementation.
I love the web platform, but this feels so unfortunate.
What's wrong with many tabs? The DB handles getting hit by many tabs correctly -- it's just as if many processes were hitting the db in a non-browser environment.
It's mostly works with OPFS but with a few edge cases. They are working with the browser vendors on improving this in OPFS.
Alternatively run it in a SharedWorker (https://developer.mozilla.org/en-US/docs/Web/API/SharedWorke...), which seems to have made a comeback after being dropped for security concerns. Or do some sort of leader election with browser tabs and use a BroadcastChannel.
SQLite WASM uses SharedArrayBuffers and Atomics to turn async OPFS apis into blocking APIs (really just _one_ api -- getSyncAccessHandle)
Given the use of SharedArrayBuffers and that SharedArrayBuffers can't be used in a shared worker (funny), SQLite WASM doesn't work in shared workers.
If getSyncAccessHandle was made synchronous then maybe everything would just work. It'd also improve SQLite WASM perf by 30% according to their measurements [1]
You can't write to SQLite from multiple processes either, so you mitigate that exactly the same way you do for a desktop app (i.e. show "Can't access DB for writing because other instance is running")
You can write from multiple processes and this is enabled by default (see PRAGMA locking_mode [1]). If you've got multiple writers you might have to occasionally handle SQLITE_BUSY and retry.
If your app cannot handle concurrency while your back-end i/o is via a machine-local singleton, that sounds like a a problem which will come up for more than just multiple tabs. If architecting around this known-fact is a challenge, an engineer may have bigger problems.
Does anyone have a good idea of when OPFS will be broadly available in all major browsers? And what's best to use as a pseudo-polyfill in the mean time?
Something relevant is absurd-sql, but unfortunately that's not even close to production ready :/
Interesting. Looks like the data persisted to disk should be available for opening via traditional SQLite GUI tools too (eg sqlitebrowser.org), though Chrome/Chromium would probably need to be not running at the same time for safety.
The sqlite.wasm is 683kB and the sqlite.js is 300kB. So is it supposed to be used in PWA context where these files are cached? Why no one is talking about CAP here. Is it real-production-ready?
It's an alternative to using IndexedDB which is a browser provided API.
On use cases, you can give your web app offline support by locally caching data in an SQL database and have it be fully queryable.
Say you are building an app like Notion, they already have "offline mode" for the mobile and desktop apps, this would enable you to build that for the web app.
This is very much one of the final jigsaw pieces needed to make PWAs (progressive web apps) competitive for the majority of use cases. We just need Apple to catch up and fill in a few other blanks too.
A design pattern that is beginning to emerge is "offline/local first". You design your app to fundamentally work offline, using things such as this, and the server component only works to synchronise clients. It's a bit like the design move to "mobile first" that happed 10 years ago, but going to another level.
So thick client applications get the last laugh in the end?
It reminds me of an old concept, that has a catchy name I can't recall, positing an endless cycle in systems evolution where local peripherals grow processing capability until someone notices and (re)centralizes it, only for the local processing capability to quietly begin growing again, and so on.
I haven't used IndexedDB in a while.. but my recollection is that it is basically SQLite. Not sure I see the advantage of this above and beyond what IndexedDB is already providing. What are the pros and cons of each approach?
> but my recollection is that it is basically SQLite.
SQLite, which has a SQL database engine, is, by definition, a relational database. IndexedDB is a non-relational, or noSQL, database. One can't be basically the same as the other :-)
It feels like you're assuming that Apple is a student here that forgot to do his homework, as opposed to a miscreant who is trying to block progress on purpose.
Apple wants to handicap web apps to encourage people to build for iOS.
That may or may not be to some extent true, however I find negativity isn't generally the best way to encourage people to do something you would like.
I think Safari has also made progress over the last couple of years, and there is indication of them being more receptive to PWAs. I'm going to cheerlead for that.
It's also worth noting that the EU and UK competition authorities are looking at forcing Apple to make changes to some of their rules around web browsers on iOS. They have a big stick, I don't, hence my nicer words.
> In our blog post Deprecating and removing Web SQL, we promised a replacement for Web SQL based on SQLite. The SQLite Wasm library with the Origin Private File System persistence backend is our fulfillment of this promise.
I'd venture to guess the best answer is "whatever the hell people used Web SQL for." Doesn't really answer the question, but alas.
Plenty. Suppose you're working on a cross-platform application, building for web will get even easier, as up until a few months ago there weren't even official WASM builds from SQLite (but some amazing community run ones such as sql.js!). As for why somebody would want that, an example: your browser and everyone's is probably always open when you're at a PC, attrition is next to none compared to having to download an application, it's great for even just trying something out. SQLite really makes user data storage trivial in most circumstances, key:value doesn't.
Perhaps this could be used to create more privacy focused apps where your data stays local and works offline. I'm not sure about the persistence though, like if you clear the cache does it wipe out the local database?
It lets you build static web applications that can manage relational data for your users that is stored on their own machines.
This is a really powerful capability. You could build the equivalent of full desktop applications - things like Word, Excel, Access, Evernote, task trackers, Blender... all without any server side storage mechanism at all.
True, but you have it backwards. Web app needs user to download and install a web browser first; desktop app does not need anything downloaded first. Both can be started by clicking a link.
Quite simple. I do not need to download a web browser at all to run the desktop app. Deployment of the desktop app is simpler and provides the same or even more functionality without adding a complex desktop browser and sandbox container to run the app. Why is this so hard to get across?
Let's hope Firefox starts to support the File System Access API soon.
For me, it is the last missing piece to make web apps as comfortable as native apps.
I use a lot of web apps I fine-tuned exactly to my liking. I do all my writing in this web app for example: https://www.gibney.org/writer - I hit F11 and boom! I am in distraction free writing heaven :)
If Firefox would support the File System Access API, it would be much more comfortable to load and save my writings.
Mozilla opposes the picker methods of the File System Access API. They are fully behind the Origin Private File System part, and actually have the SQLite demo from the article running in Firefox Nightly.
To hopefully add a bit of clarity: indeed, that page has OPFS flagged as positive under the label "File System." Further down the page is a "File System Access" entry which is flagged negative.
That's why I wrote this: "in true Google fashion Chrome team implemented and released at least three different APIs all having something to do with files." :)
I didn't want to dig through all of them on mobile when I was responding.
Both Safari and Mozilla are open (and implement/have implemented) to Origin Private File System. Because it gives web sites an access to a file system without a chance of escaping and damaging user data. Can't remember which spec introduced it, and it doesn't really matter at this point.
There was also Storage Foundation API but it was thankfully never shipped (it was like a third way of handling files or something),and also received negative responses: https://chromestatus.com/feature/5670244905385984
Yep, Chrome is the new IE. Only instead of ActiveX and small proprietary HTML additions and EEE, it's a full coup de tat on web standards and Embrace, Extend, Control.
So if people abandoned Chrome and Safari for Firefox, wouldn't Firefox be the new IE? I don't remember IE being open source and community-developed like Chromium which Chrome is based off of. I use Chromium on Linux and while there are a few things to complain about, it has been a pretty positive experience overall and worked better than Firefox.
What Mozilla/Firefox needs to understand is that end users do not care about standards purity/principles. If core functionality doesn't work in your browser, people simply aren't going to use it. This is why the web is increasingly broken on Firefox--nowadays, about 10% of the sites I visit are simply utterly broken or slow on Firefox but work fine on Chrome.
Hell, even the $LARGE_CORPORATION I work at has all its developer tooling (including documentation, custom extensions, etc.) on Chrome. I've reported countless Firefox-specific issues, often due to missing API support, because no one internally tests on anything other than Chrome. You have to meet your users where they are.
These are the same sort of arguments that surrounded ActiveX and IE, back in the day. When $LARGE_CORPORATION is stuck on Chrome years after they would like to be (and other fallout), it will have been entirely predictable. Good on you for continuing to report issues you see to FF, but to your final point, if everyone is going to a bad place, no, you do not have to go.
> What Mozilla/Firefox needs to understand is that end users do not care about standards purity/principles. If core functionality doesn't work in your browser, people simply aren't going to use it. This is why the web is increasingly broken on Firefox--nowadays, about 10% of the sites I visit are simply utterly broken or slow on Firefox but work fine on Chrome.
It's not broken because they are fussy on standards.
it's broken because their marketshare dropped so low companies that make money on developing web stuff are often not contractually obliged to even support it. Few of our clients have contracts where we're obliged to support any browser above 5% market share and FF is already below even that!
I agree. I'm developing a system which uses webapp to interact with device over USB. It works in Chrome. It's sad for me to not support Firefox but the only way is to ask user to run some binary which will open websocket on localhost to interact with USB device. This is strictly worse from security PoV. This is strictly worse from usability PoV (user have to download and install manually additional component). This introduces update hurdles (I need to develop reliable updater). So much friction and issues just because of purity stand which hurts user in the end.
> I'm developing a system which uses webapp to interact with device over USB. It works in Chrome. It's sad for me to not support Firefox
As a cautionary tale: the moment Firefox shipped support for a hardware API they immediately ran into sites abusing them. Of course, Chrome doesn't even hide them behind a user prompt. https://twitter.com/denschub/status/1582730985778556931 (and comment)
There are reasons Firefox opposes these standards. Chrome is neither your friend nor does it care for the web. Why are you so willing to give a carte blanche to a power struggle (Chrome/Web vs. Android) inside a web advertising company is beyond me.
That's why th ecautionary tale: which of the capabilities of those APIs whould be behind user consent and which shouldn't? At which point will the user just click "yes" without reading those consent pop ups?
> What Mozilla/Firefox needs to understand is that end users do not care about standards purity/principles.
They might when their files get exfiltrated or encrypted. The file system access API UX is far too easily abused to misleading users into granting access to things they didn’t intend.
Single issue user. Adblocking. Not because of some grand belief in the idea that Mozilla would be a better steward for the web like what many others here are advocating.
I have never seen a terminal solution that allows to have the terminal in fullscreen mode and still have a way to have a user defined writing area which can be resized on the fly.
I have experimented with various distraction free vim plugins but they were all fragile and cumbersome.
If you have a solution, a screenshot would be very interesting.
I haven't maintained a Vim config in a few years now (more of an Emacs man now), but I do remember using Goyo in college. Looking back at it, I think it might scratch your itch as far as Vim plugins go, it even allows you to resize the area on the fly.
Goyo is one of the plugins I experienced as brittle and cumbersome. IIRC it creates the centered text area by creating 9 windows. The outter 8 to center the one in the middle, which is used to edit the text.
In my attempts to use it, that created a bunch of issues.
does it have spellcheck? text-to-speech so you can have it read back to you? A browser window would have those things (on a mac, and I assume there's equivalents in windows)
Funny, Firefox on Mac sure doesn't integrate with the native text to speech, or offer it's own without plugins as far as I can tell. Gee it's almost like all browsers aren't exactly the same
The Web SQL standard was deprecated (by Mozilla) on the grounds of a single implementation.
The result today: A single SQL implementation with no standard (beyond SQLite) and therefore no easy way for anyone to create a compatible alternate implementation.
Thanks to Mozilla rejecting Web SQL we now get to run the exact version of SQLite we need, compiled to WASM and downloaded to the browser rather than being baked in and unable to upgrade.
SQLite not right for a project? DuckDB has a WASM build too.
A SQL standard never precluded a separate filesystem access standard and implementing databases on top of it.
But rejecting the Web SQL standard means that every SQL-using webapp will be tied to its own database - replicating the SQL incompatibility mess when we had a chance to finally enforce a little standardization based on current SQL standards - and require depending on WASM when WASM might have not been necessary.
> But rejecting the Web SQL standard means that every SQL-using webapp will be tied to its own database - replicating the SQL incompatibility mess when we had a chance to finally enforce a little standardization based on current SQL standards
Web SQL was dropped for precisely this reason:
> The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path.
Without a magic want to force multiple implementations, all of those web apps would have been tied to a single implementation anyway _and_ it would have been harder to update it so even after something was fixed you'd need to wait until the oldest browsers were updated before you could rely on it. That is possible, of course, but it's a recipe for problems with something like a SQL database.
This ignores the fact that SQL already has standards with multiple implementations and is very well established. You can have a basic portable subset - the problem was already very well specified at the offline level.
A simple SQL standard would therefore not have incurred an unnecessary dependency on SQLite (had WASM/filesystem existed at the time, I suspect we could have also supported Web SQL with it).
--
Bugs of course, exist with IndexedDb as well and will exist with these new WASM implementation.
> This ignores the fact that SQL already has standards with multiple implementations and is very well established. You can have a basic portable subset - the problem was already very well specified at the offline level.
This is only partially true, as you can see from the many apps which only support one SQL database even if they're using something like an ORM which is ostensibly portable. Sometimes that's caused by use of things like using extensions but in other cases it can be caused by differences in things like data validation and conversion or how transactions are processed. That doesn't mean that portability is impossible but everyone doing it seriously tests continuously against multiple implementations to make sure they aren't inadvertently introducing a dependency on one flavor. Both SQLite and MySQL are especially notorious for this because you'd have cases where code which was invalid would either be automagically converted (hopefully to what you wanted) or the error silently suppressed so you'd have code which only works, for example, on a database where DATE can be "0000-00-00".
As simple example, at the time Web SQL was proposed as a standard, SQLite did not support foreign key checks at all. It still does not implement full Unicode support so you'd have to require all implementations enable that extension with the same configuration or you'd have data-dependent application bugs.
I would not personally have minded terribly if SQLite was effectively standardized like this but I understand why people wanted to see n>1 implementations. The browser developers are going to be supporting things for many years and as soon as a new feature launches any and all behaviors it exposes will become an API which has to be managed carefully to avoid breaking someone's production application.
Standards are for platform implementors. Developers inevitably ignore standards, and instead code "empirically" to whatever the implementation they're developing against is willing to expose—both in terms of features (guaranteed by the standard or not), and in terms of bugs.
(Yes, those devs are doing a bad thing. They should only obey the letter of the standard, and not "look through" to what the implementation is doing. But we can't just intentionally break support for the stuff they build through this empirical-discovery-of-features-and-bugs process. That would just hurt the innocent users who just want to access their websites — which might very well be abandonware by the point that we discover and fix whatever standard-nonconformance bug such sites relied upon.)
What we can do to control developer behavior, is to impose incentives.
If there are multiple implementations of the standard in popular use in different browsers, each with different bugs, then developers will feel the need to support those different browsers, and so will test in them, find where the deviations from the standard cause breakage, and so end up coding portably.
But if all the implementations in use in different browsers have the same bugs (perhaps because they're all actually the same implementation), then even supporting multiple browsers won't do anything to break people's assumption on those bugs; and so people will write brittle code that depends on exactly those bugs being present to work, and will break even if the platform implementor just upgrades the implementation to a newer version without the bug.
This is the "bug-for-bug compatibility" that Microsoft worries so much about across Windows versions. It's a royal PITA to support, and W3C/WHATWG/etc have thusfar assiduously avoided allowing anything into the standards which has only one implementation, and therefore which devs could ever "lock onto" the bugs of with brittle implementations.
---
Also, this ignores that most of the "pragmatics" of managing a SQL DBMS are not specified by the SQL standard. SQL is not as standard as people think it is! ANSI SQL is mostly concerned with specifying how DML queries work; the DDL stuff is almost entirely a mess of implementation-defined syntax that happens to share some conventions, but with no standards guarantees.
> CREATE INDEX is a PostgreSQL language extension. There are no provisions for indexes in the SQL standard.
Do you really want to expose a SQL API where you can't create indices, because there's no portable syntax for creating indices? Would such an API even be useful?
If we really wanted a "web SQL API", the designers of said API couldn't just lean on the existing SQL standard; they would actually have to standardize what "web SQL" syntax is themselves — probably as a superset of ANSI SQL, but still laying out a bunch of additional requirements. It'd be similar in scope to WebGL. That's a big ask.
And even if they did that, that'd likely mean that they'd end up with no existing implementations of the standard! (Because it's not like you'd want to lock in all of SQLite's nonstandard DDL syntax choices as "the" WebSQL syntax, right? You'd want to think these out "from first principles", considering what's most "implementation-neutral.")
I don't think there was so much risk of relying on db behaviour much less bugs. There's a DML standard, breaking it would have had massive consequences far beyond the web platform - we could have relied on SQLite being broadly compatible within the simple INSERT/UPDATE/DELETE/SELECT subset (I think more could have been done, but this is enough for a start).
As for DDL, not much was initially required beyond CREATE TABLE, CREATE VIEW and CREATE (UNIQUE) INDEX (which in practice have a basic common syntax). IMHO the ask was not anywhere as complex as WebGL.
SQLite fits one generic model, and is a row level data store. DuckDB is a column store, and other DBs are designed to operate for specific use cases. This is open to all options.
The point is that SQLite is a brilliant genetic option, but should not be the only option. WASM and OPFS is the right route forward to enable application developers to use the best data store architecture for their use case.
Even within SQLite there are extension that would never have been an available to WebSQL that with this design are.
sqlite> create table numbers (val int);
sqlite> insert into numbers values (7);
sqlite> insert into numbers values (8);
sqlite> insert into numbers values (9);
sqlite> insert into numbers values ("hi mom, I'm a number!");
sqlite> select * from numbers;
7
8
9
hi mom, I'm a number!
I agree that SQLite is great, but do you really want to standardize on that behavior?
People will write applications that insert 41 characters into a varchar(40). And they will care when their sites break when constraints start to be enforced.
I certainly would agree that SQLite is exemplary but consider how many different things are handled under the aegis of the W3C: if you allow one exception, then you will have other people saying their favorite thing should also have one.
So how many implementations of widevine are around? Web standards are well past the point where anyone involved could claim the procedural, moral or intellectual high ground.
That provides a standard way for web developers to say “Our audio/video files are encrypted. Use this decoder to decrypt the bytes for playback.”. Wide Vine is a popular Content Decryption Module used by EME but it's not the only one (Apple devices use FairPlay, Microsoft ships PlayReady) and the standards process required interoperability to be demonstrated.
Web SQL's standardization process halted precisely because nobody was interested in building a second implementation. If Web SQL had a second implementation, it could have been standardized just like EME was.
A SQL-level standard wouldn't do much because different apps will be using different database schemas anyway. (It's similar to how two applications that both use JSON format usually aren't going to be compatible, because they use it to store different things.) Only generic, schema-independent database tools and libraries would be helped.
For a portable application file format, you'd want to pick a SQL implementation (probably SQLite), decide on a schema, and probably wrap it in a library, but once you do that it could be used in multiple apps.
Blasphemy! More fun to deal with compatibilities of running in different browsers. Seems like we have returned full circle (25 yrs later) to a new form of ActiveX plugins.
Or an alternative read of the result today: A simple implementation of filesystem access, on top of which any number of database (SQL or not) can be implemented, thus allowing for more innovation in the database space.
That's the difference between a platform and an application using that platform. If SQLite is baked into your browser, you can only use the subset of functionality supported by the oldest / weirdest browser which your users use.
Think about the DOM APIs: jQuery was super popular in the 2000s. Say all of the browsers had decided to bundle it like WebSQL bundled SQLite. Would you be happy using a copy of jQuery 1.7 until all of Apple, Microsoft, Mozilla, and Google shipped 1.8 and all of your users upgrade or would you prefer to update your site to reference the version with the features you want to use? (or switch to a different framework in 5 years)
Browser developers always have to balance the tension of adding something new with the knowledge that they're going to need to support it for long periods of time. WASM is a good compromise position because it reduces performance as the reason why something should live in the browser engine rather than being shipped as an app resource.
>jQuery was super popular in the 2000s. Say all of the browsers had decided to bundle it like WebSQL bundled SQLite.
This ignores the fact that SQL already has standards with multiple implementations and is very well established. It's not a recent-ish JS library. It's not new tech. It's an almost 50 year old tech predating the internet.
Standardizing on a basic standard subset using the current SQL standards would have been enough for nearly all apps, and would have allowed alternate implementations without any real SQLite path dependency since SQL already has standards and well trotted paths for implementing them (at least a basic subset).
[EDIT: I don't wish to spam the comments. If you wish to reply to this, please use the other thread you also replied to? I should have done it myself, sorry about that.]
The web is an open platform accessible to all and anyone. Mobile platforms in comparison are, depending on which one, either a walled garden or nearly so. Web apps are better for humanity than the closed ecosystem of other platforms.
Yes, iOS is a black box hellscape. Android less so.
And webapps are the epitome of walled garden. The programs literally require a host server to function. All you get is front end.
I take your point on accessibility. And that is the main draw for a lot of people and devs I imagine: I can access the thing I like from nearly anywhere and anything!
And I can respect that. And it has its uses.
My complaint is that this won’t be a choice eventually for most things. It’ll be “use this web version or nothing”.
You are already seeing negative aspects of a lot of this even on desktop programs. Like look at Steam. It’s convenient. People have flocked to it.
If you are offline you lose access to everything. Unless a cached credential is still working.
If you lose access to your account, you’ve lost all your games. Blah blah.
I’ve had personal experiences with this. It blows.
It won’t stop it. People are fine till it happens to them. But then they forget it five seconds later and right back at it.
Because they end up migrating more and more functionality to their web side? Deprecating desktop or mobile app functionality or entirely.
So you end up with piss poor “web apps” that half the time require a specific browser to work remotely well and have kludged “offline” modes that break every other day.
Do not want. I know I’m probably in the minority but it’s damned irritating. The UX on most is atrocious and you have to deal with a complete hodgepodge of interface styles and ideas and designs and also hope your browser keeps running smoothly after trying to use multiple more advanced webapps simultaneously.
Oh, but all these advancements on wasm and all will continue improving performance and fixing problems!!!
… you mean stuff that was already solved with native programs? “But you can work on your stuff from anywhere with just an account!”
Oh, like a native app’s data synchronization?
But but…
It would be nice if we had both and both were equals. But that isn’t the case. It won’t be the case. And most companies see this as a wonderful way to vendor lock in even more than before and implement more and more IAP and subscription crap.
I’m obviously not talking about programs who’s primary function is some sort of network usage.
But if it’s a desktop app that does desktop things but still requires a network connection for a completely inane reason I don’t use it. I hate that shit. That’s the same sort of awful as a “webapp”.
And I knew people would start with the “that’s actually no different than desktop programs requiring specific versions of the apps or toolkit. Therefore your complaints are invalid.”
Difference here is I can keep my program. And use it via emulation or virtualization if necessary.
Webapps are another trend towards a “you own nothing. You rent everything.” Economy with computer software.
It’s shit.
And I’ll add that webapps /have/ their place. And I obviously use some.
My big gripe is, like with everything with humans, there’s always this push to extremes. It’s not “let’s use webapps where they make sense” it’s “let’s try to convince everyone webapps are perfect and we need everything to be that!!” cough ChromeOS trash
So if it would stay reasonable where traditional and web and even hybrids can coexist that’s fine.
It won’t. It never does. And I am not looking forward to the inevitable future of Always Connected being a requirement for literally everything. And you have to pay monthly fees for every little thing. And you lose access to your supposed data of you let you account lapse briefly. Blah blah.
It’s already happening and if you claim otherwise you are being willfully blind.
But people love the touted “convenience” and eat it up. And poo poo naysayers.
When it is a desktop or even a mobile app I as the end user have a level of control over my environment that I don't with a webapp. In addition, my interface doesn't change at the whim of the company or developer.
There's one problem though, this new API is for the web, so the nature of this storage is temporary - obviously the user must be able to clear it when clearing site data and this what makes it currently an unviable solution for a persistent extensions data storage. https://bugs.chromium.org/p/chromium/issues/detail?id=138321...