So far, we haven't had any particular need for native support for parquet files. This type of import can be done satisfactorily via RayPy, and there is also a plugin that allows you to access kdb+ instances.
I am pleased to announce that the RayforceDB columnar database, developed by Lynx Capital Partners, is now an open source project.
RayforceDB is an implementation of the array programming language Rayfall (in the same way that kdb+ is an implementation of k/q), which inherits the ideas embodied in k and q. However, RayforceDB uses Lisp-like syntax, which, as our experience has shown, significantly lowers the entry threshold for beginners and also makes the code much more readable and easier to maintain. However, the implementation of K syntax remains an option for enthusiasts of this type of notation.
RayforceDB is written in pure C with a minimum of external dependencies, the executable file size does not exceed 1 megabyte on all platforms (tested and actively used on Linux, macOS, and Windows), and the executable file is the only thing you need to deploy to get a working instance. Additionally, it is possible to compile to Webassembly and run in a browser. However, in this case, automatic vectorization is not available.
RayforceDB was developed by a company that provides infrastructure for the most liquid financial markets. As you might expect, the company has extremely high requirements for data processing speed. The effectiveness of the tool can be determined by visiting the following link: https://rayforcedb.com/content/benchmarks/bench.html
The connection with the Python ecosystem is facilitated by an external library, which is available here: https://raypy.rayforcedb.com
RayforceDB offers all the features that users of columnar databases would expect from modern software of this kind. Please find the necessary documentation and a link to the project's GitHub page at the following address: https://rayforcedb.com
Here's a new kdb+ like time-series database and programming language. This new implementation is being tried to avoid well known kdb+ drawbacks (including its famous price tag) and to add many new features now missed by many kdb+ programmers. And, well, it's going to become an open-source project soon.
every instance can export its monitoring information as 9p virtual filesystem, which easily can be mounted from outside
we debug server code in BEAM, Erlang on Xen is a deployment platform, if instance crashed we simply restart it.
intruder has very few chances to find breaking in beneficial - there's no shell inside which gives only minimal chances to snatch control, instance simply will crash. also, having of no OS leaves no holes to dig deeper
You're correct, it's exokernel-like approach.
We gain simplicity, much better resource consumption characteristrics, manageability at large scale and much better instance mobility. And, well, security.
It mainly mean that instead of using a shell that runs /bin/sh and associated control commands (ls, cat, whatever), you've to bring your own shell code and call the functions yourself. I assume one would write such a loader in erlang that serves a webpage to query any content from the fs, database, etc.
Also, IPC seems to be mainly network based, which means latency. Some modern OS designs function with the same base ideas:
managed runtime, small codebase, fully contained processes but use system-local IPC and thus, do have multi-processing (instead of multi-nodes, or in fact, in addition to multi-nodes).
Maybe some of those should be written in a web-friendly language and ship and httpd for adoption (so far they've not been adopted as the cost of rewriting apps > using archaic OSes)
Ideally I'd see an OS with:
- above characteristics (singularity, plan9 like)
- Simple, fast, efficient filesystem (i.e. with features and performance as good as popular databases) - so you don't need a database server
- clustered resources that are language-aware: filesystem (database), cpu, memory are networked resources, but you get control from the code about what is executed on the same local instance (ie same physical system) and what can be shipped to "any instance" - this brings true, full elasticity. (all this is also a little plan9-ish but not exactly)
BEAM ("official" Erlang VM) starts much longer. Erlang On Xen has it's own VM (written from scratch), it was designed for run without OS and to srtart very fast, particularly.
When you sat in container, you're limited with libraries of host OS, and there still will be some issues with migration and resource management (on large scales)
yes, very soon we will start to give attention to writing application over this platform. there is some work still needs to be done in management stack, but it's manageable amount of work.
