Embedding User-Defined Indexes in Apache Parquet
https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/
#HackerNews #Embedding #User-Defined #Indexes #in #Apache #Parquet #ApacheParquet #UserDefinedIndexes #DataFusion #BigData #Analytics
Improvement of the disk manager configuration on #apache #datafusion by introducing a builder.
Achievement Unlocked! Opteryx now has its own dialect for DataFusion’s SQL parser! 
Why settle for one SQL flavor when you can have the best from all worlds? With our custom dialect, we’re blending the greatest hits from different SQL languages—because your queries deserve both power and flexibility. 
One parser to rule them all. Game on. 
#OpenSource #SQL #Databases #Python #FOSS #DataFusion
https://github.com/mabel-dev/opteryx
Combining relatively low accuracy drone data with old but high quality Lidar gives insights for planning walking tracks for locals. #drone #lidar #datafusion #mappingg
I wrote a blog post on how to write user defined functions in Apache DataFusion. This includes how you can write Rust backed Python functions that operate at full native speed with zero copy operations of the data structures. Switching from pure python functions to these types of UDFs can lead to 10x speed improvements.
https://datafusion.apache.org/blog/2024/11/19/datafusion-python-udf-comparisons/
The Apache Arrow ecosystem is an amazing enabler for a whole new range of database systems. Tonbo is a K/V store that uses Arrow+Parquet on storage systems ranging from OPFS (browser filesystem) to S3. And of course you get SQL for free with Apache DataFusion! #ApacheArrow #DataFusion #RustLang
https://tonbo.io/blog/introducing-tonbo
Awesome reading list about Apache DataFusion. I started diving into it lately, hacking an extension to have Elasticsearch as a data source (aka a TableProvider). It's a wonderful piece of software and an impressive ecosystem. #DataFusion #RustLang https://datafusion.apache.org/user-guide/concepts-readings-events.html
@dom @maxroser @ourworldindata
If you folks are not yet familiar with dynamicland.org's work take a look at this work (focus on 24-25mins) from @bret (in 2018)
I can see this approach having multiple synergies for the work you do.
For example in helping engage people with data and help them explore it (the point made at 25m), as well as provide a powerful way for academics and professionals to work with, integrate (#DataFusion) and understand the meaning of data.
Very nice introduction to Apache DataFusion and its internals. I started to dive into it lately, and it's really a wonderful piece of software. #datafusion #rustlang https://www.youtube.com/watch?v=iJhRbDFJjbg
And the code is up! Still an early prototype, but we can write SQL queries with DataFusion that join Elasticsearch indices with other sources like SQL databases, CVS files, Parquet data on S3, etc 
DataFusion is powerful and fun to use! #elasticsearch #datafusion #rustlang https://github.com/swallez/elasticsearch-datafusion-tableprovider
Starting a "space time" week at Elastic, where we can work on whatever we want that is loosely related to our products. For me, this will be experimenting with Apache DataFusion to see how we can use/integrate it with Elasticsearch.
#elasticsearch #datafusion
I recently worked on an update to the Apache DataFusion project. The goal of this project is to provide a fast, modular approach to building large scale data processing.
The update adds in significant improvements to the python interface, to the point where I would now recommend giving it a try to people who haven't used it or had success.
I wrote up a blog post about the changes, hosted on the Apache DataFusion site: https://datafusion.apache.org/blog/2024/08/20/python-datafusion-40.0.0/
Airbus UpNext Testing New Technologies for Automatic Taxiing, Pilot Assistance
I put in an open source PR today!
I've been playing around with Apache #DataFusion and wanted to see how it performed against a real life non-trivial problem I have. I ran into a blocker in that one of the basic functions that is exposed in the rust code underneath isn't exposed in python, so I just wrote it myself and put up a PR. I've never really been much of an open source contributor, so looking forward to seeing how this goes.
ROAPI automatically spins up read-only APIs for static datasets without requiring you to write a single line of code. It builds on top of #ApacheArrow and #Datafusion (and #rust).
Realized that I botched my #introduction (#neuhier halt)
Doing a #PhD in #biostatistics in #switzerland, focusing on experimental #design, small sample #statistics and #datafusion in #animalresearch.
Managing a #grassroots #thinktank called #Reatch! #Research. #Think. #Change.
Engaged in #scicomm, #sciencedialogue, #science and #politics relations.
Interested in #sts, #epistemology of #statistics and the use of #statistics and #science in #democracy and #democratic discourse.
