Python Bridge

Use Ibex as a DSL inside a real notebook workflow

The ibex_pyarrow bridge evaluates Ibex code and returns a pyarrow.Table. The ibex_ipython extension adds %%ibex, %ibexfile, and %ibexreset, so notebooks can bind pandas or Arrow tables and ordinary Python scalars into Ibex, keep expensive table loads alive across cells, and keep plotting in Python.

Scalar bindings currently support Python int, float, bool, str, datetime.date, and datetime.datetime, with the last two mapped to Ibex Date and Timestamp.

This keeps Ibex focused on joins, aggregations, reshaping, and time-series work, while reusing pandas, matplotlib, and the rest of the notebook ecosystem for presentation.

Build the bridge with cmake --build build-release --parallel --target ibex_pyarrow, then start IPython from the repository root and load the extension.

# IPython / Jupyter
%load_ext ibex_ipython

import pandas as pd

trades = pd.DataFrame({
    "symbol": ["AAPL", "AAPL", "MSFT"],
    "qty": [10, 15, 7],
    "px": [101.2, 101.5, 299.8],
})
offset = 10

%%ibex --bind trades=trades --bind offset=offset --as pandas --out grouped
trades[
    select { total_qty = sum(qty + offset), avg_px = mean(px) },
    by symbol,
    order symbol
];

grouped.plot(kind="bar", x="symbol", y="total_qty");

Persistent Session

Load once, reuse across notebook cells

%%ibex keeps table-valued let bindings in a hidden Ibex session. Large CSV or parquet loads do not need to be repeated in every cell.

%%ibex --quiet
import "csv";
let train = read_csv("../../kaggle/data/train.csv", "<empty>");

%%ibex --as pandas --out bucket_summary
train[select { rows = count() }, by seconds_in_bucket, order seconds_in_bucket];

%ibexreset  # clear the hidden session

R Package

Use Ibex through native source or a lazy dplyr plan

The in-repo ibex package evaluates inline Ibex queries or .ibex files from R. Results come back as a data.frame by default, so the common ggplot2 path is immediate. For lower-level interop, format = "nanoarrow" returns the Arrow-backed result.

ibex_tbl() also records a supported subset of dplyr as an immutable native plan. show_query() reveals the generated Ibex, while collect() returns a tibble. Unsupported R calls cross one explicit fallback boundary and then remain in local dplyr; R closures never execute on Ibex workers.

Like the Python notebook path, ibex supports a persistent Ibex session, so large CSV or parquet loads can happen once and then be reused across later calls.

For literate reports, ibex can also register a real {ibex} knitr engine inside R Markdown, so Ibex chunks and ordinary R / ggplot2 chunks can share one named session.

R input bindings are available by copy for named data.frame tables and named scalar lists. Scalars currently support R integer, double, logical, character, Date, and POSIXct.

# R
library(ibex)
library(ggplot2)

ibex::register_knitr_engines()

sess <- create_session(
  plugin_paths = c("build-release/tools")
)

session_eval(
  sess,
  '
  import "csv";
  let iris = read_csv("data/iris.csv", "");
  '
)

summary <- session_eval(
  sess,
  '
  iris[
      select { mean_sl = mean(Sepal_Length), n = count() },
      by Species,
      order Species
  ];
  '
)

ggplot(summary, aes(Species, mean_sl)) +
  geom_col()

query <- ibex_tbl(trades, fallback = "error") |>
  filter(price > 10) |>
  group_by(symbol) |>
  summarise(total = sum(price), .groups = "drop")

show_query(query)
result <- collect(query)

CSV Options

Read delimited files with or without headers

The bundled CSV plugin supports custom delimiters and headerless files. When has_header is false, Ibex synthesizes column names col1, col2, and so on.

SQLite is also available through the optional ADBC plugin, so the same pipeline can start from a SQL query instead of a flat file when you want database-side staging first.

import "csv";

let raw = read_csv("examples/measurements.txt", "", ";", false)
    [select { station = col1, temp = col2 }];

Compile-Time Map

Generate feature families without leaving Ibex

Braced select and update blocks support a compile-time map form. It expands from earlier string-list let bindings into ordinary named fields before lowering, so runtime execution stays unchanged.

Use backtick aliases with ${name} interpolation and get(name) inside the map body to turn compile-time names into column references.

columns(table_expr) returns a one-column metadata table named name, so generated schemas can stay in Ibex instead of being rebuilt in notebook host code.

let trades = Table {
    symbol = ["AAPL", "AAPL", "GOOG"],
    price  = [150.0, 155.0, 140.0],
    fee    = [1.0, 1.5, 2.0]
};
let measures = ["price", "fee"];

trades[select {
    symbol,
    map m in measures => `avg_${m}` = mean(get(m))
}, by symbol];

let price_cols = ["ask_price", "bid_price", "wap"];
let book = Table {
    ask_price = [101.0, 103.0],
    bid_price = [99.0, 100.0],
    wap       = [100.0, 101.5]
};

book[update {
    map (i, a) in price_cols, (j, b) in price_cols
        where i > j
        => `${a}_${b}_imb` = (get(a) - get(b)) / (get(a) + get(b))
}];

let generated = columns(book);
generated[filter name != "row_id"];