Importing Data with XplainPy

This tutorial covers getting data into Xplain as XTables — from a relational database or from CSV files and pandas DataFrames — and assembling those XTables into the object hierarchy used for analysis.

On this page

Concepts 

Before running any code it helps to understand the four building blocks of a Xplain data import:

Concept	Description
Connection	JDBC credentials for a source database (PostgreSQL, MySQL, Oracle, …). Created once, saved to a file, reused across imports. Not needed when importing from CSV or a DataFrame.
XTable	A binary fact table produced by importing one database table, SQL expression, or CSV file. Columns become dimensions; one column is the primary key.
XAttribute	A hierarchical dimension added on top of an XTable (e.g. a 3-level geography hierarchy stored in a lookup table).
XView	A startup configuration that assembles one or more XTables into an object hierarchy (parent → child), ready to load in XOE or via Python.

The Importer class is the main entry point. It provides factory methods for all four building blocks and keeps the session wired to every object it creates.

Xsession
    └── Importer
            ├── create_connection()      →  Connection
            ├── create_xtable_config()   →  Xtable_config  →  import_xtable()
            ├── create_dataframe_xtable_config() → DataFrameXtableConfig → import_xtable()
            ├── create_xattribute_config() → Xattribute_config → import_xattribute()
            └── create_xview()           →  Xview  →  save()  →  startup()

Choosing an approach 

Approach	Use when	Entry point
Database table	The data lives in a reachable database; the server pulls it directly	`importer.create_xtable_config(connection=...)`
DataFrame, one call	Quick CSV import, automatic column types are fine	`importer.import_dataframe(df, object_name=...)`
DataFrame, with config	You need to rename columns, override types, or mark time dimensions	`importer.create_dataframe_xtable_config(...)`
Saved configuration	A `.xtableconfig` already exists and should be reused	`Xtable_config(xsession=..., load_from=...)`
Chunked upload	The CSV is very large (multiple GB) and a single request would time out	`config.import_xtable_in_chunks()`
Persisted CSV import	The XOE UI should be able to reload and re-run the import later	`config.save_as_csv_import(...)`

The CSV routes all go through pandas, so anything pandas.read_csv can parse can be imported — including compressed files, custom separators and explicit dtypes.

Prerequisites 

xplain package installed (pip install xplain)
An Xplain server is running and accessible
For a database import: a source database reachable from the Xplain server

from xplain import Xsession
from xplain.tools import Importer

session = Xsession(url="http://myhost:8080", user="admin", password="secret")
importer = Importer(session)

Importing from a database 

Step 1 — Create a database connection 

Pass your JDBC credentials to create_connection(). Call test_connection() immediately to validate them before running a full import.

conn = importer.create_connection(
    databaseType="POSTGRESQL",
    url="db.example.com",
    user="analyst",
    password="dbpassword",
    databaseName="warehouse",
    portNumber=5432
)

result = conn.test_connection()
print(result)   # {'status': 'success'} or {'status': 'error', 'message': '…'}

Save the connection for reuse

Saving avoids storing credentials in every script. Future imports can load by name instead of re-entering credentials.

conn.save("warehouse_pg", "PUBLIC")

Load a saved connection

conn = importer.create_connection(load_from="warehouse_pg")

Explore the database schema

# List all tables in the connected database
tables = conn.get_tables()
print(tables)

# Preview the first 10 rows of a table
df = conn.get_table_content("orders", limit=10)
print(df)

Step 2 — Configure and import an XTable 

Create an Xtable_config by pointing it at a table. The server auto-detects column types and populates dimensionConfigurations.

config = importer.create_xtable_config(
    connection=conn,
    db_table_name="orders",
    object_name="Orders",
    primary_key="order_id",
    foreign_keys=["customer_id", "product_id"]
)

Review the auto-detected mapping

print(config.show_dimension_configurations())
#    dbColumnName   dimensionName  xplainDataType  primaryKey  foreignKey
# 0  order_id       order_id       LONG            X
# 1  customer_id    customer_id    LONG                        X
# 2  product_id     product_id     LONG                        X
# 3  order_date     order_date     CATEGORIAL
# 4  quantity       quantity       INT
# 5  total_amount   total_amount   DOUBLE

Adjust dimension names and types

Column names from the database are used as-is. Override them to match your analytical naming conventions:

from xplain.tools import XplainType

# Rename dimensions
config.set_dimension_name("order_date", "OrderDate")
config.set_dimension_name("total_amount", "Revenue")

# Override types where auto-detection is wrong
config.set_dimension_xplain_data_type("quantity", XplainType.INT)

# Mark a date column so Xplain treats it as a time axis
config.set_time_dimension("OrderDate")

# Drop a column that is not needed for analysis
config.remove_dimension("internal_flag")

See xplain.tools.XplainType for the full list of dimension types and what each is for.

Import the XTable

result = config.import_xtable()
print(result)   # {'status': 'success'}

Save the configuration for later re-imports

config.save("orders_import", "PUBLIC")

# Reload and re-run at any time:
config2 = importer.create_xtable_config(load_from="orders_import")
config2.import_xtable()

Step 3 — Import child tables 

Repeat Step 2 for every table that will appear as a child in the object hierarchy. Use foreign_keys to declare which columns link to parent tables.

# Customers table  (root's child)
customers_cfg = importer.create_xtable_config(
    connection=conn,
    db_table_name="customers",
    object_name="Customers",
    primary_key="customer_id"
)
customers_cfg.import_xtable()

# Products table  (root's child)
products_cfg = importer.create_xtable_config(
    connection=conn,
    db_table_name="products",
    object_name="Products",
    primary_key="product_id"
)
products_cfg.import_xtable()

Importing from CSV or a DataFrame 

No database connection is involved here — the CSV is the data source. Read it with pandas and hand the DataFrame to the importer:

import pandas as pd

df = pd.read_csv("payers.csv")

result = importer.import_dataframe(
    dataframe=df,
    object_name="Payers",     # the XTable name in Xplain
    primary_key="Id",
)
print(result)                 # {'status': 'success'}

This creates Payers.xtable on the server. import_dataframe returns a dict with status — either 'success' or 'error' plus a 'message' — rather than raising, so check it instead of assuming success.

For CSVs beyond a few hundred megabytes, pass use_multipart=True to send the data as a file upload instead of an inline JSON payload.

How columns become dimensions 

Column types are inferred from the DataFrame’s dtypes:

pandas dtype	Xplain type	What happens to the values
`int64`, `uint32`, `uint64`	`LONG`	unchanged (`uint32` does not fit a signed `INT`, so it widens)
`int8`, `int16`, `int32`, `uint8`, `uint16`	`INT`	unchanged
`float64`	`DOUBLE`	unchanged; `NaN` becomes an empty field
`float32`	`FLOAT`	unchanged
`bool`	`BOOLEAN`	unchanged
`datetime64`	`LONG`	converted to Unix milliseconds; a tz-aware column is converted to UTC first (`NaT` becomes an empty field)
`timedelta64`	`LONG`	converted to milliseconds
`object` / string / `category`	`CATEGORIAL`	unchanged

Pandas nullable dtypes (Int64, Float64, boolean, string) map to the same types as their plain counterparts, so a frame that went through convert_dtypes() or pd.read_csv(..., dtype_backend="numpy_nullable") imports the same way.

Inference covers the common cases; XOE itself supports more dimension types than the ones produced here — BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, BOOLEAN, STRING, CATEGORIAL, CATEGORIALBYTE and CATEGORIALSHORT (see xplain.tools.XplainType for what each one is for). Any of them can be set explicitly with set_dimension_xplain_data_type or the type_overrides argument, for example to store a low-cardinality code as CATEGORIALBYTE or a small counter as BYTE.

Important

Creating the configuration modifies the DataFrame you passed in. Only the types with no direct CSV representation are rewritten in place, so afterwards the datetime and timedelta columns of your own df hold integers instead of timestamps. Numbers, booleans and text are left alone. Pass df.copy() if you still need the original values — bearing in mind that a copy doubles the memory for a large frame.

Always review the mapping before importing:

config = importer.create_dataframe_xtable_config(
    dataframe=df, object_name="Payers", primary_key="Id"
)
print(config.show_dimension_configurations())

Controlling the mapping 

Build the configuration, adjust it, then import. This is the same object import_dataframe creates internally:

from xplain.tools import XplainType

config = importer.create_dataframe_xtable_config(
    dataframe=df,
    object_name="Payers",
    primary_key="Id",
    type_overrides={"ZipCode": "CATEGORIAL"},   # override inference upfront
)

# Rename a column for analysis (matches on the CSV column name)
config.set_dimension_name("pyr_nm", "PayerName")

# Override a type later; XplainType members and strings are interchangeable
config.set_dimension_xplain_data_type("RegionCode", XplainType.CATEGORIAL)

# Mark a column as a time instant so Xplain can build time axes on it
config.set_time_dimension("ContractStart")

result = config.import_xtable()

set_dimension_name and set_dimension_xplain_data_type both match on the CSV column name, not on a dimension name you may already have assigned.

Note

DataFrameXtableConfig can only be created from a DataFrame — it cannot load a stored configuration. That is what the next section is for.

Reusing a saved configuration 

Once a configuration is stored on the server it can be loaded back and applied to another CSV — useful for a recurring import of the same file shape. Only Xtable_config can load; the .xtableconfig extension is added automatically:

from xplain.tools import Xtable_config, XplainType

config = Xtable_config(xsession=session, load_from="trace_boards")

# Check what came back before importing anything
print(config.object_name)       # the XTable this config writes to
config.show_dimension_configurations()

config.set_dimension_xplain_data_type("PanelUId", XplainType.CATEGORIAL)

with open("trace_boards.csv", "rb") as f:
    config.import_xtable_from_csv(f)

Four things to watch for:

load() reads PUBLIC configurations only. A configuration saved as PRIVATE is silently not found, and the fields stay unchanged.
The stored column mapping is sent as it is — it is not re-matched against the CSV header. If the CSV’s column names changed since the configuration was saved, the import fails or maps columns wrongly.
object_name decides which XTable is written. Override it before importing if this CSV should populate a different table.
Open the file in binary mode ("rb").

To go the other way and import a DataFrame through an existing configuration, use config.import_xtable_from_dataframe(df).

Linking several CSVs into a hierarchy 

Xplain’s analytical power comes from parent-child relationships between objects. Import each CSV as its own XTable, declaring which column links it to its parent. The parent’s primary_key and the child’s foreign_keys must name the same identifier:

customers = pd.read_csv("customers.csv")   # CustomerId, Name, City
orders    = pd.read_csv("orders.csv")      # OrderId, CustomerId, Amount

importer.import_dataframe(customers, object_name="Customers",
                          primary_key="CustomerId")

importer.import_dataframe(orders, object_name="Orders",
                          primary_key="OrderId",
                          foreign_keys=["CustomerId"])

Both XTables now exist but nothing yet says which is the parent. That is the XView’s job — continue with Building the XView below, which applies identically no matter whether the XTables came from a database or from CSV files.

Building the XView 

An Xview assembles the XTables into the parent–child object hierarchy that Xplain uses during analysis. insert_xtable with as_root=True sets the root; subsequent calls attach children. The parent’s primary_key and the child’s foreign_keys must refer to the same identifier — that link is what makes the hierarchy work, and it is declared during import, not here.

view = importer.create_xview()

# Root object
view.insert_xtable(
    xtable="Orders",
    as_root=True,
    auto_generate_attributes=True
)

# Child objects (linked via foreign keys declared during import)
view.insert_xtable(xtable="Customers", as_root=False, parent="Orders")
view.insert_xtable(xtable="Products",  as_root=False, parent="Orders")

# Save the XView as a startup configuration
view.save("orders_analysis", "PUBLIC")   # → orders_analysis.xstartup

The example uses the tables from the database walkthrough. It is identical for CSV-sourced XTables — with the two from Linking several CSVs into a hierarchy it would read:

view = importer.create_xview()
view.insert_xtable(xtable="Customers", as_root=True)
view.insert_xtable(xtable="Orders", as_root=False, parent="Customers")
view.save("customer_orders", "PUBLIC")

Key columns are deliberately not turned into analysable attributes — they identify rows rather than describe them.

Optional: add numeric range attributes

view.add_attribute_intervals(
    intervals_dict={"Revenue": (0, 100, 500, 1000, 5000, 50000)},
    object_name="Orders",
    attribute_name="_Group"
)
view.save("orders_analysis", "PUBLIC")

Load the XView for analysis

session.startup("orders_analysis")
session.show_tree()

To load a view that has not been saved, hand the object straight to the session instead:

session.http_post("/init_xplain_session")     # clear the current session
session.startup_from_xview_config(xview_config=view)

See Data Import Tools Reference for the full XView API, including normalized dimensions and hierarchical interval attributes.

Advanced options 

Filter rows during import 

Use where_clause to import only the rows you need. The clause is applied server-side so no unnecessary data is transferred.

config = importer.create_xtable_config(
    connection=conn,
    db_table_name="orders",
    object_name="RecentOrders",
    primary_key="order_id",
    foreign_keys=["customer_id"],
    where_clause="order_date >= '2023-01-01' AND status = 'completed'"
)
config.import_xtable()

Import from a SQL expression 

Replace the table name with an arbitrary SQL query. Useful for views, joins, or computed columns that do not exist as a plain table.

config = importer.create_xtable_config(
    connection=conn,
    object_name="CustomerRevenue",
    primary_key="customer_id"
)
config.add_db_table_sql_expression("""
    SELECT
        c.customer_id,
        c.country,
        SUM(o.total_amount) AS lifetime_revenue,
        COUNT(o.order_id)   AS order_count
    FROM customers c
    JOIN orders o ON c.customer_id = o.customer_id
    GROUP BY c.customer_id, c.country
""")
config.import_xtable()

Add a derived dimension 

Use add_dimension_with_sql_expression to compute a new column on the fly without modifying the source database.

config = importer.create_xtable_config(
    connection=conn,
    db_table_name="orders",
    primary_key="order_id"
)
config.add_dimension_with_sql_expression(
    "OrderYear",
    "EXTRACT(YEAR FROM order_date)",
    "LONG"
)
config.import_xtable()

Sample rows for testing 

Import a fixed number of rows to validate the configuration before a full run.

config = importer.create_xtable_config(
    connection=conn,
    db_table_name="events",
    primary_key="event_id",
    sampling_rate=10000
)
config.import_xtable()

Very large CSV files 

A single request has to complete within the server’s timeout. For CSVs of several gigabytes, upload in chunks instead: the configuration is registered once, the data is sent in ordered pieces, and the XTable is built at the end.

config = importer.create_dataframe_xtable_config(
    dataframe=df, object_name="Measurements", primary_key="Id"
)

def on_progress(index, total):
    print(f"  chunk {index + 1}/{total}")

result = config.import_xtable_in_chunks(
    chunk_size_mb=100,
    progress_callback=on_progress,
)

Each chunk is a separate request, so any individual request stays well within the timeout regardless of total size. Chunks must arrive in order, which means this is sequential by design and not parallelizable.

Note that the DataFrame is serialized to CSV in memory before chunking, so the machine running the import still needs to hold the data — chunking solves the request timeout, not local memory pressure.

Persist a CSV import for the XOE UI 

A DataFrame import creates the XTable but leaves nothing behind for the XOE UI to reload. To make it reproducible from the UI, three artifacts must exist on the server:

the raw CSV file, in XOE’s IMPORT_CSV storage,
a CsvConnection file (.xdbconnection) describing the CSV’s format,
an .xtableconfig that references that connection file.

from xplain.tools import CsvConnection

df = pd.read_csv("payers.csv")
config = importer.create_dataframe_xtable_config(
    dataframe=df, object_name="Payers", primary_key="Id"
)
config.import_xtable()

# 1 — upload the raw CSV
with open("payers.csv", "rb") as f:
    session.upload_csv_file("payers.csv", f.read(), ownership="PUBLIC")

# 2 — describe the CSV format
CsvConnection(xsession=session, dataframe=df).save("CSV_payers", "PUBLIC")

# 3 — save the xtableconfig referencing that connection
config.save_as_csv_import(
    file_name="payers",
    csv_filename="payers",
    connection_file_name="CSV_payers.xdbconnection",
    ownership="PUBLIC",
)                                   # → payers.xtableconfig

The naming convention CSV_<table>.xdbconnection is what the UI expects. Use plain config.save(...) instead if you only need the configuration back in Python — it omits the CSV connection reference, so the UI cannot re-run it. The configuration written here is exactly what Reusing a saved configuration loads back.

Add a hierarchical attribute (XAttribute)

XAttributes add a multi-level dimension sourced from a lookup table — for example, a three-level product category hierarchy stored in a product_hierarchy table.

attr_cfg = importer.create_xattribute_config(
    connection_file="warehouse_pg",
    db_table_name="product_hierarchy",
    attribute_name="ProductCategory",
    primary_key="product_id",
    hierarchy_level_names=["Department", "Category", "SubCategory"],
    hierarchy_level_column_names=["dept_name", "cat_name", "subcat_name"]
)
importer.import_xattribute(attr_cfg)

The same hierarchy can come from a CSV instead, via CsvAttributeConfig. The CSV and its connection file must be uploaded first, exactly as in Persist a CSV import for the XOE UI:

from xplain.tools import CsvAttributeConfig

levels = pd.read_csv("product_hierarchy.csv")

attr = CsvAttributeConfig(
    xsession=session,
    dataframe=levels,
    attribute_name="ProductHierarchy",
    primary_key="ProductId",
    hierarchy_level_column_names=["division", "group", "product"],
)
attr.import_attribute(
    csv_stem="product_hierarchy",
    connection_file_name="CSV_product_hierarchy.xdbconnection",
)

Inspect what is already imported 

# List saved connections
print(importer.list_saved_connections())

# List imported XTables
print(importer.list_xtables())

# List saved XTable import configurations
print(importer.list_xtable_configs())

# List imported XAttributes
print(importer.list_xattributes())

# Retrieve the original config used to create a specific XTable
print(importer.get_config_of_xtable("orders.xtable"))

Troubleshooting 

XplainInvalidRequestException: Key dimensions (parameter dbForeignKeys) are missing: The configuration reached the server without its key columns. Check that primary_key (or foreign_keys) is actually set — print(config.to_json()) shows what will be sent, where the keys appear as dbKey and dbForeignKeys.
Could not convert database column name ...: A column named in the configuration does not exist in the data. When reusing a saved configuration, compare config.show_dimension_configurations() against df.columns — the stored mapping is not re-matched automatically.
AttributeError: 'str' object has no attribute 'get': Something treated a raw HTTP response as a parsed dict. http_post returns the body as a str and http_get as bytes; decode with json.loads first. perform returns a parsed dict already — see Core API Reference for the contract.
Import reports success but the data looks wrong: Check the dimension types first — config.show_dimension_configurations() lists the type of every dimension. A numeric column that was inferred as CATEGORIAL (because the CSV had it as text) is the most common cause; convert the column with pd.to_numeric before importing.
Numbers cannot be summed in the UI: The dimension is CATEGORIAL or STRING, which can only be grouped and counted. Override it with a numeric type, for example set_dimension_xplain_data_type(column, XplainType.DOUBLE).

Complete end-to-end example 

from xplain import Xsession
from xplain.tools import Importer, XplainType

# 1. Connect to Xplain
session = Xsession(url="http://myhost:8080", user="admin", password="secret")
importer = Importer(session)

# 2. Create and save a database connection
conn = importer.create_connection(
    databaseType="POSTGRESQL",
    url="db.example.com",
    user="analyst",
    password="dbpassword",
    databaseName="warehouse",
    portNumber=5432
)
conn.test_connection()
conn.save("warehouse_pg", "PUBLIC")

# 3. Import root XTable
orders = importer.create_xtable_config(
    connection=conn,
    db_table_name="orders",
    object_name="Orders",
    primary_key="order_id",
    foreign_keys=["customer_id", "product_id"],
    where_clause="order_date >= '2022-01-01'"
)
orders.set_dimension_name("total_amount", "Revenue")
orders.set_time_dimension("order_date")
orders.import_xtable()

# 4. Import child XTables
importer.create_xtable_config(
    connection=conn,
    db_table_name="customers",
    object_name="Customers",
    primary_key="customer_id"
).import_xtable()

importer.create_xtable_config(
    connection=conn,
    db_table_name="products",
    object_name="Products",
    primary_key="product_id"
).import_xtable()

# 5. Build and save the XView
view = importer.create_xview()
view.insert_xtable(xtable="Orders", as_root=True, auto_generate_attributes=True)
view.insert_xtable(xtable="Customers", as_root=False, parent="Orders")
view.insert_xtable(xtable="Products",  as_root=False, parent="Orders")
view.save("orders_analysis", "PUBLIC")

# 6. Load and start analysing
session.startup("orders_analysis")
session.show_tree()