Tutorial: CrossRegistry¶

The CrossRegistry allows you to conveniently get, aggregate, and label data stored at the CROSS data platform

Packages and data¶

In [1]:

# to manage your .env file, you can use the python-dotenv package. 
# Install it with pip if you haven't already:
from dotenv import load_dotenv
import os

# Import the CrossRegistry class from the crosscontract package
from crosscontract import CrossRegistry

# to manage your .env file, you can use the python-dotenv package. # Install it with pip if you haven't already: from dotenv import load_dotenv import os # Import the CrossRegistry class from the crosscontract package from crosscontract import CrossRegistry

Creating the CrossRegistry¶

To create the registry, you simply provide your username and password. Here we assume that your credentials are stored in a .env file and we extract them from there.

Note Do not store your credentials in GitHub!

In [2]:

# load the environment variables from the .env file
load_dotenv(".env")
username = os.getenv("CROSSUSER")

# create the registry using the environment variables
my_registry = CrossRegistry(
    username=os.getenv("CROSSUSER"), 
    password=os.getenv("PASSWORD")
)

# load the environment variables from the .env file load_dotenv(".env") username = os.getenv("CROSSUSER") # create the registry using the environment variables my_registry = CrossRegistry( username=os.getenv("CROSSUSER"), password=os.getenv("PASSWORD") )

Getting a variable¶

To get a variable, you need to know the name of the contract. To get on overview over your available contracts, you can use the contract_overview property.

In [3]:

my_registry.contract_overview.query("name.str.startswith('result_')")

my_registry.contract_overview.query("name.str.startswith('result_')")

Out[3]:

	name	title	description	contract_type
0	result_electricity_supply	Annual electricity supply	Annual electricity supply by year, technology,...	ValueVariable
1	result_storage_installed_volume	Installed storage volume	Installed storage volume of storage technologi...	ValueVariable
2	result_passenger_road_public_fec	Passenger road public transport final energy c...	Final energy consumption of passenger road pub...	ValueVariable
3	result_methane_consumption	Methane final energy consumption (FEC)	Methane final energy consumption by year, end-...	ValueVariable
5	result_total_system_costs	Total annual system costs	Total annual system costs by year, model and s...	ValueVariable
6	result_space_heat_energy_supply	Space heat useful energy supply	Useful energy supply of space heat by year, te...	ValueVariable
7	result_district_heat_energy_production	District Heat useful energy production	Useful energy production of distric heat by ye...	ValueVariable
8	result_passenger_road_private_fec	Passenger road private transport final energy ...	Final energy consumption of passenger road pri...	ValueVariable
11	result_liquids_consumption	Liquid fuels final energy consumption (FEC)	Liquid fuels final energy consumption by year,...	ValueVariable
15	result_elec_cons_typical_day	Hourly electricity consumption	Hourly electricity consumption by hours, end-u...	ValueVariable
16	result_carbon_price	Carbon price	Carbon price (marginal cost) by year, model an...	ValueVariable
21	result_h2_fec	Hydrogen final energy consumption (FEC)	Hydrogen final energy consumption by year, end...	ValueVariable
30	result_installed_capacity	Installed capacity of electricity generation t...	Installed capacity of electricity generation t...	ValueVariable
33	result_elec_supply_typical_day_net	Hourly electricity supply	Hourly electricity supply by hour, technology,...	ValueVariable
34	result_elec_cons_typical_day_net	Hourly electricity consumption	Hourly electricity consumption by hours, end-u...	ValueVariable
36	result_process_heat_energy_production	Process heat useful energy production	Useful energy production of process heat by ye...	ValueVariable
38	result_electricity_supply_monthly_net	Monthly electricity supply	Monthly electricity supply by year, technology...	ValueVariable
47	result_elec_supply_typical_day	Hourly electricity supply	Hourly electricity supply by hour, technology,...	ValueVariable
48	result_storage_output	Storage annual energy output	Storage energy output by year, technology, mod...	ValueVariable
52	result_electricity_consumption_monthly_net	Monthly electricity consumption	Monthly electricity consumption by year, end-u...	ValueVariable
56	result_liquids_supply	Liquid fuels supply	Liquid fuels supply by year, technology, model...	ValueVariable
58	result_electricity_consumption	Annual electricity consumption	Annual electricity consumption by year, end-us...	ValueVariable
60	result_h2_supply	Hydrogen supply	Hydrogen supply by year, technology, model and...	ValueVariable
61	result_freight_road_fec	Freight road transport final energy consumptio...	Final energy consumption of freight road trans...	ValueVariable
62	result_electricity_consumption_monthly	Monthly electricity consumption	Monthly electricity consumption by year, end-u...	ValueVariable
63	result_electricity_supply_monthly	Monthly electricity supply	Monthly electricity supply by year, technology...	ValueVariable
64	result_total_system_costs_nuclear	Total annual system costs	Total annual system costs by year, model and s...	ValueVariable
65	result_electricity_consumption_net_nuclear	Annual electricity consumption	Annual electricity consumption by year, end-us...	ValueVariable
66	result_carbon_emissions	Carbon emissions	Carbon emissions by year, end-use sector, mode...	ValueVariable
67	result_methane_supply	Methane supply	Methane supply by year, technology, model and ...	ValueVariable
68	result_electricity_consumption_net	Annual electricity consumption	Annual electricity consumption by year, end-us...	ValueVariable
69	result_electricity_supply_net	Annual electricity supply	Annual electricity supply by year, technology,...	ValueVariable

Given the name, you can add the variable to the registry or simply use dot notation. If you use dot notation, the registry will automatically add the variable to the registry.

In [10]:

res_elec_supply = my_registry.result_electricity_supply
res_elec_supply

res_elec_supply = my_registry.result_electricity_supply res_elec_supply

Out[10]:

CrossDataVariable(name=result_electricity_supply, filters=None)

Assessing data¶

Now that you have the variable, you can access the data by using its data attribute. Using the data attribute provides you the data stored at the platform as pandas dataframe.

In [5]:

res_elec_supply.data.head()

res_elec_supply.data.head()

Out[5]:

	model	scenario_group	scenario_name	scenario_variant	technology	country	year	unit	value
0	zengarden	cross202506	abroad-nores-full	reference	wind_on	CH	2035	TWh	0.013612
1	zengarden	cross202506	abroad-nores-full	reference	wind_on	CH	2040	TWh	0.006385
2	zengarden	cross202506	abroad-nores-full	reference	wind_on	CH	2045	TWh	0.000009
3	zengarden	cross202506	abroad-nores-full	reference	wind_on	CH	2050	TWh	0.000009
4	zengarden	cross202506	abroad-nores-full	reference	imports	CH	2025	TWh	43.057953

While the .data property provides access to the full dataset, the get_data method allows you to filter and aggregate the data, and to label items based on the information in the contract (and the references to the Cross Dimensions).

• Filtering is based on a dictionary with the key being the name of the column and the value a list with the allowed values
• Aggregation is also dictionary based. The key is the name of the column over which to aggregate and the entry is an integer to specify the aggregation level. 0 is the highest aggregation level, i.e., the level with as little as possible details.
• Labeling is based on the use_titles parameter. If set to true all columns will be automatically relabelled.
• Columns allow to narrow the list of columns in the dataframe provided. Note that the filter does not drop colums at all. Columns are always applied at the very end of the transformation.

In [6]:

res_elec_supply.get_data(
    filters={
        "scenario_name": ["abroad-res-full"],
        "year": [2050],
        "scenario_variant": ["reference"],
    },
    aggregation={"technology": 0},
    use_titles=True,
    columns=["model", "technology", "value"]
).pivot_table(index="model", columns="technology", values="value").round(1)

res_elec_supply.get_data( filters={ "scenario_name": ["abroad-res-full"], "year": [2050], "scenario_variant": ["reference"], }, aggregation={"technology": 0}, use_titles=True, columns=["model", "technology", "value"] ).pivot_table(index="model", columns="technology", values="value").round(1)

Out[6]:

technology	Curtailment	Electrochemical	Imports	Renewables	Storage	Thermal	Vehicle to grid
model
PowerCheck	NaN	0.2	10.7	64.9	6.0	3.8	NaN
SES	NaN	NaN	18.1	80.9	0.0	1.7	NaN
SES-ETH	1.2	0.0	13.3	77.8	6.8	4.0	0.0
STEM	1.6	0.2	13.6	78.4	7.2	6.2	NaN
SecMOD	NaN	NaN	32.4	81.0	8.0	18.3	NaN
SwissX	NaN	0.0	15.0	81.5	4.8	6.9	NaN
ZEN-Garden	NaN	0.0	47.1	76.9	11.6	5.6	NaN

Aggregation¶

Aggregation is more flexible than only using one aggregation level. In principle there are three ways to aggregate:

1. Provide a single level of aggregation for the aggregation level (as above): aggregation={"technology": 0}
2. Aggregate to given set of identifiers: E.g. aggregation={"technology": ["renewable", "thermal"]}
3. Aggregate everything to a given level except some identifiers that should be kept: {"technology": {"level": 0, "keep": ["hydro_dam", "hydro_run"]}}

Note that the list of identifiers has to include the original identifiers and not the label or title of the column items as they appears after use_titles=True.

For the aggregation by title assume the example with aggregation={"technology": ["renewable", "thermal"]}. This aggregates all sub-categories of renewable and thermal but leaves the remaining items untouched:

In [7]:

res_elec_supply.get_data(
    filters={
        "scenario_name": ["abroad-res-full"],
        "year": [2050],
        "scenario_variant": ["reference"],
    },
    aggregation={"technology": ["renewable", "thermal"]},
    use_titles=True,
    columns=["model", "technology", "value"]
).pivot_table(index="model", columns="technology", values="value").round(1)

res_elec_supply.get_data( filters={ "scenario_name": ["abroad-res-full"], "year": [2050], "scenario_variant": ["reference"], }, aggregation={"technology": ["renewable", "thermal"]}, use_titles=True, columns=["model", "technology", "value"] ).pivot_table(index="model", columns="technology", values="value").round(1)

Out[7]:

technology	Batteries	Curtailment	Hydrogen fuel cell	Imports	Methane fuel cell	Pumped hydro	Renewables	Thermal	Vehicle to grid
model
PowerCheck	3.8	NaN	0.2	10.7	0.0	2.2	64.9	3.8	NaN
SES	0.0	NaN	NaN	18.1	NaN	0.0	80.9	1.7	NaN
SES-ETH	1.3	1.2	0.0	13.3	0.0	5.4	77.8	4.0	0.0
STEM	3.4	1.6	0.2	13.6	0.0	3.8	78.4	6.2	NaN
SecMOD	3.5	NaN	NaN	32.4	NaN	4.6	81.0	18.3	NaN
SwissX	3.2	NaN	0.0	15.0	0.0	1.6	81.5	6.9	NaN
ZEN-Garden	5.1	NaN	0.0	47.1	NaN	6.5	76.9	5.6	NaN

Now suppose you want to aggregate everything to level 0 but want to have hydro technologies more disaggregated: {"technology": {"level": 0, "keep": ["hydro_dam", "hydro_run"]}}

In [8]:

res_elec_supply.get_data(
    filters={
        "scenario_name": ["abroad-res-full"],
        "year": [2050],
        "scenario_variant": ["reference"],
    },
    aggregation={"technology": {"level": 0, "keep": ["hydro_dam", "hydro_run"]}},
    use_titles=True,
    columns=["model", "technology", "value"]
).pivot_table(index="model", columns="technology", values="value").round(1)

res_elec_supply.get_data( filters={ "scenario_name": ["abroad-res-full"], "year": [2050], "scenario_variant": ["reference"], }, aggregation={"technology": {"level": 0, "keep": ["hydro_dam", "hydro_run"]}}, use_titles=True, columns=["model", "technology", "value"] ).pivot_table(index="model", columns="technology", values="value").round(1)

Out[8]:

technology	Curtailment	Electrochemical	Hydro dams	Imports	Renewables	Storage	Thermal	Vehicle to grid
model
PowerCheck	NaN	0.2	18.1	10.7	46.8	6.0	3.8	NaN
SES	NaN	NaN	20.0	18.1	60.9	0.0	1.7	NaN
SES-ETH	1.2	0.0	19.5	13.3	58.3	6.8	4.0	0.0
STEM	1.6	0.2	20.8	13.6	57.6	7.2	6.2	NaN
SecMOD	NaN	NaN	16.9	32.4	64.1	8.0	18.3	NaN
SwissX	NaN	0.0	18.4	15.0	63.1	4.8	6.9	NaN
ZEN-Garden	NaN	0.0	19.5	47.1	57.3	11.6	5.6	NaN

Examine dimensions¶

To use the flexible aggregation, must know the identifiers and the hierarchy within the dimensions. One way is to look it up at the CROSS webpage.

Alternatively, you can inspect the dimension associated with a column from the given variable:

In [9]:

(
    res_elec_supply
    .dimensions["technology"]
    .data
    [["id", "level", "parent_id"]]
    .pivot(index="id", values="parent_id", columns="level")
    .sort_index()
    .fillna("")
)

( res_elec_supply .dimensions["technology"] .data [["id", "level", "parent_id"]] .pivot(index="id", values="parent_id", columns="level") .sort_index() .fillna("") )

Out[9]:

level	0	1	2	3
id
battery_out		storage_elec
coal_cc			coal_pp
coal_cc_ccs				coal_cc
coal_cc_other				coal_cc
coal_cc_woccs				coal_cc
...	...	...	...	...
wood_chp_ccs				wood_chp
wood_chp_other				wood_chp
wood_chp_woccs				wood_chp
wood_pp		thermal
wood_pp_other			wood_pp

98 rows × 4 columns

In [ ]: