API Reference#

Dashboard lanucher interface#

Todo

Implement functionality to run dashboard with supabase, local postgres, and nemosis cache backends.

PostgresSQL interfaces#

A set of functions to allow the user to store and retrieve bidding and operational data from a PostgresSQL database.

build_postgres_db#

nem_bidding_dashboard.build_postgres_db.create_db_functions(connection_string)[source]#

Creates the functions needed to retreive data in a PostgresSQL database. This function should be run after creating an empty database, then functions in the nem_bidding_dashboard.populate_postgres_db can be used to add data to the database.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> create_db_functions(con_string)

Parameters:: connection_string – str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used

nem_bidding_dashboard.build_postgres_db.create_db_tables(connection_string)[source]#

Creates the tables needed to store data in a PostgresSQL database. This function should be run after creating an empty database, then functions in the nem_bidding_dashboard.populate_postgres_db can be used to add data to the database.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> create_db_tables(con_string)

Parameters:

connection_string –

str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used

populate_postgres_db#

nem_bidding_dashboard.populate_postgres_db.all_tables_two_most_recent_market_days(connection_string, cache)[source]#

Load data to postgres database for a window starting at 4 am of the current day and going back 48 hrs. Loading is performed for all tables except price_bin_edges.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> all_tables_two_most_recent_market_days(
...  con_string
... "D:/nemosis_data_cache")

Parameters:

connection_string – str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
cache – str the directory to use for caching data prior to upload.

nem_bidding_dashboard.populate_postgres_db.bid_data(connection_string, raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing bidding data by unit. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.bid_data().

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> bid_data(
... con_string,
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_postgres_db.duid_info(connection_string, raw_data_cache)[source]#

Function to populate database table containing bidding data by unit. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.duid_info().

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> duid_info(
...  con_string,
... "D:/nemosis_data_cache",)

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_postgres_db.price_bin_edges_table(connection_string)[source]#

Function to populate database table containing bin definitions for aggregating bids. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.region_data().

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> price_bin_edges_table(
... con_string)

Parameters:

connection_string –

nem_bidding_dashboard.populate_postgres_db.region_data(connection_string, raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing electricity demand and price data by region. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.define_and_return_price_bins().

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> region_data(
... con_string,
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_postgres_db.unit_dispatch(connection_string, raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing unit time series metrics. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.unit_dispatch().

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> unit_dispatch(
... con_string,
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

query_postgres_db#

nem_bidding_dashboard.query_postgres_db.aggregate_bids(connection_string, start_time, end_time, regions, dispatch_type, tech_types, resolution, adjusted)[source]#

Function to query and aggregate bidding data from postgres database. Data is filtered according to the regions, dispatch type, tech types and time window provided, it is then aggregated into a set of predefined bins. Data can be queried at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> aggregate_bids(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... 'hourly',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME   BIDVOLUME
0   2022-01-01 02:00:00   [-1000, -100)  9158.02700
1   2022-01-01 02:00:00       [-100, 0)   299.74405
2   2022-01-01 02:00:00         [0, 50)  1142.00000
3   2022-01-01 02:00:00       [50, 100)  1141.00000
4   2022-01-01 02:00:00      [100, 200)   918.00000
5   2022-01-01 02:00:00      [200, 300)  1138.00000
6   2022-01-01 02:00:00      [300, 500)   920.00000
7   2022-01-01 02:00:00     [500, 1000)   273.00000
8   2022-01-01 02:00:00    [1000, 5000)   210.00000
9   2022-01-01 02:00:00   [5000, 10000)   125.00000
10  2022-01-01 02:00:00  [10000, 15500)  7009.00000

>>> aggregate_bids(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... '5-min',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME   BIDVOLUME
0   2022-01-01 01:05:00   [-1000, -100)  9642.26100
1   2022-01-01 01:05:00       [-100, 0)   361.94458
2   2022-01-01 01:05:00         [0, 50)  1348.00000
3   2022-01-01 01:05:00       [50, 100)  1415.00000
4   2022-01-01 01:05:00      [100, 200)   912.00000
5   2022-01-01 01:05:00      [200, 300)  1188.00000
6   2022-01-01 01:05:00      [300, 500)   903.00000
7   2022-01-01 01:05:00     [500, 1000)   272.00000
8   2022-01-01 01:05:00    [1000, 5000)   210.00000
9   2022-01-01 01:05:00   [5000, 10000)   125.00000
10  2022-01-01 01:05:00  [10000, 15500)  6853.00000

Parameters:

connection_string – str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, BIN_NAME (upper and lower limits of price bin) and BIDVOLUME (total volume bid by units within price bin).

nem_bidding_dashboard.query_postgres_db.aggregated_dispatch_data(connection_string, column_name, start_time, end_time, regions, dispatch_type, tech_types, resolution)[source]#

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> aggregated_dispatch_data(
... con_string,
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'],
... 'Generator',
... [],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00     10402.473

>>> aggregated_dispatch_data(
... con_string,
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['NSW'],
... 'Generator',
... [],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00     10440.107

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_postgres_db.aggregated_dispatch_data_by_duids(connection_string, column_name, start_time, end_time, duids, resolution)[source]#

Function to query dispatch and aggregate data from a postgres database. Data is filter according to the DUIDs, and time window. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only data for 5 minute interval ending on the hour are returned.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> aggregated_dispatch_data_by_duids(
... con_string,
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00         234.0

>>> aggregated_dispatch_data_by_duids(
... con_string,
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['AGLHAL', 'BASTYAN'],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00         234.0

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_postgres_db.aggregated_vwap(connection_string, start_time, end_time, regions)[source]#

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> aggregated_vwap(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'])
         SETTLEMENTDATE      PRICE
0   2022-01-01 01:05:00  107.80005
1   2022-01-01 01:10:00  107.80005
2   2022-01-01 01:15:00   91.92056
3   2022-01-01 01:20:00  107.80005
4   2022-01-01 01:25:00   91.37289
5   2022-01-01 01:30:00   91.38851
6   2022-01-01 01:35:00   92.14760
7   2022-01-01 01:40:00  100.27929
8   2022-01-01 01:45:00   91.90742
9   2022-01-01 01:50:00  100.30000
10  2022-01-01 01:55:00   85.00000
11  2022-01-01 02:00:00   85.00005

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
regions – list[str] of region to aggregate.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time

Returns:

pd.DataFrame with columns SETTLEMENTDATE, TOTALDEMAND and RRP (volume weighted avergae of energy price at regional reference nodes).

nem_bidding_dashboard.query_postgres_db.duid_bids(connection_string, start_time, end_time, duids, resolution, adjusted)[source]#

Function to query bidding data from a postgres database. Data is filter according to the DUID list and time window provided, and returned on a duid basis. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> duid_bids(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 02:00:00   AGLHAL        7       32.0    557.39
1  2022-01-01 02:00:00   AGLHAL       10      121.0  14541.30
2  2022-01-01 02:00:00  BASTYAN        2       53.0    -55.64
3  2022-01-01 02:00:00  BASTYAN        4       28.0     -0.91
4  2022-01-01 02:00:00  BASTYAN       10        0.0  14021.86

>>> duid_bids(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 02:00:00   AGLHAL        7       32.0    557.39
1  2022-01-01 02:00:00   AGLHAL       10      121.0  14541.30
2  2022-01-01 02:00:00  BASTYAN        2       53.0    -55.64
3  2022-01-01 02:00:00  BASTYAN        4       28.0     -0.91
4  2022-01-01 02:00:00  BASTYAN       10        0.0  14021.86

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME, and BIDPRICE

nem_bidding_dashboard.query_postgres_db.region_demand(connection_string, start_time, end_time, regions)[source]#

Query demand and price data from a postgres database. To aggregate demand data is summed.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> region_demand(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 01:30:00",
... ['NSW'])
        SETTLEMENTDATE  TOTALDEMAND
0  2022-01-01 01:05:00      6631.21
1  2022-01-01 01:10:00      6655.52
2  2022-01-01 01:15:00      6496.85
3  2022-01-01 01:20:00      6520.86
4  2022-01-01 01:25:00      6439.22
5  2022-01-01 01:30:00      6429.13

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.

Returns:

pd.DataFrame with columns SETTLEMENTDATE, and TOTALDEMAND (demand to be meet by schedualed and semischedualed generators, not including schedualed loads).

nem_bidding_dashboard.query_postgres_db.stations_and_duids_in_regions_and_time_window(connection_string, start_time, end_time, regions, dispatch_type, tech_types)[source]#

Function to query units from given regions with bids available in the given time window, with the the given dispatch and technology type. Data returned is DUIDs and corresponding Station Names.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> stations_and_duids_in_regions_and_time_window(
... con_string,
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'],
... "Generator",
... [])
        DUID             STATION NAME
0   BANGOWF1      Bango 973 Wind Farm
1   BANGOWF2      Bango 999 Wind Farm
2   BERYLSF1         Beryl Solar Farm
3   BLOWERNG  Blowering Power Station
4   BOCORWF1      Boco Rock Wind Farm
..       ...                      ...
69  WALGRVG1         Wallgrove BESS 1
70   WELLSF1    Wellington Solar Farm
71  WOODLWN1       Woodlawn Wind Farm
72     WRSF1    White Rock Solar Farm
73     WRWF1     White Rock Wind Farm

[74 rows x 2 columns]

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
regions – list[str] regions to filter, should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame with columns DUID and STATION NAME

nem_bidding_dashboard.query_postgres_db.unit_types(connection_string, regions, dispatch_type)[source]#

Function to query distinct unit types from postgres database.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> unit_types(con_string,
... ['NSW'],
... 'Generator')
            UNIT TYPE
           Bagasse
 Battery Discharge
        Black Coal
              CCGT
            Engine
             Hydro
              OCGT
Run of River Hydro
             Solar
              Wind

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
dispatch_type – str ‘Generator’ or ‘Load’
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.

Returns:

pd.DataFrame column UNIT TYPE (this is the unit type as determined by the function nem_bidding_dashboard.preprocessing.tech_namer_by_row())

postgres_helpers#

nem_bidding_dashboard.postgres_helpers.build_connection_string(hostname, dbname, username, password, port, timeout_seconds=None)[source]#

Creates a properly formatted connection string for connecting to a PostgresSQL database.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

Parameters:

hostname – str where the database is hosted
dbname – str name of database to connect to
username – str name of user that has read and write permission on database
password – str password of user
port – int or str port that you can connect to the database on
timeout_seconds – int number of seconds before timeout

Returns: str in formatted required for connecting to postgres database.

nem_bidding_dashboard.postgres_helpers.drop_tables_and_functions(connection_string)[source]#

Drop all the tables and functions created by build_postgres.create_db_tables_and_functions. Intended for to help developement when testing the creation of tables and functions.

Examples:

>>> con_string = build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> drop_tables_and_functions(con_string)

Parameters:: connection_string – str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used

nem_bidding_dashboard.postgres_helpers.insert_data_into_postgres(connection_string, table_name, data)[source]#

Insert data into the postgres database.

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
table_name – str which is the name of the table in the postgres database
data – pd dataframe of data to be uploaded

nem_bidding_dashboard.postgres_helpers.run_query(connection_string, query, autocommit=False)[source]#

Run a genric query in the database which isn’t inserting or retrieving data. For example, creating and dropping tables, functions and indexes.

Examples:

>>> import os

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname=os.environ.get("SUPABASEADDRESS"),
... dbname='postgres',
... username='postgres',
... password=os.environ.get("SUPABASEPASSWORD"),
... port=5432,
... timeout_seconds=6000)

# >>> run_query(con_string, “DROP INDEX unit_dispatch_time_hour_duid_index;”) # # >>> run_query(con_string, “DROP INDEX unit_dispatch_time_duid_index;”)

>>> run_query(con_string, "CREATE INDEX unit_dispatch_time_hour_duid_index ON unit_dispatch (interval_datetime DESC, onhour, duid);")

>>> run_query(con_string, "CREATE INDEX unit_dispatch_time_duid_index ON unit_dispatch (interval_datetime DESC, duid);")

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
query – str query to run in supabase database
autocommit – boolean, set to True to run queries that must be run outside a transaction such as vaccum

nem_bidding_dashboard.postgres_helpers.run_query_return_dataframe(connection_string, query)[source]#

Sends an arbitary query to the specified postgres database and return the result as a pd.DataFrame. Should only be used for queries that return a result as a table. Handles opening and closing connection to database.

Examples:

>>> from nem_bidding_dashboard import postgres_helpers

>>> con_string = postgres_helpers.build_connection_string(
... hostname='localhost',
... dbname='bidding_dashboard_db',
... username='bidding_dashboard_maintainer',
... password='1234abcd',
... port=5433)

>>> run_query_return_dataframe(con_string, "select * from duid_info limit 10;")
       DUID REGION  ...           UNIT TYPE                 STATION NAME
 ADPBA1G     SA  ...   Battery Discharge  Adelaide Desalination Plant
 ADPBA1L     SA  ...      Battery Charge  Adelaide Desalination Plant
  ADPMH1     SA  ...  Run of River Hydro  Adelaide Desalination Plant
  ADPPV3     SA  ...               Solar  Adelaide Desalination Plant
  ADPPV2     SA  ...               Solar  Adelaide Desalination Plant
  ADPPV1     SA  ...               Solar  Adelaide Desalination Plant
AGLSITA1    NSW  ...              Engine              Agl Kemps Creek
 ANGAST1     SA  ...              Engine       Angaston Power Station
   APPIN    NSW  ...              Engine            Appin Power Plant
   ARWF1    VIC  ...                Wind             Ararat Wind Farm

[10 rows x 7 columns]

Parameters:

connection_string –
str for connecting to PostgresSQL database, the function nem_bidding_dashboard.postgres_helpers.build_connection_string() can be used to build a properly formated connection string, or alternative any string that matches the format allowed by PostgresSQL can be used
query – str which is postgres select query

Returns:

pd.DataFrame column as per the query provided

Supabase interfaces#

A set of functions to allow the user to store and retrieve bidding and operational data from a database hosted by Supabase. To use these functions a Supabase database must have been created with the necessary tables and functions and the supabase url and keys need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL, SUPABASE_BIDDING_DASHBOARD_WRITE_KEY, and SUPABASE_BIDDING_DASHBOARD_KEY.

populate_supabase_db#

nem_bidding_dashboard.populate_supabase_db.all_tables_two_most_recent_market_days(cache)[source]#

Upload data to supabase for a window starting at 4 am of the current day and going back 48 hrs. Upload is performed for all tables except price_bin_edges.

Examples:

>>> all_tables_two_most_recent_market_days(
... "D:/nemosis_data_cache")

Parameters:: cache – str the directory to use for caching data prior to upload.

nem_bidding_dashboard.populate_supabase_db.bid_data(raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing bidding data by unit. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_WRITE_KEY respectively. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.bid_data().

Examples:

>>> bid_data(
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_supabase_db.duid_info(raw_data_cache)[source]#

Examples:

>>> duid_info(
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00",
... "D:/nemosis_cache")

Parameters:

start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_supabase_db.insert_data_into_supabase(table_name, data, rows_per_chunk=1000)[source]#

Insert data into the supabase database. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_WRITE_KEY respectively.

Parameters:

table_name – str which is the name of the table in the supabase database
data – pd dataframe of data to be uploaded

nem_bidding_dashboard.populate_supabase_db.price_bin_edges_table()[source]#

Function to populate database table containing bin definitions for aggregating bids. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_WRITE_KEY respectively. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.region_data().

Examples:

>>> price_bin_edges_table()

nem_bidding_dashboard.populate_supabase_db.region_data(raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing electricity demand and price data by region. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_WRITE_KEY respectively. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.region_data().

Examples:

>>> region_data(
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

nem_bidding_dashboard.populate_supabase_db.unit_dispatch(raw_data_cache, start_time, end_time)[source]#

Function to populate database table containing unit time series metrics. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_WRITE_KEY respectively. Data is prepped for loading by the function nem_bidding_dashboard.fetch_and_preprocess.unit_dispatch().

Examples:

>>> unit_dispatch(
... "D:/nemosis_cache",
... "2020/01/01 00:00:00",
... "2020/01/02 00:00:00")

Parameters:

start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for storing data that is fetched

query_supabase_db#

nem_bidding_dashboard.query_supabase_db.aggregate_bids(start_time, end_time, regions, dispatch_type, tech_types, resolution, adjusted)[source]#

Function to query bidding data from supabase. Data is filtered according to the regions, dispatch type, tech types and time window provided, it is then aggregated into a set of predefined bins. Data can queried at hourly or 5 minute resolution. If a hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> aggregate_bids(
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... 'hourly',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME  BIDVOLUME
0   2022-01-01 02:00:00   [-1000, -100)   9158.030
1   2022-01-01 02:00:00       [-100, 0)    299.744
2   2022-01-01 02:00:00         [0, 50)   1142.000
3   2022-01-01 02:00:00       [50, 100)   1141.000
4   2022-01-01 02:00:00      [100, 200)    918.000
5   2022-01-01 02:00:00      [200, 300)   1138.000
6   2022-01-01 02:00:00      [300, 500)    920.000
7   2022-01-01 02:00:00     [500, 1000)    273.000
8   2022-01-01 02:00:00    [1000, 5000)    210.000
9   2022-01-01 02:00:00   [5000, 10000)    125.000
10  2022-01-01 02:00:00  [10000, 15500)   7009.000

>>> aggregate_bids(
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... '5-min',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME  BIDVOLUME
0   2022-01-01 01:05:00   [-1000, -100)   9642.260
1   2022-01-01 01:05:00       [-100, 0)    361.945
2   2022-01-01 01:05:00         [0, 50)   1348.000
3   2022-01-01 01:05:00       [50, 100)   1415.000
4   2022-01-01 01:05:00      [100, 200)    912.000
5   2022-01-01 01:05:00      [200, 300)   1188.000
6   2022-01-01 01:05:00      [300, 500)    903.000
7   2022-01-01 01:05:00     [500, 1000)    272.000
8   2022-01-01 01:05:00    [1000, 5000)    210.000
9   2022-01-01 01:05:00   [5000, 10000)    125.000
10  2022-01-01 01:05:00  [10000, 15500)   6853.000

Parameters:

regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, BIN_NAME (upper and lower limits of price bin) and BIDVOLUME (total volume bid by units within price bin).

nem_bidding_dashboard.query_supabase_db.aggregated_dispatch_data(column_name, start_time, end_time, regions, dispatch_type, tech_types, resolution)[source]#

Function to query dispatch and aggregate data from a postgres database. Data is filter according to the regions, time window, dispatch type, and technology type provided. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only data for 5 minute interval ending on the hour are returned. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> aggregated_dispatch_data(
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'],
... 'Generator',
... [],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00       10402.5

>>> aggregated_dispatch_data(
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['NSW'],
... 'Generator',
... [],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00       10440.1

Parameters:

column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_supabase_db.aggregated_dispatch_data_by_duids(column_name, start_time, end_time, duids, resolution)[source]#

Function to query dispatch data from supabase. Data is filter according to the duids and time window provided, and returned on a duid basis. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> aggregated_dispatch_data_by_duids(
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00           234

>>> aggregated_dispatch_data_by_duids(
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['AGLHAL', 'BASTYAN'],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00           234

Parameters:

column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_supabase_db.aggregated_vwap(start_time, end_time, regions)[source]#

Function to query aggregated Volume Weighted Average Price from supabase. Data is filter according to the regions and time window provided. Data can queryed at hourly or 5 minute resolution. Prices are weighted by demand in each region selected. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> aggregated_vwap(
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'])
         SETTLEMENTDATE      PRICE
 2022-01-01 01:05:00  107.80005
 2022-01-01 01:10:00  107.80005
 2022-01-01 01:15:00   91.92056
 2022-01-01 01:20:00  107.80005
 2022-01-01 01:25:00   91.37289
 2022-01-01 01:30:00   91.38851
 2022-01-01 01:35:00   92.14760
 2022-01-01 01:40:00  100.27929
 2022-01-01 01:45:00   91.90742
 2022-01-01 01:50:00  100.30000
2022-01-01 01:55:00   85.00000
2022-01-01 02:00:00   85.00005

Parameters:

regions – list[str] of region to aggregate.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time

Returns:

pd.DataFrame with column SETTLEMENTDATE and PRICE

nem_bidding_dashboard.query_supabase_db.duid_bids(start_time, end_time, duids, resolution, adjusted)[source]#

Function to query bidding data from supabase. Data is filter according to the DUID list and time window provided, and returned on a duid basis. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> duid_bids(
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 02:00:00   AGLHAL        7         32    557.39
1  2022-01-01 02:00:00   AGLHAL       10        121  14541.30
2  2022-01-01 02:00:00  BASTYAN        2         53    -55.64
3  2022-01-01 02:00:00  BASTYAN        4         28     -0.91
4  2022-01-01 02:00:00  BASTYAN       10          0  14021.90

>>> duid_bids("2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 02:00:00   AGLHAL        7         32    557.39
1  2022-01-01 02:00:00   AGLHAL       10        121  14541.30
2  2022-01-01 02:00:00  BASTYAN        2         53    -55.64
3  2022-01-01 02:00:00  BASTYAN        4         28     -0.91
4  2022-01-01 02:00:00  BASTYAN       10          0  14021.90

Parameters:

duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME, and BIDPRICE

nem_bidding_dashboard.query_supabase_db.region_demand(start_time, end_time, regions)[source]#

Query demand data from supabase. To aggregate demand data is summed. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> region_demand("2022/01/01 01:00:00","2022/01/01 01:30:00",['NSW'])
        SETTLEMENTDATE  TOTALDEMAND
2022-01-01 01:05:00      6631.21
2022-01-01 01:10:00      6655.52
2022-01-01 01:15:00      6496.85
2022-01-01 01:20:00      6520.86
2022-01-01 01:25:00      6439.22
2022-01-01 01:30:00      6429.13

Parameters:

start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.

Returns:

pd.DataFrame with columns SETTLEMENTDATE, REGIONID, and TOTALDEMAND (demand to be meet by schedualed and semischedualed generators, not including schedualed loads)

nem_bidding_dashboard.query_supabase_db.stations_and_duids_in_regions_and_time_window(start_time, end_time, regions, dispatch_type, tech_types)[source]#

Function to query units from given regions with bids available in the given time window. Data returned is DUIDs and corresponding Station Names. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> stations_and_duids_in_regions_and_time_window(
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... "Generator",
... [])
        DUID             STATION NAME
0   BANGOWF1      Bango 973 Wind Farm
1   BANGOWF2      Bango 999 Wind Farm
2   BERYLSF1         Beryl Solar Farm
3   BLOWERNG  Blowering Power Station
4   BOCORWF1      Boco Rock Wind Farm
..       ...                      ...
69  WALGRVG1         Wallgrove BESS 1
70   WELLSF1    Wellington Solar Farm
71  WOODLWN1       Woodlawn Wind Farm
72     WRSF1    White Rock Solar Farm
73     WRWF1     White Rock Wind Farm

[74 rows x 2 columns]

Parameters:

regions – list[str] regions to filter, should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology

Returns:

pd.DataFrame with columns DUID and STATION NAME

nem_bidding_dashboard.query_supabase_db.unit_types(regions, dispatch_type)[source]#

Function to query distinct unit types from supabase. For this function to run the supabase url and key need to be configured as environment variables labeled SUPABASE_BIDDING_DASHBOARD_URL and SUPABASE_BIDDING_DASHBOARD_KEY respectively.

Examples:

>>> unit_types(
... ['NSW'],
... 'Generator')
            UNIT TYPE
           Bagasse
 Battery Discharge
        Black Coal
              CCGT
            Engine
             Hydro
              OCGT
Run of River Hydro
             Solar
              Wind

Returns:: pd.DataFrame column UNIT TYPE (this is the unit type as determined by the function nem_bidding_dashboard.preprocessing.tech_namer_by_row())

Interfaces for accessing raw AEMO data#

A set of function to allow the user to access raw bidding and operation data from AEMO. These interfaces wrap the functionality provided by the NEMOSIS package. For certain datasets they also combined data from different AEMO tables, and combine data stored by AEMO in there ‘current’ and ‘archive’ datasets.

fetch_data#

nem_bidding_dashboard.fetch_data.duid_availability_data(start_time, end_time, raw_data_cache)[source]#

Fetch unit availability and other dispatch values using NEMOSIS. Attempts to pull data from AEMO monthly archive table DISPATCHLOAD, if all the required data cannot be fetched from this table then the AEMO current table PUBLIC_NEXT_DAY_DISPATCH is also queried. This should allow all historical AEMO data to fetched including data from the previous day. Data is filtered for INTERVENTION values equal to 1 where an intervention dispatch run is present, such that the values returned are those assocaited with the dispatch run used to set unit dispatch targets.

Examples:

>>> duid_availability_data(
... '2022/01/01 00:00:00',
... '2022/01/01 00:05:00',
... 'D:/nemosis_data_cache')
         SETTLEMENTDATE     DUID  AVAILABILITY  TOTALCLEARED  INITIALMW  RAMPDOWNRATE  RAMPUPRATE
0   2022-01-01 00:05:00  ADPBA1G         0.000         0.000    0.00000         93.00       93.00
1   2022-01-01 00:05:00  ADPBA1L         0.000         0.000    0.00000         93.00       93.00
2   2022-01-01 00:05:00   ADPPV1         0.000         0.000    0.00000        120.00      120.00
3   2022-01-01 00:05:00   AGLHAL       153.000         0.000   -0.00763        720.00      720.00
4   2022-01-01 00:05:00   AGLSOM       160.000         0.000    0.00000        480.00      480.00
..                  ...      ...           ...           ...        ...           ...         ...
385 2022-01-01 00:05:00  YENDWF1        17.201        17.201   16.66000       1680.00     1680.00
386 2022-01-01 00:05:00    YWPS1       380.000       380.000  376.69113        178.88      178.88
387 2022-01-01 00:05:00    YWPS2         0.000         0.000    0.00000        180.00      180.00
388 2022-01-01 00:05:00    YWPS3       350.000       350.000  348.94431        180.00      180.00
389 2022-01-01 00:05:00    YWPS4       370.000       370.000  368.98608        180.00      180.00

[390 rows x 7 columns]

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns INTERVAL_DATETIME, DUID, AVAILABILITY, TOTALCLEARED, INITIALMW, RAMPDOWNRATE, RAMPUPRATE

nem_bidding_dashboard.fetch_data.duid_data(raw_data_cache)[source]#

Fetch unit data using NEMOSIS. Data is sourced from AEMO’s NEM Registration and Exemption List workbook (Generators and Scheduled Loads tab). This only includes currently registered generator, so if historical analysis is being conducted care needs to taken that data for some generators is not missing.

Examples:

>>> duid_data('D:/nemosis_data_cache')
                       STATION NAME REGION DISPATCH TYPE FUEL SOURCE - DESCRIPTOR       TECHNOLOGY TYPE - DESCRIPTOR      DUID
     Adelaide Desalination Plant    SA1     Generator                     Grid               Battery and Inverter   ADPBA1G
     Adelaide Desalination Plant    SA1          Load                     Grid               Battery and Inverter   ADPBA1L
     Adelaide Desalination Plant    SA1     Generator                    Water                       Run of River    ADPMH1
     Adelaide Desalination Plant    SA1     Generator                    Solar            Photovoltaic Flat panel    ADPPV3
     Adelaide Desalination Plant    SA1     Generator                    Solar            Photovoltaic Flat panel    ADPPV2
..                              ...    ...           ...                      ...                                ...       ...
Yarrawonga Hydro Power Station   VIC1     Generator                    Water                    Hydro - Gravity  YWNGAHYD
          Yarwun Power Station   QLD1     Generator              Natural Gas  Combined Cycle Gas Turbine (CCGT)  YARWUN_1
            Yatpool Solar Farm   VIC1     Generator                    Solar   Photovoltaic Tracking Flat panel    YATSF1
              Yawong Wind Farm   VIC1     Generator                     Wind                     Wind - Onshore    YAWWF1
              Yendon Wind Farm   VIC1     Generator                     Wind                     Wind - Onshore   YENDWF1

[488 rows x 6 columns]

Parameters:: raw_data_cache – Filepath to directory for caching files downloaded from AEMO
Returns:: pandas dataframe with columns DUID, REGIONID, “FUEL SOURCE - DESCRIPTOR”, “DISPATCH TYPE”, “TECHNOLOGY TYPE - DESCRIPTOR”, “STATION NAME”

nem_bidding_dashboard.fetch_data.price_bids(start_time: str, end_time: str, raw_data_cache: str)[source]#

Fetch unit price bid data using NEMOSIS. Data source from AEMO monthly archive tables BIDPDAYOFFER_D and current tables BIDMOVE_COMPETE. This should allow all historical AEMO data to fetched including data from the previous day.

Examples:

>>> price_bids(
... '2022/01/01 00:00:00',
... '2022/01/01 00:05:00',
... 'D:/nemosis_data_cache')
     SETTLEMENTDATE     DUID     BIDTYPE  PRICEBAND1  PRICEBAND2  PRICEBAND3  PRICEBAND4  PRICEBAND5  PRICEBAND6  PRICEBAND7  PRICEBAND8  PRICEBAND9  PRICEBAND10
0        2021-12-31  ADPBA1L    RAISEREG        5.00        8.00       12.00       18.00       24.00       47.00       98.00      268.00       498.0     12000.00
1        2021-12-31    ARWF1      ENERGY     -898.70     -224.68     -179.74     -157.27     -145.59     -125.82      -44.94      224.68      2696.1     13570.37
2        2021-12-31  ASNENC1   RAISE6SEC        0.03        0.30        0.73        0.99        1.98        5.00        9.90       17.70       100.0     10000.00
3        2021-12-31   ASSEL1  LOWER60SEC        1.00        2.00        3.00        4.00        5.00        6.00        7.00        8.00         9.0     13000.00
4        2021-12-31  ASSENC1  RAISE60SEC        0.00        1.00        2.00        4.00        8.00       16.00       32.00       64.00       128.0       256.00
...             ...      ...         ...         ...         ...         ...         ...         ...         ...         ...         ...         ...          ...
1284     2021-12-31    YWPS3   LOWER5MIN        0.08        0.17        0.79        1.19        4.40        9.99       29.99       99.99       249.9      8999.99
1285     2021-12-31    YWPS3   RAISE6SEC        0.48        1.75        4.90       20.70       33.33       99.90      630.00     1999.00      6000.0     12299.00
1286     2021-12-31    YWPS4   LOWER6SEC        0.03        0.05        0.16        0.30        1.90       25.04       30.04       99.00      4600.0      9899.00
1287     2021-12-31    YWPS4    LOWERREG        0.05        1.90        4.78        9.40       14.00       29.00       64.90      240.90     11990.0     14600.00
1288     2021-12-31    YWPS4  RAISE60SEC        0.17        1.80        4.80       10.01       21.00       39.00       52.00      102.00      4400.0     11999.00

[1289 rows x 13 columns]

Parameters:

start_time (str) – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time (str) – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache (str) – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns INTERVAL_DATETIME, SETTLEMENTDATE, DUID, BIDTYPE, PRICEBAND1, PRICEBAND2, PRICEBAND3, PRICEBAND4, PRICEBAND5, PRICEBAND6, PRICEBAND7, PRICEBAND8, PRICEBAND9, PRICEBAND10

nem_bidding_dashboard.fetch_data.region_data(start_time, end_time, raw_data_cache)[source]#

Fetch electricity price and demand data using NEMOSIS. Attempts to pull data from AEMO monthly archive tables DISPATCHPRICE and DISPATCHREGIONSUM, if all the required data cannot be fetched from these tables then AEMO current table PUBLIC_DAILY is also queried. This should allow all historical AEMO data to fetched including data from the previous day.

Examples:

>>> region_data(
... '2022/01/01 00:00:00',
... '2022/01/01 00:05:00',
... 'D:/nemosis_data_cache')
  REGIONID      SETTLEMENTDATE  TOTALDEMAND        RRP
0     NSW1 2022-01-01 00:05:00      7206.03  124.85631
1     QLD1 2022-01-01 00:05:00      5982.85  118.73008
2      SA1 2022-01-01 00:05:00      1728.03  133.94970
3     TAS1 2022-01-01 00:05:00      1148.93   40.34000
4     VIC1 2022-01-01 00:05:00      5005.34  114.80312

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns: pandas dataframe with columns SETTLEMENTDATE, REGIONID, TOTALDEMAND (the operational demand AEMO dispatches: generation to meet), and RRP (the regional reference price for energy).

nem_bidding_dashboard.fetch_data.volume_bids(start_time, end_time, raw_data_cache)[source]#

Fetch unit volume bid data using NEMOSIS. Data source from AEMO monthly archive tables BIDPEROFFER_D and current tables BIDMOVE_COMPETE. This should allow all historical AEMO data to fetched including data from the previous day.

Examples:

>>> volume_bids(
... '2022/01/01 00:00:00',
... '2022/01/01 01:00:00',
... 'D:/nemosis_data_cache')
       SETTLEMENTDATE     DUID     BIDTYPE  MAXAVAIL  ROCUP  ROCDOWN  BANDAVAIL1  BANDAVAIL2  BANDAVAIL3  BANDAVAIL4  BANDAVAIL5  BANDAVAIL6  BANDAVAIL7  BANDAVAIL8  BANDAVAIL9  BANDAVAIL10  PASAAVAILABILITY   INTERVAL_DATETIME
309360     2021-12-31  ADPBA1G      ENERGY         0    2.0      2.0           0           0           0           0           0           0           0           6           0            0               6.0 2022-01-01 00:05:00
309361     2021-12-31  ADPBA1G    LOWERREG         0    NaN      NaN           0           0           0           0           0           0           0           6           0            0               NaN 2022-01-01 00:05:00
309362     2021-12-31  ADPBA1G   RAISE5MIN         0    NaN      NaN           0           0           0           0           0           0           0           0           0            2               NaN 2022-01-01 00:05:00
309363     2021-12-31  ADPBA1G  RAISE60SEC         0    NaN      NaN           0           0           0           0           0           0           0           0           0            2               NaN 2022-01-01 00:05:00
309364     2021-12-31  ADPBA1G   RAISE6SEC         0    NaN      NaN           0           0           0           0           0           0           0           0           0            2               NaN 2022-01-01 00:05:00
...               ...      ...         ...       ...    ...      ...         ...         ...         ...         ...         ...         ...         ...         ...         ...          ...               ...                 ...
324823     2021-12-31    YWPS4    LOWERREG        20    NaN      NaN           0           0           0           0           0           0           0          20           0            0               NaN 2022-01-01 01:00:00
324824     2021-12-31    YWPS4   RAISE5MIN         0    NaN      NaN           0           0           0           5           5           0           0           0           0           10               NaN 2022-01-01 01:00:00
324825     2021-12-31    YWPS4  RAISE60SEC         5    NaN      NaN           0           0           0           0           0           0           5          10           0            5               NaN 2022-01-01 01:00:00
324826     2021-12-31    YWPS4   RAISE6SEC         5    NaN      NaN           0           0           0           0           0           5          10           0           0           10               NaN 2022-01-01 01:00:00
324827     2021-12-31    YWPS4    RAISEREG        15    NaN      NaN           0           0           0           0           0           0           5          10           0            5               NaN 2022-01-01 01:00:00

[15468 rows x 18 columns]

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns INTERVAL_DATETIME, SETTLEMENTDATE, DUID, BIDTYPE, BANDAVAIL1, BANDAVAIL2, BANDAVAIL3, BANDAVAIL4, BANDAVAIL5, BANDAVAIL6, BANDAVAIL7, BANDAVAIL8, BANDAVAIL9, BANDAVAIL10, MAXAVAIL, ROCUP, ROCDOWN, PASAAVAILABILITY

Interfaces for processing raw data#

A set of function for processing raw AEMO data before loading into a database. The general purpose of these processing steps is to limit the data manipulation that must be performed later, and so improve the responsiveness of dashboard applications that pull data from the database, or speed up runtimes of analysis scripts that use the database.

preprocessing#

nem_bidding_dashboard.preprocessing.adjust_bids_for_availability(stacked_bids, unit_availability)[source]#

Adjust bid volumes where the unit availability would restrict a bid from actually being fully dispatched. Starting from the highest bid bands bid volumes are adjusted down until the total bid volume is equal to the availability, if the total bid volume is already equal to or less than availability no adjustments are made.

Examples:

>>> bid_data = pd.DataFrame(
... columns=["INTERVAL_DATETIME", "DUID", "BIDBAND", "BIDVOLUME", "BIDPRICE"],
... data=[('2020/01/01 00:45:00', 'AGLHAL', 1, 50, 120),
...       ('2020/01/01 00:45:00', 'AGLHAL', 2, 50, 200)])

>>> bid_data
     INTERVAL_DATETIME    DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2020/01/01 00:45:00  AGLHAL        1         50       120
1  2020/01/01 00:45:00  AGLHAL        2         50       200

>>> unit_availability = pd.DataFrame(
... columns=["SETTLEMENTDATE", "DUID", "AVAILABILITY"],
... data=[('2020/01/01 00:45:00', 'AGLHAL', 80),])

>>> unit_availability
        SETTLEMENTDATE    DUID  AVAILABILITY
0  2020/01/01 00:45:00  AGLHAL            80

>>> adjust_bids_for_availability(bid_data, unit_availability)
     INTERVAL_DATETIME    DUID  BIDBAND  BIDVOLUME  BIDVOLUMEADJUSTED  BIDPRICE
0  2020/01/01 00:45:00  AGLHAL        1         50                 50       120
1  2020/01/01 00:45:00  AGLHAL        2         50                 30       200

Parameters:

stacked_bids – pd.dataframe containing matched quantity and price pairs on a 5 minutely basis with bid bands on row basis. Should have columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME and BIDPRICE
unit_availability – pd.dataframe containing unit availability values on 5 minutely basis. Should have columns SETTLEMENTDATE, DUID, AVAILABILITY

Returns:

pd.dataframe containing matched quantity and price pairs on a 5 minutely basis with bid bands on row basis. An extra column is added in which did volumes have been adjusted so total bid volume doesn’t exceed unit availability. Should have columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME, BIDVOLUMEADJUSTED and BIDPRICE

nem_bidding_dashboard.preprocessing.calculate_unit_time_series_metrics(as_bid_metrics, after_dispatch_metrics)[source]#

Calculate some additional values associated with unit dispatch. These are used to populate the database table unit_dispatch and should be helpful in understanding unit dispatch outcomes, e.g. show when a plant is limited by its ramp rate, or if a unit’s availability is the same as the availability submitted for the PASA process indicating that output is probably limited by a technical constraint.

Examples:

>>> as_bid_metrics = pd.DataFrame({
... 'INTERVAL_DATETIME': ['2022/01/01 00:00:00', '2022/01/01 00:05:00', '2022/01/01 00:10:00'],
... 'DUID': ['AGLHAL', 'AGLHAL', 'AGLHAL'],
... 'ROCUP': [10, 15, 60],
... 'ROCDOWN': [20, 25, 30],
... 'MAXAVAIL': [100, 90,  105],
... 'PASAAVAILABILITY': [120, 90, 110],
... })

>>> as_bid_metrics['INTERVAL_DATETIME'] = pd.to_datetime(as_bid_metrics['INTERVAL_DATETIME'])

>>> as_bid_metrics
    INTERVAL_DATETIME    DUID  ROCUP  ROCDOWN  MAXAVAIL  PASAAVAILABILITY
0 2022-01-01 00:00:00  AGLHAL     10       20       100               120
1 2022-01-01 00:05:00  AGLHAL     15       25        90                90
2 2022-01-01 00:10:00  AGLHAL     60       30       105               110

>>> after_dispatch_metrics = pd.DataFrame({
... 'SETTLEMENTDATE': ['2022/01/01 00:00:00', '2022/01/01 00:05:00', '2022/01/01 00:10:00'],
... 'DUID': ['AGLHAL', 'AGLHAL', 'AGLHAL'],
... 'RAMPUPRATE': [9*60, 14*60, 50*60],
... 'RAMPDOWNRATE': [18*60, 24*60, 28*60],
... 'AVAILABILITY': [100, 90,  105],
... 'INITIALMW': [80.1, 88.9,  84.5],
... 'TOTALCLEARED': [80, 90,  85],
...  })

>>> after_dispatch_metrics['SETTLEMENTDATE'] = pd.to_datetime(after_dispatch_metrics['SETTLEMENTDATE'])

>>> after_dispatch_metrics
       SETTLEMENTDATE    DUID  RAMPUPRATE  RAMPDOWNRATE  AVAILABILITY  INITIALMW  TOTALCLEARED
0 2022-01-01 00:00:00  AGLHAL         540          1080           100       80.1            80
1 2022-01-01 00:05:00  AGLHAL         840          1440            90       88.9            90
2 2022-01-01 00:10:00  AGLHAL        3000          1680           105       84.5            85

>>> calculate_unit_time_series_metrics(as_bid_metrics, after_dispatch_metrics)
    INTERVAL_DATETIME    DUID  AVAILABILITY  TOTALCLEARED  FINALMW  ASBIDRAMPUPMAXAVAIL  ASBIDRAMPDOWNMINAVAIL  RAMPUPMAXAVAIL  RAMPDOWNMINAVAIL  PASAAVAILABILITY  MAXAVAIL
0 2022-01-01 00:00:00  AGLHAL           100            80     88.9                130.1                  -19.9           125.1              -9.9               120       100
1 2022-01-01 00:05:00  AGLHAL            90            90     84.5                163.9                  -36.1           158.9             -31.1                90        90

Parameters:

as_bid_metrics – pd.DataFrame containing values submitted by unit’s as part of the bidding process. Should contain columns INTERVAL_DATETIME, DUID, ROCUP (ramp up rate in MW per min), ROCDOWN (ramp down rate in MW per min), MAXAVAIL (limit the unit can be dispatched up to), PASAAVAILABILITY (The technical maximum availability of unit given 24h notice, not used in dispatch)
after_dispatch_metrics – pd.DataFrame containing values calculated by AEMO as part of the dispatch process. Should contain columns SETTLEMENTDATE, DUID, AVAILABILITY (presumed to be the lesser of the unit bid availability (MAXAVAIL column) and unit forecast availability for variable renewables), RAMPUPRATE, RAMPDOWNRATE (lesser of bid and telemetry ramp rates, MW per hour), INITIALMW (operating level of unit at start of dispatch interval), TOTALCLEARED (dispatch target for unit to ramp to by end of dispatch interval).

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, DUID, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate), RAMPUPMAXAVAIL ( upper dispatch limit based lesser of as bid and telemetry ramp rates), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).

nem_bidding_dashboard.preprocessing.hard_code_fix_fuel_source_and_tech_errors(duid_data)[source]#

Where NA values occur in columns FUEL SOURCE - DESCRIPTOR or TECHNOLOGY TYPE - DESCRIPTOR replace these with the value ‘-’. This function is used to clean data before passing to nem_bidding_dashboard.preprocessing.tech_namer().

Parameters:: duid_data – pd.DataFrame containing at least the columns FUEL SOURCE - DESCRIPTOR and TECHNOLOGY TYPE - DESCRIPTOR
Returns:: pd.DataFrame with the same columns as input data. Columns FUEL SOURCE - DESCRIPTOR and TECHNOLOGY TYPE - DESCRIPTOR have NA values replaced with ‘-‘.

nem_bidding_dashboard.preprocessing.remove_number_from_region_names(region_column, data)[source]#

Removes the trailing 1 from region names in the specified column. Names in input data are expected to be of the format NSW1, QLD1 etc. and the names in the output format will be NSW, QLD etc.

Examples:

>>> region_data = pd.DataFrame({
... 'region': ['QLD1', 'TAS1'],
... 'dummy_values': [55.7, 102.9]})

>>> region_data
  region  dummy_values
0   QLD1          55.7
1   TAS1         102.9

>>> remove_number_from_region_names('region', region_data)
  region  dummy_values
0    QLD          55.7
1    TAS         102.9

Parameters:

region_column – str the name of the column containing the region names.
data – pd dataframe with a column called the value of region_column.

Returns:

pd dataframe with names in region_column modified.

nem_bidding_dashboard.preprocessing.stack_unit_bids(volume_bids, price_bids)[source]#

Combine volume and price components of energy market offers and reformat them such that each price quantity pair is on a separate row of the dataframe.

Examples:

>>> volume_bids = pd.DataFrame(
... columns=["INTERVAL_DATETIME", "SETTLEMENTDATE", "DUID", "BANDAVAIL1", "BANDAVAIL2", "BANDAVAIL3",
...          "BANDAVAIL4", "BANDAVAIL5", "BANDAVAIL6", "BANDAVAIL7", "BANDAVAIL8", "BANDAVAIL9", "BANDAVAIL10"],
... data=[('2020/01/01 00:45:00', '2019/12/31 00:00:00', 'AGLHAL', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)])

>>> volume_bids
     INTERVAL_DATETIME       SETTLEMENTDATE  ... BANDAVAIL9  BANDAVAIL10
0  2020/01/01 00:45:00  2019/12/31 00:00:00  ...          9           10

[1 rows x 13 columns]

>>> price_bids = pd.DataFrame(
... columns=["SETTLEMENTDATE", "DUID", "PRICEBAND1", "PRICEBAND2", "PRICEBAND3", "PRICEBAND4", "PRICEBAND5",
...          "PRICEBAND6", "PRICEBAND7", "PRICEBAND8", "PRICEBAND9", "PRICEBAND10"],
... data=[('2019/12/31 00:00:00', 'AGLHAL', 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)])

>>> price_bids
        SETTLEMENTDATE    DUID  PRICEBAND1  ...  PRICEBAND8  PRICEBAND9  PRICEBAND10
0  2019/12/31 00:00:00  AGLHAL           1  ...           8           9           10

[1 rows x 12 columns]

>>> stack_unit_bids(volume_bids, price_bids)
     INTERVAL_DATETIME    DUID  BIDBAND  BIDVOLUME  BIDPRICE
2020/01/01 00:45:00  AGLHAL        1          1         1
2020/01/01 00:45:00  AGLHAL        2          2         2
2020/01/01 00:45:00  AGLHAL        3          3         3
2020/01/01 00:45:00  AGLHAL        4          4         4
2020/01/01 00:45:00  AGLHAL        5          5         5
2020/01/01 00:45:00  AGLHAL        6          6         6
2020/01/01 00:45:00  AGLHAL        7          7         7
2020/01/01 00:45:00  AGLHAL        8          8         8
2020/01/01 00:45:00  AGLHAL        9          9         9
2020/01/01 00:45:00  AGLHAL       10         10        10

Parameters:

volume_bids – pd dataframe containing quantity of bids on a 5 minutely basis. Should have columns INTERVAL_DATETIME, SETTLEMENTDATE, DUID, BANDAVAIL1 … . BANDAVAIL10
price_bids – pd dataframe containing quantity of bids on a daily basis. Should have columns SETTLEMENTDATE, DUID, PRICEBAND1 … . PRICEBAND10

Returns:

pd dataframe containing matched quantity and price pairs on a 5 minutely basis with bid bands on row basis. Should have columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME and BIDPRICE

nem_bidding_dashboard.preprocessing.tech_namer(duid_info)[source]#

Create a name for generation and load unit using custom logic applied to the fuel type, technology descriptor, and dispatch type supplied by AEMO in the NEM Registration and Exemption List.xls file. See the source code for logic that this function applies:

Parameters:: duid_info – pd dataframe with columns ‘FUEL SOURCE - DESCRIPTOR’, ‘TECHNOLOGY TYPE - DESCRIPTOR’ and ‘DISPATCH TYPE’
Returns:: pd dataframe with additional column ‘UNIT_TYPE’

nem_bidding_dashboard.preprocessing.tech_namer_by_row(fuel, tech_descriptor, dispatch_type)[source]#

Create a name for generation and loads using custom logic that considers the fuel type, technology descriptor, and dispatch type supplied by AEMO in the NEM Registration and Exemption List.xls file. See the source code for logic that this function applies:

Examples:

>>> tech_namer_by_row("Natural Gas", "Steam Sub-Critical", "Generator")
'Gas Thermal'

>>> tech_namer_by_row("solar", "PV - Tracking", "Generator")
'Solar'

>>> tech_namer_by_row("-", "Battery", "Load")
'Battery Charge'

Parameters:

fuel – str should be the value from the column ‘Fuel Source - Descriptor’
tech_descriptor – str should be the value from the column ‘Technology Type - Descriptor’
dispatch_type – str should be the value from the column ‘Dispatch Type’

Returns:

str a value categorising the technology of the generation or load unit

fetch_and_preprocess#

nem_bidding_dashboard.fetch_and_preprocess.bid_data(start_time, end_time, raw_data_cache)[source]#

Wrapper for fetching and preprocessing bid data.

Calls nem_bidding_dashboard.fetch_data.get_volume_bids(), nem_bidding_dashboard.fetch_data.get_price_bids(), and nem_bidding_dashboard.fetch_data.get_duid_availability_data() to get raw bidding and availabilty data
Volume and price bids are filtered to get only bids for the energy spot market
Volume and price bids are combined using nem_bidding_dashboard.preprocessing.stack_unit_bids()
Bids are filtered to remove those with zero volume.
The function nem_bidding_dashboardpreprocessing.adjust_bids_for_availability() to calculate the bid volume adjusted so that the bid volume does not exceed the unit availablity
Finally datetime columns are converted to string format.

Used for preparing data to load into dashboard backend PostgresSQL database, or can be used for compiling regional data directly if the user does not have a database, but compling data using this function will be considerably slower than from a PostgresSQL database.

Examples:

>>> bid_data(
... '2022/01/01 00:00:00',
... '2022/01/01 00:05:00',
... 'D:/nemosis_data_cache')
       INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDVOLUMEADJUSTED  BIDPRICE  ONHOUR
0    2022-01-01 00:05:00  ADPBA1G        8          6              0.000    998.00   False
462  2022-01-01 00:05:00   REECE1        2         45             45.000    -55.03   False
463  2022-01-01 00:05:00   REECE1        4         74             74.000     -0.85   False
464  2022-01-01 00:05:00   REECE2        2         35             35.000    -54.77   False
465  2022-01-01 00:05:00   REECE2        4         84             84.000     -0.86   False
..                   ...      ...      ...        ...                ...       ...     ...
235  2022-01-01 00:05:00  GUTHEGA       10         80             68.000  13599.06   False
236  2022-01-01 00:05:00  HALLWF1        1         95             10.840   -963.00   False
237  2022-01-01 00:05:00  HALLWF2        1         71              4.408   -960.60   False
229  2022-01-01 00:05:00  GSTONE6        8          5              5.000   4806.84   False
700  2022-01-01 00:05:00    YWPS4       10          9              0.000  14489.96   False

[701 rows x 7 columns]

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns INTERVAL_DATETIME, DUID, BIDPRICE ($/MWh), BIDVOLUME (MW), BIDVOLUMEADJUSTED (MW)

nem_bidding_dashboard.fetch_and_preprocess.define_and_return_price_bins()[source]#

Defines the bins for aggregating bidding data.

Examples:

>>> define_and_return_price_bins()
          BIN_NAME  LOWER_EDGE  UPPER_EDGE
  [-1000, -100)       -2000        -100
      [-100, 0)        -100           0
        [0, 50)           0          50
      [50, 100)          50         100
     [100, 200)         100         200
     [200, 300)         200         300
     [300, 500)         300         500
    [500, 1000)         500        1000
   [1000, 5000)        1000        5000
  [5000, 10000)        5000       10000
[10000, 15500)       10000       16000

Returns:: pandas dataframe with column bin_name, lower_edge and upper_edge

nem_bidding_dashboard.fetch_and_preprocess.duid_info(raw_data_cache)[source]#

Wrapper for fetching and preprocessing duid summary data.

Calls nem_bidding_dashboard.fetch_data.get_duid_data() to get raw duid info
Calls nem_bidding_dashboard.preprocessing.hard_code_fix_fuel_source_and_tech_errors() to replace NA values with ‘-’
Calls nem_bidding_dashboard.preprocessing.remove_number_from_region_names() to remove trailing ‘1’ from REGIONIDs
Calls nem_bidding_dashboard.preprocessing.tech_namer() to determine a more concise and consistent technology type for units. This value is stored in the column “UNIT TYPE”.

Used for preparing data to load into dashboard backend PostgresSQL database, or can be used for compiling regional data directly if the user does not have a database.

Examples:

>>> duid_info('D:/nemosis_data_cache')
                       STATION NAME REGION DISPATCH TYPE FUEL SOURCE - DESCRIPTOR       TECHNOLOGY TYPE - DESCRIPTOR      DUID           UNIT TYPE
     Adelaide Desalination Plant     SA     Generator                     Grid               Battery and Inverter   ADPBA1G   Battery Discharge
     Adelaide Desalination Plant     SA          Load                     Grid               Battery and Inverter   ADPBA1L      Battery Charge
     Adelaide Desalination Plant     SA     Generator                    Water                       Run of River    ADPMH1  Run of River Hydro
     Adelaide Desalination Plant     SA     Generator                    Solar            Photovoltaic Flat panel    ADPPV3               Solar
     Adelaide Desalination Plant     SA     Generator                    Solar            Photovoltaic Flat panel    ADPPV2               Solar
..                              ...    ...           ...                      ...                                ...       ...                 ...
Yarrawonga Hydro Power Station    VIC     Generator                    Water                    Hydro - Gravity  YWNGAHYD               Hydro
          Yarwun Power Station    QLD     Generator              Natural Gas  Combined Cycle Gas Turbine (CCGT)  YARWUN_1                CCGT
            Yatpool Solar Farm    VIC     Generator                    Solar   Photovoltaic Tracking Flat panel    YATSF1               Solar
              Yawong Wind Farm    VIC     Generator                     Wind                     Wind - Onshore    YAWWF1                Wind
              Yendon Wind Farm    VIC     Generator                     Wind                     Wind - Onshore   YENDWF1                Wind

[488 rows x 7 columns]

Parameters:: raw_data_cache – Filepath to directory for caching files downloaded from AEMO
Returns:: pandas dataframe with columns DUID, REGIONID, “FUEL SOURCE - DESCRIPTOR”, “DISPATCH TYPE”, “TECHNOLOGY TYPE - DESCRIPTOR”, “UNIT TYPE”, “STATION NAME”

nem_bidding_dashboard.fetch_and_preprocess.region_data(start_time, end_time, raw_data_cache)[source]#

Wrapper for fetching and preprocessing regional demand and price data. Calls nem_bidding_dashboard.fetch_data.get_region_data() to get regional data, re-formats REGIONID by removing the trailing ‘1’ using nem_bidding_dashboard.preprocessing.remove_number_from_region_names(), and finally converting datetime columns to string format. Used for preparing data to load into dashboard backend PostgresSQL database, or can be used for compiling regional data directly if the user does not have database, but compling data using this function will be considerably slower than from a PostgresSQL database.

Examples:

>>> region_data(
... '2022/01/01 00:00:00',
... '2022/01/01 00:05:00',
... 'D:/nemosis_data_cache')
  REGIONID       SETTLEMENTDATE  TOTALDEMAND        RRP
0      NSW  2022-01-01 00:05:00      7206.03  124.85631
1      QLD  2022-01-01 00:05:00      5982.85  118.73008
2       SA  2022-01-01 00:05:00      1728.03  133.94970
3      TAS  2022-01-01 00:05:00      1148.93   40.34000
4      VIC  2022-01-01 00:05:00      5005.34  114.80312

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns SETTLEMENTDATE, REGIONID, TOTALDEMAND (the operational demand AEMO dispatches generation to meet), and RRP (the regional reference price for energy).

nem_bidding_dashboard.fetch_and_preprocess.unit_dispatch(start_time, end_time, raw_data_cache)[source]#

Wrapper for fetching and preprocessing unit dispatch data.

Calls nem_bidding_dashboard.fetch_data.get_volume_bids() and nem_bidding_dashboard.fetch_data.get_duid_availability_data() to get raw bidding and availabilty data
Volume are filtered to get only bids for the energy spot market
Calls nem_bidding_dashboard.preprocessing.calculate_unit_time_series_metrics()
Finally datetime columns are converted to string format.

Used for preparing data to load into dashboard backend PostgresSQL database, or can be used for compiling regional data directly if the user does not have database, but compling data using this function will be considerably slower than from a PostgresSQL database.

Examples:

>>> unit_dispatch(
... '2022/01/01 00:55:00',
... '2022/01/01 01:05:00',
... 'D:/nemosis_data_cache')
       INTERVAL_DATETIME      DUID  AVAILABILITY  TOTALCLEARED    FINALMW  ASBIDRAMPUPMAXAVAIL  ASBIDRAMPDOWNMINAVAIL  RAMPUPMAXAVAIL  RAMPDOWNMINAVAIL  PASAAVAILABILITY  MAXAVAIL  ONHOUR
0    2022-01-01 01:00:00   ADPBA1G         0.000         0.000    0.00000             10.00000              -10.00000         7.75000          -7.75000               6.0         0    True
250  2022-01-01 01:00:00     RRSF1         0.000         0.000    0.00000           3480.00000            -3480.00000      3480.00000       -3480.00000             116.0       116    True
249  2022-01-01 01:00:00    ROMA_8        34.000         0.000    0.00000             40.00000              -40.00000        40.00000         -40.00000              34.0        34    True
248  2022-01-01 01:00:00    ROMA_7        34.000         0.000    0.00000             40.00000              -40.00000        40.00000         -40.00000              34.0        34    True
247  2022-01-01 01:00:00    REECE2       119.000       118.000  117.53001            266.89001              -33.10999       266.89001         -33.10999             119.0       119    True
..                   ...       ...           ...           ...        ...                  ...                    ...             ...               ...               ...       ...     ...
118  2022-01-01 01:00:00   GSTONE3       260.000       170.000  170.45000            193.87500              143.87500       193.35000         144.40000             260.0       260    True
117  2022-01-01 01:00:00   GSTONE2         0.000         0.000    0.00000             25.00000              -25.00000        25.00000         -25.00000             270.0         0    True
116  2022-01-01 01:00:00   GSTONE1         0.000         0.000    0.00000             15.00000              -15.00000        15.00000         -15.00000               0.0         0    True
125  2022-01-01 01:00:00  GUNNING1        10.042        10.042    7.97946             25.02899               -4.97101        25.02899          -4.97101              47.0        47    True
368  2022-01-01 01:00:00     YWPS4       370.000       370.000  370.11520            385.39746              355.39746       385.39746         355.39746             396.0       370    True

[369 rows x 12 columns]

Parameters:

start_time – str formatted “DD/MM/YYYY HH:MM:SS”, data with date times greater than start_time are returned
end_time – str formatted identical to start_time, data with date times less than or equal to end_time are returned
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pandas dataframe with columns INTERVAL_DATETIME, DUID, AVAILABILITY, TOTALCLEARED, FINALMW, ASBIDRAMPUPMAXAVAIL, ASBIDRAMPDOWNMINAVAIL, RAMPUPMAXAVAIL, RAMPDOWNMINAVAIL, PASAAVAILABILITY, MAXAVAIL (see unit_dispatch in Database Guide for column definitions)

Interface for aggregating from NEMOSIS cache#

nem_bidding_dashboard.query_cached_data.aggregate_bids(raw_data_cache, start_time, end_time, regions, dispatch_type, tech_types, resolution, adjusted)[source]#

Function to query and aggregate bidding data from raw data cache database. Data is filter according to the regions, dispatch type, tech types and time window provided, it is then aggregated into a set of predefined bins. Data can be queried at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned.

Examples:

>>> aggregate_bids(
... 'D:/nemosis_data_cache',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... 'hourly',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME   BIDVOLUME
0   2022-01-01 02:00:00   [-1000, -100)  9158.02765
1   2022-01-01 02:00:00       [-100, 0)   299.74402
2   2022-01-01 02:00:00         [0, 50)  1142.00000
3   2022-01-01 02:00:00       [50, 100)  1141.00000
4   2022-01-01 02:00:00      [100, 200)   918.00000
5   2022-01-01 02:00:00      [200, 300)  1138.00000
6   2022-01-01 02:00:00      [300, 500)   920.00000
7   2022-01-01 02:00:00     [500, 1000)   273.00000
8   2022-01-01 02:00:00    [1000, 5000)   210.00000
9   2022-01-01 02:00:00   [5000, 10000)   125.00000
10  2022-01-01 02:00:00  [10000, 15500)  7009.00000

>>> aggregate_bids(
... 'D:/nemosis_data_cache',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['QLD', 'NSW', 'SA'],
... 'Generator',
... [],
... '5-min',
... 'adjusted')
      INTERVAL_DATETIME        BIN_NAME   BIDVOLUME
0   2022-01-01 01:05:00   [-1000, -100)  9642.26127
1   2022-01-01 01:05:00       [-100, 0)   361.94456
2   2022-01-01 01:05:00         [0, 50)  1348.00000
3   2022-01-01 01:05:00       [50, 100)  1415.00000
4   2022-01-01 01:05:00      [100, 200)   912.00000
5   2022-01-01 01:05:00      [200, 300)  1188.00000
6   2022-01-01 01:05:00      [300, 500)   903.00000
7   2022-01-01 01:05:00     [500, 1000)   272.00000
8   2022-01-01 01:05:00    [1000, 5000)   210.00000
9   2022-01-01 01:05:00   [5000, 10000)   125.00000
10  2022-01-01 01:05:00  [10000, 15500)  6853.00000

Parameters:

regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, BIN_NAME (upper and lower limits of price bin) and BIDVOLUME (total volume bid by units within price bin).

nem_bidding_dashboard.query_cached_data.aggregated_dispatch_data(raw_data_cache, column_name, start_time, end_time, regions, dispatch_type, tech_types, resolution)[source]#

Function to query dispatch data from a raw data cache. Data is filter according to the regions, time window, dispatch type, and technology type provided, and returned on a SETTLEMENTDATE basis. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only data for 5 minute interval ending on the hour are returned.

Examples:

>>> aggregated_dispatch_data(
... 'D:/nemosis_data_cache',
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'],
... 'Generator',
... [],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00   10402.47408

>>> aggregated_dispatch_data(
... 'D:/nemosis_data_cache',
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['NSW'],
... 'Generator',
... [],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00   10440.10679

Parameters:

column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_cached_data.aggregated_dispatch_data_by_duids(raw_data_cache, column_name, start_time, end_time, duids, resolution)[source]#

Function to query dispatch data from a raw data cahce. Data is filter according to the duids and time window provided, and returned on a duid basis. Data can queryed at hourly or 5 minute resolution. If a hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned.

Examples:

>>> aggregated_dispatch_data_by_duids(
... 'D:/nemosis_data_cache',
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 02:00:00         234.0

>>> aggregated_dispatch_data_by_duids(
... 'D:/nemosis_data_cache',
... 'AVAILABILITY',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['AGLHAL', 'BASTYAN'],
... '5-min')
     INTERVAL_DATETIME  COLUMNVALUES
0  2022-01-01 01:05:00         234.0

Parameters:

column_name – str, which column of dispatch data to aggregate and return. Should be one of NTERVAL_DATETIME, ASBIDRAMPUPMAXAVAIL (upper dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot exceed MAXAVAIL), ASBIDRAMPDOWNMINAVAIL (lower dispatch limit based on as bid ramp rate, when aggregated unit contribution cannot be less than zero), RAMPUPMAXAVAIL (upper dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot exceed AVAILABILITY), RAMPDOWNMINAVAIL (lower dispatch limit based lesser of as bid and telemetry ramp rates, when aggregated unit contribution cannot be less than zero), AVAILABILITY, TOTALCLEARED (as for after_dispatch_metrics), PASAAVAILABILITY, MAXAVAIL (as for as_bid_metrics), and FINALMW (the unit operating level at the end of the dispatch interval).
duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame containing columns INTERVAL_DATETIME, COLUMNVALUES (aggregate of column specified in input)

nem_bidding_dashboard.query_cached_data.aggregated_vwap(raw_data_cache, start_time, end_time, regions)[source]#

Function to query and aggregate price data from the raw data cache. To aggregate price data volume weighted averaged.

Examples:

>>> aggregated_vwap(
... raw_data_cache="D:/nemosis_data_cache",
... start_time="2022/01/01 01:00:00",
... end_time="2022/01/01 02:00:00",
... regions=['NSW'])
         SETTLEMENTDATE      PRICE
0   2022-01-01 01:05:00  107.80005
1   2022-01-01 01:10:00  107.80005
2   2022-01-01 01:15:00   91.92056
3   2022-01-01 01:20:00  107.80005
4   2022-01-01 01:25:00   91.37289
5   2022-01-01 01:30:00   91.38851
6   2022-01-01 01:35:00   92.14760
7   2022-01-01 01:40:00  100.27929
8   2022-01-01 01:45:00   91.90742
9   2022-01-01 01:50:00  100.30000
10  2022-01-01 01:55:00   85.00000
11  2022-01-01 02:00:00   85.00005

Parameters:

regions – list[str] of region to aggregate.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame with columns SETTLEMENTDATE, TOTALDEMAND and RRP (volume weighted avergae of energy price at regional reference nodes).

nem_bidding_dashboard.query_cached_data.duid_bids(raw_data_cache, start_time, end_time, duids, resolution, adjusted)[source]#

Function to query bidding data from a raw data cache. Data is filter according to the duids and time window provided, and returned on a duid basis. Data can queryed at hourly or 5 minute resolution. If an hourly resolution is chosen only bid for 5 minute interval ending on the hour are returned.

Examples:

>>> duid_bids(
... 'D:/nemosis_data_cache',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['AGLHAL', 'BASTYAN'],
... 'hourly',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 02:00:00   AGLHAL        7       32.0    557.39
1  2022-01-01 02:00:00   AGLHAL       10      121.0  14541.30
2  2022-01-01 02:00:00  BASTYAN        2       53.0    -55.64
3  2022-01-01 02:00:00  BASTYAN        4       28.0     -0.91
4  2022-01-01 02:00:00  BASTYAN       10        0.0  14021.86

>>> duid_bids(
... 'D:/nemosis_data_cache',
... "2022/01/01 01:00:00",
... "2022/01/01 01:05:00",
... ['AGLHAL', 'BASTYAN'],
... '5-min',
... 'adjusted')
     INTERVAL_DATETIME     DUID  BIDBAND  BIDVOLUME  BIDPRICE
0  2022-01-01 01:05:00   AGLHAL        7       32.0    557.39
1  2022-01-01 01:05:00   AGLHAL       10      121.0  14541.30
2  2022-01-01 01:05:00  BASTYAN        2       53.0    -55.64
3  2022-01-01 01:05:00  BASTYAN        4       28.0     -0.91
4  2022-01-01 01:05:00  BASTYAN       10        0.0  14021.86

Parameters:

duids – list[str] of duids to return in result.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
resolution – str ‘hourly’ or ‘5-min’
adjusted – str which bid data to use aggregate ‘raw’ or ‘adjusted’. Adjusted bid data has been adjusted down so the total bid does not exceed the unit availability.
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame with columns INTERVAL_DATETIME, DUID, BIDBAND, BIDVOLUME, and BIDPRICE

nem_bidding_dashboard.query_cached_data.region_demand(raw_data_cache, start_time, end_time, regions)[source]#

Query demand and price data from the raw data cache. To aggregate demand data is summed.

Examples:

>>> region_demand(
... raw_data_cache="D:/nemosis_data_cache",
... start_time="2022/01/01 01:00:00",
... end_time="2022/01/01 01:30:00",
... regions=['NSW'])
        SETTLEMENTDATE  TOTALDEMAND
0  2022-01-01 01:05:00      6631.21
1  2022-01-01 01:10:00      6655.52
2  2022-01-01 01:15:00      6496.85
3  2022-01-01 01:20:00      6520.86
4  2022-01-01 01:25:00      6439.22
5  2022-01-01 01:30:00      6429.13

Parameters:

regions – regions to aggregate demand and price from
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame with columns SETTLEMENTDATE and TOTALDEMAND (demand to be meet by schedualed and semischedualed generators, not including schedualed loads)

nem_bidding_dashboard.query_cached_data.stations_and_duids_in_regions_and_time_window(raw_data_cache, start_time, end_time, regions, dispatch_type, tech_types)[source]#

Function to query units from given regions with bids available in the given time window, with the given dispatch and technology type. Data returned is DUIDs and corresponding Station Names.

Examples:

>>> stations_and_duids_in_regions_and_time_window(
... 'D:/nemosis_data_cache',
... "2022/01/01 01:00:00",
... "2022/01/01 02:00:00",
... ['NSW'],
... "Generator",
... [])
        DUID             STATION NAME
0   BANGOWF1      Bango 973 Wind Farm
1   BANGOWF2      Bango 999 Wind Farm
2   BERYLSF1         Beryl Solar Farm
3   BLOWERNG  Blowering Power Station
4   BOCORWF1      Boco Rock Wind Farm
..       ...                      ...
69  WALGRVG1         Wallgrove BESS 1
70   WELLSF1    Wellington Solar Farm
71  WOODLWN1       Woodlawn Wind Farm
72     WRSF1    White Rock Solar Farm
73     WRWF1     White Rock Wind Farm

[74 rows x 2 columns]

Parameters:

regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
start_time – Initial datetime, formatted “DD/MM/YYYY HH:MM:SS”
end_time – Ending datetime, formatted identical to start_time
dispatch_type – str ‘Generator’ or ‘Load’
tech_types – list[str] the technology types to filter for e.g. Solar, Black Coal, CCGT. An empty list will result in no filtering by technology
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame with columns DUID and STATION NAME

nem_bidding_dashboard.query_cached_data.unit_types(raw_data_cache, regions, dispatch_type)[source]#

Function to query distinct unit types from raw data cache.

Examples:

>>> unit_types(
... 'D:/nemosis_data_cache',
... ['NSW'],
... 'Generator')
            UNIT TYPE
0             Bagasse
1   Battery Discharge
2          Black Coal
3                CCGT
4              Engine
5               Hydro
6                OCGT
7  Run of River Hydro
8               Solar
9                Wind

Parameters:

dispatch_type – str ‘Generator’ or ‘Load’
regions – list[str] regions to aggregate should only be QLD, NSW, VIC, SA or TAS.
raw_data_cache – Filepath to directory for caching files downloaded from AEMO

Returns:

pd.DataFrame column UNIT TYPE (this is the unit type as determined by the function nem_bidding_dashboard.preprocessing.tech_namer_by_row())