Developer setup#

Prerequisites#

While VEDA needs to be run in Windows, we recommend running the scripted data processing pipelines in a WSL Linux environment.

Ensure the following prerequisites are installed:

  • Windows 10 or later.

  • Windows Subsystem for Linux (WSL) with Ubuntu 22.04 or later.

  • Python 3.12 or later installed within WSL.

  • GAMS installed on Windows.

  • A valid GAMS license placed at C:\GAMS\44\gamslice.txt.

  • A working VEDA (VEDA 2.0 localhost) installation in your Windows user directory. See instructions here.

  • Poetry for Python dependency management. Install Poetry within WSL by running: curl -sSL https://install.python-poetry.org | python3 -

Getting the Code#

Clone the repository with submodules:

git clone git@github.com:EECA-NZ/TIMES-NZ-Model-Files.git

Workflow Overview#

All commands below assume you are using a WSL (Ubuntu) shell.

Step 1: Model Preparation#

Navigate into the PREPARE-TIMES-NZ directory and install Python dependencies using Poetry:

cd PREPARE-TIMES-NZ
poetry install --with dev

It’s a good idea to execute:

poetry env activate

from within PREPARE-TIMES-NZ or other modules managed by poetry. This returns a command which you can copy-paste and execute to activate the poetry environment shell. Doing this allows your terminal to run directly from the poetry shell, meaning that poetry run prefixes are not required to run scripts, doit, or pre-commit.

Building model files#

Run the data preparation pipeline (doit):

poetry run doit

doit runs the entire data preparation method and creates TIMES model files in PREPARE-TIMES-NZ/output.

Alternatively, run a specific script:

poetry run python scripts/prepare_times_nz.py

Running Tests#

Run tests within PREPARE-TIMES-NZ:

poetry run pytest

Note that these are unit tests for code, not data validation tests.

Poetry and VSCode#

Poetry creates a virtual environment that may not be in your project directory, so VSCode might not pick it up automatically. We need to set the VSCode interpreter to align with Poetry.

If VSCode fails to automatically register the env, poetry env activate from within the vscode terminal still works to move into the poetry shell. The VSCode python interpreter is slightly different, and should register automatically. It sometimes struggles if you are moving between different modules, like PREPARE-TIMES-NZ and TIMES-NZ-INTERNAL-QA. If so, you may need to follow the steps below:

  1. Run the following in your terminal to find the virtual environment path: poetry env info --path. This will give you something like /home/YourName/.cache/pypoetry/virtualenvs/prepare-times-nz-abc123

  2. Press Ctrl+Shift+P (or Cmd+Shift+P on Mac)

  3. Search for and select: Python: Select Interpreter

  4. Click Enter interpreter path

  5. Paste the path from step 1 and append /bin/python (or Scripts/python.exe on Windows), e.g.: /home/YourName/.cache/pypoetry/virtualenvs/prepare-times-nz-abc123/bin/python

Step 2: Running the Model Using VEDA#

The excel files for VEDA are generated in TIMES-NZ by the PREPARE-TIMES-NZ module by running doit. At this stage, it’s recommended that you move these to a separate area to work on in VEDA separately. Future iterations will hopefully automate this stage better.

This project assumes your VEDA installation is stored in Windows, under C:/Users/yourusername/Veda. VEDA is currently used for synchronising files, running the TIMES model, and generating output .vd files.

Step 3: Result post-processing#

Assuming the VEDA installation is in the expected location, you can automatically retrive VEDA model results and apply post-processing using the below script:

cd TIMES-NZ-INTERNAL-QA/src/times-nz-internal-qa/postprocessing
poetry run python run_all_postprocessing.py

Note that some components of this script rely on data generated in PREPARE-TIMES-NZ to assign TIMES process and commodity codes to the relevant categories and add labels. If you don’t have these, then you can skip the define_data steps of this script, and simply run the get_data and process_data functions.

If anything in the results is not defined, this will be flagged in the console.

Step 5: Visualization (internal qa method)#

Visualize using the TIMES-NZ visualization tools.

cd TIMES-NZ-INTERNAL-QA/
poetry run python run_local.py

Builds the explorer app to investigate results locally.

On this page
Show Source