Generation data being collected when it falls below 0kw

[ADIMANV]

Describe the bug

The data being collected from the SLDC website provided some faulty data. The data contained negative generation, which caused the error calculations for September to be a very high negative percentage. A fix in the backend should be implemented to ignore numbers that are below 0 kW so that our calculations are intact.
This issue is similar to : openclimatefix/ruvnl-consumer-app#20

Expected behavior

The generation data that is seen above where they cross the 0 kW threshold should be completely ignored.

[suvanbanerjee]

@peterdudfield @ADIMANV I’d like to work on this issue as well. Since openclimatefix/ruvnl-consumer-app#20 has already been merged, I believe I can complete this quickly.

[suvanbanerjee]

@peterdudfield @ADIMANV I have tried setting up the project locally, but the password is always wrong, first i encountered an error below

✔ analysis-dashboard                                 Built                                                                             0.0s 
 ✔ Network analysis-dashboard_analysis-net            Created                                                                           0.2s 
 ✔ Container analysis-dashboard-analysis-dashboard-1  Created                                                                           0.1s 
Attaching to analysis-dashboard-1
analysis-dashboard-1  | 2025-01-22 10:55:50.530 
analysis-dashboard-1  | Warning: the config option 'server.enableCORS=false' is not compatible with 'server.enableXsrfProtection=true'.
analysis-dashboard-1  | As a result, 'server.enableCORS' is being overridden to 'true'.
analysis-dashboard-1  | 
analysis-dashboard-1  | More information:
analysis-dashboard-1  | In order to protect against CSRF attacks, we send a cookie with each request.
analysis-dashboard-1  | To do so, we must specify allowable origins, which places a restriction on
analysis-dashboard-1  | cross-origin resource sharing.
analysis-dashboard-1  | 
analysis-dashboard-1  | If cross origin resource sharing is required, please disable server.enableXsrfProtection.
analysis-dashboard-1  |             
analysis-dashboard-1  | 
analysis-dashboard-1  | Collecting usage statistics. To deactivate, set browser.gatherUsageStats to false.
analysis-dashboard-1  | 
analysis-dashboard-1  | 
analysis-dashboard-1  |   You can now view your Streamlit app in your browser.
analysis-dashboard-1  | 
analysis-dashboard-1  |   URL: http://0.0.0.0:8501
analysis-dashboard-1  | 
analysis-dashboard-1  | 2025-01-22 10:55:50.643 Failed to load secrets.toml file
analysis-dashboard-1  | Traceback (most recent call last):
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 511, in loads
analysis-dashboard-1  |     ret = decoder.load_line(line, currentlevel, multikey,
analysis-dashboard-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 778, in load_line
analysis-dashboard-1  |     value, vtype = self.load_value(pair[1], strictly_valid)
analysis-dashboard-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 918, in load_value
analysis-dashboard-1  |     v = int(v, 0)
analysis-dashboard-1  |         ^^^^^^^^^
analysis-dashboard-1  | ValueError: invalid literal for int() with base 0: 'example'
analysis-dashboard-1  | 
analysis-dashboard-1  | During handling of the above exception, another exception occurred:
analysis-dashboard-1  | 
analysis-dashboard-1  | Traceback (most recent call last):
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/streamlit/web/bootstrap.py", line 114, in _on_server_start
analysis-dashboard-1  |     secrets.load_if_toml_exists()
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/streamlit/runtime/secrets.py", line 226, in load_if_toml_exists
analysis-dashboard-1  |     self._parse()
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/streamlit/runtime/secrets.py", line 382, in _parse
analysis-dashboard-1  |     path_secrets, found_secrets_file_in_path = self._parse_file_path(path)
analysis-dashboard-1  |                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/streamlit/runtime/secrets.py", line 340, in _parse_file_path
analysis-dashboard-1  |     return self._parse_toml_file(path)
analysis-dashboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/streamlit/runtime/secrets.py", line 280, in _parse_toml_file
analysis-dashboard-1  |     secrets.update(toml.loads(secrets_file_str))
analysis-dashboard-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
analysis-dashboard-1  |   File "/usr/local/lib/python3.12/site-packages/toml/decoder.py", line 514, in loads
analysis-dashboard-1  |     raise TomlDecodeError(str(err), original, pos)
analysis-dashboard-1  | toml.decoder.TomlDecodeError: invalid literal for int() with base 0: 'example' (line 1 column 1 char 0)

on changing the password to an int like 1234 the app runs, but I can't login even with correct password

Later just for testing I remove the check in auth.py and got the page below

can you also tell how to setup a local db for testing or do we have an db that we can use for testing and development

[peterdudfield]

@MAYANK12SHARMA could advive how you can run it locally? We should add this to the readme. You have to spin up a local database based on nowcastin_datamodel

[MAYANK12SHARMA]

I've already begun working on it, but it will take some time as the dashboard requires a fake database and a complete schema. I'm in the process of converting them into a .sql file that includes both the schema and the fake data. So far, I've completed about 40% of the work and will try to upload it within a week.

[suvanbanerjee]

@MAYANK12SHARMA any updates ?