Sinoptik removes valid data from AQI source #23

New Issue

admar · 2022-05-24T00:33:05+02:00

admar commented

2022-05-24 00:33:05 +02:00

It is now 2022-05-23T22:20+00:00.

JSON data from luchtmeetnet starts at 2022-05-23T21:00:00+00:00 (ignoring the first 5 elements which are stuck in the past) and increments in steps of 1 hour. Last element is 2022-05-24T15:00:00+00:00.

AQI data from Sinoptik starts at 1653368400 which is 2022-05-24T05:00:00+00:00 and increments in steps of 1 hour. Note that this is 7 hours Last element is 1653404400 which is 2022-05-24T03:00:00+00:00.

Pollen data from Sinoptik starts at 1653343200 which is 2022-05-23T22:00+00:00 and increments in steps of 1 hour. Last element is at 1653426000 which is 2022-05-24T21:00+00:00.

It looks like Sinoptik has correct data from pollen, but misinterprets timestamps from luchtmeetnet.

See attatched JSON documents for raw data from luchtmeetnet and from Sinoptik.

It is now 2022-05-23T22:20+00:00. JSON data from luchtmeetnet starts at 2022-05-23T21:00:00+00:00 (ignoring the first 5 elements which are stuck in the past) and increments in steps of 1 hour. Last element is 2022-05-24T15:00:00+00:00. AQI data from Sinoptik starts at 1653368400 which is 2022-05-24T05:00:00+00:00 and increments in steps of 1 hour. Note that this is 7 hours Last element is 1653404400 which is 2022-05-24T03:00:00+00:00. Pollen data from Sinoptik starts at 1653343200 which is 2022-05-23T22:00+00:00 and increments in steps of 1 hour. Last element is at 1653426000 which is 2022-05-24T21:00+00:00. It looks like Sinoptik has correct data from pollen, but misinterprets timestamps from luchtmeetnet. See attatched JSON documents for raw data from luchtmeetnet and from Sinoptik.

admar commented

2022-05-24 00:42:49 +02:00

Upon closer inspection it looks like Sinoptik removes all AQI data before 1653368400 (2022-05-24T05:00:00+00:00) while it should only remove data from before 2022-05-24T00:00:00+00:00. I'll update the title of this issue.

admar changed title from ~~Timestamps of AQI and PAQI are incorrect~~ to Sinoptik removes valid data from AQI source

2022-05-24 00:43:38 +02:00

paul added the

bug

label 2022-05-24 12:18:08 +02:00

paul self-assigned this 2022-05-24 12:18:14 +02:00

paul commented

2022-05-24 12:30:33 +02:00

Cannot reproduce for now with both my development instance and production instance that has been running for days.

It is now 2022-05-24T12:26:47+0200.
The first AQI item is at 1653386400 which is 2022-05-24T12:00:00+0200 and the last item is at 1653454800 which is 022-05-25T07:00:00+0200. Thus a total of 19 items.

The filter is really simple, so it can be caching (or a weird answer cached) but that only lasts for 1800.

Probably have to try later at night then?

Cannot reproduce for now with both my development instance and production instance that has been running for days. It is now 2022-05-24T12:26:47+0200. The first AQI item is at 1653386400 which is 2022-05-24T12:00:00+0200 and the last item is at 1653454800 which is 022-05-25T07:00:00+0200. Thus a total of 19 items. The filter is [really simple](https://git.luon.net/paul/sinoptik/src/commit/3451445de1d293f28882b1de059cc17f06ff2d03/src/providers/luchtmeetnet.rs#L90), so it can be caching (or a weird answer cached) but that only lasts for 1800. Probably have to try later at night then?

admar commented

2022-06-04 09:33:13 +02:00

Right now Sinoptik again removes valid data. AQI according to Sinoptik is from 1654326000 (Saturday, June 4, 2022 9:00:00 AM GMT+02:00 DST) to 1654390800 (Sunday, June 5, 2022 3:00:00 AM GMT+02:00 DST). AQI according to Luchtmeetnet is from 2022-06-04T09:00:00+00:00 to 2022-06-05T01:00:00+00:00 (ignoring first 7 entries since they are in the past)

See attached json files.

Right now Sinoptik again removes valid data. AQI according to Sinoptik is from 1654326000 (Saturday, June 4, 2022 9:00:00 AM GMT+02:00 DST) to 1654390800 (Sunday, June 5, 2022 3:00:00 AM GMT+02:00 DST). AQI according to Luchtmeetnet is from 2022-06-04T09:00:00+00:00 to 2022-06-05T01:00:00+00:00 (ignoring first 7 entries since they are in the past) See attached json files.

paul commented

2022-06-05 17:01:31 +02:00

This seems to be related to (map) caching. We probably need to address #5 first to find out if and why the cache is not updated.
Refresh log from before the moment mentioned above:

Jun 04 00:20:29 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 00:50:30 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 01:20:31 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 01:50:33 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 02:20:35 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 02:50:36 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 03:20:38 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 03:50:39 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 04:20:40 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 04:50:42 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 06:40:55 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 07:10:57 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 07:40:59 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 08:11:00 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 08:41:01 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 09:11:03 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545
Jun 04 09:34:21 target sinoptik[23893]: ▶️  Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.43926&longitude=5.47863

This seems to be related to (map) caching. We probably need to address #5 first to find out if and why the cache is not updated. Refresh log from before the moment mentioned above: ``` Jun 04 00:20:29 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 00:50:30 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 01:20:31 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 01:50:33 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 02:20:35 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 02:50:36 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 03:20:38 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 03:50:39 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 04:20:40 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 04:50:42 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 06:40:55 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 07:10:57 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 07:40:59 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 08:11:00 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 08:41:01 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 09:11:03 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.44540&longitude=5.51545 Jun 04 09:34:21 target sinoptik[23893]: ▶️ Retrieving Luchtmeetnet data from: https://api.luchtmeetnet.nl/open_api/concentrations?formula=lki&latitude=51.43926&longitude=5.47863 ```

paul added the

question

label 2023-05-26 20:51:44 +02:00

paul commented

2023-05-26 20:52:00 +02:00

Is this still a thing?

Sign in to join this conversation.