kweather: Improve station list processing

+ Add a README file with information about the processing stations
+ Automatically generate stations.dat and weather_stations.desktop
  from data/nsd_cccc.txt before cmake checks
+ File downloaded from https://tgftp.nws.noaa.gov/data/nsd_cccc.txt
+ Clean up weather_stations.desktop.in and moved relevant information
  to data/station_names.txt
+ Added GG to EU and NU, PE to Canada
+ Mapped most of African countries to AF region
+ Mapped few other countries to their regions
+ Algeria is not in the ME region
+ Christmas Islands code is CX
+ Added Greenland to Atlantic
+ Added some more states to Oceania
+ Added some more states to Cenetral and South Americas
+ Antarctica as new region
+ Renamed Australiasia to Australia and Oceania
+ Region abbreviations in alphabetic order
+ Removed Atlantic and Mexico regions and added North America
+ Mapped most of the outstanding stations in Canada and elsewhere
+ J Leslie Turriff provided information on the stations in Canada

Signed-off-by: Emanoil Kotsev <deloptes@gmail.com>

Modify the generation of weather_stations.desktop so that it is possible
to merge translations into a generated file.

Signed-off-by: Slávek Banko <slavek.banko@axis.cz>
pull/10/head
Emanoil Kotsev 3 years ago committed by Slávek Banko
parent f89d317e76
commit f48efce561
No known key found for this signature in database
GPG Key ID: 608F5293A04BE668

@ -11,7 +11,7 @@ tde_l10n_create_template(
tde_l10n_create_template(
CATALOG "desktop_files/kweather-stations/"
SOURCES weather_stations.desktop
SOURCES_DESKTOP weather_stations.desktop.in
KEYWORDS name
DESTINATION "${CMAKE_SOURCE_DIR}/translations"
)

@ -45,6 +45,21 @@ tde_add_executable( kweatherservice AUTOMOC
)
##### generate stations.dat and weather_stations.desktop
if( NOT PERL_EXECUTABLE )
include( FindPerl )
endif( )
execute_process(
COMMAND ${PERL_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/genstations.pl
${CMAKE_CURRENT_SOURCE_DIR}/data/nsd_cccc.txt
${CMAKE_CURRENT_SOURCE_DIR}/data/station_names.txt
${CMAKE_CURRENT_SOURCE_DIR}/weather_stations.desktop.in
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
)
##### sun_test (test) ###########################
# FIXME: add correct test targets support
#
@ -118,13 +133,14 @@ tde_install_icons( kweather )
##### other data ################################
tde_create_translated_desktop(
SOURCE weather_stations.desktop
SOURCE ${CMAKE_CURRENT_BINARY_DIR}/weather_stations.desktop
KEYWORDS name
DESTINATION ${DATA_INSTALL_DIR}/kweatherservice
PO_DIR kweather-stations
)
install( FILES stations.dat
install(
FILES ${CMAKE_CURRENT_BINARY_DIR}/stations.dat
DESTINATION ${DATA_INSTALL_DIR}/kweatherservice
)

@ -0,0 +1,77 @@
January 2021
====================
Stations are retrieved from https://tgftp.nws.noaa.gov/data/nsd_cccc.txt
wget -O data/nsd_cccc.txt https://tgftp.nws.noaa.gov/data/nsd_cccc.txt
dos2unix data/nsd_cccc.txt
data/station_names.txt is used to provide correct orthography of the area
where the station is located, which is used in the configuration dialog
of kweather applet.
The format of station_names.txt is ICAO;status;region_state;name;INFO1;INFO2
If status is one of none|false|na|decomissioned the station is ignored.
INFO1 is what was previously the 5th position in weather_stations.desktop
makred with ------
INFO2 is what was previously the 6th position in weather_stations.desktop
makred with ---
The source nsd_cccc.txt has illed information and structure that needs to be
corrected in the files used in kweather. Some of the problems are:
1. decomissioned stations
2. names of station locations
The genstations.pl is based on following idea
Based on the input nsd_cccc.txt and the station_names.txt file we can
generate stations.dat like following:
for each station in nsd_cccc.txt do
check if station ID matches one in station_names.txt
if it matches then
use the name of the station
else
use the name from nsd_cccc.txt
write entry/line in stations.dat
Based on stations.dat and weather_stations.desktop.in we can generate
weather_stations.desktop.
weather_stations.desktop will be updated with the names from stations.dat
and the additional information will be preserved like following.
for each station in stations.dat
for each region from weather_stations.desktop.in
if state is provided in stations.dat
use the state to build a key with region and state
else
for each state in the states of the region
use the state to build a key
if section for this key exists
and
if name of the section matches
stop processing
proper section was found
we add the station here
write entry for the station
There are few problems with this approach and the ill provided nsd_cccc.txt data
1. States are missing and thus correlation between station and state can not be done
2. When State is missing, based on the country/state name some entries are ambig
Example Georgia and US state of Georgia
Country list seemed to be used for the country names and as convention here
wget ftp://ftp.ncdc.noaa.gov/pub/data/noaa/country-list.txt but the FIPS notation was
dropped in 2008 as a standard, however continiues to be used.
"The Census Bureau decided that, based on decades of using the terminology FIPS to
describe its codes, it would continue to use the FIPS name for its updated codes,
where FIPS now stood for FIP "Series", since there no longer existed an official
FIP "Standard". [https://en.wikipedia.org/wiki/FIPS_county_code]
A mapping to ISO 3166 ALPHA-2 https://laendercode.net/en/2-letter-list.html
seems more appropriate.
TODO: check the country code mapping in weather_stations.desktop.in

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

@ -0,0 +1,190 @@
# This file is part of KWeather.
# Copyright (c) 2021 Emanoil Kotsev <deloptes@gmail.com>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
# As a special exception, permission is given to link this program
# with any edition of TQt, and distribute the resulting executable,
# without including the source code for TQt in the source distribution.
use strict;
use warnings;
use utf8;
#use Data::Dumper qw(Dumper);
######## pick arguments
my $stationsNSD = $ARGV[0]; # data/nsd_cccc.txt
my $stationNames = $ARGV[1]; # data/station_names.txt
my $desktopIn = $ARGV[2]; # weather_stations.desktop.in
######## generate stations.dat
#input format ICAO;status;location;name;INFO1;INFO2
my %stnames;
open(FDSN, "<", $stationNames) || die "can not open file $!";
while (<FDSN>) {
my $line = $_; chomp $line;
my ($key,$status,$loc,$name,$v1,$v2) = split(/;/,$line);
$status = '' if $status =~ /none|false|na|decomissioned/;
$stnames{$key} = { 'status' => $status,
'loc' => $loc,
'name' => $name,
'v1' => $v1,
'v2' => $v2
};
}
close(FDSN);
open(FDNSD, "<", $stationsNSD) || die "can not open file $!";
open(FDOUT, ">", "stations.dat") || die "can not open file $!";
while (<FDNSD>) {
my $line = $_; chomp $line;
my @entries = split(/;/,$line,6);
if ( exists $stnames{$entries[0]} ) {
next if ! exists $stnames{$entries[0]}{'status'};
$entries[3] = $stnames{$entries[0]}{'name'};
}
print FDOUT join(';', @entries) , "\n";
# print join(';', @entries) , "\n" ;
}
close(FDOUT);
close(FDNSD);
#print Dumper \%stnames and exit;
######## generate weather_stations.desktop
my $section;
my %data;
## parse weather_stations.desktop.in
open(FDWSTIN, "<", $desktopIn) || die "can not open file $!";
while (my $line = <FDWSTIN>) {
if ($line =~ /^\s*#/) {
next; # skip comments
}
if ($line =~ /^\s*$/) {
next; # skip empty lines
}
if ($line =~ /^\[(.*)\]\s*$/) {
$section = $1;
next;
}
if ($line =~ /^([^=]+?)\s*=\s*(.*?)\s*$/) {
my ($field, $value) = ($1, $2);
if (not defined $section) {
warn "Line outside of seciton '$line'\n";
next;
}
$data{$section}{$field} = $value;
}
}
close(FDWSTIN);
#print Dumper \%data and exit;
open(FDST, "<", "stations.dat") || die "can not open file $!";
while(<FDST>) {
my $line = $_; chomp $line;
my @entries = split(/;/,$line);
#print Dumper \@entries and exit;
my $k;
my $found = 0;
if ( ! exists $stnames{$entries[0]}{'loc'} ) {
my @regions = split (/\s/, $data{'Main'}{'regions'});
foreach my $r (@regions) {
if (!$entries[4]) {
my @states = split (/\s/, $data{$r}{'states'});
foreach my $s (@states) {
$k = "$r" . "_" . "$s";
if ( exists $data{$k}{'name'}
and $data{$k}{'name'} eq $entries[5]) { # state/country matches
$found = 1;
last;
}
}
} else {
$k = "$r" . "_" . $entries[4];
$found = 1;
}
last if $found;
}
} else {
$k = $stnames{$entries[0]}{'loc'};
$found = 1;
}
if ( ! $found ) {
warn "not found in weather_stations.desktop.in: $entries[0]\t$entries[5]\t$entries[3]\n";
next ;
}
my $name = $entries[3] ;
$name =~ s/\s+$//; # remove last space from the name
$name =~ s/\s/\\\\\ /g; # replace space with \\space
my ($v1,$v2) ;
$v1 = $stnames{$entries[0]}{'v1'} if exists $stnames{$entries[0]}{'v1'};
$v2 = $stnames{$entries[0]}{'v2'} if exists $stnames{$entries[0]}{'v2'};
$v1 = "-------" if !$v1;
$v2 = "---" if !$v2;
$data{$k}{'loc'}[++$#{$data{$k}{'loc'}}] = $name ." ".$entries[0] ." $v1 $v2";
}
close(FDST);
#print Dumper \%data;
## write output to weather_stations.desktop
open(FDWSTOUT, ">", "weather_stations.desktop") || die "can not open file $!";
print FDWSTOUT "[Main]" . "\n";
print FDWSTOUT "regions=" . $data{'Main'}{'regions'} . "\n";
print FDWSTOUT "\n";
my @regions = split (/\s+/, $data{'Main'}{'regions'});
foreach my $r (@regions) {
print FDWSTOUT "[$r]" . "\n";
print FDWSTOUT "name=" . $data{$r}{'name'} . "\n";
foreach my $f (keys %{$data{$r}}) {
next if $f eq 'states';
next if $f eq 'name';
print FDWSTOUT $f . "=" . $data{$r}{$f} . "\n";
}
print FDWSTOUT "states=" . $data{$r}{'states'} . "\n";
print FDWSTOUT "\n";
my @states = split (/\s+/, $data{$r}{'states'});
foreach my $s (@states) {
my $k = "$r" . "_" . "$s";
print FDWSTOUT "[" . $k . "]\n";
print FDWSTOUT "name=" . $data{$k}{'name'} . "\n";
for ( my $i=0; $i <= $#{$data{$k}{'loc'}}; $i++ ) {
print FDWSTOUT "loc" . $i . "=" . $data{$k}{'loc'}[$i] . "\n";
}
print FDWSTOUT "\n";
}
}
close(FDWSTOUT);

File diff suppressed because it is too large Load Diff

@ -0,0 +1,877 @@
[Main]
regions=AF AN AS CA EU ME M_ NA OZ US
[ME]
name=Middle East
states=AE AF BH IR IL IQ JO KW LB OM QA SA SY YE
[ME_AE]
name=United Arab Emirates
[ME_AF]
name=Afghanistan
[ME_BH]
name=Bahrain
[ME_QA]
name=Qatar
[ME_OM]
name=Oman
[ME_YE]
name=Yemen
[ME_SA]
name=Saudi Arabia
[ME_IL]
name=Israel
[ME_LB]
name=Lebanon
[ME_IR]
name=Iran
[ME_IQ]
name=Iraq
[ME_JO]
name=Jordan
[ME_KW]
name=Kuwait
[ME_SY]
name=Syria
[US]
name=United States
states=AL AK AZ AR CA CO CT DE DC FL GA HI ID IL IN IA KS KY LA ME MA MD MI MS MN MO MT NE NH NM NV NY ND NJ NC OH OK OR PA RI SC SD TN TX UT VT VA WA WI WV WY
[US_AK]
name=Alaska
[US_AR]
name=Arkansas
[US_DC]
name=District of Columbia
[US_ID]
name=Idaho
[US_IN]
name=Indiana
[US_IA]
name=Iowa
[US_KS]
name=Kansas
[US_KY]
name=Kentucky
[US_LA]
name=Lousiana
[US_ME]
name=Maine
[US_MS]
name=Mississippi
[US_MO]
name=Missouri
[US_MT]
name=Montana
[US_NE]
name=Nebraska
[US_NH]
name=New Hampshire
[US_NM]
name=New Mexico
[US_ND]
name=North Dakota
[US_OK]
name=Oklahoma
[US_SD]
name=South Dakota
[US_UT]
name=Utah
[US_VT]
name=Vermont
[US_WY]
name=Wyoming
[US_PA]
name=Pennsylvania
[US_CA]
name=California
[US_AL]
name=Alabama
[US_AZ]
name=Arizona
[US_CO]
name=Colorado
[US_CT]
name=Connecticut
[US_DE]
name=Delaware
[US_FL]
name=Florida
[US_GA]
name=Georgia
[US_HI]
name=Hawaii
[US_IL]
name=Illinois
[US_IN]
name=Indiana
[US_MD]
name=Maryland
[US_MI]
name=Michigan
[US_MN]
name=Minnesota
[US_NV]
name=Nevada
[US_NJ]
name=New Jersey
[US_NC]
name=North Carolina
[US_OH]
name=Ohio
[US_OR]
name=Oregon
[US_RI]
name=Rhode Island
[US_SC]
name=South Carolina
[US_TN]
name=Tennessee
[US_TX]
name=Texas
[US_VA]
name=Virginia
[US_WV]
name=West Virginia
[US_WI]
name=Wisconsin
[US_MA]
name=Massachusetts
[US_NY]
name=New York
[US_WA]
name=Washington
[CA]
name=Canada
states=AB BC MB NB NF NS NT NU ON PE QC SK YK
[CA_AB]
name=Alberta
[CA_BC]
name=British Columbia
[CA_MB]
name=Manitoba
[CA_NB]
name=New Brunswick
[CA_NF]
name=Newfoundland
[CA_NS]
name=Nova Scotia
[CA_NT]
name=Northwest Territories
[CA_NU]
name=Nunavut
[CA_ON]
name=Ontario
[CA_PE]
name=Prince Edward Island
[CA_QC]
name=Quebec
[CA_SK]
name=Saskatchewan
[CA_YK]
name=Yukon
[NA]
name=North America
states=GL MX PM
[NA_GL]
name=Greenland
[NA_MX]
name=Mexico
[NA_PM]
name=Saint Pierre and Miquelon
[EU]
name=Europe
states=AB OS BE BA BY BG CZ HR CY DK EE FI FR MK DE GG GI GR HU IE IS IT LV LT LU MT MD NL NO PL PT RO RU SK SI SP SE CH TR UA UK YU
[EU_AB]
name=Albania
[EU_OS]
name=Austria
[EU_BE]
name=Belgium
[EU_BA]
name=Bosnia and Herzegovina
[EU_BG]
name=Bulgaria
[EU_BY]
name=Belarus
[EU_CZ]
name=Czech Republic
[EU_HR]
name=Croatia
[EU_DK]
name=Denmark
[EU_EE]
name=Estonia
[EU_GI]
name=Gibraltar
[EU_GG]
name=Georgia
[EU_IS]
name=Iceland
[EU_LV]
name=Latvia
[EU_LT]
name=Lithuania
[EU_LU]
name=Luxembourg
[EU_MK]
name=Macedonia, The Republic of
[EU_MT]
name=Malta
[EU_MD]
name=Moldova
[EU_NO]
name=Norway
[EU_PL]
name=Poland
[EU_PT]
name=Portugal
[EU_RO]
name=Romania
[EU_RU]
name=Russia
[EU_UA]
name=Ukraine
[EU_SK]
name=Slovakia
[EU_SI]
name=Slovenia
[EU_SE]
name=Sweden
[EU_CH]
name=Switzerland
[EU_TR]
name=Turkey
[EU_CY]
name=Cyprus
[EU_FI]
name=Finland
[EU_FR]
name=France
[EU_DE]
name=Germany
[EU_GR]
name=Greece
[EU_HU]
name=Hungary
[EU_IE]
name=Ireland
[EU_IT]
name=Italy
[EU_NL]
name=Netherlands
[EU_SP]
name=Spain
[EU_UK]
name=United Kingdom
[EU_YU]
name=Serbia and Montenegro
[AF]
name=Africa
states=AO BF BI BJ BW CI CD CF CG CM CV DJ DZ EG EH ET GA GH GQ GN GM GS GW KE KM LR LS LY MG ML MR MO MW MU MZ NG NE NA RE RW SC SD SH SL SN SO ST SZ TD TG TN TZ UG ZA ZM ZW
[AF_DZ]
name=Algeria
[AF_AO]
name=Angola
[AF_BJ]
name=Benin
[AF_BW]
name=Botswana
[AF_BF]
name=Burkina Faso
[AF_BI]
name=Burundi
[AF_CV]
name=Cape Verde
[AF_TD]
name=Chad
[AF_KM]
name=Comoros
[AF_CI]
name=Cote d'Ivoire
[AF_CD]
name=Congo, Democratic Republic of the
[AF_CF]
name=Central African Republic
[AF_CG]
name=Congo, Republic of the
[AF_CM]
name=Cameroon
[AF_DJ]
name=Djibouti
[AF_EG]
name=Egypt
[AF_ET]
name=Ethiopia
[AF_GQ]
name=Equatorial Guinea
[AF_GA]
name=Gabon
[AF_GM]
name=Gambia, The
[AF_GH]
name=Ghana
[AF_GN]
name=Guinea
[AF_GW]
name=Guinea-Bissau
[AF_KE]
name=Kenya
[AF_LS]
name=Lesotho
[AF_LR]
name=Liberia
[AF_LY]
name=Libya
[AF_MG]
name=Madagascar
[AF_MW]
name=Malawi
[AF_ML]
name=Mali
[AF_MR]
name=Mauritania
[AF_MU]
name=Mauritius
[AF_MO]
name=Morocco
[AF_MZ]
name=Mozambique
[AF_NA]
name=Namibia
[AF_NG]
name=Nigeria
[AF_NE]
name=Niger
[AF_RE]
name=Reunion
[AF_RW]
name=Rwanda
[AF_SH]
name=Saint Helena
[AF_ST]
name=Sao Tome and Principe
[AF_SN]
name=Senegal
[AF_SC]
name=Seychelles
[AF_SL]
name=Sierra Leone
[AF_SO]
name=Somalia
[AF_ZA]
name=South Africa
[AF_GS]
name=South Georgia and the Islands
[AF_SD]
name=Sudan
[AF_SZ]
name=Swaziland
[AF_TZ]
name=Tanzania
[AF_TG]
name=Togo
[AF_TN]
name=Tunisia
[AF_UG]
name=Uganda
[AF_EH]
name=Western Sahara
[AF_ZM]
name=Zambia
[AF_ZW]
name=Zimbabwe
[OZ]
name=Australia and Oceania
states=AU BN CK CX FJ FM IN KI MA MH MP NC NR NZ PF PI PG PW SB TO TV UM VU WS
[OZ_AU]
name=Australia
[OZ_BN]
name=Brunei
[OZ_CK]
name=Cook Islands
[OZ_CX]
name=Christmas Island
[OZ_FJ]
name=Fiji
[OZ_PF]
name=French Polynesia
[OZ_KI]
name=Kiribati
[OZ_MA]
name=Malaysia
[OZ_MH]
name=Marshall Islands
[OZ_FM]
name=Micronesia, Federated States of
[OZ_NC]
name=New Caledonia
[OZ_NR]
name=Nauru
[OZ_NZ]
name=New Zealand
[OZ_MP]
name=NORTHERN MARIANA ISLAND
[OZ_IN]
name=Indonesia
[OZ_PW]
name=Palau
[OZ_PG]
name=Papua New Guinea
[OZ_PI]
name=Philippines
[OZ_SB]
name=Solomon Islands
[OZ_WS]
name=Samoa
[OZ_TO]
name=Tonga
[OZ_TV]
name=Tuvalu
[OZ_UM]
name=United States Minor Outlying Islands
[OZ_VU]
name=Vanuatu
[AS]
name=Asia
states=AM AZ BA CN ET HK IN JP KG KH KR KP KZ LA MN MO MV MY NE PK SG SL TJ TH TM TW UZ VN
[AS_AM]
name=Armenia
[AS_AZ]
name=Azerbaijan
[AS_BA]
name=Bangladesh
[AS_KH]
name=Cambodia
[AS_CN]
name=China
[AS_ET]
name=East Timor
[AS_HK]
name=Hong Kong
[AS_IN]
name=India
[AS_JP]
name=Japan
[AS_KZ]
name=Kazakhstan
[AS_KR]
name=Korea, South
[AS_KP]
name=Korea, North
[AS_KG]
name=Kyrgyzstan
[AS_LA]
name=Laos
[AS_MO]
name=Macau
[AS_MV]
name=Maldives
[AS_MN]
name=Mongolia
[AS_MY]
name=Myanmar
[AS_NE]
name=Nepal
[AS_PK]
name=Pakistan
[AS_SG]
name=Singapore
[AS_SL]
name=Sri Lanka
[AS_TJ]
name=Tajikistan
[AS_TW]
name=Taiwan
[AS_TH]
name=Thailand
[AS_TM]
name=Turkmenistan
[AS_UZ]
name=Uzbekistan
[AS_VN]
name=Vietnam
[M_]
name=Central and South America
states=AI AG AR AW BB BR BS BZ BO EC CL CO CR CU DM DO FK GD GF GP GT GY HT HN JM KN KY LC MQ NI PA PE PR PY SR SV TC VC VE VG VI UY
[M__AI]
name=Anguilla
[M__AG]
name=Antigua and Barbuda
[M__AR]
name=Argentina
[M__AW]
name=Aruba
[M__BB]
name=Barbados
[M__BS]
name=Bahamas, The
[M__BZ]
name=Belize
[M__BM]
name=Bermuda
[M__BO]
name=Bolivia
[M__BR]
name=Brazil
[M__VG]
name=British Virgin Islands
[M__KY]
name=Cayman Islands
[M__CL]
name=Chile
[M__CR]
name=Costa Rica
[M__CO]
name=Colombia
[M__CU]
name=Cuba
[M__DM]
name=Dominica
[M__DO]
name=Dominican Republic
[M__EC]
name=Ecuador
[M__SV]
name=El Salvador
[M__GF]
name=French Guiana
[M__FK]
name=Falkland Islands, Islas Malvinas
[M__GD]
name=Grenada
[M__GT]
name=Guatemala
[M__GP]
name=Guadeloupe
[M__GY]
name=Guyana
[M__HT]
name=Haiti
[M__HN]
name=Honduras
[M__JM]
name=Jamaica
[M__MQ]
name=Martinique
[M__AN]
name=Netherlands Antilles
[M__NI]
name=Nicaragua
[M__PA]
name=Panama
[M__PE]
name=Peru
[M__PR]
name=Puerto Rico
[M__PY]
name=Paraguay
[M__KN]
name=Saint Kitts and Nevis
[M__LC]
name=Saint Lucia
[M__VC]
name=Saint Vincent and the Grenadines
[M__SR]
name=Suriname
[M__TT]
name=Trinidad and Tobago
[M__TC]
name=Turks and Caicos Islands
[M__UY]
name=Uruguay
[M__VE]
name=Venezuela
[M__VI]
name=Virgin Islands
[AN]
name=Antarctica
states=AN
[AN_AN]
name=Antarctica
Loading…
Cancel
Save