Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Launching differentiated policy data #131

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open

Launching differentiated policy data #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
dd95e8b
update for differentiated policies
TobyPhillips Jul 25, 2022
a7ba6db
Update README.md
TobyPhillips Jul 25, 2022
9540268
update codebook for differentiated policies
TobyPhillips Jul 25, 2022
faf14fe
Update with differentiated policies
TobyPhillips Jul 25, 2022
468739b
update for differentiated policies
TobyPhillips Jul 25, 2022
d09f5de
Update get_data.yml
TobyPhillips Jul 25, 2022
963de41
Update get_data.yml
TobyPhillips Jul 25, 2022
4319793
Update get_data.yml
TobyPhillips Jul 25, 2022
7a09926
Update get_data.yml
TobyPhillips Jul 25, 2022
da5b1a4
Update get_data.yml
TobyPhillips Jul 25, 2022
9045017
Update get_data.yml
TobyPhillips Jul 25, 2022
310b762
Update get_data.yml
TobyPhillips Jul 25, 2022
90e9546
OxCGRT Data Update
actions-user Jul 25, 2022
1b0c5b4
Update get_data.yml
TobyPhillips Jul 25, 2022
afe2b4d
add mailing list details
TobyPhillips Jul 26, 2022
5008c22
add note that V means different things
TobyPhillips Jul 26, 2022
435a396
update paras that describe data
TobyPhillips Jul 26, 2022
f0ede0e
Update README.md
TobyPhillips Jul 26, 2022
c54adb3
Create README.md
TobyPhillips Jul 26, 2022
733d40c
change heading level
TobyPhillips Jul 26, 2022
fa68c1e
Update README.md
TobyPhillips Jul 26, 2022
172f18b
Update README.md
TobyPhillips Jul 26, 2022
759abdf
link to new R script
TobyPhillips Jul 26, 2022
2b956a6
add sentence about non-current CSV formats
TobyPhillips Jul 27, 2022
bf9a23a
Update get_data.yml
TobyPhillips Jul 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/analysis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: An R workflow to automate daily updates of main repo graphs and timeseries csv
name: An R workflow to automate daily updates of main repo timeseries csv
on:
schedule:
- cron: '45 7 * * *'
Expand Down Expand Up @@ -29,12 +29,12 @@ jobs:
ls
source <(curl -s https://raw.githubusercontent.com/saptahash/test_repo/master/mainrepo/sysrequirements.sh)

- name: Run automated script for graphs and timeseries
- name: Run automated script for timeseries
run: |
ls
cd master
ls
Rscript -e 'source("https://raw.githubusercontent.com/saptahash/test_repo/master/mainrepo/githubroutine.r")'
Rscript -e 'source("https://raw.githubusercontent.com/OxCGRT/covid-policy-scratchpad/master/r-code/create_main_timeseries.R")'

- name: Commit changes
run: |
Expand All @@ -44,5 +44,5 @@ jobs:
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
git add .
git diff-index --quiet HEAD || git commit -m "OxCGRT Graphs and timeseries Update" -a
git diff-index --quiet HEAD || git commit -m "OxCGRT timeseries update" -a
git push origin master:master
59 changes: 44 additions & 15 deletions .github/workflows/get_data.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Data link to OxCGRT
on:
schedule:
- cron: '1 2-23 * * *'
push
# schedule:
# - cron: '1 2-23 * * *'

jobs:
build:
Expand All @@ -12,25 +13,53 @@ jobs:
fetch-depth: 1
- name: Download raw csv from OxCGRT database
run: |
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload?type=Compressed' -s -f -o 'data/OxCGRT_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload?type=changes_responses' -s -f -o 'data/OxCGRT_latest_responses.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload?type=changes_all' -s -f -o 'data/OxCGRT_latest_allchanges.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload?type=combined' -s -f -o 'data/OxCGRT_latest_combined.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new' -s -f -o 'data/OxCGRT_nat_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=vaccine_full' -s -f -o 'data/OxCGRT_vaccines_full.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload' -s -f -o 'data/OxCGRT_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/OxCGRT_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/OxCGRT_withnotes_" year ".csv")}' 'data/OxCGRT_withnotes.csv'
rm 'data/OxCGRT_withnotes.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes' -s -f -o 'data/OxCGRT_nat_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/OxCGRT_nat_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/OxCGRT_nat_differentiated_withnotes_" year ".csv")}' 'data/OxCGRT_nat_differentiated_withnotes.csv'
rm 'data/OxCGRT_nat_differentiated_withnotes.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new&countryCode=AUS' -s -f -o 'data/Australia/OxCGRT_AUS_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes&countryCode=AUS' -s -f -o 'data/Australia/OxCGRT_AUS_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/Australia/OxCGRT_AUS_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/Australia/OxCGRT_AUS_differentiated_withnotes_" year ".csv")}' 'data/Australia/OxCGRT_AUS_differentiated_withnotes.csv'
rm 'data/Australia/OxCGRT_AUS_differentiated_withnotes.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload?type=Compressed' -s -f -o 'data/OxCGRT_latest LEGACY.csv'
awk --lint -F "," 'NR==1 {header=$0}; NF>1 && NR>1 && ($2=="BRA" || $2=="IND") {countrycode=$2; if(! files[countrycode]) {print header > ("data/OxCGRT_" countrycode "_latest LEGACY.csv"); files[countrycode]=1}; print $0 > ("data/OxCGRT_" countrycode "_latest LEGACY.csv")}' 'data/OxCGRT_latest LEGACY.csv'
mv 'data/OxCGRT_BRA_latest LEGACY.csv' 'data/Brazil/OxCGRT_BRA_latest LEGACY.csv'
mv 'data/OxCGRT_IND_latest LEGACY.csv' 'data/India/OxCGRT_IND_latest LEGACY.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/CSVDownload' -s -f -o 'data/OxCGRT_withnotes LEGACY.csv'
awk --lint -F "," 'NR==1 {header=$0}; NF>1 && NR>1 && ($2=="BRA" || $2=="IND") {countrycode=$2; if(! files[countrycode]) {print header > ("data/OxCGRT_" countrycode "_withnotes LEGACY.csv"); files[countrycode]=1}; print $0 > ("data/OxCGRT_" countrycode "_withnotes LEGACY.csv")}' 'data/OxCGRT_withnotes LEGACY.csv'
mv 'data/OxCGRT_BRA_withnotes LEGACY.csv' 'data/Brazil/OxCGRT_BRA_withnotes LEGACY.csv'
mv 'data/OxCGRT_IND_withnotes LEGACY.csv' 'data/India/OxCGRT_IND_withnotes LEGACY.csv'
rm 'data/OxCGRT_latest LEGACY.csv'
rm 'data/OxCGRT_withnotes LEGACY.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new&countryCode=CAN' -s -f -o 'data/Canada/OxCGRT_CAN_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes&countryCode=CAN' -s -f -o 'data/Canada/OxCGRT_CAN_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/Canada/OxCGRT_CAN_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/Canada/OxCGRT_CAN_differentiated_withnotes_" year ".csv")}' 'data/Canada/OxCGRT_CAN_differentiated_withnotes.csv'
rm 'data/Canada/OxCGRT_CAN_differentiated_withnotes.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new&countryCode=CHN' -s -f -o 'data/China/OxCGRT_CHN_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes&countryCode=CHN' -s -f -o 'data/China/OxCGRT_CHN_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/China/OxCGRT_CHN_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/China/OxCGRT_CHN_differentiated_withnotes_" year ".csv")}' 'data/China/OxCGRT_CHN_differentiated_withnotes.csv'
rm 'data/China/OxCGRT_CHN_differentiated_withnotes.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new&countryCode=GBR' -s -f -o 'data/United Kingdom/OxCGRT_GBR_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes&countryCode=GBR' -s -f -o 'data/United Kingdom/OxCGRT_GBR_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/United Kingdom/OxCGRT_GBR_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/United Kingdom/OxCGRT_GBR_differentiated_withnotes_" year ".csv")}' 'data/United Kingdom/OxCGRT_GBR_differentiated_withnotes.csv'
rm 'data/United Kingdom/OxCGRT_GBR_differentiated_withnotes.csv'

curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=compressed_new&countryCode=USA' -s -f -o 'data/United States/OxCGRT_USA_latest.csv'
curl 'https://oxcgrtportal.azurewebsites.net/api/csvdownload?type=differentiated_notes&countryCode=USA' -s -f -o 'data/United States/OxCGRT_USA_differentiated_withnotes.csv'
awk -F "," 'NR==1 {header=$0}; NF>1 && NR>1 {year=substr($6,1,4); if(! files[year]) {print header > ("data/United States/OxCGRT_USA_differentiated_withnotes_" year ".csv"); files[year]=1}; print $0 > ("data/United States/OxCGRT_USA_differentiated_withnotes_" year ".csv")}' 'data/United States/OxCGRT_USA_differentiated_withnotes.csv'
rm 'data/United States/OxCGRT_USA_differentiated_withnotes.csv'


- name: Commit changes
run: |
git config --local user.email "action@github.com"
git config --local user.name "GitHub Action"
git add .
git diff-index --quiet HEAD || git commit -m "OxCGRT Data Update" -a
git push origin master:master

# - name: Push changes
# uses: ad-m/github-push-action@master
# with:
# github_token: ${{ secrets.GITHUB_TOKEN }}
# branch: master
Loading