IO data updates

[WesIngwersen]

Discuss strategies and issues related to updating useeior for updated input data

[WesIngwersen]

The BEA benchmark IO and underlying data release which are every 5 yrs is the most significant regular data update required. This update comes with a schema change with new commodities and industries.

Functions to retrieve the data are in data-raw. In BEAData. R they are currently tied to a single year in name. Also they are very repetitive.. more like a copy and paste of each other.

[bl-young]

First pass at consolidating the functions (detail tables to start) is in cc91a02. They can be consolidated further but doing it piecemeal.

[WesIngwersen]

Might want a letter(s) appended to the year representing the schema in that name like 2017sch to distinguish it from the data year but otherwise a good proposal

[bl-young]

yes ok, in fact I was also thinking we could abbreviate the schema to two digits, so:
Summary_Make_2019_AfterRedef_17sch.01.rda

[bl-young]

@WesIngwersen one hiccup with this schema - I went ahead and renamed the 2012 schema file as planned, e.g., Summary_Make_2014_AfterRedef_12sch.rda. However, these .rda files are called by the name of the object when they are saved, not by the name of the file apparently. So after renaming, you still have to use: get('Summary_Make_2014_AfterRedef.rda'). I think there are a few options:

1. keep the name change of the 2012 files, so in the repo you can see which data file is which based on the file name, knowing that this will cause a discrepancy between the filename and what is called with get()
2. revert the name change for the 2012 files so they are the same as before, when we adjust the code for getting IO data, append the schema only when the schema is not 2012.
3. I could probably write a script to load all the 2012 schema files, rename the object, and write back to RDA with the new schema name.

[bl-young]

discussed ok to stick with option 1

[bl-young]

Regarding the data objects and metadata files that get created. Currently they get stored in data/ and inst/ext/metadata respectively. Storing the data files uses usethis::use_data

useeior/R/UtilityFunctions.R

Lines 378 to 383 in c884903

	writeDatatoRDA <- function(data, data_name) {
	assign(data_name, data)
	do.call(eval(str2expression("usethis::use_data")),
	list(as.name(data_name), overwrite = TRUE))
	}

Would there be any benefit to storing these data objects on data commons / appdirs. Given their small size I'm not seeing major benefit to making that shift, especially once the new schema are in place.

In fact I'm even reconsidering whether we need to indicate the "version number" in the file naming system (which would result in new files instead of replacing data files upon data updates). The benefit to version number would be if we want to be able to recreate prior versions of the model moving forward without having to back in the package history, otherwise we have all that information in the file metadata.

[bl-young]

Discussed that keeping in the repo and replacing files makes sense and is the path forward. Will want to be explicit when data are updated to note those updates in package releases.

[bl-young]

Another question: do we want to enable access to be able to freshly obtain 2012 files from the BEA archive? Or can we assume that we won't need to do that anymore and just keep our processed files in the repository?

The former would require making further updates to enable pointing to the archive and adjusting some steps for 2012 schema tables.

[bl-young]

Confirmed - no explicit need to access the 2012 archive, though models for 2012 schema can still be built.

[bl-young]

There is now a 2017 schema sample model (w/ GHGs) on that branch. You can see in the testing that it successfully creates the model, though there are some validation issues.

There are a few spots where I took some quick measures to get it working that I will need to go back to review (specifically around some mapping files when pulling the CPI and GO data, e.g. here). I expect those may be causing the validation errors but i will be looking closer.

[bl-young]

last remaining validation issue is with 333914/US Pumping Equipment Manufacturing. May be related to a mapping issue. is this a new BEA sector??

[WesIngwersen]

Confirmed yes

[bl-young]

the validation issue is now resolved, was due to mis-aligned sectors because commodity names were missing for two new BEA sectors

[bl-young]

Regarding differences in NAICS schema for satellite tables compared to IO schema, for example, the 2017 Model being tested using the published GHG datasets, which were built with 2012 NAICS schema

see the new warning in model build: WARNING::Lost sectors due to missing mapping: 333911, 333913, 335221, 335222, 335224, 335228, 5171, 5172
In this case, 333911 and 333913 have been updated to 333914 in the 2017 NAICS (and identical BEA code). Because 333911 and 333913 no longer exist in the 2017 schema, those data points get dropped from the TbS.

Do we want to allow satellite tables w/ 2012 NAICS to be used in 2017 BEA models? If so we will need a function to do that conversion, similar to mapFlowTotalsbySectorfromBEASchema2007to2012

[WesIngwersen]

It would be cleaner not to allow this ... to require the schemas to be the same or else through an error. I understand this would require rebuilding fbs models in some cases.. let's evaluate the trade-offs

[bl-young]

I think that's a fine approach for now. I'll note that because the new 2017 schema is currently only available from 2017-2022, many of our FBS wouldn't work anyway (e.g., water, land, etc.) that are from 2016 or earlier (there is no industry output for those years).