set_raad_filenames(): sign each file_db.tab by size+mtime rather than md5 of
every byte (run_build_raad_cache rewrites files wholesale, so both move)
keep verbatim local copies of the text caches under R_user_dir(cache) and point vroom at them, refreshed only when the signature changes; vroom stays lazy so nothing is materialised on load. Option raadfiles.local.cache = FALSE to disable
get_raad_filenames(): apply the data_deprecated/PRIVATE root filters to the
unique root values instead of a regex over every row
set_raad_data_roots(): show mtime of file_db.tab (was looking for file_db.rds)
Add nsidc_cdr_daily_files and nsidc_cdr_monthly_files.
updated OISST monthly to new file.
*Switch amsr2 3k ice to netcdf not the geotiffs, so the missing value is consistent.
Sort Altimetry files on processing_date descending so that NRT gets filter out preferentially.
Switch ghrsst to the online COGs.
Modify nsidc daily to pick up the new earthdatacloud series 25km (monthly unchanged).
fasticefiles has been hard-renamed to fraser_fasticefiles.
Fix fasticefiles order.
Re memoize get_raad_file_names.
Begin ccmp v03.
Fix problematic old UTC flush (thanks Dale Maschette #33).
Avoid oldformat in Copernicus altimetry files.
New function gebco23_files() for our local copy.
srtm_files() now defunct, use opentopography or similar (one example is in gh:hypertidy/sds::cop30()).
Bumped required version of fs to dev, see https://github.com/r-lib/fs/issues/447.
GHRSST now uses our own COGs.
GHRSST now picks up new files (since 2023) https://github.com/AustralianAntarcticDivision/raadtools/issues/139#issuecomment-1678118793.
Made AMSR 3k file finding a bit more deliberate, and can now access the HDF4 files as well.
Fixed a bunch of stuff, and now passing check!
Removed rema_tiles_polygons data set because we no longer have sp (via raster). We can easily do this other ways, and the REMA here is superseded anyway see https://github.com/AustralianAntarcticDivision/remav2.
raadfiles no longer imports the raster package.
srtm_files() no longer adds the longlat columns.
OISST monthly files no longer has a valid date (raadtools does this).
Do not expand file list for SOSE or fasticefiles.
Fixed fastice files.
New handling for v2 NSIDC files in NetCDF format. The daily/monthly north/south variants use a new function 'nsidc_files_v2'. (The old daily/monthly files still have the binary files, but raadtools has moved to using the new scheme.)
nsidc_south_daily_files() and nsidc_south_daily_files() now return all daily and nrt files, but chooses final (not nrt in preference). In raadtools we cull out the dates that don't have data in them (there are the 2 daily steps in the beginning, and a few weeks gap).
Fix regression in argo_files() that was limiting to 'usgodae' from sold old unused code, thanks @KimBaldry and @RaymondBen for troubleshooting.
New function 'par_files()' for ocean colour PAR, contributed by Kimberlee Baldry @KimBaldry.
BREAKING CHANGES
CHANGES
Big flush of file searching, new internal function .find_files_generic() to replace old style and to make topography file find consistent.
Now all REMA 8m functions have a 'tile' column with the two element column_row index, so we can easily join when
we've detected the tile/s we want.
New IBCSO v2 files, for 'ibcso_files', 'ibcso_bed_files', 'ibcso_digital_chart_files' and 'ibcso_rid_files' and 'ibcso_tid_files'. Older 'background' and 'sid' currently set still to v1, with new functions for 'rid' and 'tid' (we haven't investigated/consolidated details yet).
Fixed CCMP files, we were missing files prior to 2015 because of an assumption about file naming.
options("raadfiles.env")$raadfiles.filename.database which makes package load a few seconds faster.New function gebco21_files() for GEBCO 2021.
New function ccmp_6hourly_files() RSS CCMP_RT V2.1 derived surface winds (Level 3.0).
Files functions now check there are no zero-size files, emitting a message to tell maintainers ( this is done along with setting the timezone).
Altimetry Antarctic files.
Consistency fixes for load up configuration, see https://github.com/AustralianAntarcticDivision/raadfiles/issues/23.
New REMA functions, and data set for the polygons. WIP
Slight speedup for package load by using text file and vroom and read/write the file database.
Move to REMA 1.1, 8m, 100m, 200m.
New functions amsre_daily_files(), amsr2_daily_files(), amsr2_3k_daily_files() to round out
AMSRE and AMSR2 support. The original function amsr_daily_files() provides a mix of AMSRE and AMSR2
6.25km files to give a full temporal coverage (2002- current), but are different products.
Add leads files from Reiser et al, new functions iceclim_north_leadsfiles() iceclim_south_leadsfiles().
Move to using 'circum_fast_ice' files.
Now fasticefiles() moved here from raadtools.
New function polarview_files() for Polarview images in JPEG or the tarball GeoTIFFs.
Updated path to OISST files https://github.com/AustralianAntarcticDivision/raadfiles/issues/21.
Add CERSAT 12.5 km ice files cersat_daily_files().
New function geoid_files() for Earth Gravitation Model.
New function sose_monthly_files() for Southern Ocean State Estimate (SOSE).
Add MODIS "cafe" files in cafe_monthly_files().
Use memoization on all files functions.
Fix ncep2 files to use base time at midnight UTC (so we can speed up raadtools getting the time for each band).
New altimetry_currents_polar_files() for local derived product.
New all argument to get_raad_filenames(), for internal use.
Fix export of gebco_14_files().
New SEAPODYM function seapodym_weekly_files(), from the private collection onr
ly.
New GEBCO 2019 function gebco19_files().
New function srtm_files().
New function for SMAP salinity files.
New function for fsle files().
'raadfiles' now uses a new approach to file cache management, so easier to use from scratch, relevant topics are in raadfiles-admin. New
functions include getters and setters for the file cache, as well as options to control whether the file cache is loaded, and how often
it is refreshed.
Internal function run_this_function_to_build_raad_cache is now deprecated, please use the newly exported run_build_raad_cache.
New rema_8m_files, rema_100m_files, rema_200m_files, and rema_1km_files for the Reference Elevation Model of Antarctica (REMA) topography.
New amps_d2files and amps_d1files (to replace raadtools version) functions for AMPs files.
Added RAMP topography in ramp_files.
each 'topography' related files function now uses an internal generic pattern form, and each separate data variant has its own function. In
contrast to other types, these files now have an argument all for exploratory use, to return the (non-exclusive) set of all files available with the
data set
new functions for Macquarie Island region topography macquarie100m_57S_files and macquarie100m_58S_files, these are two UTM forms for
the alternative zones 57 or 58.
new functions for Smith and Sandwell topography smith_sandwell_files, smith_sandwell_lon180_files for the Pacific-view and Atlantic-view
variants and their analogues smith_sandwell_unpolished_files, smith_sandwell_lon180_unpolished_files. All Smith and Sandwell data now
defaults to version 18.1, older versions may be found with smith_sandwell_files(all = TRUE).
new functions for AAD George V Terre Adelie topography at various resolutions george_v_terre_adelie_1000m_files,
george_v_terre_adelie_500m_files, george_v_terre_adelie_250m_files, and george_v_terre_adelie_100m_files
new functions for IBCSO data and images ibcso_files, ibcso_background_files, ibcso_bed_files, ibcso_digital_chart_files, ibcso_sid_files
new functions etopo1_files and etopo2_files for for NGDC NOAA global topographic data
new function lakesuperior_files for NGDC NOAA Lake Superior bathymetry data
new function cryosat2_files for Cryosat2 topography data
new functions gebco14_files and gebco08_files for GEBCO topography data
new home for 'NOAA OI 1/4 Degree Daily SST AVHRR' now in ncei.noaa.gov
new function amsr_daily_files for the AMSR 6.25km data
new global file cache not bound to a single root
new ncep2_uwnd_6hr_files and ncep2_vwnd_6hr_files, note this is quite
different from the old raadtools::windfiles - expectation is that downstream read will explicitly deal with uwnd, vwnd separately
new altimetry_daily_files, bom_tmax_daily_files, cmip5_files, ghrsst_daily_files,
ncep2_uwnd_6hr_files, ncep2_vwnd_6hr_files, nsidc_monthly_files, nsidc_north_monthly_files,
nsidc_south_monthly_files, oisst_daily_files, oisst_monthly_files, thelist_files
new NSIDC monthly files, separates north, south, daily, monthly and provider at the function level, also supports fix https://github.com/AustralianAntarcticDivision/raadtools/issues/54
standard cache now separates "root" and "file" as separate columns
general scheme for file cache that can incorporate other collections
separated out from raadtools