Products
collekt ships a source-adapter for each supported provider. An adapter knows how to fetch source-native files for a request and how to plan (dry-run) the same work. Adapters are registered on import (collekt.sources) and selected by a source’s kind.
Built-in adapters
| Kind | Produces | Access | Format |
|---|---|---|---|
cmems |
Gridded ocean fields (currents, SST, waves, biogeochemistry) | open | NetCDF |
ecmwf_open_data |
Gridded forecast wind | open | NetCDF (cropped from global GRIB2) |
era5 |
Gridded reanalysis wind | credentials (CDS) | NetCDF |
era5_arco |
The same ERA5 reanalysis, read from Google’s public ARCO-ERA5 Zarr store | open | NetCDF (cropped from global hourly chunks) |
gfs |
Gridded NOAA GFS analysis wind (NOMADS GRIB filter, last ~10 days) | open | NetCDF (daily, from GRIB2 analysis cycles) |
copernicus_dataspace |
Sentinel / Landsat product files | credentials | product (e.g. .zip) |
skytruth |
Oil-slick detections (features) | open | Parquet |
hozint |
Threat-intelligence reports (features) | credentials | Parquet |
gfw |
Fishing / encounter / port-visit / loitering / AIS-gap events (features) | credentials | Parquet |
geonorge |
Norwegian maritime reference vectors: fairways, navigation aids, aquaculture, restricted zones, … (features) | open | GeoParquet |
eodyn |
eOdyn surface currents / drifters | open (preview archive) | NetCDF |
local |
Row-filtered subset of a local damast-managed tabular archive (features) |
none — local filesystem | Parquet |
file_archive |
Row-filtered subset of a remote archive of zipped or compressed CSVs, mirrored locally (AIS positions: AISDK, Marine Cadastre) | open | Parquet |
Open sources need no secrets; credential-gated sources resolve their secrets from the environment or provider tools — see Credentials. The eOdyn adapter serves a historical preview archive (mode: archive) until the production API ships (mode: api). The local adapter reads an archive that already exists on disk rather than fetching from a remote provider — see Local data collections. The file_archive adapter downloads a provider’s files into a local mirror first — see Remote file archives (AIS).
era5 and era5_arco serve the same ERA5 reanalysis: both select variables by their long ERA5 names and write them under the CDS short names (u10, t2m, …), so their files are interchangeable apart from the time coordinate (time for ARCO, valid_time for the CDS), which the Assembler accepts either way. Choosing between them is up to you, collekt never switches on its own. The CDS (era5) crops on the server, so it transfers little but each request waits in a queue, from minutes to hours. ARCO-ERA5 (era5_arco) has no queue and no account, but every Zarr chunk is one hour of one variable for the whole globe (about 3 MB), so the volume grows with hours x variables whatever the region size. Prefer era5_arco for short windows and era5 for long time series over small regions. ARCO trails real time by about a week (the CDS by about five days); downloads read the store’s own valid_time_stop_era5t attribute and skip later days.
Bundled catalog
collekt ships a curated catalog of concrete datasets — the datasets AI4COPSEC uses, grouped into catalogs under collekt/conf/source/:
| Catalog | Sources |
|---|---|
cmems_global |
cmems_glorys_nrt, cmems_glorys_nrt_2d_hourly, cmems_glorys_nrt_3d_6h, cmems_glorys_nrt_total_currents, cmems_glorys_my, cmems_duacs_nrt, cmems_duacs_my, cmems_global_sst_nrt, cmems_global_sst_my, cmems_global_waves, cmems_global_chl_obs_nrt, cmems_global_chl_obs_my, cmems_global_bgc_model, cmems_global_wind_nrt |
cmems_eur |
cmems_duacs_eur_nrt, cmems_duacs_eur_my |
cmems_med |
cmems_med_currents_nrt, cmems_med_currents_nrt_15min, cmems_med_currents_nrt_2d_hourly, cmems_med_currents_nrt_3d_hourly, cmems_med_currents_my, cmems_med_currents_my_2d_hourly, cmems_med_waves, cmems_med_chl_obs, cmems_med_bgc_model |
cmems_ibi |
cmems_ibi_currents, cmems_ibi_currents_2d_hourly, cmems_ibi_currents_3d_hourly, cmems_ibi_waves, cmems_atl_chl_obs_nrt, cmems_atl_chl_obs_my, cmems_ibi_bgc_model |
cmems_nws |
cmems_nws_currents, cmems_nws_currents_2d_hourly, cmems_nws_currents_3d_hourly, cmems_nws_waves, cmems_nws_chl_obs, cmems_nws_bgc_model |
ecmwf |
ecmwf_open_data_forecast, era5_reanalysis, era5_arco_reanalysis |
gfs |
gfs_analysis |
eodyn |
eodyn_osmose_currents |
skytruth |
skytruth |
hozint |
hozint |
gfw |
gfw |
aisdk |
aisdk_ais |
marine_cadastre |
marine_cadastre_ais, marine_cadastre_ais_west |
geonorge |
geonorge_depth_contours, geonorge_fairways, geonorge_navigation_aids, geonorge_emergency_harbours, geonorge_aquaculture, geonorge_coral_reefs, geonorge_coral_reef_closures, geonorge_military_firing_ranges, geonorge_military_restricted_areas, geonorge_trawling_closures |
dataspace |
copernicus_s1_grd, copernicus_s2_l2a, copernicus_landsat |
Browse the catalog programmatically instead of reading the tables below:
import collekt
print(collekt.describe_datasets()) # human-readable, grouped by provider
collekt.available_datasets()["cmems_glorys_my"]["variables"]
# ['bottomT', 'mlotst', 'siconc', 'sithick', 'so', 'thetao', 'uo', 'usi', 'vo', 'vsi', 'zos']Both accept an optional conf_dir= to include a downstream catalog overlay (see Extending the catalog downstream).
CMEMS Dataset Snapshot
The table below mirrors the bundled src/collekt/conf/source/cmems_*.yaml catalog files as of 2026-09-21. Temporal coverage is the offline coverage declared in the repository on that date; rolling/NRT products intentionally keep an open end and can be refreshed manually with scripts/update_cmems_coverage.py. Resolution is inferred from the CMEMS dataset identifier when the catalog does not declare it as a separate field. Both this table and the catalog listing above are generated — run scripts/update_products_table.py rather than editing them.
| Source key | Dataset ID | Resolution | Native sampling | Spatial coverage | Temporal coverage | Variables | Depth | DOI / product |
|---|---|---|---|---|---|---|---|---|
cmems_glorys_nrt |
cmems_mod_glo_phy-cur_anfc_0.083deg_P1D-m |
0.083 deg | 24h |
-180.0/179.9 -80.0/90.0 | from 2022-06-01 (rolling/open on 2026-09-21) |
uo, vo |
yes | 10.48670/moi-00016 product |
cmems_glorys_nrt_2d_hourly |
cmems_mod_glo_phy_anfc_0.083deg_PT1H-m |
0.083 deg | 1h |
-180.0/179.9 -80.0/90.0 | from 2022-06-01 (rolling/open on 2026-09-21) |
so, thetao, uo, vo, zos |
no | 10.48670/moi-00016 product |
cmems_glorys_nrt_3d_6h |
cmems_mod_glo_phy-cur_anfc_0.083deg_PT6H-i |
0.083 deg | 6h |
-180.0/179.9 -80.0/90.0 | from 2022-06-01 (rolling/open on 2026-09-21) |
uo, vo |
yes | 10.48670/moi-00016 product |
cmems_glorys_nrt_total_currents |
cmems_mod_glo_phy_anfc_merged-uv_PT1H-i |
not declared | 1h |
-180.0/179.9 -80.0/90.0 | from 2020-11-01 (rolling/open on 2026-09-21) |
uo, utide, utotal, vo, vsdx, vsdy, vtide, vtotal |
no | 10.48670/moi-00016 product |
cmems_glorys_my |
cmems_mod_glo_phy_my_0.083deg_P1D-m |
0.083 deg | 24h |
-180.0/179.9 -80.0/90.0 | 1993-01-01 to 2026-06-23 |
bottomT, mlotst, siconc, sithick, so, thetao, uo, usi, vo, vsi, zos |
yes | 10.48670/moi-00021 product |
cmems_duacs_nrt |
cmems_obs-sl_glo_phy-ssh_nrt_allsat-l4-duacs-0.125deg_P1D |
0.125 deg | 24h |
-179.9/179.9 -89.9/89.9 | from 2024-07-01 (rolling/open on 2026-09-21) |
adt, err_sla, err_ugosa, err_vgosa, flag_ice, sla, ugos, ugosa, vgos, vgosa |
no | 10.48670/moi-00149 product |
cmems_duacs_my |
cmems_obs-sl_glo_phy-ssh_my_allsat-l4-duacs-0.125deg_P1D |
0.125 deg | 24h |
-179.9/179.9 -89.9/89.9 | 1993-01-01 to 2026-01-16 |
adt, err_sla, err_ugosa, err_vgosa, flag_ice, sla, tpa_correction, ugos, ugosa, vgos, vgosa |
no | 10.48670/moi-00148 product |
cmems_global_sst_nrt |
METOFFICE-GLO-SST-L4-NRT-OBS-SST-V2 |
not declared | 24h |
-180.0/180.0 -90.0/90.0 | from 2024-01-17 (rolling/open on 2026-09-21) |
analysed_sst, analysis_error, mask, sea_ice_fraction |
no | 10.48670/moi-00165 product |
cmems_global_sst_my |
METOFFICE-GLO-SST-L4-REP-OBS-SST |
not declared | 24h |
-180.0/180.0 -90.0/90.0 | 1981-10-01 to 2026-03-31 |
analysed_sst, analysis_error, mask, sea_ice_fraction |
no | 10.48670/moi-00168 product |
cmems_global_waves |
cmems_mod_glo_wav_anfc_0.083deg_PT3H-i |
0.083 deg | 3h |
-180.0/179.9 -80.0/90.0 | from 2022-11-01 (rolling/open on 2026-09-21) |
VCMX, VHM0, VHM0_SW1, VHM0_SW2, VHM0_WW, VMDR, VMDR_SW1, VMDR_SW2, VMDR_WW, VMXL, VPED, VSDX, VSDY, VTM01_SW1, VTM01_SW2, VTM01_WW, VTM02, VTM10, VTPK |
no | 10.48670/moi-00017 product |
cmems_global_chl_obs_nrt |
cmems_obs-oc_glo_bgc-plankton_nrt_l4-gapfree-multi-4km_P1D |
4 km | 24h |
-180.0/180.0 -90.0/90.0 | from 2026-09-05 (rolling/open on 2026-09-21) |
CHL, CHL_uncertainty, flags |
no | 10.48670/moi-00279 product |
cmems_global_chl_obs_my |
cmems_obs-oc_glo_bgc-plankton_my_l3-multi-4km_P1D |
4 km | 24h |
-180.0/180.0 -90.0/90.0 | 1997-09-04 to 2026-09-13 |
CHL, CHL_uncertainty, DIATO, DIATO_uncertainty, DINO, DINO_uncertainty, GREEN, GREEN_uncertainty, HAPTO, HAPTO_uncertainty, MICRO, MICRO_uncertainty, NANO, NANO_uncertainty, PICO, PICO_uncertainty, PROCHLO, PROCHLO_uncertainty, PROKAR, PROKAR_uncertainty, flags |
no | 10.48670/moi-00280 product |
cmems_global_bgc_model |
cmems_mod_glo_bgc-pft_anfc_0.25deg_P1D-m |
0.25 deg | 24h |
-180.0/179.8 -80.0/90.0 | from 2021-11-01 (rolling/open on 2026-09-21) |
chl, phyc |
yes | 10.48670/moi-00015 product |
cmems_global_wind_nrt |
cmems_obs-wind_glo_phy_nrt_l4_0.125deg_PT1H |
0.125 deg | 1h |
-179.9/179.9 -89.9/89.9 | from 2024-06-13 (rolling/open on 2026-09-21) |
air_density, eastward_stress, eastward_stress_bias, eastward_stress_sdd, eastward_wind, eastward_wind_bias, eastward_wind_sdd, northward_stress, northward_stress_bias, northward_stress_sdd, northward_wind, northward_wind_bias, northward_wind_sdd, number_of_observations, number_of_observations_divcurl, stress_curl, stress_curl_bias, stress_curl_dv, stress_divergence, stress_divergence_bias, stress_divergence_dv, wind_curl, wind_curl_bias, wind_curl_dv, wind_divergence, wind_divergence_bias, wind_divergence_dv |
no | 10.48670/moi-00305 product |
| Source key | Dataset ID | Resolution | Native sampling | Spatial coverage | Temporal coverage | Variables | Depth | DOI / product |
|---|---|---|---|---|---|---|---|---|
cmems_duacs_eur_nrt |
cmems_obs-sl_eur_phy-ssh_nrt_allsat-l4-duacs-0.0625deg_P1D |
0.0625 deg | 24h |
-30.0/42.0 20.0/66.0 | from 2024-07-01 (rolling/open on 2026-09-21) |
adt, err_sla, err_ugosa, err_vgosa, flag_ice, sla, ugos, ugosa, vgos, vgosa |
no | 10.48670/moi-00142 product |
cmems_duacs_eur_my |
cmems_obs-sl_eur_phy-ssh_my_allsat-l4-duacs-0.0625deg_P1D |
0.0625 deg | 24h |
-30.0/42.0 20.0/66.0 | 1993-01-01 to 2026-01-16 |
adt, err_sla, err_ugosa, err_vgosa, flag_ice, sla, tpa_correction, ugos, ugosa, vgos, vgosa |
no | 10.48670/moi-00141 product |
| Source key | Dataset ID | Resolution | Native sampling | Spatial coverage | Temporal coverage | Variables | Depth | DOI / product |
|---|---|---|---|---|---|---|---|---|
cmems_ibi_currents |
cmems_mod_ibi_phy_anfc_0.027deg-3D_P1D-m |
0.027 deg | 24h |
-19.1/5.1 26.2/56.1 | from 2022-11-23 (rolling/open on 2026-09-21) |
bottomT, mlotst, so, thetao, uo, vo, zos |
yes | 10.48670/moi-00027 product |
cmems_ibi_currents_2d_hourly |
cmems_mod_ibi_phy_anfc_0.027deg-2D_PT1H-m |
0.027 deg | 1h |
-19.1/5.1 26.2/56.1 | from 2022-11-23 (rolling/open on 2026-09-21) |
mlotst, thetao, ubar, uo, vbar, vo, zos |
no | 10.48670/moi-00027 product |
cmems_ibi_currents_3d_hourly |
cmems_mod_ibi_phy_anfc_0.027deg-3D_PT1H-m |
0.027 deg | 1h |
-19.1/5.1 26.2/56.1 | from 2024-11-10 (rolling/open on 2026-09-21) |
so, thetao, uo, vo |
yes | 10.48670/moi-00027 product |
cmems_ibi_waves |
cmems_mod_ibi_wav_anfc_0.027deg_PT1H-i |
0.027 deg | 1h |
-19.0/5.0 26.0/56.0 | from 2022-11-26 (rolling/open on 2026-09-21) |
VCMX, VHM0, VHM0_SW1, VHM0_SW2, VHM0_WW, VMDR, VMDR_SW1, VMDR_SW2, VMDR_WW, VMXL, VPED, VSDX, VSDY, VTM01_SW1, VTM01_SW2, VTM01_WW, VTM02, VTM10, VTPK |
no | 10.48670/moi-00025 product |
cmems_atl_chl_obs_nrt |
cmems_obs-oc_atl_bgc-plankton_nrt_l3-multi-1km_P1D |
1 km | 24h |
-46.0/13.0 20.0/66.0 | from 2026-09-04 (rolling/open on 2026-09-21) |
CHL, CHL_uncertainty, DIATO, DIATO_uncertainty, DINO, DINO_uncertainty, GREEN, GREEN_uncertainty, HAPTO, HAPTO_uncertainty, MICRO, MICRO_uncertainty, NANO, NANO_uncertainty, PICO, PICO_uncertainty, PROCHLO, PROCHLO_uncertainty, PROKAR, PROKAR_uncertainty, flags |
no | 10.48670/moi-00284 product |
cmems_atl_chl_obs_my |
cmems_obs-oc_atl_bgc-plankton_my_l3-multi-1km_P1D |
1 km | 24h |
-46.0/13.0 20.0/66.0 | 1997-09-04 to 2026-09-13 |
CHL, CHL_uncertainty, DIATO, DIATO_uncertainty, DINO, DINO_uncertainty, GREEN, GREEN_uncertainty, HAPTO, HAPTO_uncertainty, MICRO, MICRO_uncertainty, NANO, NANO_uncertainty, PICO, PICO_uncertainty, PROCHLO, PROCHLO_uncertainty, PROKAR, PROKAR_uncertainty, flags |
no | 10.48670/moi-00286 product |
cmems_ibi_bgc_model |
cmems_mod_ibi_bgc_anfc_0.027deg-3D_P1D-m |
0.027 deg | 24h |
-19.1/5.1 26.2/56.1 | from 2022-11-23 (rolling/open on 2026-09-21) |
chl, dissic, fe, nh4, no3, nppv, o2, ph, phyc, po4, si, spco2, zeu, zooc |
yes | 10.48670/moi-00026 product |
| Source key | Dataset ID | Resolution | Native sampling | Spatial coverage | Temporal coverage | Variables | Depth | DOI / product |
|---|---|---|---|---|---|---|---|---|
cmems_med_currents_nrt |
cmems_mod_med_phy-cur_anfc_4.2km_P1D-m |
4.2 km | 24h |
-17.3/36.3 30.2/46.0 | from 2024-08-27 (rolling/open on 2026-09-21) |
uo, vo |
yes | 10.48670/mds-00359 product |
cmems_med_currents_nrt_15min |
cmems_mod_med_phy-cur_anfc_4.2km_PT15M-i |
4.2 km | 15min |
-17.3/36.3 30.2/46.0 | from 2024-08-23 (rolling/open on 2026-09-21) |
uo, vo |
yes | 10.48670/mds-00359 product |
cmems_med_currents_nrt_2d_hourly |
cmems_mod_med_phy-cur_anfc_4.2km-2D_PT1H-m |
4.2 km | 1h |
-17.3/36.3 30.2/46.0 | from 2024-08-25 (rolling/open on 2026-09-21) |
uo, vo |
no | 10.48670/mds-00359 product |
cmems_med_currents_nrt_3d_hourly |
cmems_mod_med_phy-cur_anfc_4.2km-3D_PT1H-m |
4.2 km | 1h |
-17.3/36.3 30.2/46.0 | from 2025-10-06 (rolling/open on 2026-09-21) |
uo, vo |
yes | 10.48670/mds-00359 product |
cmems_med_currents_my |
cmems_mod_med_phy-cur_my_4.2km_P1D-m |
4.2 km | 24h |
-6.0/36.3 30.2/46.0 | 1987-01-01 to 2026-08-31 |
uo, vo |
yes | 10.48670/mds-00375 product |
cmems_med_currents_my_2d_hourly |
cmems_mod_med_phy-cur_my_4.2km_PT1H-m |
4.2 km | 1h |
-6.0/36.3 30.2/46.0 | 1987-01-01 to 2026-08-31 |
uo, vo |
no | 10.48670/mds-00375 product |
cmems_med_waves |
cmems_mod_med_wav_anfc_4.2km_PT1H-i |
4.2 km | 1h |
-18.1/36.3 30.2/46.0 | from 2021-11-30 (rolling/open on 2026-09-21) |
VCMX, VHM0, VHM0_SW1, VHM0_SW2, VHM0_WW, VMDR, VMDR_SW1, VMDR_SW2, VMDR_WW, VMXL, VPED, VSDX, VSDY, VTM01_SW1, VTM01_SW2, VTM01_WW, VTM02, VTM10, VTPK |
no | 10.48670/mds-00373 product |
cmems_med_chl_obs |
cmems_obs-oc_med_bgc-plankton_nrt_l3-multi-1km_P1D |
1 km | 24h |
-6.0/36.5 30.0/46.0 | from 2026-09-13 (rolling/open on 2026-09-21) |
CHL, CRYPTO, DIATO, DINO, GREEN, HAPTO, MICRO, NANO, PICO, PROKAR, QI_CHL, SENSORMASK, WTM |
no | 10.48670/moi-00297 product |
cmems_med_bgc_model |
cmems_mod_med_bgc-bio_anfc_4.2km_P1D-m |
4.2 km | 24h |
-5.5/36.3 30.2/46.0 | from 2024-07-23 (rolling/open on 2026-09-21) |
nppv, o2 |
yes | 10.48670/mds-00358 product |
| Source key | Dataset ID | Resolution | Native sampling | Spatial coverage | Temporal coverage | Variables | Depth | DOI / product |
|---|---|---|---|---|---|---|---|---|
cmems_nws_currents |
cmems_mod_nws_phy-cur_anfc_1.5km-3D_P1D-m |
1.5 km | 24h |
-16.0/13.0 46.0/62.7 | from 2024-08-02 (rolling/open on 2026-09-21) |
ubar, uo, vbar, vo, wo |
yes | 10.48670/moi-00054 product |
cmems_nws_currents_2d_hourly |
cmems_mod_nws_phy-cur_anfc_1.5km-2D_PT1H-i |
1.5 km | 1h |
-16.0/13.0 46.0/62.7 | from 2024-08-04 (rolling/open on 2026-09-21) |
uo, vo |
no | 10.48670/moi-00054 product |
cmems_nws_currents_3d_hourly |
cmems_mod_nws_phy-cur_anfc_1.5km-3D_PT1H-i |
1.5 km | 1h |
-16.0/13.0 46.0/62.7 | from 2024-09-17 (rolling/open on 2026-09-21) |
ubar, uo, vbar, vo, wo |
yes | 10.48670/moi-00054 product |
cmems_nws_waves |
cmems_mod_nws_wav_anfc_1.5km_PT1H-i |
1.5 km | 1h |
-16.0/13.0 46.0/62.7 | from 2024-08-06 (rolling/open on 2026-09-21) |
VCMX, VHM0, VHM0_SW1, VHM0_SW2, VHM0_WW, VMDR, VMDR_SW1, VMDR_SW2, VMDR_WW, VMXL, VPED, VSDX, VSDY, VTM01_SW1, VTM01_SW2, VTM01_WW, VTM02, VTM10, VTPK, forecast_period |
no | 10.48670/moi-00055 product |
cmems_nws_chl_obs |
cmems_obs_oc_nws_bgc_tur-spm-chl_nrt_l3-hr-mosaic_P1D-m |
not declared | 24h |
-12.0/13.0 48.0/62.0 | from 2020-01-01 (rolling/open on 2026-09-21) |
CHL, SPM, TUR |
no | 10.48670/moi-00118 product |
cmems_nws_bgc_model |
cmems_mod_nws_bgc-chl_anfc_7km-3D_P1D-m |
7 km | 24h |
-19.9/13.0 40.1/65.0 | from 2024-07-29 (rolling/open on 2026-09-21) |
chl |
yes | 10.48670/moi-00056 product |
Select concrete datasets with DatasetConfig, using provider-specific dataclasses whose fields match the YAML preset form:
config = collekt.DatasetConfig(
collekt.CMEMS("cmems_glorys_my", variables=["uo", "vo"], depth=[1.0, 1.1]),
collekt.CMEMS("cmems_duacs_my", variables=["ugos", "vgos"]),
collekt.SkyTruth(),
)The equivalent YAML is:
datasets:
- provider: cmems
key: cmems_glorys_my
variables: [uo, vo]
depth: [1.0, 1.1]
- provider: cmems
key: cmems_duacs_my
variables: [ugos, vgos]
- provider: skytruth
key: skytruthDatasetConfig resolves each key against the bundled catalog, validates selected variables against available_variables, and validates required provider parameters such as CMEMS depth ranges. Fetcher takes the output directory separately; it is runtime state, not dataset configuration.
The catalog is curated, not exhaustive — dataset IDs drift, so collekt doctor --online validates the CMEMS entries against the provider catalogue.
Each gridded source declares one temporal field: temporal_sampling, the native cadence of that dataset (15min, 1h, 3h, 6h, or 24h). Datasets are always fetched at that cadence — a request carries no sampling of its own, so thinning or aligning to a coarser common axis is a downstream Assembler step. When a provider offers multiple cadences or dimensionalities, model them as separate source entries (for example cmems_med_currents_nrt_15min, cmems_med_currents_nrt_2d_hourly, and cmems_med_currents_nrt) so each source maps to one provider dataset and one cadence, and the caller picks the cadence by picking the dataset.
Every model-currents product in the catalog now carries its subdaily variants alongside the daily one:
| Region | Daily | Subdaily |
|---|---|---|
| Global NRT | cmems_glorys_nrt |
cmems_glorys_nrt_2d_hourly, cmems_glorys_nrt_3d_6h, cmems_glorys_nrt_total_currents |
| Mediterranean NRT | cmems_med_currents_nrt |
cmems_med_currents_nrt_15min, cmems_med_currents_nrt_2d_hourly, cmems_med_currents_nrt_3d_hourly |
| Mediterranean reanalysis | cmems_med_currents_my |
cmems_med_currents_my_2d_hourly |
| IBI | cmems_ibi_currents |
cmems_ibi_currents_2d_hourly, cmems_ibi_currents_3d_hourly |
| North-West Shelf | cmems_nws_currents |
cmems_nws_currents_2d_hourly, cmems_nws_currents_3d_hourly |
The daily mean is the unsuffixed base entry, and every subdaily variant of it carries both its dimensionality and its cadence (_2d_hourly, _3d_6h). Dimensionality is in the key because it decides whether a caller must pass a depth: CMEMS ships its -2D datasets as surface fields, so cmems_med_currents_my needs a depth while cmems_med_currents_my_2d_hourly does not. A dataset holding a different quantity rather than another cadence of the same field is named for the quantity instead (cmems_glorys_nrt_total_currents). cmems_med_currents_nrt_15min predates this rule and keeps its released name.
Switching between cadences of the same product is then a one-key change: the sources share a path, resolve the same region on the same grid, and each output file is prefixed with its source name, so a daily and an hourly variant can also be collected side by side. Only the depth dimension differs, and the assembled dataset gains or loses its depth axis accordingly.
Regional Atlantic products use the CMEMS regional names: IBI (cmems_ibi) and European North-West Shelf (cmems_nws) are separate catalogs. Basin-level Atlantic ocean-colour observations keep cmems_atl in their source names.
Sources may also declare a coverage block:
coverage:
longitude: [-19.08, 5.08]
latitude: [26.17, 56.08]
temporal:
start: "2022-11-23"
end: null
kind: rollingThis is an offline planning hint. Static spatial bounds and stable temporal starts are shipped in the catalog; rolling/NRT products keep end: null because their latest date moves. When online catalogue access is available, CMEMS planning still prefers Copernicus Marine describe() and collekt doctor --online compares declared coverage with the provider catalogue.
Presets
Presets are plain YAML versions of DatasetConfig:
collekt fetch --bbox -6 20 35 45 --start 2023-06-15 \
--dataset-config westmed.yaml --output-dir data/collectionsThey name only desired datasets and provider parameters. The full source catalog remains package-maintained metadata under collekt/conf/source/.
Extending the catalog downstream
The bundled catalog is not exhaustive. A downstream project — for example a technological brick with its own ocean datasets — adds or overrides datasets with a configuration directory, passed as --conf-dir (CLI) or conf_dir= (Fetcher / DatasetConfig.resolve). It overlays the bundled configuration rather than replacing it: its default.yaml declares only additions, its source_catalogs are appended to the shipped ones, and its source/*.yaml files define new datasets or override bundled ones by key.
my_brick/conf/
default.yaml # source_catalogs: [ocean] (bundled catalogs stay available)
source/ocean.yaml # defines cmems / era5 / ... datasets by key
collekt fetch --bbox -6 20 35 45 --start 2023-06-15 \
--conf-dir my_brick/conf --dataset-config presets/currents.yaml \
--output-dir data/collections--conf-dir is the catalog (what datasets exist); --dataset-config is the selection (which of them to fetch, and with which variables and depth). A selected key must exist in the merged catalog, so a brick’s presets can reference both bundled and its own datasets. Adding a new provider (a new adapter kind) is a larger extension and is not configured this way.
Local data collections
The local adapter serves a tabular archive that already exists on disk as a collekt source, without fetching anything remotely. Each requested day’s rows are filtered to the request’s region and written out as Parquet; a day with no matching file, or no in-region rows, is skipped rather than treated as an error. Two ways to locate a day’s rows are supported, depending on whether the archive is already day-partitioned.
Because archive_root is a machine-local path, local sources are typically declared in a downstream conf_dir rather than the bundled catalog (see Extending the catalog downstream).
Day-partitioned archives (layout)
If the archive is already split into one file (or file set) per day — e.g. built with damast convert --save-as or AnnotatedDataFrame.export_partitioned — point layout at its partitioning spec; a day’s file(s) are then resolved from the filename alone, with nothing read until a match is found:
# my_brick/conf/source/ais.yaml
sources:
ais_archive:
kind: local
path: "local/ais"
dataset_id: "ais-daily-archive"
archive_root: "/data/ais/daily"
layout: "time:timestamp+daily:%Y-%m-%d"
region_columns: [lat, lon]
temporal_sampling: 24hpath— like any source, where collekt writes this source’s output, relative to the collection’s output directory (request_dir / path). It has nothing to do with the archive being read.archive_root— root of the pre-existing on-disk archive to read from. Unrelated topath:pathis collekt’s own output layout,archive_rootis the source’s input.layout— the archive’s partitioning spec, as produced bydamast.core.SaveAs/AnnotatedDataFrame.export_partitioned(atime:ortime+column:spec;localreads the time column out of it to scope each request day).region_columns—[lat_column, lon_column]used to filter rows to the request’s bounding box.
Rows are filtered by time as well as region in both modes, so layout only decides which files are opened — a partitioning coarser than the request (one file per month, say) still yields just that day’s rows for each day. If the archive’s files carry a compression suffix (AIS_2026-01-01.zst.parquet), they are matched despite expected_paths naming only AIS_2026-01-01.parquet. A region_columns or time_column entry the archive doesn’t actually have raises rather than skipping every day, which would otherwise be indistinguishable from an archive holding no data.
Unpartitioned archives (file_pattern + time_column)
If the archive is a flat set of files with no time-encoded naming, use file_pattern (a glob, resolved once per fetch rather than once per day) and time_column instead of layout; a day’s rows are then selected by filtering time_column, the same way region_columns filters by region:
sources:
ais_flat:
kind: local
path: "local/ais"
dataset_id: "ais-flat-archive"
archive_root: "/data/ais/flat"
file_pattern: "*.parquet"
time_column: timestamp
region_columns: [lat, lon]
temporal_sampling: 24hlayout and file_pattern are mutually exclusive; exactly one is required. Every matching file is scanned for every request day, so this mode leans on Parquet row-group statistics on time_column to skip irrelevant data — an unsorted archive, or one made of very few, very large files, is scanned in full for each day. Prefer layout when the archive can be (re-)partitioned (e.g. damast convert --save-as "time:timestamp+daily:%Y-%m-%d"); reach for file_pattern when it can’t.
That pushdown is Parquet-only: damast reads Parquet through polars.scan_parquet, but its NetCDF, CSV and HDF readers load each file in full, and file_pattern re-reads every matched file once per request day. For a non-Parquet archive of any size — especially NetCDF holding a dense, padded grid rather than a row per observation — convert it once (damast convert --save-as "time:<time_column>+daily:%Y-%m-%d") and point layout at the result.
Select either the same way:
config = collekt.DatasetConfig(
collekt.Local("ais_archive", variables=["mmsi", "timestamp", "lat", "lon"]),
)or as a preset:
datasets:
- provider: local
key: ais_archive
variables: [mmsi, timestamp, lat, lon]collekt fetch --bbox -6 20 35 45 --start 2026-01-01 \
--conf-dir my_brick/conf --dataset-config presets/ais.yaml \
--output-dir data/collectionsRemote file archives (AIS)
The file_archive adapter serves providers that publish tabular data as CSV files over HTTP, one file per day or month, either zipped or as a single (possibly compressed, e.g. .csv.zst) CSV. Two AIS archives are bundled:
| Source | Provider | Files | Coverage |
|---|---|---|---|
aisdk_ais |
Danish Maritime Authority | monthly zips until 2024-02, daily from 2024-03 (bucket root from 2025-02-27) | Danish EEZ, 2006-03-01 to a few days ago |
marine_cadastre_ais |
NOAA Marine Cadastre | daily zstd-compressed CSVs (Broadcast Points) | U.S. EEZ east of 180°, 2015-01-01 to the latest published day |
marine_cadastre_ais_west |
NOAA Marine Cadastre | same files and mirror | U.S. EEZ west of 180° (western Aleutians, far end of the Hawaiian chain) |
collekt.DatasetConfig(collekt.FileArchive("aisdk_ais", variables=["# Timestamp", "MMSI", "Latitude", "Longitude"]))Each remote file is downloaded once into a persistent mirror that every request shares. Where mirrors live is set by the COLLEKT_ARCHIVE_ROOT environment variable, in the shell or in the project’s .env (which collekt loads on import, see Credentials):
# .env
COLLEKT_ARCHIVE_ROOT=/data/collekt/archiveEach catalog entry’s relative archive_root (aisdk, marine_cadastre_points) is a directory under it; without the variable, mirrors go to ~/.cache/collekt/archive/<provider>. An absolute archive_root is used as is. Mirrors can be large (one AISDK day is ~0.6 GB zipped), so point the variable at a disk with room.
Every request day is then read from the mirror, cut to the day and the request region, and written as one Parquet file into the request directory. Each catalog entry declares a coarse coverage box (the provider’s EEZ) and date range; a request outside it is skipped before anything is downloaded, and an entry only returns rows inside its box. Because neither boxes nor request regions can cross the antimeridian, the U.S. EEZ is split into two entries that share one mirror and differ only in their region; select both to cover the whole EEZ (a day is still downloaded only once). Columns keep the provider’s own names.
A period can name an index: a listing page of the published files, such as Marine Cadastre’s yearly file lists. The latest file it lists then ends the entry’s date range, so a request for a later day is skipped with the last published day in the message (... is outside the available date range (2015-01-01 to 2026-06-30)), both when fetching and in a dry run. A dry run of such an entry therefore reads the index over the network. If the index cannot be read (offline, say), the static coverage applies.
convert decides what the mirror keeps:
parquet(default): each CSV is converted to annotated Parquet with a parsed UTC time column and written with damast’sexport_partitionedinto a time-partitioned archive followinglayout(default: one file per UTC day,time:<time_column>+daily:%Y/%m/%Y-%m-%d); the CSV is removed. The mirror is a regular damast archive, so alocalsource with the samelayout— or damast’s own tools — can read it directly. Rows that two remote files contribute to the same partition are merged, not overwritten.spec_only: the original CSVs stay byte-identical, one directory per remote file, with only a damast<file>.spec.yamlsidecar written next to each. The time column is parsed at read time. A single compressed CSV (.csv.gz,.csv.zst) is kept compressed, since damast reads it directly.
Column metadata (descriptions, units) comes from the provider’s bundled damast spec (aisdk.spec.yaml, marine_cadastre.spec.yaml, next to the adapter); a source without spec infers it from the data.
A further archive of the same shape needs only a catalog entry, declared downstream like any other source:
sources:
my_ais:
kind: file_archive
path: "ais/my_ais"
filename_pattern: "my_ais_{date:%Y%m%d}_{bbox_hash}.parquet"
archive_root: "my_ais" # under $COLLEKT_ARCHIVE_ROOT
convert: spec_only
periods:
# str.format templates over `date`; the first period containing a day wins.
- every: month # one download serves every day of the month
until: "2019-12-31"
url: "https://example.org/{date:%Y}/ais-{date:%Y-%m}.zip"
member: "ais-{date:%Y-%m-%d}.csv" # the day's CSV inside the monthly zip
- every: day
start: "2020-01-01"
url: "https://example.org/{date:%Y}/ais-{date:%Y-%m-%d}.zip"
# Optional yearly listing; its latest file ends the date range.
index: "https://example.org/{date:%Y}/index.html"
time_column: timestamp
time_format: "%Y-%m-%d %H:%M:%S"
region_columns: [lat, lon]Conversion loads one CSV at a time into memory, which suits a day of AIS data (a few GB) — the scale this adapter targets.
Dependencies
collekt installs all provider clients and the gridded Assembler by default, so every source works out of the box after uv sync. Clients are imported lazily, so importing collekt stays fast and a source only pulls its client into memory when it actually runs. The ECMWF Open Data forecast subset, Skytruth, and eOdyn (in its current preview-archive mode) need no credentials; the other providers do — see Credentials.