Model Source Utilities¶
Utilities for inspecting and recovering delivered model archives live under
ras_commander.sources.federal.
Streaming ZIP Recovery¶
StreamingZipReader walks ZIP local-file headers in order, so it can survey or
extract archives whose central directory is missing. It verifies each extracted
member against its recorded CRC and reports truncation and unreadable members
instead of silently skipping them.
from pathlib import Path
from ras_commander.sources.federal import StreamingZipReader
archive = StreamingZipReader(Path("delivered-model.zip"))
survey = archive.probe()
print(len(survey.members), survey.truncated)
When an archive has neither a central directory nor sizes in its local headers,
probe() must scan compressed member data to locate each trailing descriptor.
The per-member scan is bounded by max_probe_member_size (64 GiB by default).
Deflate and stored members use the Python standard library. FEMA deliveries compressed with Deflate64 require the optional dependency:
The ebfe extra installs the decoder on Python 3.10, the newest interpreter
for which its publisher provides wheels. It is intentionally excluded from
the broad all extra so Python 3.11+ installations do not depend on an
unqualified native source build.
Deflate64 members require an authoritative compressed size from a local or central-directory record. Deferred-size Deflate64 members in archives with no central directory are reported as unsupported because the dependency does not expose the exact compressed boundary.
This is a direct recovery API. RasEbfeModels validates every central-directory
path before extracting. When a complete ZIP contains Deflate64 members and the
optional decoder is absent, its verified extraction path can use a locally
installed 7-Zip CLI, then performs the same size and CRC32 audit before atomic
promotion. Without either decoder it fails without promoting files.
Archive member names are untrusted input. A sink_factory must normalize each
name and verify that its output path remains below the intended destination.
Audit Rendering¶
ebfe_audit turns an existing _audit.json bundle and its JSONL sidecars into
a consistent review document covering the delivery verdict, inventory,
supporting data, repair actions, missing inputs, acquisition needs, and
provenance. It renders captured audit evidence; it does not generate the audit
bundle itself.
from pathlib import Path
from ras_commander.sources.federal.ebfe_audit import (
load_audit_bundle,
render_audit_markdown,
)
audit_folder = Path("model-audit")
bundle = load_audit_bundle(audit_folder)
(audit_folder / "AUDIT.md").write_text(
render_audit_markdown(bundle),
encoding="utf-8",
)
Louisiana LWI EnDMC¶
LouisianaEndmc lists the public LWI EnDMC catalog, classifies each
resource's S3 objects, downloads selected components with resumable parallel
range requests, and returns /vsicurl/ paths for range-reading rasters. See
Louisiana Watershed Initiative EnDMC.
Louisiana Watershed Initiative EnDMC catalog and public S3 file access.
All methods are static. Network reads are anonymous: the public
/search JSON endpoint for the catalog and the public
lwi-model-data bucket for files. Credentials are never accepted,
stored, or required.
Example
from ras_commander.sources.state import LouisianaEndmc df = LouisianaEndmc.list_resources(region="Region 5") rasters = LouisianaEndmc.get_raster_urls("685d772c2ecdbd05b0cb03d1") rasters[["component", "vsicurl"]].head()
list_resources
staticmethod
¶
list_resources(kinds: Optional[Sequence[str]] = None, search: Optional[str] = None, software: Optional[str] = None, region: Optional[Union[str, int]] = None, source: str = 'auto', refresh: bool = False, timeout: float = REQUEST_TIMEOUT) -> pd.DataFrame
Return the EnDMC catalog as a DataFrame without logging in.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kinds
|
Optional[Sequence[str]]
|
Resource kinds to return. Defaults to the downloadable
kinds |
None
|
search
|
Optional[str]
|
Full-text query passed to the site's |
None
|
software
|
Optional[str]
|
|
None
|
region
|
Optional[Union[str, int]]
|
LWI region number or label, for example |
None
|
source
|
str
|
|
'auto'
|
refresh
|
bool
|
Bypass the in-process live catalog cache. |
False
|
timeout
|
float
|
HTTP timeout in seconds. |
REQUEST_TIMEOUT
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns |
DataFrame
|
|
DataFrame
|
|
DataFrame
|
record date from the bundled snapshot (None for resources newer |
than the snapshot; use
|
meth: |
list_s3_resource_ids
staticmethod
¶
list_s3_resource_ids(kinds: Optional[Sequence[str]] = None, timeout: float = REQUEST_TIMEOUT) -> pd.DataFrame
Inventory resource ids directly from the public bucket.
The bucket can hold prefixes that are absent from the public catalog
(for example unpublished or withdrawn records). Compare this result
with :meth:list_resources to find them.
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with |
get_resource_metadata
staticmethod
¶
get_resource_metadata(resource_id: str, kind: Optional[str] = None, timeout: float = REQUEST_TIMEOUT) -> Dict[str, Any]
Read the public metadata page for one resource.
Parses https://lwi.endmc.org/view/<kind>/<id>, which is served
without login. Returns the labelled metadata fields (for example
Application date, Coordinate system, Common software
version), containing packages, and associated simulations. Author
and owner contact fields are omitted. Parsing is best effort and
follows the page markup observed in November 2026.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict with |
Dict[str, Any]
|
|
Dict[str, Any]
|
|
list_objects
staticmethod
¶
list_objects(resource_id: str, kind: Optional[str] = None, components: Optional[Sequence[str]] = None, patterns: Optional[Sequence[str]] = None, timeout: float = REQUEST_TIMEOUT) -> pd.DataFrame
List a resource's S3 objects with component classification.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resource_id
|
str
|
24-character hexadecimal EnDMC id. |
required |
kind
|
Optional[str]
|
|
None
|
components
|
Optional[Sequence[str]]
|
Keep only these components (see |
None
|
patterns
|
Optional[Sequence[str]]
|
Keep only relative paths matching any of these
case-insensitive glob patterns ( |
None
|
timeout
|
float
|
HTTP timeout in seconds. |
REQUEST_TIMEOUT
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with |
DataFrame
|
|
DataFrame
|
|
classify_object
staticmethod
¶
Classify one relative object path into a COMPONENTS value.
object_url
staticmethod
¶
Return the anonymous HTTPS URL for a bucket key.
vsicurl_path
staticmethod
¶
Return a GDAL /vsicurl/ path for range-reading a bucket object.
get_raster_urls
staticmethod
¶
get_raster_urls(resource_id: str, kind: Optional[str] = None, components: Sequence[str] = OUTPUT_RASTER_COMPONENTS, timeout: float = REQUEST_TIMEOUT) -> pd.DataFrame
Return HTTPS and /vsicurl/ paths for a resource's rasters.
GDAL-based readers (rasterio, GDAL, ras2cng) can open the
vsicurl column directly and read windows or overviews by HTTP
range without downloading the file. Raster sidecars are excluded.
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with |
DataFrame
|
|
read_range
staticmethod
¶
Read length bytes from start of a bucket object by HTTP range.
download_resource
staticmethod
¶
download_resource(resource_id: str, output_folder: Union[str, Path], kind: Optional[str] = None, components: Optional[Sequence[str]] = None, patterns: Optional[Sequence[str]] = None, max_workers: int = DEFAULT_MAX_WORKERS, chunk_size: int = DEFAULT_CHUNK_SIZE, overwrite: bool = False, verify: str = 'size', max_path_length: Optional[int] = DEFAULT_MAX_PATH_LENGTH, max_bytes: Optional[int] = None, dry_run: bool = False, timeout: float = REQUEST_TIMEOUT) -> pd.DataFrame
Download selected objects from one resource with resumable ranges.
Files are written below output_folder at their path relative to
the resource prefix. Each file is assembled in <name>.part;
files larger than chunk_size are fetched as parallel byte ranges
and record completed ranges in <name>.part.json so an
interrupted download resumes where it stopped. Ranges are requested
with If-Match on the listed ETag, so an object replaced on the
server restarts instead of mixing versions. A receipt named
_endmc_<resource_id>.json records every selected object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resource_id
|
str
|
24-character hexadecimal EnDMC id. |
required |
output_folder
|
Union[str, Path]
|
Destination root. Keep it short: resource paths
are already long, and the default |
required |
kind
|
Optional[str]
|
|
None
|
components
|
Optional[Sequence[str]]
|
Components to download (see |
None
|
patterns
|
Optional[Sequence[str]]
|
Case-insensitive relative-path globs; combined with
|
None
|
max_workers
|
int
|
Parallel HTTP streams (default 4). |
DEFAULT_MAX_WORKERS
|
chunk_size
|
int
|
Range size in bytes (default 64 MiB). |
DEFAULT_CHUNK_SIZE
|
overwrite
|
bool
|
Re-download files that already exist with the expected size. |
False
|
verify
|
str
|
|
'size'
|
max_path_length
|
Optional[int]
|
Refuse the download when any destination path
(plus the partial-file suffix) exceeds this length. |
DEFAULT_MAX_PATH_LENGTH
|
max_bytes
|
Optional[int]
|
Refuse when the selected total exceeds this size. |
None
|
dry_run
|
bool
|
Return the plan without downloading. |
False
|
timeout
|
float
|
HTTP timeout per request in seconds. |
REQUEST_TIMEOUT
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with |
DataFrame
|
|
DataFrame
|
|
DataFrame
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
Unsafe object paths, path length or size limits. |
ImportError
|
|
list_models
staticmethod
¶
list_models(location: Optional[str] = None, model_type: Optional[ModelType] = None, hecras_version: Optional[str] = None, tags: Optional[List[str]] = None, limit: Optional[int] = None, software: Optional[str] = 'HEC-RAS', **kwargs) -> List[ModelMetadata]
List EnDMC model applications as ModelMetadata for ModelCatalog.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Optional[str]
|
Case-insensitive substring of the location, title, or
region (for example |
None
|
model_type
|
Optional[ModelType]
|
Applied when given; EnDMC metadata does not state
steady/unsteady, so entries are |
None
|
hecras_version
|
Optional[str]
|
Version keyword such as |
None
|
tags
|
Optional[List[str]]
|
Required tags (AND logic). |
None
|
limit
|
Optional[int]
|
Maximum results. |
None
|
software
|
Optional[str]
|
Software filter; defaults to |
'HEC-RAS'
|
download_model
staticmethod
¶
download_model(model_id: str, output_folder: Union[str, Path], extract: bool = True, overwrite: bool = False, credentials: Optional[dict] = None, **kwargs) -> DownloadResult
Download a resource for ModelCatalog.
credentials is ignored; EnDMC files are public. Keyword arguments
are passed to :meth:download_resource. max_bytes defaults to
50 GiB here so an unfiltered catalog download cannot silently pull a
400 GiB design-storm resource; pass components or
max_bytes=None deliberately. extract is ignored (EnDMC serves
individual files, not archives).
get_source_status
staticmethod
¶
Return AVAILABLE when the public bucket answers an anonymous listing.
Complete source reference¶
RasEbfeModels source reference¶
Organization functions for specific eBFE/BLE models.
Static class providing deterministic organization for eBFE models used in example notebooks. Each function was generated by the organizing-ebfe-models agent skill after analyzing the model's specific archive structure.
Example
from ras_commander.ebfe_models import RasEbfeModels from pathlib import Path
Organize Spring Creek¶
organized = RasEbfeModels.organize_spring_creek( ... Path(r"D:/downloads/12040102_Spring_Models_extracted") ... )
Use with ras-commander¶
from ras_commander import init_ras_project init_ras_project(organized / "RAS Model", "5.0.7")
available_models
staticmethod
¶
List organizers and both delivered and qualified RAS versions.
download_model
staticmethod
¶
download_model(model_id: str, output_folder: Union[str, Path], extract: bool = True, overwrite: bool = False, credentials: Optional[dict] = None, **kwargs) -> DownloadResult
Thin ModelSource-compatible wrapper around organize_model().
download_source_asset
staticmethod
¶
Download and verify one catalogued eBFE source asset.
Catalog entries default to ZIP extraction for backward compatibility.
Entries with extract=False are retained as loose source files and
returned directly.
get_model_metadata
staticmethod
¶
Return normalized source metadata for a registered eBFE study.
get_source_status
staticmethod
¶
Always available — registry is built-in, no network required.
list_models
staticmethod
¶
list_models(location: Optional[str] = None, model_type: Optional[ModelType] = None, hecras_version: Optional[str] = None, tags: Optional[List[str]] = None, limit: Optional[int] = None, **kwargs) -> List[ModelMetadata]
Return ModelMetadata for all registered eBFE models, with optional filtering.
normalize_model_key
staticmethod
¶
Return the canonical eBFE model slug for a name, alias, or HUC8.
organize_amite
staticmethod
¶
organize_amite(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, skip_output: bool = False, validate_dss: bool = False, ras_version: str = '5.0.7', rebuild_mismatched_terrain: bool = True, terrain_create_ras_version: str = '6.6', terrain_rebuild_gdal_version: str = '6.6', terrain_rebuild_timeout: int = 7200) -> Path
Organize Amite (08070202) eBFE model with automatic download.
Pattern 2: URL links file with separate Input/Terrain/LandUse/Output zips.
Automatic Download: Downloads multiple component zips from eBFE S3 bucket. Total download: ~31 GB (or ~4 GB with skip_output=True).
Model characteristics: - 5 cascaded 2D watershed models (WA1-WA5) - HEC-RAS version 5.0.1 / 5.0.3 - Shared Terrain and LandCover across all watersheds - Per-watershed Output HDF results - HEC-HMS boundary conditions via DSS files - State: Louisiana
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Path where data should be (will auto-download if missing). Default: ./ebfe_downloads/08070202_Amite |
None
|
output_folder
|
Optional[Path]
|
Output location. Default: ./ebfe_organized/Amite_08070202/ |
None
|
skip_output
|
bool
|
If True, skip downloading Output zips (saves ~27 GB). Input/Terrain/LandUse still downloaded (~4 GB). |
False
|
validate_dss
|
bool
|
Run DSS validation checks (default: False) |
False
|
ras_version
|
str
|
HEC-RAS version used when initializing projects for path repair metadata. |
'5.0.7'
|
rebuild_mismatched_terrain
|
bool
|
Reproject and rebuild terrains when a watershed's RAS projection does not match the delivered shared terrain. Default True for Amite's LA North/LA South gap. |
True
|
terrain_create_ras_version
|
str
|
HEC-RAS version used for RasProcess terrain creation when a rebuild is required. |
'6.6'
|
terrain_rebuild_gdal_version
|
str
|
HEC-RAS version used for bundled GDAL reprojection tools when a rebuild is required. |
'6.6'
|
terrain_rebuild_timeout
|
int
|
Timeout in seconds for each reprojection and terrain creation process. |
7200
|
Returns:
| Type | Description |
|---|---|
Path to organized model with structure
|
Amite_08070202/ ├── HMS Model/ (README - no HMS, BC via DSS) ├── RAS Model/ │ ├── WA1/ (WA1_LA project + Terrain + Landcover + Output) │ ├── WA2/ (WA2 project + Terrain + Landcover + Output) │ ├── WA3/ (Amite_WA3 project + Terrain + Landcover + Output) │ ├── WA4/ (WA4 project + Terrain + Landcover + Output) │ └── WA5/ (WA5 project + Terrain + Landcover + Output) ├── Spatial Data/ (shared Input_DSS) ├── Documentation/ └── agent/model_log.md |
Example
from ras_commander.sources.federal.ebfe_models import RasEbfeModels from pathlib import Path
Download Input/Terrain/LandUse only (4 GB)¶
organized = RasEbfeModels.organize_amite(skip_output=True)
Full download with Output results (31 GB)¶
organized = RasEbfeModels.organize_amite()
Use with ras-commander¶
from ras_commander import init_ras_project init_ras_project(organized / "RAS Model" / "WA1", "5.0.7")
organize_austin_oyster
staticmethod
¶
organize_austin_oyster(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None) -> Path
Reassemble the audited Austin--Oyster 2D delivery without computing.
The retained 22.4 GB FEMA object is identity-checked by its exact size and multipart ETag sidecar. Its nested archives are CRC-audited, then a concise active workspace is assembled in a sibling temporary directory. All 28 audited HDF association attributes are checked. Only absent or empty attributes are written; path-equivalent delivered values are preserved. The audited RAS Mapper projection relocation/rewrite is applied separately and actual mutations are reported in the manifest.
organize_bayou_darbonne
staticmethod
¶
organize_bayou_darbonne(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, download_input: bool = True, include_hydrology: bool = True, validate_dss: bool = False, ras_version: str = '6.6', require_projects: bool = False) -> Path
Organize Bayou D'Arbonne (08040206) eBFE model.
Pattern 2 (URL-index). Unlike Amite, the published results are bundled
inside the 22 GiB BayouDArbonne_Hydraulic_Input.zip as
Input/BayouDarbonne.p0#.hdf (plans p01-p07), so there is no separate
Output download step. Terrain and LandUse ship as their own component zips.
Pass download_input=False to build a delivery shell without the 22 GiB
download (inventory/metadata only).
organize_boeuf
staticmethod
¶
organize_boeuf(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, download_components: bool = False, components: Optional[List[str]] = None, include_hydrology: bool = True, validate_dss: bool = False, ras_version: str = '6.6', require_projects: bool = False) -> Path
Organize Boeuf (08050001) eBFE model.
Pattern 2 split delivery: six HydraulicModels_{N}of6.zip parts
(~30 GiB each, ~175 GiB total) covering multiple work areas (BO1..BOn),
plus a small HydrologicModels.zip (HMS). Each work area is a standalone
2D project with published plan results (p01-p07). CRS is NAD83 LA State
Plane North.
By default this builds a delivery shell only (download_components=False)
to avoid an accidental ~175 GiB download; pass download_components=True
and optionally components=['part1', ...] to fetch parts.
organize_cibolo
staticmethod
¶
organize_cibolo(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None) -> Path
Organize Cibolo's nested 2D unsteady delivery.
organize_east_galveston_bay
staticmethod
¶
organize_east_galveston_bay(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None, extract_ras_nested: bool = False, include_results: bool = False, validate_dss: bool = True) -> Path
Organize the public East Galveston Bay eBFE/BLE delivery.
The public outer archive contains an HEC-HMS project and a large nested
RAS_Submittal.zip. The outer inventory establishes a 2D delivery,
but it does not establish an HEC-RAS version, plan identifiers, or
runtime readiness. This organizer copies the HMS and documentation
assets immediately. It extracts only the Input, Terrain, and LandCover
archives from the nested RAS submission when extract_ras_nested=True.
The delivered Output archive remains unexpanded unless
include_results=True. Organization does not execute HEC-RAS.
organize_double_mountain_fork_brazos
staticmethod
¶
organize_double_mountain_fork_brazos(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = False) -> Path
Assemble the four-project Double Mountain Fork Brazos delivery.
The FEMA delivery is split across four project ZIPs and three loose documentation files. Archives are extracted directly into the organized tree, leaving the downloaded source objects unchanged. The specialized standardization pass preserves the complete outer sibling terrain, land-cover, and infiltration bundles while applying shared projection, DSS, plan, and output repairs.
The canonical 1% AEP dependency order is DMF1 p04, DMF2 p01, DMF3 p01, then DMF4 p01. This organizer prepares and audits the files; it does not execute HEC-RAS.
organize_eleven_point
staticmethod
¶
organize_eleven_point(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True) -> Path
Organize Eleven Point (11010011) eBFE model with automatic download.
organize_lake_maurepas
staticmethod
¶
organize_lake_maurepas(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True) -> Path
Organize Lake Maurepas (08070204) eBFE model with automatic download.
organize_lower_brazos
staticmethod
¶
organize_lower_brazos(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, download_components: bool = False, components: Optional[List[str]] = None, validate_dss: bool = False) -> Path
Organize Lower Brazos (12070104) eBFE model components.
Lower Brazos is a Pattern 2 delivery with a small inventory and three
very large model component zips (about 188 GB, 229 GB, and 190 GB).
By default this function downloads only the inventory/manifest and
creates a clear delivery shell. Pass download_components=True and
optionally components=['lb_ma01'] to download/extract model zips.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Local cache folder for component downloads.
Default: |
None
|
output_folder
|
Optional[Path]
|
Output location.
Default: |
None
|
download_components
|
bool
|
Download/extract the huge model components. Default False to avoid accidental 600+ GB downloads. |
False
|
components
|
Optional[List[str]]
|
Component keys to process. Valid keys are
|
None
|
validate_dss
|
bool
|
Run DSS validation after organization. |
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the organized Lower Brazos delivery folder. |
organize_lower_colorado_cummins
staticmethod
¶
organize_lower_colorado_cummins(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, river: Optional[str] = None, reach: Optional[str] = None, validate: bool = False, validation_ras_version: str = '6.6') -> Path
Organize Lower Colorado-Cummins (12090301) eBFE model with automatic download.
Pattern 1D-BLE: Multiple 1D steady-state reach models.
Automatic Download: If source data not found, automatically downloads 290.7 MB from eBFE S3 bucket. Download includes progress tracking and resumes interrupted downloads when the server supports byte-range requests.
Model characteristics: - 2,378 separate 1D steady-state reach models, including 46 projects nested below intermediate reach-group folders - Organized as Model/{River Name}/{Reach Name}/ inside the archive - Each reach is a standalone HEC-RAS project (.prj, .p01, .f01, .g01, etc.) - 7 steady flow profiles per reach: 10-year, 25-year, 50-year, 1pct_min, 100-year, 1pct_plu, 500-year - HEC-RAS 4.x era (needs re-run with 6.x for HDF output) - No Output/, Terrain/, or DSS files
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Path where data should be (will auto-download if missing). Default: ./ebfe_downloads/12090301_LowerColoradoCummins |
None
|
output_folder
|
Optional[Path]
|
Output location. Default: ./ebfe_organized/LowerColoradoCummins_12090301/ |
None
|
river
|
Optional[str]
|
If specified, only extract reaches from this river (e.g., "Rabbs Creek-Colorado River") |
None
|
reach
|
Optional[str]
|
If specified, only extract this specific reach (e.g., "SHILOH BRANCH"). Requires river to also be specified. |
None
|
validate
|
bool
|
If True, validate with init_ras_project() on the first (or specified) reach |
False
|
validation_ras_version
|
str
|
HEC-RAS version used for optional initialization validation. The delivery itself is version 4.1.0. |
'6.6'
|
Returns:
| Type | Description |
|---|---|
Path to organized output folder with structure
|
LowerColoradoCummins_12090301/ ├── HMS Model/ (empty - no HMS for 1D steady models) ├── RAS Model/ (reach models: {River}/{Reach}/) ├── Spatial Data/ (empty - no terrain for 1D models) ├── Documentation/ (empty placeholder) └── agent/model_log.md |
Example
from ras_commander.sources.federal.ebfe_models import RasEbfeModels from pathlib import Path
Extract single reach (fast, good for testing)¶
organized = RasEbfeModels.organize_lower_colorado_cummins( ... river="Rabbs Creek-Colorado River", ... reach="SHILOH BRANCH" ... )
Extract all 2,378 reaches¶
organized = RasEbfeModels.organize_lower_colorado_cummins()
organize_lower_ouachita
staticmethod
¶
organize_lower_ouachita(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, download_output: bool = True, output_plans: Optional[List[str]] = None, validate_dss: bool = False, ras_version: str = '5.0.7', require_projects: bool = False) -> Path
Organize Lower Ouachita (08040207) eBFE model.
Pattern 2 (URL-index). Downloads Input/LandUse/Terrain, stages the single 2D project, and (optionally) downloads the loose per-plan Output HDFs.
Only three results are published on S3: p21 (1pct / 100yr),
p25 (10pct / 10yr) and p26 (02pct / 500yr). The 25yr (4pct, p04)
and 50yr (2pct, p03) plans exist in the project but have no published
Output HDF and must be re-run with RasCmdr to obtain those grids.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
download_output
|
bool
|
Download the (very large, ~25 GB each) published
Output HDFs listed in |
True
|
output_plans
|
Optional[List[str]]
|
Subset of published plans to fetch. Valid: p21/p25/p26. |
None
|
organize_lower_ouachita_bayou_deloutre
staticmethod
¶
organize_lower_ouachita_bayou_deloutre(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True, include_hydrology: bool = True, ras_version: str = '5.0.7', require_projects: bool = False) -> Path
Organize Lower Ouachita-Bayou De Loutre (08040202) eBFE model.
Pattern 3: a single 08040202_Models.zip (3.43 GB) whose RAS content is
a nested zip with a space in its name
(LowerOuachitaBayouDLoutre_Hydraulic Models.zip). The outer archive is
downloaded/extracted, the nested hydraulic (and optional hydrology) zips
are extracted, then the projects are staged and standardized.
Note: HUC 08040202's FEMA eBFE archive ships Terrain and LandUse only
- the RAS model and Output HDFs are absent from the eBFE S3 bucket - so a
successful organize here produces a delivery with zero runnable RAS
projects. A clear WARNING is printed in that case; pass
require_projects=True to raise a ValueError instead.
organize_medina
staticmethod
¶
organize_medina(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None) -> Path
Organize Medina's recoverable delivery and preserve its source gap.
organize_model
staticmethod
¶
organize_model(model_key: str, download_root: Optional[Path] = None, output_root: Optional[Path] = None, downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, verbose: bool = False, show_progress: bool = False, **kwargs: Any) -> Path
Download, extract, and organize a known eBFE model by slug.
This is the common entry point used by examples and end-to-end audit
scripts. It preserves each model's specialized organizer while giving
callers a stable model_key interface.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_key
|
str
|
Model slug, alias, or HUC8. Use
:meth: |
required |
download_root
|
Optional[Path]
|
Base cache folder for raw downloads/extractions. |
None
|
output_root
|
Optional[Path]
|
Base folder for organized delivery folders. |
None
|
downloaded_folder
|
Optional[Path]
|
Explicit raw/extracted source folder. Overrides
|
None
|
output_folder
|
Optional[Path]
|
Explicit organized output folder. Overrides
|
None
|
verbose
|
bool
|
If True, emit organizer progress messages to stdout. Default False keeps example notebooks concise. |
False
|
show_progress
|
bool
|
If True, display download/extraction progress bars. |
False
|
**kwargs
|
Any
|
Additional arguments passed to the model-specific
organizer, such as |
{}
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the organized delivery folder. |
organize_models
staticmethod
¶
organize_models(model_keys: List[str], download_root: Optional[Path] = None, output_root: Optional[Path] = None, model_kwargs: Optional[Dict[str, Dict[str, Any]]] = None, verbose: bool = False, show_progress: bool = False) -> Dict[str, Path]
Organize multiple known eBFE models sequentially.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_keys
|
List[str]
|
Model slugs, aliases, or HUC8 values. |
required |
download_root
|
Optional[Path]
|
Base cache folder used by each organizer. |
None
|
output_root
|
Optional[Path]
|
Base delivery folder used by each organizer. |
None
|
model_kwargs
|
Optional[Dict[str, Dict[str, Any]]]
|
Optional per-model kwargs keyed by canonical slug or alias. |
None
|
verbose
|
bool
|
If True, emit organizer progress messages to stdout. |
False
|
show_progress
|
bool
|
If True, display download/extraction progress bars. |
False
|
Returns:
| Type | Description |
|---|---|
Dict[str, Path]
|
Mapping of canonical model slug to organized delivery folder. |
organize_north_galveston_bay
staticmethod
¶
organize_north_galveston_bay(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, extract_ras_nested: bool = False, validate_dss: bool = True) -> Path
Organize North Galveston Bay (12040203) eBFE model with automatic download.
Pattern 4: Compound archive with HMS hydrologic + RAS hydraulic models.
Automatic Download: If source data not found, automatically downloads 8.2 GB from eBFE S3 bucket. Download includes progress tracking and resumes interrupted downloads when the server supports byte-range requests.
Model characteristics: - Contains both HMS (NorthGalvestonBay.hms) and RAS models - HMS: 7 storm frequencies (10yr through 500yr) + sensitivity variants - RAS: 2D coastal model in nested 6.1 GB zip (may require manual extraction) - Total size: 8.2 GB
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Path where data should be (will auto-download if missing) |
None
|
output_folder
|
Optional[Path]
|
Output location (default: ./ebfe_organized/NorthGalvestonBay_12040203/) |
None
|
extract_ras_nested
|
bool
|
Attempt auto-extraction of RAS_Submittal.zip (may fail, default: False) |
False
|
validate_dss
|
bool
|
Run DSS validation checks (default: True) |
True
|
Returns:
| Type | Description |
|---|---|
Path to organized model with structure
|
NorthGalvestonBay_12040203/ ├── HMS Model/ (NorthGalvestonBay HMS project) ├── RAS Model/ (2D coastal model, may need manual extraction) ├── Spatial Data/ (pending RAS extraction) ├── Documentation/ (BLE reports + metadata) └── agent/model_log.md |
Note
RAS_Submittal.zip (6.1 GB) often requires manual extraction via Windows Explorer due to file locking issues with command-line tools.
Example
from ras_commander.ebfe_models import RasEbfeModels from pathlib import Path
Data will auto-download if not present (8.2 GB)¶
organized = RasEbfeModels.organize_north_galveston_bay( ... downloaded_folder=Path(r"D:/eBFE/12040203_Models_extracted"), ... extract_ras_nested=False # Manual extraction recommended ... )
HMS model ready immediately¶
hms = organized / "HMS Model/NorthGalvestonBay/NorthGalvestonBay.hms"
organize_pedernales
staticmethod
¶
organize_pedernales(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None) -> Path
Organize the complete 530-project Pedernales 1D BLE corpus.
organize_rio_hondo
staticmethod
¶
organize_rio_hondo(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, documents_folder: Optional[Path] = None, validate: bool = False) -> Path
Organize Rio Hondo (13060008) eBFE 1D steady-state model collection.
Pattern 1D-BLE: hundreds of separate reach models nested as
{Watershed}/{Project}/Model in the source archive. This organizer
downloads/extracts the model and document archives when needed, flattens
each raw Model folder into an openable HEC-RAS project folder, and
preserves the source watershed/project grouping for traceability.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Folder containing the extracted models, or the
parent folder containing |
None
|
output_folder
|
Optional[Path]
|
Output location.
Default: |
None
|
documents_folder
|
Optional[Path]
|
Optional folder containing extracted documents. If missing, the eBFE documents zip is downloaded and extracted. |
None
|
validate
|
bool
|
If True, validate the first organized reach with
|
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the organized Rio Hondo delivery folder. |
organize_salt_draw
staticmethod
¶
organize_salt_draw(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None, output_plans: Optional[List[str]] = None, include_hms: bool = True) -> Path
Organize Salt Draw (13070004), extracting only selected result HDFs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Union[str, Path]]
|
|
None
|
output_folder
|
Optional[Union[str, Path]]
|
Organized destination. |
None
|
output_plans
|
Optional[List[str]]
|
Plan result HDFs to extract, from |
None
|
include_hms
|
bool
|
Copy the delivered HEC-HMS project. |
True
|
Returns:
| Type | Description |
|---|---|
Path
|
Organized folder; the project is |
Path
|
|
organize_san_gabriel
staticmethod
¶
organize_san_gabriel(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None, validate_dss: bool = True) -> Path
Organize the five-project San Gabriel (12070205) eBFE delivery.
The delivery is a DSS fan-in system: LBSG_501 through LBSG_504
provide boundary records to LBSG_505. Every project references one
shared compiled Terrain/Terrain.hdf that FEMA did not provide.
This organizer preserves the delivered HDEMs and compiled geometry
tables, records the terrain deficiency, and does not automatically
rebuild terrain. An explicitly authorized HDEM-only reconstruction
must be recorded in the generated Record of Deficiencies (ROD).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Union[str, Path]]
|
Extracted |
None
|
output_folder
|
Optional[Union[str, Path]]
|
Organized delivery destination. Defaults to
|
None
|
validate_dss
|
bool
|
Catalog the organized DSS files after path repair. |
True
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the organized San Gabriel delivery. |
organize_spring_creek
staticmethod
¶
organize_spring_creek(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True) -> Path
Organize Spring Creek (12040102) eBFE model with automatic download.
Pattern 3a: Single 2D model in nested zip with self-contained terrain.
Automatic Download: If source data not found, automatically downloads 9.7 GB from eBFE S3 bucket. Download includes progress tracking and resumes interrupted downloads when the server supports byte-range requests.
Model characteristics: - Single HEC-RAS 2D unsteady flow model - Nested _Final.zip (9.67 GB) requires two-level extraction - Terrain self-contained in Terrain/ subfolder - 8 plans with pre-computed results (~1.2 GB per plan) - HEC-RAS version 5.0.7
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
downloaded_folder
|
Optional[Path]
|
Path where data should be (will auto-download if missing) |
None
|
output_folder
|
Optional[Path]
|
Output location (default: ./ebfe_organized/SpringCreek_12040102/) |
None
|
validate_dss
|
bool
|
Run DSS validation checks (default: True) |
True
|
Returns:
| Type | Description |
|---|---|
Path to organized model with structure
|
SpringCreek_12040102/ ├── HMS Model/ (README - no HMS for Pattern 3a) ├── RAS Model/ (Spring 2D model + terrain) ├── Spatial Data/ (terrain + shapefiles) ├── Documentation/ (inventory spreadsheet) └── agent/model_log.md |
Example
from ras_commander.ebfe_models import RasEbfeModels from pathlib import Path
Data will auto-download if not present (9.7 GB)¶
organized = RasEbfeModels.organize_spring_creek( ... downloaded_folder=Path(r"D:/eBFE/12040102_Models_extracted"), ... validate_dss=True ... )
Use with ras-commander¶
from ras_commander import init_ras_project init_ras_project(organized / "RAS Model", "5.0.7")
organize_spring_river
staticmethod
¶
organize_spring_river(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True) -> Path
Organize Spring River (11010010) eBFE model with automatic download.
organize_tickfaw
staticmethod
¶
organize_tickfaw(downloaded_folder: Optional[Path] = None, output_folder: Optional[Path] = None, validate_dss: bool = True, include_ble_surfaces: bool = False, ras_version: str = '5.0.7') -> Path
Organize Tickfaw (08070203) eBFE model with automatic download.
organize_upper_guadalupe
staticmethod
¶
organize_upper_guadalupe(downloaded_folder: Optional[Union[str, Path]] = None, output_folder: Optional[Union[str, Path]] = None, validate_dss: bool = False) -> Path
Reassemble the four-project Upper Guadalupe 2D cascade.
The immutable FEMA ZIP is verified by exact size and retained multipart
ETag, CRC-extracted into a temporary build, and atomically promoted. A
legacy extracted source tree is also accepted. The active layout stays
flat (RAS Model/UPGU#/UPGU#.prj), and upstream boundary conditions
reference the preceding project's live DSS file rather than a copied
snapshot. This method prepares the model but does not execute HEC-RAS.
repair_project_paths
staticmethod
¶
repair_project_paths(project_folder: Path, search_roots: Optional[List[Path]] = None, ras_version: Optional[str] = None) -> Dict[str, int]
Repair local DSS and rasmap references for an organized RAS project.
StreamingZipReader source reference¶
Walk a ZIP archive forward through its local file headers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
archive_path
|
Union[str, Path]
|
Archive to read. Never modified. |
required |
chunk_size
|
int
|
Read granularity in bytes. |
_DEFAULT_CHUNK
|
max_probe_member_size
|
Optional[int]
|
Maximum uncompressed bytes that :meth: |
_DEFAULT_MAX_PROBE_MEMBER_SIZE
|
Attributes:
| Name | Type | Description |
|---|---|---|
stats |
Populated by :meth: |
probe
¶
Enumerate members and identify recoverable archive content.
Uses the central directory when one is readable, else walks local
headers forward (the only option when the directory was never written).
Deferred-size compressed members have no boundary in their local
header, so this fallback inflates them without retaining their output,
up to max_probe_member_size per member, to find and validate the
trailing descriptor.
Returns:
| Type | Description |
|---|---|
ArchiveSurvey
|
members found, plus whether the archive is truncated. |
supported_methods
staticmethod
¶
Compression methods this reader can decode in the current environment.
walk
¶
walk(want: Optional[Callable[[ZipMemberInfo], bool]] = None, sink_factory: Optional[Callable[[ZipMemberInfo], Optional[BinaryIO]]] = None, survey: Optional[ArchiveSurvey] = None) -> Iterator[Tuple[ZipMemberInfo, bool]]
Walk members, extracting the ones want accepts.
want is evaluated before extraction. On archives with a central
directory, skipped member data is not read. Without one, :meth:probe
may already have scanned deferred-size compressed members to recover
their boundaries.
sink_factory returns a writable binary handle for a member, or None
to skip it. Taking a factory rather than a destination directory keeps
path policy -- flattening, projection, long-path budgets -- with the
caller, where it belongs. member.name is untrusted archive input;
callers must normalize it and verify the destination remains beneath
their intended extraction root.
Yields:
| Type | Description |
|---|---|
ZipMemberInfo
|
(member, extracted) for every member considered. The flag makes a |
bool
|
deliberate skip distinguishable from a member that could not be read; |
failures are recorded in
|
attr: |
ArchiveSurvey source reference¶
What an archive survey found.
A missing central directory can require inflating deferred-size members to
locate their trailing descriptors. See :meth:StreamingZipReader.probe.
complete_members
property
¶
Members whose data lies wholly inside the file.
projected_bytes
property
¶
Uncompressed total of recoverable members -- the space gate's input.
Without a central directory this is an estimate: members with deferred sizes report zero here, so allow margin rather than treating it as exact.
ExtractStats source reference¶
Verification outcome. Part of the contract, not debug output.
crc_ok / crc_fail are the substitute for hashing: the ZIP format
already carries a CRC-32 per member, so checking it costs nothing beyond the
decompression we are doing anyway and proves byte-correctness without a
second full read of the data.
The counters do not overlap, and conflating them understates or overstates integrity in opposite directions:
crc_failcounts only CRC mismatches on members that were read.truncatedcounts members whose data lies past the end of the archive. They are never read, so they can never fail a CRC. They are also listed in :attr:ArchiveSurvey.truncated_members-- the same members seen from the header side -- so counting both surfaces double-counts them.unreadableis the total that could not be extracted for any reason and includestruncated.
failures carries (name, kind, reason); filter on kind rather than
matching substrings in reason.
failures_of
¶
Failures of one kind: truncated, unsupported, read, crc, size.
ZipMemberInfo source reference¶
One member, as described by its own local header.
sizes_deferred
property
¶
True when sizes live in a trailing descriptor rather than the header.