Data Cleansing and Records Calculations

Part 2 of 3

This notebook follows sequentially from NOAA-CO-OPS-data in which we downloaded the latest data for a particular NOAA CO-OPS weather and tide station. The data record and corresponding metadata were written to file. Here we use those data to calculate several daily and monthly statistics and records. This is done in two steps:

1. Filter the data: We do not perform any quality assurance or quality control checks, but we do remove from the records any days missing a specified amount of data and any months missing a specified number of days of data.

2. Calculate records:

   • Daily and monthly averages
   • Record high daily and monthly averages^*
   • Record low daily and monthly averages^*
   • Average daily and monthly high
   • Lowest daily and monthly high^*
   • Record daily and monthly high^*
   • Average daily and monthly low
   • Highest daily and monthly low^*
   • Record daily and monthly low^*

Years are also noted for those records marked by an asterisk (*).

Packages and configurations

First we import the packages we need.

import pandas as pd
import xarray as xr
import numpy as np
import calendar
import yaml
import os

Functions

Next, we define a number of functions that will come in handy later:

• camel: This will use the site location to create a directory name in camelCase (e.g., “virginiaKeyFl”) so that we do not have to do it manually

• DOY: Determine day of year (DOY) out of 366 for each day in the data record

• count_missing_hours: Return the number of hours of missing data for a given day

• count_missing_days: Return the number of days of missing data for a given month

• filter_hours: Filter out days with too many hours of missing data

• filter_days: Filter out months with too many days of missing data

• Functions for all of the highs, lows, and averages (see their docstrings for more information)

Helper functions

def camel(text):
    """Convert 'text' to camel case"""
    s = text.replace(',','').replace("-", " ").replace("_", " ")
    s = s.split()
    if len(text) == 0:
        return text
    return s[0].lower() + ''.join(i.capitalize() for i in s[1:])

def DOY(df):
    """Determine year day out of 366"""
    # Day of year as integer
    df['YearDay'] = df.index.day_of_year.astype(int)
    # Years that are NOT leap years
    leapInd = [not calendar.isleap(i) for i in df.index.year]
    mask = (leapInd) & (df.index.month > 2)
    # Advance by one day everything after February 28 
    df.loc[mask, 'YearDay'] += 1
    return df

Filtering data

def count_missing_hours(group, threshold=4):
    """Return True if the number of hours in a day with good data is 
    greater than or equal to 24-'threshold' (i.e., a 'good' day) and False 
    otherwise.
    """
    num_obs = (~group.resample('1h').mean().isna()).sum()
    good_threshold = 24 - threshold
    return num_obs >= good_threshold

def count_missing_days(group, threshold=2):
    """Return True if the number of days in a month with good data 
    is greater than or equal to the number of days in the month minus 'theshold' (i.e., a 'good' month) and False
    otherwise.
    """
    try:
        days_in_month = pd.Period(group.index[0].strftime(format='%Y-%m-%d')).days_in_month
        good_days = (~group.resample('1D').mean().isna()).sum()
        good_threshold = days_in_month - threshold
        missing_days_flag = good_days > good_threshold
        return good_days >= good_threshold
    except IndexError:
        pass

def filter_hours(data, hr_threshold=4):
    """Filter data to remove days with more than 'hr_threshold' missing
    hours of data.
    """
    # Filter out fillVals==31.8
    filtered = data.replace(31.8, np.nan)
    # Filter out days missing more than <hr_threshold> hours
    filtered = filtered.groupby(pd.Grouper(freq='1D')).filter(
        lambda x: count_missing_hours(group=x, threshold=hr_threshold))
    return filtered

def filter_days(data, hr_threshold=4, day_threshold=2):
    """Filter months with more than 'day_threshold' days of missing
    data by first filtering data to remove days with more than 
    'hr_threshold' missing hours of data.
    """
    # Filter out fillVals==31.8
    filtered = data.replace(31.8, np.nan)
    # Filter out days missing more than <hr_threshold> hours
    filtered = filter_hours(filtered, hr_threshold=hr_threshold)
    # Filter out months missing more than <day_threshold> days
    filtered = filtered.groupby(pd.Grouper(freq='1M')).filter(
        lambda x: count_missing_days(group=x, threshold=day_threshold))
    return filtered

Calculate records

def daily_highs(df):
    """Daily highs"""
    return df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).max(numeric_only=True)
              
def daily_lows(df):
    """Daily lows"""
    return df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).min(numeric_only=True)

def daily_avgs(df, true_average=False):
    """Daily averages by calendar day. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    if true_average:
        results = df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).mean(numeric_only=True)
    else:
        dailyHighs = daily_highs(df)
        dailyLows = daily_lows(df)
        results = (dailyHighs + dailyLows) / 2
    return results

def mon_daily_highs(df):
    """Daily highs using data filtered by days"""
    return df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).max(numeric_only=True)

def mon_daily_lows(df):
    """Daily lows using data filtered by days"""
    return df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).min(numeric_only=True)

def mon_daily_avgs(df, true_average=False):
    """Daily averages by calendar day using data filtered by day. If
    'true_average' is True, all measurements from each 24-hour day will be
    used to calculate the average. Otherwise, only the maximum and minimum
    observations are used. Defaults to False (meteorological standard).
    """
    if true_average:
        return df.groupby(pd.Grouper(freq='1D', closed='left', label='left', dropna=True)).mean(numeric_only=True)
    else:
        dailyHighs = mon_daily_highs(df)
        dailyLows = mon_daily_lows(df)
        results = (dailyHighs + dailyLows) / 2
        return results

def daily_avg(df, true_average=False):
    """Daily averages. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, true_average=true_average)
    dailyAvg = dailyAvgs.groupby('YearDay').mean(numeric_only=True)
    dailyAvg.index = dailyAvg.index.astype(int)
    results = xr.DataArray(dailyAvg, dims=['yearday', 'variable'])
    results.name = 'Daily Average'
    return results

def monthly_highs(df, true_average=False):
    """Monthly highs. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, true_average=true_average)
    monthHighs = dailyAvgs.groupby(pd.Grouper(freq='M')).max(numeric_only=True)
    return monthHighs
  
def monthly_lows(df, true_average=False):
    """Monthly lows. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    dailyAvgs = daily_avgs(df, true_average=true_average)
    monthLows = dailyAvgs.groupby(pd.Grouper(freq='M')).min(numeric_only=True)
    return monthLows
    
def monthly_avg(df, true_average=False):
    """Monthly averages. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological
    standard).
    """
    dailyAvgs = mon_daily_avgs(df, true_average=true_average)
    monthlyMeans = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True)
    monthlyMeans.drop('YearDay', axis=1, inplace=True)
    monthlyAvg = monthlyMeans.groupby(monthlyMeans.index.month).mean(numeric_only=True)
    monthlyAvg.index = monthlyAvg.index.astype(int)
    results = xr.DataArray(monthlyAvg, dims=['month', 'variable'])
    results.name = 'Monthly Average'
    return results

def record_high_daily_avg(df, true_average=False):
    """Record high daily averages. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the records
    dailyAvgs = daily_avgs(df=df, true_average=true_average)
    recordHighDailyAvg = dailyAvgs.groupby('YearDay').max(numeric_only=True)
    recordHighDailyAvg.index = recordHighDailyAvg.index.astype(int)
    # Record years
    recordHighDailyAvgYear = dailyAvgs.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    recordHighDailyAvgYear.drop('YearDay', axis=1, inplace=True)
    recordHighDailyAvgYear.index = recordHighDailyAvgYear.index.astype(int)
    recordHighDailyAvgYear.columns = [i+' Year' for i in recordHighDailyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordHighDailyAvg, recordHighDailyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record High Daily Average'
    return results
    
def record_high_monthly_avg(df, true_average=False):
    """Record high monthly averages. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the records
    dailyAvgs = mon_daily_avgs(df, true_average=true_average)
    monthlyAvgs = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True)
    monthlyAvgs.drop('YearDay', axis=1, inplace=True)
    recordHighMonthlyAvg = monthlyAvgs.groupby(monthlyAvgs.index.month).max(numeric_only=True)
    recordHighMonthlyAvg.index = recordHighMonthlyAvg.index.astype(int)
    # Record years
    recordHighMonthlyAvgYear = monthlyAvgs.groupby(monthlyAvgs.index.month).apply(
                lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    recordHighMonthlyAvgYear.index = recordHighMonthlyAvgYear.index.astype(int)
    recordHighMonthlyAvgYear.columns = [i+' Year' for i in recordHighMonthlyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordHighMonthlyAvg, recordHighMonthlyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record High Monthly Average'
    return results

def record_low_daily_avg(df, true_average=False):
    """Record low daily averages.  If 'true_average' True, all measurements from each 24-hour day will be used to calculate the average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard)."""
    # Calculate the records
    dailyAvgs = daily_avgs(df=df, true_average=true_average)
    recordLowDailyAvg = dailyAvgs.groupby('YearDay').min(numeric_only=True)
    recordLowDailyAvg.index = recordLowDailyAvg.index.astype(int)
    # Record years
    recordLowDailyAvgYear = dailyAvgs.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    recordLowDailyAvgYear.drop('YearDay', axis=1, inplace=True)
    recordLowDailyAvgYear.index = recordLowDailyAvgYear.index.astype(int)
    recordLowDailyAvgYear.columns = [i+' Year' for i in recordLowDailyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordLowDailyAvg, recordLowDailyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Low Daily Average'
    return results

def record_low_monthly_avg(df, true_average=False):
    """Record low monthly averages. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the records
    dailyAvgs = mon_daily_avgs(df, true_average=true_average)
    monthlyAvgs = dailyAvgs.groupby(pd.Grouper(freq='M')).mean(numeric_only=True)
    monthlyAvgs.drop('YearDay', axis=1, inplace=True)
    recordLowMonthlyAvg = monthlyAvgs.groupby(monthlyAvgs.index.month).min(numeric_only=True)
    recordLowMonthlyAvg.index = recordLowMonthlyAvg.index.astype(int)
    # Record years
    recordLowMonthlyAvgYear = \
            monthlyAvgs.groupby(monthlyAvgs.index.month).apply(
                lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    recordLowMonthlyAvgYear.index = recordLowMonthlyAvgYear.index.astype(int)
    recordLowMonthlyAvgYear.columns = [i+' Year' for i in recordLowMonthlyAvgYear.columns]
    # Create xarray
    results = pd.concat((recordLowMonthlyAvg, recordLowMonthlyAvgYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Low Monthly Average'
    return results

def avg_daily_high(df):
    """Average daily highs."""        
    dailyHighs = daily_highs(df)
    results = dailyHighs.groupby('YearDay').mean(numeric_only=True)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Average Daily High'
    return results

def avg_monthly_high(df, true_average=False):
    """Average monthly highs. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    monthlyHighs = monthly_highs(df, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    avgMonthlyHighs = monthlyHighs.groupby(monthlyHighs.index.month).mean(numeric_only=True)
    results = xr.DataArray(avgMonthlyHighs, dims=['month', 'variable'])
    results.name = 'Average Monthly High'
    return results

def lowest_daily_high(df):
    """Lowest daily highs."""
    # Calculate the record
    dailyHighs = daily_highs(df)
    lowestHigh = dailyHighs.groupby('YearDay').min(numeric_only=True)
    lowestHigh.index = lowestHigh.index.astype(int)
    # Record years
    lowestHighYear = dailyHighs.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    lowestHighYear.drop('YearDay', axis=1, inplace=True)
    lowestHighYear.index = lowestHighYear.index.astype(int)
    lowestHighYear.columns = [i+' Year' for i in lowestHighYear.columns]
    # Create xarray
    results = pd.concat((lowestHigh, lowestHighYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Lowest Daily High'
    return results
    
def lowest_monthly_high(df, true_average=False):
    """Lowest monthly highs. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyHighs = monthly_highs(df, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    lowMonthlyHigh = monthlyHighs.groupby(monthlyHighs.index.month).min(numeric_only=True)
    lowMonthlyHigh.index = lowMonthlyHigh.index.astype(int)
    # Record years
    lowMonthlyHighYear = \
            monthlyHighs.groupby(monthlyHighs.index.month).apply(
                lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    lowMonthlyHighYear.index = lowMonthlyHighYear.index.astype(int)
    lowMonthlyHighYear.columns = [i+' Year' for i in lowMonthlyHighYear.columns]
    # Create xarray
    results = pd.concat((lowMonthlyHigh, lowMonthlyHighYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Lowest Monthly High'
    return results

def record_daily_high(df):
    """Record daily highs."""
    # Calculate the record
    dailyHighs = daily_highs(df)
    recordHigh = dailyHighs.groupby('YearDay').max(numeric_only=True)
    recordHigh.index = recordHigh.index.astype(int)
    # Record years
    recordHighYear = dailyHighs.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    recordHighYear.drop('YearDay', axis=1, inplace=True)
    recordHighYear.index = recordHighYear.index.astype(int)
    recordHighYear.columns = [i+' Year' for i in recordHighYear.columns]
    # Create xarray
    results = pd.concat((recordHigh, recordHighYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Daily High'
    return results

def record_monthly_high(df, true_average=False):
    """Record monthly highs. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyHighs = monthly_highs(df, true_average=true_average)
    monthlyHighs.drop('YearDay', axis=1, inplace=True)
    recordMonthlyHigh = monthlyHighs.groupby(monthlyHighs.index.month).max(numeric_only=True)
    recordMonthlyHigh.index = recordMonthlyHigh.index.astype(int)
    # Record years
    recordMonthlyHighYear = \
            monthlyHighs.groupby(monthlyHighs.index.month).apply(
                lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    recordMonthlyHighYear.index = recordMonthlyHighYear.index.astype(int)
    recordMonthlyHighYear.columns = [i+' Year' for i in recordMonthlyHighYear.columns]
    # Create xarray
    results = pd.concat((recordMonthlyHigh, recordMonthlyHighYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Monthly High'
    return results

def avg_daily_low(df):
    """Average daily lows."""        
    dailyLows = daily_lows(df)
    results = dailyLows.groupby('YearDay').mean(numeric_only=True)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Average Daily Low'
    return results

def avg_monthly_low(df, true_average=False):
    """Average monthly lows. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    monthlyLows = monthly_lows(df, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    avgMonthlyLows = monthlyLows.groupby(monthlyLows.index.month).mean(numeric_only=True)
    results = xr.DataArray(avgMonthlyLows, dims=['month', 'variable'])
    results.name = 'Average Monthly Low'
    return results

def highest_daily_low(df):
    """Highest daily lows."""
    # Calculate the record
    dailyLows = daily_lows(df)
    highestLow = dailyLows.groupby('YearDay').max(numeric_only=True)
    highestLow.index = highestLow.index.astype(int)
    # Record years
    highestLowYear = dailyLows.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    highestLowYear.drop('YearDay', axis=1, inplace=True)
    highestLowYear.index = highestLowYear.index.astype(int)
    highestLowYear.columns = [i+' Year' for i in highestLowYear.columns]
    # Create xarray
    results = pd.concat((highestLow, highestLowYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Highest Daily Low'
    return results
    
def highest_monthly_low(df, true_average=False):
    """Highest monthly lows. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyLows = monthly_lows(df, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    highestMonthlyLow = monthlyLows.groupby(monthlyLows.index.month).max(numeric_only=True)
    highestMonthlyLow.index = highestMonthlyLow.index.astype(int)
    # Record years
    highestMonthlyLowYear = \
            monthlyLows.groupby(monthlyLows.index.month).apply(
                lambda x: x.sort_index().idxmax(numeric_only=True).dt.year)
    highestMonthlyLowYear.index = highestMonthlyLowYear.index.astype(int)
    highestMonthlyLowYear.columns = [i+' Year' for i in highestMonthlyLowYear.columns]
    # Create xarray
    results = pd.concat((highestMonthlyLow, highestMonthlyLowYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Highest Monthly Low'
    return results

def record_daily_low(df):
    """Record daily lows."""
    # Calculate the record
    dailyLows = daily_lows(df)
    recordLow = dailyLows.groupby('YearDay').min(numeric_only=True)
    recordLow.index = recordLow.index.astype(int)
    # Record years
    recordLowYear = dailyLows.groupby('YearDay').apply(
            lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    recordLowYear.drop('YearDay', axis=1, inplace=True)
    recordLowYear.index = recordLowYear.index.astype(int)
    recordLowYear.columns = [i+' Year' for i in recordLowYear.columns]
    # Create xarray
    results = pd.concat((recordLow, recordLowYear), axis=1)
    results = xr.DataArray(results, dims=['yearday', 'variable'])
    results.name = 'Record Daily Low'
    return results

def record_monthly_low(df, true_average=False):
    """Record monthly lows. If 'true_average' is True, all measurements from each 24-hour day will be used to calculate the daily average. Otherwise, only the maximum and minimum observations are used. Defaults to False (meteorological standard).
    """
    # Calculate the record
    monthlyLows = monthly_lows(df, true_average=true_average)
    monthlyLows.drop('YearDay', axis=1, inplace=True)
    recordMonthlyLow = monthlyLows.groupby(monthlyLows.index.month).min(numeric_only=True)
    recordMonthlyLow.index = recordMonthlyLow.index.astype(int)
    # Record years
    recordMonthlyLowYear = \
            monthlyLows.groupby(monthlyLows.index.month).apply(
                lambda x: x.sort_index().idxmin(numeric_only=True).dt.year)
    recordMonthlyLowYear.index = recordMonthlyLowYear.index.astype(int)
    recordMonthlyLowYear.columns = [i+' Year' for i in recordMonthlyLowYear.columns]
    # Create xarray
    results = pd.concat((recordMonthlyLow, recordMonthlyLowYear), axis=1)
    results = xr.DataArray(results, dims=['month', 'variable'])
    results.name = 'Record Monthly Low'
    return results

def number_of_years_byday(df):
    """Number of years in the historical data records by day of year."""
    numYears = pd.concat([df[[v, 'YearDay']].dropna().groupby('YearDay')\
                                        .apply(lambda x: len(x.index.year.unique())) \
                         for v in filtered_hours.columns if v != 'YearDay'], axis=1)
    numYears.columns = [v for v in df.columns if v != 'YearDay']
    results = xr.DataArray(numYears, dims=['yearday', 'variable'])
    results.name = 'Number of Years'
    return results

def number_of_years_bymonth(df):
    """Number of years in the historical data records by month."""
    numYears = pd.concat([df[v].dropna().groupby(df[v].dropna().index.month)\
                                        .apply(lambda x: len(x.index.year.unique())) \
                         for v in filtered_hours.columns if v != 'YearDay'], axis=1)
    numYears.columns = [v for v in df.columns if v != 'YearDay']
    results = xr.DataArray(numYears, dims=['month', 'variable'])
    results.name = 'Number of Years'
    return results

def generate_yeardays():
    return pd.date_range(start='2020-01-01',end='2020-12-31', freq='1D').strftime('%d-%b')

Data cleaning

First we need to load in the data and metadata for the desired station. As before, stationname is the custom human-readable “City, ST” string for the station. This will be used to determine the directory from which to load the data. Since we are not downloading data, we do not need the NOAA-COOPS station ID number.

stationname = 'Virginia Key, FL'

Derive the local directory name containing the data from the station name. This is the same way the directory was created when the data were downloaded.

dirname = camel(stationname)
outdir = f'../{dirname}'

print(f"Station folder: {dirname}")
print(f"Full directory: {outdir}")

Station folder: virginiaKeyFl
Full directory: ../virginiaKeyFl

Next, load the data and metadata.

# Metadata
with open(os.path.join(outdir, 'metadata.yml')) as m:
    meta = yaml.safe_load(m)

# Data types
dtypeDict = {k: float for k in meta['variables']}
dtypeDict['Water Level QC'] = str

# Observational data
data = pd.read_csv(os.path.join(outdir, 'observational_data_record.csv.gz'),
                   index_col=f'time_{meta["tz"]}', parse_dates=True,
                   compression='infer', dtype=dtypeDict)

Now we filter the data to remove days with more than 4 hours of missing data and months with more than 2 days of missing data. These thresholds are stored in meta and can easily be changed. We have to do this one variable at a time because this is sensor-dependent, so it takes a short while to run.

filtered_hours = pd.concat([filter_hours(data[var],
                                       hr_threshold=meta['hr_threshold'])
                                       for var in meta['variables']], axis=1)

filtered_days = pd.concat([filter_days(data[var],
                                       hr_threshold=meta['hr_threshold'],
                                       day_threshold=meta['day_threshold'])
                                       for var in meta['variables']], axis=1)

Confirm that the data were filtered:

print(data.shape)
print(filtered_hours.shape)
print(filtered_days.shape)

(2745205, 4)
(2720854, 3)
(2657974, 3)

Calculate records

Now we’re ready to determine the records using all of the functions above. We’ll store these in an xarray dataset and add the appropriate metadata for convenience. But first, we need to add a day of year (DOY) column so that we can calculate daily records. We’ve used a function to do this because accounting for leap years is not trivial.

filtered_hours = DOY(filtered_hours)
filtered_days = DOY(filtered_days)

daily_records = \
    xr.Dataset({'Daily Average': daily_avg(filtered_hours),
                'Record High Daily Average': record_high_daily_avg(filtered_hours),
                'Record Low Daily Average': record_low_daily_avg(filtered_hours),
                'Average High': avg_daily_high(filtered_hours),
                'Lowest High': lowest_daily_high(filtered_hours),
                'Record High': record_daily_high(filtered_hours),
                'Average Low': avg_daily_low(filtered_hours),
                'Highest Low': highest_daily_low(filtered_hours),
                'Record Low': record_daily_low(filtered_hours),
                'Years': number_of_years_byday(filtered_hours)},
               attrs={k:v for k, v in meta.items() if k not in ['outdir', 'variables', 'units']})

monthly_records = \
    xr.Dataset({'Monthly Average': monthly_avg(filtered_days),
                'Record High Monthly Average': record_high_monthly_avg(filtered_days),
                'Record Low Monthly Average': record_low_monthly_avg(filtered_days),
                'Average High': avg_monthly_high(filtered_days),
                'Lowest High': lowest_monthly_high(filtered_days),
                'Record High': record_monthly_high(filtered_days),
                'Average Low': avg_monthly_low(filtered_days),
                'Highest Low': highest_monthly_low(filtered_days),
                'Record Low': record_monthly_low(filtered_days),
                'Years': number_of_years_bymonth(filtered_days)},
               attrs={k:v for k, v in meta.items() if k not in ['outdir', 'variables', 'units']})

Add data units and time series ranges for each variable to the arrays as metadata attributes.

for k, v in meta['units'].items():
    daily_records.attrs[k+' units'] = v

for var in daily_records.coords['variable'].values:
    if 'Year' not in var:
        daily_records.attrs[var+' data range'] = \
            (filtered_hours[var].first_valid_index().strftime('%Y-%m-%d'),
             filtered_hours[var].last_valid_index().strftime('%Y-%m-%d'))

for k, v in meta['units'].items():
    monthly_records.attrs[k+' units'] = v

for var in monthly_records.coords['variable'].values:
    if 'Year' not in var:
        monthly_records.attrs[var+' data range'] = \
            (filtered_days[var].first_valid_index().strftime('%Y-%m-%d'),
             filtered_days[var].last_valid_index().strftime('%Y-%m-%d'))

What do we have now? Let’s take a look:

daily_records

<xarray.Dataset> Size: 179kB
Dimensions:                    (yearday: 366, variable: 6)
Coordinates:
  * yearday                    (yearday) int64 3kB 1 2 3 4 5 ... 363 364 365 366
  * variable                   (variable) object 48B 'Air Temperature' ... 'W...
Data variables:
    Daily Average              (yearday, variable) float64 18kB 72.06 ... nan
    Record High Daily Average  (yearday, variable) float64 18kB 78.7 ... 2.02...
    Record Low Daily Average   (yearday, variable) float64 18kB 54.4 ... 2.01...
    Average High               (yearday, variable) float64 18kB 75.49 ... nan
    Lowest High                (yearday, variable) float64 18kB 63.3 ... 2.01...
    Record High                (yearday, variable) float64 18kB 80.8 ... 2.02...
    Average Low                (yearday, variable) float64 18kB 68.64 ... nan
    Highest Low                (yearday, variable) float64 18kB 77.2 ... 2.02...
    Record Low                 (yearday, variable) float64 18kB 45.5 ... 2.01...
    Years                      (yearday, variable) float64 18kB 27.0 nan ... nan
Attributes: (12/14)
    datum:                         MHHW
    day_threshold:                 2
    dirname:                       virginiaKeyFl
    hr_threshold:                  4
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    ...                            ...
    Air Temperature units:         °F
    Water Level units:             ft
    Water Temperature units:       °F
    Air Temperature data range:    ('1994-01-29', '2025-05-30')
    Water Level data range:        ('1994-01-29', '2025-05-29')
    Water Temperature data range:  ('1994-01-29', '2025-05-30')

xarray.Dataset

• Dimensions:
  • yearday: 366
  • variable: 6
• Coordinates: (2)
  • yearday
    (yearday)
    int64
    1 2 3 4 5 6 ... 362 363 364 365 366

    array([  1,   2,   3, ..., 364, 365, 366])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype=object)

• Data variables: (10)
  • Daily Average
    (yearday, variable)
    float64
    72.06 nan -1.108 ... nan 72.48 nan

    array([[72.06481481,         nan, -1.10846774,         nan, 72.78392857,
                    nan],
           [72.12222222,         nan, -1.13256452,         nan, 73.23214286,
                    nan],
           [70.36296296,         nan, -1.11587097,         nan, 73.35      ,
                    nan],
           ...,
           [70.63076923,         nan, -0.97168333,         nan, 72.45      ,
                    nan],
           [69.73653846,         nan, -1.03998387,         nan, 72.47777778,
                    nan],
           [70.84038462,         nan, -1.05217742,         nan, 72.48333333,
                    nan]])

  • Record High Daily Average
    (yearday, variable)
    float64
    78.7 2.016e+03 ... 80.5 2.021e+03

    array([[ 7.870e+01,  2.016e+03, -6.695e-01,  2.022e+03,  8.040e+01,
             2.022e+03],
           [ 7.780e+01,  2.022e+03, -6.775e-01,  2.022e+03,  8.025e+01,
             2.022e+03],
           [ 7.800e+01,  2.015e+03, -3.575e-01,  2.018e+03,  8.070e+01,
             2.017e+03],
           ...,
           [ 7.945e+01,  2.015e+03, -3.900e-01,  1.994e+03,  8.260e+01,
             2.016e+03],
           [ 7.950e+01,  2.015e+03, -4.400e-01,  1.994e+03,  8.195e+01,
             2.016e+03],
           [ 7.920e+01,  2.015e+03, -6.525e-01,  2.022e+03,  8.050e+01,
             2.021e+03]])

  • Record Low Daily Average
    (yearday, variable)
    float64
    54.4 2.001e+03 ... 66.1 2.01e+03

    array([[ 5.4400e+01,  2.0010e+03, -1.7765e+00,  1.9980e+03,  6.6450e+01,
             2.0110e+03],
           [ 5.6650e+01,  2.0100e+03, -1.5780e+00,  2.0110e+03,  6.6850e+01,
             2.0110e+03],
           [ 5.1700e+01,  2.0120e+03, -1.7915e+00,  2.0110e+03,  6.7300e+01,
             2.0110e+03],
           ...,
           [ 5.8700e+01,  2.0090e+03, -1.6585e+00,  2.0080e+03,  6.6550e+01,
             1.9950e+03],
           [ 5.4750e+01,  2.0000e+03, -1.5370e+00,  1.9960e+03,  6.6400e+01,
             2.0100e+03],
           [ 4.9900e+01,  2.0000e+03, -1.5765e+00,  1.9970e+03,  6.6100e+01,
             2.0100e+03]])

  • Average High
    (yearday, variable)
    float64
    75.49 nan 0.04435 ... nan 73.71 nan

    array([[7.54925926e+01,            nan, 4.43548387e-02,            nan,
            7.40178571e+01,            nan],
           [7.53962963e+01,            nan, 3.09677419e-03,            nan,
            7.44607143e+01,            nan],
           [7.44370370e+01,            nan, 1.67419355e-02,            nan,
            7.46000000e+01,            nan],
           ...,
           [7.44307692e+01,            nan, 1.53700000e-01,            nan,
            7.38444444e+01,            nan],
           [7.33230769e+01,            nan, 9.42903226e-02,            nan,
            7.37962963e+01,            nan],
           [7.44500000e+01,            nan, 7.68709677e-02,            nan,
            7.37074074e+01,            nan]])

  • Lowest High
    (yearday, variable)
    float64
    63.3 2.001e+03 ... 67.8 2.01e+03

    array([[ 6.330e+01,  2.001e+03, -6.460e-01,  2.004e+03,  6.780e+01,
             2.011e+03],
           [ 6.420e+01,  2.010e+03, -6.430e-01,  2.012e+03,  6.800e+01,
             2.011e+03],
           [ 5.700e+01,  2.012e+03, -7.780e-01,  2.012e+03,  6.890e+01,
             2.011e+03],
           ...,
           [ 6.710e+01,  2.010e+03, -6.690e-01,  2.008e+03,  6.940e+01,
             2.003e+03],
           [ 5.950e+01,  2.000e+03, -6.100e-01,  1.996e+03,  6.840e+01,
             2.010e+03],
           [ 5.560e+01,  2.000e+03, -6.070e-01,  1.996e+03,  6.780e+01,
             2.010e+03]])

  • Record High
    (yearday, variable)
    float64
    80.8 2.025e+03 ... 81.7 2.021e+03

    array([[8.080e+01, 2.025e+03, 8.100e-01, 2.022e+03, 8.110e+01, 2.022e+03],
           [7.970e+01, 2.016e+03, 9.020e-01, 2.018e+03, 8.100e+01, 2.022e+03],
           [7.920e+01, 2.019e+03, 1.198e+00, 2.018e+03, 8.150e+01, 2.017e+03],
           ...,
           [8.060e+01, 2.015e+03, 9.060e-01, 1.994e+03, 8.330e+01, 2.016e+03],
           [8.060e+01, 2.013e+03, 9.150e-01, 1.994e+03, 8.310e+01, 2.016e+03],
           [8.010e+01, 2.015e+03, 7.250e-01, 1.994e+03, 8.170e+01, 2.021e+03]])

  • Average Low
    (yearday, variable)
    float64
    68.64 nan -2.261 ... nan 71.26 nan

    array([[68.63703704,         nan, -2.26129032,         nan, 71.55      ,
                    nan],
           [68.84814815,         nan, -2.26822581,         nan, 72.00357143,
                    nan],
           [66.28888889,         nan, -2.24848387,         nan, 72.1       ,
                    nan],
           ...,
           [66.83076923,         nan, -2.09706667,         nan, 71.05555556,
                    nan],
           [66.15      ,         nan, -2.17425806,         nan, 71.15925926,
                    nan],
           [67.23076923,         nan, -2.18122581,         nan, 71.25925926,
                    nan]])

  • Highest Low
    (yearday, variable)
    float64
    77.2 2.016e+03 ... 79.3 2.021e+03

    array([[ 7.720e+01,  2.016e+03, -1.791e+00,  2.024e+03,  7.970e+01,
             2.022e+03],
           [ 7.630e+01,  2.017e+03, -1.516e+00,  2.024e+03,  7.950e+01,
             2.022e+03],
           [ 7.700e+01,  2.015e+03, -1.339e+00,  2.024e+03,  7.990e+01,
             2.017e+03],
           ...,
           [ 7.830e+01,  2.015e+03, -1.594e+00,  2.019e+03,  8.190e+01,
             2.016e+03],
           [ 7.840e+01,  2.015e+03, -1.519e+00,  2.019e+03,  8.080e+01,
             2.016e+03],
           [ 7.830e+01,  2.015e+03, -1.677e+00,  2.019e+03,  7.930e+01,
             2.021e+03]])

  • Record Low
    (yearday, variable)
    float64
    45.5 2.001e+03 ... 64.4 2.01e+03

    array([[  45.5  , 2001.   ,   -3.159, 1998.   ,   63.9  , 2001.   ],
           [  49.1  , 2010.   ,   -2.835, 2006.   ,   64.4  , 2001.   ],
           [  46.4  , 2012.   ,   -3.035, 2011.   ,   65.5  , 2001.   ],
           ...,
           [  50.1  , 2009.   ,   -2.648, 2005.   ,   62.2  , 2010.   ],
           [  50.   , 2000.   ,   -2.766, 2005.   ,   64.4  , 2010.   ],
           [  44.2  , 2000.   ,   -2.822, 1997.   ,   64.4  , 2010.   ]])

  • Years
    (yearday, variable)
    float64
    27.0 nan 31.0 nan ... nan 27.0 nan

    array([[27., nan, 31., nan, 28., nan],
           [27., nan, 31., nan, 28., nan],
           [27., nan, 31., nan, 28., nan],
           ...,
           [26., nan, 30., nan, 27., nan],
           [26., nan, 31., nan, 27., nan],
           [26., nan, 31., nan, 27., nan]])

• Indexes: (2)
  • yearday
    PandasIndex

    PandasIndex(Int64Index([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,
                ...
                357, 358, 359, 360, 361, 362, 363, 364, 365, 366],
               dtype='int64', name='yearday', length=366))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (14)

  datum :

  MHHW

  day_threshold :

  2

  dirname :

  virginiaKeyFl

  hr_threshold :

  4

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  °F

  Water Level units :

  ft

  Water Temperature units :

  °F

  Air Temperature data range :

  ('1994-01-29', '2025-05-30')

  Water Level data range :

  ('1994-01-29', '2025-05-29')

  Water Temperature data range :

  ('1994-01-29', '2025-05-30')

monthly_records

<xarray.Dataset> Size: 6kB
Dimensions:                      (month: 12, variable: 6)
Coordinates:
  * month                        (month) int64 96B 1 2 3 4 5 6 7 8 9 10 11 12
  * variable                     (variable) object 48B 'Air Temperature' ... ...
Data variables:
    Monthly Average              (month, variable) float64 576B 68.66 ... nan
    Record High Monthly Average  (month, variable) float64 576B 72.57 ... 2.0...
    Record Low Monthly Average   (month, variable) float64 576B 63.04 ... 2.0...
    Average High                 (month, variable) float64 576B 76.05 ... nan
    Lowest High                  (month, variable) float64 576B 72.95 ... 2.0...
    Record High                  (month, variable) float64 576B 78.05 ... 2.0...
    Average Low                  (month, variable) float64 576B 55.55 ... nan
    Highest Low                  (month, variable) float64 576B 63.5 ... 2.01...
    Record Low                   (month, variable) float64 576B 48.3 ... 2.01...
    Years                        (month, variable) float64 576B 23.0 nan ... nan
Attributes: (12/14)
    datum:                         MHHW
    day_threshold:                 2
    dirname:                       virginiaKeyFl
    hr_threshold:                  4
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    ...                            ...
    Air Temperature units:         °F
    Water Level units:             ft
    Water Temperature units:       °F
    Air Temperature data range:    ('1994-02-01', '2025-05-30')
    Water Level data range:        ('1994-02-01', '2025-04-30')
    Water Temperature data range:  ('1994-02-01', '2025-05-30')

xarray.Dataset

• Dimensions:
  • month: 12
  • variable: 6
• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype=object)

• Data variables: (10)
  • Monthly Average
    (month, variable)
    float64
    68.66 nan -1.124 ... nan 74.04 nan

    array([[68.65629476,         nan, -1.12401237,         nan, 71.62517419,
                    nan],
           [70.89272269,         nan, -1.18149129,         nan, 73.14671022,
                    nan],
           [72.27276989,         nan, -1.07706229,         nan, 75.15274624,
                    nan],
           [75.63306437,         nan, -1.02213989,         nan, 78.47779399,
                    nan],
           [78.72849056,         nan, -0.95087643,         nan, 82.13263705,
                    nan],
           [81.53800571,         nan, -0.98853085,         nan, 85.23606117,
                    nan],
           [82.90998167,         nan, -1.02524941,         nan, 87.13986958,
                    nan],
           [83.29827764,         nan, -0.88312497,         nan, 87.34393222,
                    nan],
           [82.09382299,         nan, -0.5381523 ,         nan, 85.7543954 ,
                    nan],
           [79.63475582,         nan, -0.36203893,         nan, 82.04959733,
                    nan],
           [75.04279933,         nan, -0.52922935,         nan, 77.38016782,
                    nan],
           [71.4254462 ,         nan, -0.91394324,         nan, 74.03937634,
                    nan]])

  • Record High Monthly Average
    (month, variable)
    float64
    72.57 2.013e+03 ... 82.55 2.016e+03

    array([[ 7.25709677e+01,  2.01300000e+03, -5.27451613e-01,
             2.02200000e+03,  7.88870968e+01,  2.01700000e+03],
           [ 7.49000000e+01,  2.01800000e+03, -6.95672414e-01,
             2.02400000e+03,  7.63017857e+01,  2.02100000e+03],
           [ 7.76166667e+01,  2.00300000e+03, -5.01596774e-01,
             2.01900000e+03,  7.99854839e+01,  2.00300000e+03],
           [ 7.93833333e+01,  2.02000000e+03, -5.01650000e-01,
             2.02300000e+03,  8.33933333e+01,  2.02000000e+03],
           [ 8.19306452e+01,  2.02400000e+03, -5.12403226e-01,
             2.02200000e+03,  8.48129032e+01,  2.02400000e+03],
           [ 8.36133333e+01,  2.01000000e+03, -5.02616667e-01,
             2.02300000e+03,  8.76333333e+01,  2.01000000e+03],
           [ 8.49967742e+01,  2.02300000e+03, -6.88129032e-01,
             2.01900000e+03,  8.94919355e+01,  2.02300000e+03],
           [ 8.59064516e+01,  2.02200000e+03, -4.23822581e-01,
             2.01900000e+03,  9.01096774e+01,  2.02100000e+03],
           [ 8.37400000e+01,  2.02400000e+03, -2.12833333e-02,
             2.01900000e+03,  8.94866667e+01,  2.02100000e+03],
           [ 8.12387097e+01,  2.02000000e+03,  1.75935484e-01,
             2.02300000e+03,  8.59064516e+01,  2.02100000e+03],
           [ 7.86233333e+01,  2.01500000e+03,  9.05500000e-02,
             2.02300000e+03,  8.14666667e+01,  2.01600000e+03],
           [ 7.68709677e+01,  2.01500000e+03, -2.97661290e-01,
             2.02200000e+03,  8.25467742e+01,  2.01600000e+03]])

  • Record Low Monthly Average
    (month, variable)
    float64
    63.04 2.001e+03 ... 68.2 2.01e+03

    array([[ 6.30387097e+01,  2.00100000e+03, -1.55930645e+00,
             2.00200000e+03,  6.75096774e+01,  2.00100000e+03],
           [ 6.54655172e+01,  1.99600000e+03, -1.64225000e+00,
             1.99500000e+03,  6.77392857e+01,  2.00500000e+03],
           [ 6.60790323e+01,  2.01000000e+03, -1.47172581e+00,
             1.99600000e+03,  6.92209677e+01,  2.01000000e+03],
           [ 7.28133333e+01,  2.00400000e+03, -1.48520000e+00,
             1.99400000e+03,  7.50883333e+01,  2.00400000e+03],
           [ 7.69741379e+01,  2.00700000e+03, -1.32648276e+00,
             1.99600000e+03,  7.86403226e+01,  2.00100000e+03],
           [ 7.98400000e+01,  2.01400000e+03, -1.41310000e+00,
             1.99600000e+03,  8.32568966e+01,  2.00200000e+03],
           [ 8.09806452e+01,  2.01300000e+03, -1.42170968e+00,
             1.99400000e+03,  8.45145161e+01,  2.01300000e+03],
           [ 8.18209677e+01,  1.99400000e+03, -1.27498387e+00,
             1.99600000e+03,  8.54354839e+01,  1.99500000e+03],
           [ 8.05716667e+01,  2.00100000e+03, -1.05141667e+00,
             1.99600000e+03,  8.27333333e+01,  2.00400000e+03],
           [ 7.75516129e+01,  2.00000000e+03, -8.86080645e-01,
             1.99800000e+03,  7.95629032e+01,  2.00000000e+03],
           [ 7.13733333e+01,  2.01200000e+03, -1.11588333e+00,
             1.99800000e+03,  7.45900000e+01,  2.01200000e+03],
           [ 6.20966667e+01,  2.01000000e+03, -1.34004839e+00,
             2.00800000e+03,  6.82016129e+01,  2.01000000e+03]])

  • Average High
    (month, variable)
    float64
    76.05 nan -0.7102 ... nan 77.61 nan

    array([[ 7.60500000e+01,             nan, -7.10209677e-01,
                        nan,  7.54460000e+01,             nan],
           [ 7.64854167e+01,             nan, -7.48650000e-01,
                        nan,  7.70181818e+01,             nan],
           [ 7.84900000e+01,             nan, -6.95586207e-01,
                        nan,  7.92420000e+01,             nan],
           [ 8.07400000e+01,             nan, -6.88650000e-01,
                        nan,  8.18076923e+01,             nan],
           [ 8.24807692e+01,             nan, -6.04620690e-01,
                        nan,  8.52923077e+01,             nan],
           [ 8.47547619e+01,             nan, -6.88844828e-01,
                        nan,  8.79041667e+01,             nan],
           [ 8.57903846e+01,             nan, -6.96016667e-01,
                        nan,  8.91807692e+01,             nan],
           [ 8.58458333e+01,             nan, -4.56645161e-01,
                        nan,  8.94750000e+01,             nan],
           [ 8.51660000e+01,             nan, -1.05483871e-02,
                        nan,  8.80340000e+01,             nan],
           [ 8.38125000e+01,             nan,  1.30224138e-01,
                        nan,  8.56300000e+01,             nan],
           [ 7.98416667e+01,             nan,  1.90666667e-02,
                        nan,  8.06300000e+01,             nan],
           [ 7.74700000e+01,             nan, -4.77350000e-01,
                        nan,  7.76060000e+01,             nan]])

  • Lowest High
    (month, variable)
    float64
    72.95 2.011e+03 ... 72.95 2.003e+03

    array([[ 7.2950e+01,  2.0110e+03, -1.3290e+00,  2.0090e+03,  7.0500e+01,
             2.0110e+03],
           [ 7.4200e+01,  2.0000e+03, -1.4140e+00,  1.9950e+03,  7.3700e+01,
             2.0160e+03],
           [ 7.4150e+01,  2.0100e+03, -1.1480e+00,  2.0080e+03,  7.3300e+01,
             2.0100e+03],
           [ 7.7300e+01,  2.0040e+03, -1.1860e+00,  2.0010e+03,  7.8450e+01,
             2.0100e+03],
           [ 7.9350e+01,  2.0070e+03, -1.1025e+00,  2.0040e+03,  8.2500e+01,
             2.0180e+03],
           [ 8.2750e+01,  2.0140e+03, -1.1155e+00,  2.0030e+03,  8.5300e+01,
             1.9960e+03],
           [ 8.4200e+01,  2.0120e+03, -1.1500e+00,  1.9940e+03,  8.6450e+01,
             2.0130e+03],
           [ 8.4050e+01,  2.0030e+03, -9.8900e-01,  2.0010e+03,  8.7250e+01,
             2.0000e+03],
           [ 8.3900e+01,  2.0000e+03, -6.6800e-01,  1.9960e+03,  8.5200e+01,
             1.9970e+03],
           [ 8.0950e+01,  2.0100e+03, -5.2650e-01,  1.9980e+03,  8.2700e+01,
             2.0040e+03],
           [ 7.6900e+01,  2.0120e+03, -6.2000e-01,  1.9950e+03,  7.6750e+01,
             2.0010e+03],
           [ 7.2500e+01,  2.0100e+03, -1.0285e+00,  2.0080e+03,  7.2950e+01,
             2.0030e+03]])

  • Record High
    (month, variable)
    float64
    78.05 2.022e+03 ... 84.55 2.016e+03

    array([[ 7.8050e+01,  2.0220e+03,  1.5000e-02,  2.0200e+03,  8.1700e+01,
             2.0170e+03],
           [ 7.8550e+01,  2.0210e+03, -1.5250e-01,  2.0240e+03,  8.1500e+01,
             2.0210e+03],
           [ 8.2850e+01,  2.0030e+03, -5.6000e-02,  2.0190e+03,  8.3200e+01,
             2.0210e+03],
           [ 8.5800e+01,  2.0200e+03, -8.0000e-03,  2.0230e+03,  8.5750e+01,
             2.0200e+03],
           [ 8.7250e+01,  2.0240e+03,  4.3150e-01,  2.0220e+03,  8.8600e+01,
             2.0240e+03],
           [ 8.7650e+01,  2.0090e+03, -4.1000e-02,  2.0230e+03,  9.0400e+01,
             2.0100e+03],
           [ 8.8700e+01,  2.0180e+03, -1.9200e-01,  2.0190e+03,  9.2000e+01,
             2.0210e+03],
           [ 8.8500e+01,  2.0220e+03,  1.4500e-02,  2.0140e+03,  9.2200e+01,
             2.0210e+03],
           [ 8.6700e+01,  2.0210e+03,  1.8965e+00,  2.0170e+03,  9.1150e+01,
             2.0210e+03],
           [ 8.6750e+01,  2.0230e+03,  7.2000e-01,  2.0170e+03,  8.9050e+01,
             2.0160e+03],
           [ 8.2050e+01,  2.0200e+03,  8.7900e-01,  2.0220e+03,  8.5000e+01,
             2.0200e+03],
           [ 7.9650e+01,  1.9940e+03,  4.8100e-01,  2.0230e+03,  8.4550e+01,
             2.0160e+03]])

  • Average Low
    (month, variable)
    float64
    55.55 nan -1.507 ... nan 70.24 nan

    array([[55.55      ,         nan, -1.5068871 ,         nan, 67.71      ,
                    nan],
           [59.3625    ,         nan, -1.5467    ,         nan, 69.18863636,
                    nan],
           [63.334     ,         nan, -1.44827586,         nan, 71.098     ,
                    nan],
           [68.398     ,         nan, -1.3832    ,         nan, 74.68461538,
                    nan],
           [73.95192308,         nan, -1.26675862,         nan, 78.20192308,
                    nan],
           [77.63571429,         nan, -1.29075862,         nan, 82.23958333,
                    nan],
           [79.01923077,         nan, -1.36201667,         nan, 84.58076923,
                    nan],
           [79.51041667,         nan, -1.25251613,         nan, 84.41346154,
                    nan],
           [78.336     ,         nan, -0.97132258,         nan, 83.102     ,
                    nan],
           [72.88333333,         nan, -0.82972414,         nan, 77.872     ,
                    nan],
           [65.97291667,         nan, -0.99663333,         nan, 74.172     ,
                    nan],
           [59.406     ,         nan, -1.29768333,         nan, 70.244     ,
                    nan]])

  • Highest Low
    (month, variable)
    float64
    63.5 2.013e+03 ... 79.65 2.016e+03

    array([[ 6.350e+01,  2.013e+03, -7.970e-01,  2.022e+03,  7.590e+01,
             2.017e+03],
           [ 7.000e+01,  2.018e+03, -1.127e+00,  2.019e+03,  7.340e+01,
             2.023e+03],
           [ 7.200e+01,  1.997e+03, -9.550e-01,  2.019e+03,  7.540e+01,
             1.997e+03],
           [ 7.260e+01,  2.015e+03, -9.910e-01,  2.024e+03,  8.105e+01,
             2.020e+03],
           [ 7.710e+01,  2.003e+03, -7.855e-01,  2.024e+03,  8.035e+01,
             1.994e+03],
           [ 8.075e+01,  2.004e+03, -8.070e-01,  2.024e+03,  8.520e+01,
             2.004e+03],
           [ 8.230e+01,  2.022e+03, -1.091e+00,  2.019e+03,  8.715e+01,
             2.023e+03],
           [ 8.355e+01,  2.022e+03, -7.610e-01,  2.019e+03,  8.725e+01,
             2.021e+03],
           [ 8.160e+01,  2.024e+03, -5.235e-01,  2.008e+03,  8.710e+01,
             2.021e+03],
           [ 7.780e+01,  2.020e+03, -1.020e-01,  2.023e+03,  8.340e+01,
             2.021e+03],
           [ 7.445e+01,  2.020e+03, -2.755e-01,  2.022e+03,  8.005e+01,
             2.016e+03],
           [ 7.050e+01,  2.015e+03, -6.525e-01,  2.022e+03,  7.965e+01,
             2.016e+03]])

  • Record Low
    (month, variable)
    float64
    48.3 1.997e+03 ... 63.7 2.01e+03

    array([[ 4.8300e+01,  1.9970e+03, -2.0160e+00,  2.0010e+03,  6.3950e+01,
             2.0030e+03],
           [ 4.7900e+01,  1.9960e+03, -2.0225e+00,  2.0080e+03,  6.3150e+01,
             2.0050e+03],
           [ 5.5100e+01,  1.9960e+03, -2.0045e+00,  1.9960e+03,  6.4850e+01,
             2.0100e+03],
           [ 6.1200e+01,  2.0090e+03, -1.9000e+00,  1.9940e+03,  6.9400e+01,
             2.0030e+03],
           [ 6.7950e+01,  1.9990e+03, -1.6075e+00,  2.0110e+03,  7.4100e+01,
             2.0010e+03],
           [ 7.5100e+01,  1.9950e+03, -1.8310e+00,  2.0060e+03,  7.9950e+01,
             2.0140e+03],
           [ 7.6100e+01,  2.0130e+03, -1.8160e+00,  1.9940e+03,  8.0950e+01,
             2.0130e+03],
           [ 7.7000e+01,  1.9940e+03, -1.6095e+00,  1.9960e+03,  8.1050e+01,
             2.0120e+03],
           [ 7.4300e+01,  2.0010e+03, -1.3995e+00,  2.0040e+03,  7.9500e+01,
             2.0040e+03],
           [ 6.4600e+01,  2.0050e+03, -1.5915e+00,  2.0100e+03,  7.3200e+01,
             2.0050e+03],
           [ 5.7350e+01,  2.0060e+03, -1.5830e+00,  1.9980e+03,  7.0800e+01,
             2.0000e+03],
           [ 4.8750e+01,  2.0100e+03, -1.7110e+00,  2.0040e+03,  6.3700e+01,
             2.0100e+03]])

  • Years
    (month, variable)
    float64
    23.0 nan 31.0 nan ... nan 25.0 nan

    array([[23., nan, 31., nan, 25., nan],
           [24., nan, 30., nan, 22., nan],
           [25., nan, 29., nan, 25., nan],
           [25., nan, 30., nan, 26., nan],
           [26., nan, 29., nan, 26., nan],
           [21., nan, 29., nan, 24., nan],
           [26., nan, 30., nan, 26., nan],
           [24., nan, 31., nan, 26., nan],
           [25., nan, 31., nan, 25., nan],
           [24., nan, 29., nan, 25., nan],
           [24., nan, 30., nan, 25., nan],
           [25., nan, 30., nan, 25., nan]])

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (14)

  datum :

  MHHW

  day_threshold :

  2

  dirname :

  virginiaKeyFl

  hr_threshold :

  4

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  °F

  Water Level units :

  ft

  Water Temperature units :

  °F

  Air Temperature data range :

  ('1994-02-01', '2025-05-30')

  Water Level data range :

  ('1994-02-01', '2025-04-30')

  Water Temperature data range :

  ('1994-02-01', '2025-05-30')

How are these stored? Let’s consider the monthly stats. Each statistic is its own variable within the dataset. Take Record High for example:

monthly_records['Record High']

<xarray.DataArray 'Record High' (month: 12, variable: 6)> Size: 576B
array([[ 7.8050e+01,  2.0220e+03,  1.5000e-02,  2.0200e+03,  8.1700e+01,
         2.0170e+03],
       [ 7.8550e+01,  2.0210e+03, -1.5250e-01,  2.0240e+03,  8.1500e+01,
         2.0210e+03],
       [ 8.2850e+01,  2.0030e+03, -5.6000e-02,  2.0190e+03,  8.3200e+01,
         2.0210e+03],
       [ 8.5800e+01,  2.0200e+03, -8.0000e-03,  2.0230e+03,  8.5750e+01,
         2.0200e+03],
       [ 8.7250e+01,  2.0240e+03,  4.3150e-01,  2.0220e+03,  8.8600e+01,
         2.0240e+03],
       [ 8.7650e+01,  2.0090e+03, -4.1000e-02,  2.0230e+03,  9.0400e+01,
         2.0100e+03],
       [ 8.8700e+01,  2.0180e+03, -1.9200e-01,  2.0190e+03,  9.2000e+01,
         2.0210e+03],
       [ 8.8500e+01,  2.0220e+03,  1.4500e-02,  2.0140e+03,  9.2200e+01,
         2.0210e+03],
       [ 8.6700e+01,  2.0210e+03,  1.8965e+00,  2.0170e+03,  9.1150e+01,
         2.0210e+03],
       [ 8.6750e+01,  2.0230e+03,  7.2000e-01,  2.0170e+03,  8.9050e+01,
         2.0160e+03],
       [ 8.2050e+01,  2.0200e+03,  8.7900e-01,  2.0220e+03,  8.5000e+01,
         2.0200e+03],
       [ 7.9650e+01,  1.9940e+03,  4.8100e-01,  2.0230e+03,  8.4550e+01,
         2.0160e+03]])
Coordinates:
  * month     (month) int64 96B 1 2 3 4 5 6 7 8 9 10 11 12
  * variable  (variable) object 48B 'Air Temperature' ... 'Water Temperature ...

xarray.DataArray

'Record High'

• month: 12
• variable: 6

• 78.05 2.022e+03 0.015 2.02e+03 ... 0.481 2.023e+03 84.55 2.016e+03

  array([[ 7.8050e+01,  2.0220e+03,  1.5000e-02,  2.0200e+03,  8.1700e+01,
           2.0170e+03],
         [ 7.8550e+01,  2.0210e+03, -1.5250e-01,  2.0240e+03,  8.1500e+01,
           2.0210e+03],
         [ 8.2850e+01,  2.0030e+03, -5.6000e-02,  2.0190e+03,  8.3200e+01,
           2.0210e+03],
         [ 8.5800e+01,  2.0200e+03, -8.0000e-03,  2.0230e+03,  8.5750e+01,
           2.0200e+03],
         [ 8.7250e+01,  2.0240e+03,  4.3150e-01,  2.0220e+03,  8.8600e+01,
           2.0240e+03],
         [ 8.7650e+01,  2.0090e+03, -4.1000e-02,  2.0230e+03,  9.0400e+01,
           2.0100e+03],
         [ 8.8700e+01,  2.0180e+03, -1.9200e-01,  2.0190e+03,  9.2000e+01,
           2.0210e+03],
         [ 8.8500e+01,  2.0220e+03,  1.4500e-02,  2.0140e+03,  9.2200e+01,
           2.0210e+03],
         [ 8.6700e+01,  2.0210e+03,  1.8965e+00,  2.0170e+03,  9.1150e+01,
           2.0210e+03],
         [ 8.6750e+01,  2.0230e+03,  7.2000e-01,  2.0170e+03,  8.9050e+01,
           2.0160e+03],
         [ 8.2050e+01,  2.0200e+03,  8.7900e-01,  2.0220e+03,  8.5000e+01,
           2.0200e+03],
         [ 7.9650e+01,  1.9940e+03,  4.8100e-01,  2.0230e+03,  8.4550e+01,
           2.0160e+03]])

• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype=object)

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (0)

Here, the rows are months and the columns are the records or corresponding year. Let’s see what the variables are:

monthly_records['Record High'].coords['variable']

<xarray.DataArray 'variable' (variable: 6)> Size: 48B
array(['Air Temperature', 'Air Temperature Year', 'Water Level',
       'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
      dtype=object)
Coordinates:
  * variable  (variable) object 48B 'Air Temperature' ... 'Water Temperature ...

xarray.DataArray

'variable'

• variable: 6

• 'Air Temperature' 'Air Temperature Year' ... 'Water Temperature Year'

  array(['Air Temperature', 'Air Temperature Year', 'Water Level',
         'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
        dtype=object)

• Coordinates: (1)
  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype=object)

• Indexes: (1)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Air Temperature Year', 'Water Level',
           'Water Level Year', 'Water Temperature', 'Water Temperature Year'],
          dtype='object', name='variable'))

• Attributes: (0)

Alternatively, we can select a specific variable and see all of its stats (converting to a dataframe makes it easier to see):

monthly_records.sel(variable='Air Temperature').to_dataframe().drop('variable', axis=1)

	Monthly Average	Record High Monthly Average	Record Low Monthly Average	Average High	Lowest High	Record High	Average Low	Highest Low	Record Low	Years
month
1	68.656295	72.570968	63.038710	76.050000	72.95	78.05	55.550000	63.50	48.30	23.0
2	70.892723	74.900000	65.465517	76.485417	74.20	78.55	59.362500	70.00	47.90	24.0
3	72.272770	77.616667	66.079032	78.490000	74.15	82.85	63.334000	72.00	55.10	25.0
4	75.633064	79.383333	72.813333	80.740000	77.30	85.80	68.398000	72.60	61.20	25.0
5	78.728491	81.930645	76.974138	82.480769	79.35	87.25	73.951923	77.10	67.95	26.0
6	81.538006	83.613333	79.840000	84.754762	82.75	87.65	77.635714	80.75	75.10	21.0
7	82.909982	84.996774	80.980645	85.790385	84.20	88.70	79.019231	82.30	76.10	26.0
8	83.298278	85.906452	81.820968	85.845833	84.05	88.50	79.510417	83.55	77.00	24.0
9	82.093823	83.740000	80.571667	85.166000	83.90	86.70	78.336000	81.60	74.30	25.0
10	79.634756	81.238710	77.551613	83.812500	80.95	86.75	72.883333	77.80	64.60	24.0
11	75.042799	78.623333	71.373333	79.841667	76.90	82.05	65.972917	74.45	57.35	24.0
12	71.425446	76.870968	62.096667	77.470000	72.50	79.65	59.406000	70.50	48.75	25.0

Reorganize

For the sake of convenience later, let’s rearrange these data arrays before saving them. It will be more useful to have records and years as data variables instead of a dimension, but we have to do some renaming in order to pull this off.

First, separate the records and years into smaller xarrays:

day_records = daily_records.sel(variable=[i for i in daily_records.coords['variable'].values if 'Year' not in i])
day_years = daily_records.sel(variable=[i for i in daily_records.coords['variable'].values if 'Year' in i])

mon_records = monthly_records.sel(variable=[i for i in monthly_records.coords['variable'].values if 'Year' not in i])
mon_years = monthly_records.sel(variable=[i for i in monthly_records.coords['variable'].values if 'Year' in i])

Next, add “Year” to all of the variable names and remove it from the coordinate name:

day_years = day_years.rename_vars({i:i+' Year' for i in day_years.data_vars})
day_years.coords['variable'] = [i.removesuffix(' Year') for i in day_years.coords['variable'].values]

mon_years = mon_years.rename_vars({i:i+' Year' for i in mon_years.data_vars})
mon_years.coords['variable'] = [i.removesuffix(' Year') for i in mon_years.coords['variable'].values]

Now we can merge these two xarrays together, rearrange the order of the variables, and drop those that do not contain a year, such as daily average.

daily_records = xr.merge([day_records, day_years])
daily_records = daily_records[[item for items in zip(day_records.data_vars, day_years.data_vars) for item in items]]
daily_records = daily_records.drop_vars([x for x in daily_records.data_vars if daily_records[x].isnull().all()])

monthly_records = xr.merge([mon_records, mon_years])
monthly_records = monthly_records[[item for items in zip(mon_records.data_vars, mon_years.data_vars) for item in items]]
monthly_records = monthly_records.drop_vars([x for x in monthly_records.data_vars if monthly_records[x].isnull().all()])

monthly_records

<xarray.Dataset> Size: 5kB
Dimensions:                           (month: 12, variable: 3)
Coordinates:
  * month                             (month) int64 96B 1 2 3 4 5 ... 9 10 11 12
  * variable                          (variable) object 24B 'Air Temperature'...
Data variables: (12/16)
    Monthly Average                   (month, variable) float64 288B 68.66 .....
    Record High Monthly Average       (month, variable) float64 288B 72.57 .....
    Record High Monthly Average Year  (month, variable) float64 288B 2.013e+0...
    Record Low Monthly Average        (month, variable) float64 288B 63.04 .....
    Record Low Monthly Average Year   (month, variable) float64 288B 2.001e+0...
    Average High                      (month, variable) float64 288B 76.05 .....
    ...                                ...
    Average Low                       (month, variable) float64 288B 55.55 .....
    Highest Low                       (month, variable) float64 288B 63.5 ......
    Highest Low Year                  (month, variable) float64 288B 2.013e+0...
    Record Low                        (month, variable) float64 288B 48.3 ......
    Record Low Year                   (month, variable) float64 288B 1.997e+0...
    Years                             (month, variable) float64 288B 23.0 ......
Attributes: (12/14)
    datum:                         MHHW
    day_threshold:                 2
    dirname:                       virginiaKeyFl
    hr_threshold:                  4
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    ...                            ...
    Air Temperature units:         °F
    Water Level units:             ft
    Water Temperature units:       °F
    Air Temperature data range:    ('1994-02-01', '2025-05-30')
    Water Level data range:        ('1994-02-01', '2025-04-30')
    Water Temperature data range:  ('1994-02-01', '2025-05-30')

xarray.Dataset

• Dimensions:
  • month: 12
  • variable: 3
• Coordinates: (2)
  • month
    (month)
    int64
    1 2 3 4 5 6 7 8 9 10 11 12

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)

• Data variables: (16)
  • Monthly Average
    (month, variable)
    float64
    68.66 -1.124 ... -0.9139 74.04

    array([[68.65629476, -1.12401237, 71.62517419],
           [70.89272269, -1.18149129, 73.14671022],
           [72.27276989, -1.07706229, 75.15274624],
           [75.63306437, -1.02213989, 78.47779399],
           [78.72849056, -0.95087643, 82.13263705],
           [81.53800571, -0.98853085, 85.23606117],
           [82.90998167, -1.02524941, 87.13986958],
           [83.29827764, -0.88312497, 87.34393222],
           [82.09382299, -0.5381523 , 85.7543954 ],
           [79.63475582, -0.36203893, 82.04959733],
           [75.04279933, -0.52922935, 77.38016782],
           [71.4254462 , -0.91394324, 74.03937634]])

  • Record High Monthly Average
    (month, variable)
    float64
    72.57 -0.5275 ... -0.2977 82.55

    array([[ 7.25709677e+01, -5.27451613e-01,  7.88870968e+01],
           [ 7.49000000e+01, -6.95672414e-01,  7.63017857e+01],
           [ 7.76166667e+01, -5.01596774e-01,  7.99854839e+01],
           [ 7.93833333e+01, -5.01650000e-01,  8.33933333e+01],
           [ 8.19306452e+01, -5.12403226e-01,  8.48129032e+01],
           [ 8.36133333e+01, -5.02616667e-01,  8.76333333e+01],
           [ 8.49967742e+01, -6.88129032e-01,  8.94919355e+01],
           [ 8.59064516e+01, -4.23822581e-01,  9.01096774e+01],
           [ 8.37400000e+01, -2.12833333e-02,  8.94866667e+01],
           [ 8.12387097e+01,  1.75935484e-01,  8.59064516e+01],
           [ 7.86233333e+01,  9.05500000e-02,  8.14666667e+01],
           [ 7.68709677e+01, -2.97661290e-01,  8.25467742e+01]])

  • Record High Monthly Average Year
    (month, variable)
    float64
    2.013e+03 2.022e+03 ... 2.016e+03

    array([[2013., 2022., 2017.],
           [2018., 2024., 2021.],
           [2003., 2019., 2003.],
           [2020., 2023., 2020.],
           [2024., 2022., 2024.],
           [2010., 2023., 2010.],
           [2023., 2019., 2023.],
           [2022., 2019., 2021.],
           [2024., 2019., 2021.],
           [2020., 2023., 2021.],
           [2015., 2023., 2016.],
           [2015., 2022., 2016.]])

  • Record Low Monthly Average
    (month, variable)
    float64
    63.04 -1.559 67.51 ... -1.34 68.2

    array([[63.03870968, -1.55930645, 67.50967742],
           [65.46551724, -1.64225   , 67.73928571],
           [66.07903226, -1.47172581, 69.22096774],
           [72.81333333, -1.4852    , 75.08833333],
           [76.97413793, -1.32648276, 78.64032258],
           [79.84      , -1.4131    , 83.25689655],
           [80.98064516, -1.42170968, 84.51451613],
           [81.82096774, -1.27498387, 85.43548387],
           [80.57166667, -1.05141667, 82.73333333],
           [77.5516129 , -0.88608065, 79.56290323],
           [71.37333333, -1.11588333, 74.59      ],
           [62.09666667, -1.34004839, 68.2016129 ]])

  • Record Low Monthly Average Year
    (month, variable)
    float64
    2.001e+03 2.002e+03 ... 2.01e+03

    array([[2001., 2002., 2001.],
           [1996., 1995., 2005.],
           [2010., 1996., 2010.],
           [2004., 1994., 2004.],
           [2007., 1996., 2001.],
           [2014., 1996., 2002.],
           [2013., 1994., 2013.],
           [1994., 1996., 1995.],
           [2001., 1996., 2004.],
           [2000., 1998., 2000.],
           [2012., 1998., 2012.],
           [2010., 2008., 2010.]])

  • Average High
    (month, variable)
    float64
    76.05 -0.7102 ... -0.4773 77.61

    array([[ 7.60500000e+01, -7.10209677e-01,  7.54460000e+01],
           [ 7.64854167e+01, -7.48650000e-01,  7.70181818e+01],
           [ 7.84900000e+01, -6.95586207e-01,  7.92420000e+01],
           [ 8.07400000e+01, -6.88650000e-01,  8.18076923e+01],
           [ 8.24807692e+01, -6.04620690e-01,  8.52923077e+01],
           [ 8.47547619e+01, -6.88844828e-01,  8.79041667e+01],
           [ 8.57903846e+01, -6.96016667e-01,  8.91807692e+01],
           [ 8.58458333e+01, -4.56645161e-01,  8.94750000e+01],
           [ 8.51660000e+01, -1.05483871e-02,  8.80340000e+01],
           [ 8.38125000e+01,  1.30224138e-01,  8.56300000e+01],
           [ 7.98416667e+01,  1.90666667e-02,  8.06300000e+01],
           [ 7.74700000e+01, -4.77350000e-01,  7.76060000e+01]])

  • Lowest High
    (month, variable)
    float64
    72.95 -1.329 70.5 ... -1.028 72.95

    array([[72.95  , -1.329 , 70.5   ],
           [74.2   , -1.414 , 73.7   ],
           [74.15  , -1.148 , 73.3   ],
           [77.3   , -1.186 , 78.45  ],
           [79.35  , -1.1025, 82.5   ],
           [82.75  , -1.1155, 85.3   ],
           [84.2   , -1.15  , 86.45  ],
           [84.05  , -0.989 , 87.25  ],
           [83.9   , -0.668 , 85.2   ],
           [80.95  , -0.5265, 82.7   ],
           [76.9   , -0.62  , 76.75  ],
           [72.5   , -1.0285, 72.95  ]])

  • Lowest High Year
    (month, variable)
    float64
    2.011e+03 2.009e+03 ... 2.003e+03

    array([[2011., 2009., 2011.],
           [2000., 1995., 2016.],
           [2010., 2008., 2010.],
           [2004., 2001., 2010.],
           [2007., 2004., 2018.],
           [2014., 2003., 1996.],
           [2012., 1994., 2013.],
           [2003., 2001., 2000.],
           [2000., 1996., 1997.],
           [2010., 1998., 2004.],
           [2012., 1995., 2001.],
           [2010., 2008., 2003.]])

  • Record High
    (month, variable)
    float64
    78.05 0.015 81.7 ... 0.481 84.55

    array([[ 7.8050e+01,  1.5000e-02,  8.1700e+01],
           [ 7.8550e+01, -1.5250e-01,  8.1500e+01],
           [ 8.2850e+01, -5.6000e-02,  8.3200e+01],
           [ 8.5800e+01, -8.0000e-03,  8.5750e+01],
           [ 8.7250e+01,  4.3150e-01,  8.8600e+01],
           [ 8.7650e+01, -4.1000e-02,  9.0400e+01],
           [ 8.8700e+01, -1.9200e-01,  9.2000e+01],
           [ 8.8500e+01,  1.4500e-02,  9.2200e+01],
           [ 8.6700e+01,  1.8965e+00,  9.1150e+01],
           [ 8.6750e+01,  7.2000e-01,  8.9050e+01],
           [ 8.2050e+01,  8.7900e-01,  8.5000e+01],
           [ 7.9650e+01,  4.8100e-01,  8.4550e+01]])

  • Record High Year
    (month, variable)
    float64
    2.022e+03 2.02e+03 ... 2.016e+03

    array([[2022., 2020., 2017.],
           [2021., 2024., 2021.],
           [2003., 2019., 2021.],
           [2020., 2023., 2020.],
           [2024., 2022., 2024.],
           [2009., 2023., 2010.],
           [2018., 2019., 2021.],
           [2022., 2014., 2021.],
           [2021., 2017., 2021.],
           [2023., 2017., 2016.],
           [2020., 2022., 2020.],
           [1994., 2023., 2016.]])

  • Average Low
    (month, variable)
    float64
    55.55 -1.507 67.71 ... -1.298 70.24

    array([[55.55      , -1.5068871 , 67.71      ],
           [59.3625    , -1.5467    , 69.18863636],
           [63.334     , -1.44827586, 71.098     ],
           [68.398     , -1.3832    , 74.68461538],
           [73.95192308, -1.26675862, 78.20192308],
           [77.63571429, -1.29075862, 82.23958333],
           [79.01923077, -1.36201667, 84.58076923],
           [79.51041667, -1.25251613, 84.41346154],
           [78.336     , -0.97132258, 83.102     ],
           [72.88333333, -0.82972414, 77.872     ],
           [65.97291667, -0.99663333, 74.172     ],
           [59.406     , -1.29768333, 70.244     ]])

  • Highest Low
    (month, variable)
    float64
    63.5 -0.797 75.9 ... -0.6525 79.65

    array([[63.5   , -0.797 , 75.9   ],
           [70.    , -1.127 , 73.4   ],
           [72.    , -0.955 , 75.4   ],
           [72.6   , -0.991 , 81.05  ],
           [77.1   , -0.7855, 80.35  ],
           [80.75  , -0.807 , 85.2   ],
           [82.3   , -1.091 , 87.15  ],
           [83.55  , -0.761 , 87.25  ],
           [81.6   , -0.5235, 87.1   ],
           [77.8   , -0.102 , 83.4   ],
           [74.45  , -0.2755, 80.05  ],
           [70.5   , -0.6525, 79.65  ]])

  • Highest Low Year
    (month, variable)
    float64
    2.013e+03 2.022e+03 ... 2.016e+03

    array([[2013., 2022., 2017.],
           [2018., 2019., 2023.],
           [1997., 2019., 1997.],
           [2015., 2024., 2020.],
           [2003., 2024., 1994.],
           [2004., 2024., 2004.],
           [2022., 2019., 2023.],
           [2022., 2019., 2021.],
           [2024., 2008., 2021.],
           [2020., 2023., 2021.],
           [2020., 2022., 2016.],
           [2015., 2022., 2016.]])

  • Record Low
    (month, variable)
    float64
    48.3 -2.016 63.95 ... -1.711 63.7

    array([[48.3   , -2.016 , 63.95  ],
           [47.9   , -2.0225, 63.15  ],
           [55.1   , -2.0045, 64.85  ],
           [61.2   , -1.9   , 69.4   ],
           [67.95  , -1.6075, 74.1   ],
           [75.1   , -1.831 , 79.95  ],
           [76.1   , -1.816 , 80.95  ],
           [77.    , -1.6095, 81.05  ],
           [74.3   , -1.3995, 79.5   ],
           [64.6   , -1.5915, 73.2   ],
           [57.35  , -1.583 , 70.8   ],
           [48.75  , -1.711 , 63.7   ]])

  • Record Low Year
    (month, variable)
    float64
    1.997e+03 2.001e+03 ... 2.01e+03

    array([[1997., 2001., 2003.],
           [1996., 2008., 2005.],
           [1996., 1996., 2010.],
           [2009., 1994., 2003.],
           [1999., 2011., 2001.],
           [1995., 2006., 2014.],
           [2013., 1994., 2013.],
           [1994., 1996., 2012.],
           [2001., 2004., 2004.],
           [2005., 2010., 2005.],
           [2006., 1998., 2000.],
           [2010., 2004., 2010.]])

  • Years
    (month, variable)
    float64
    23.0 31.0 25.0 ... 25.0 30.0 25.0

    array([[23., 31., 25.],
           [24., 30., 22.],
           [25., 29., 25.],
           [25., 30., 26.],
           [26., 29., 26.],
           [21., 29., 24.],
           [26., 30., 26.],
           [24., 31., 26.],
           [25., 31., 25.],
           [24., 29., 25.],
           [24., 30., 25.],
           [25., 30., 25.]])

• Indexes: (2)
  • month
    PandasIndex

    PandasIndex(Int64Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], dtype='int64', name='month'))

  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Level', 'Water Temperature'], dtype='object', name='variable'))

• Attributes: (14)

  datum :

  MHHW

  day_threshold :

  2

  dirname :

  virginiaKeyFl

  hr_threshold :

  4

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  °F

  Water Level units :

  ft

  Water Temperature units :

  °F

  Air Temperature data range :

  ('1994-02-01', '2025-05-30')

  Water Level data range :

  ('1994-02-01', '2025-04-30')

  Water Temperature data range :

  ('1994-02-01', '2025-05-30')

Finally, let’s convert years to integers since we do not need decimal years.

daily_records[[i for i in daily_records.data_vars if "Year" in i]] = \
    daily_records[[i for i in daily_records.data_vars if "Year" in i]].astype(int)

monthly_records[[i for i in monthly_records.data_vars if "Year" in i]] = \
    monthly_records[[i for i in monthly_records.data_vars if "Year" in i]].astype(int)

‘yearday’ is not intuitive, so we can change it to calendar day instead and rename the coordinate. Similarly, we can use month names instead of numbers for the sake of clarity.

daily_records.coords['yearday'] = pd.date_range(start='2020-01-01', end='2020-12-31', freq='1D').strftime('%d-%b')
daily_records = daily_records.rename({'yearday':'Date'})

monthly_records.coords['month'] = pd.date_range(start='2020-01-01', end='2020-12-31', freq='1m').strftime('%b')
monthly_records = monthly_records.rename({'month': 'Month'})

Now take a look at the final products

daily_records

<xarray.Dataset> Size: 143kB
Dimensions:                         (Date: 366, variable: 3)
Coordinates:
  * variable                        (variable) object 24B 'Air Temperature' ....
  * Date                            (Date) object 3kB '01-Jan' ... '31-Dec'
Data variables: (12/16)
    Daily Average                   (Date, variable) float64 9kB 72.06 ... 72.48
    Record High Daily Average       (Date, variable) float64 9kB 78.7 ... 80.5
    Record High Daily Average Year  (Date, variable) int64 9kB 2016 ... 2021
    Record Low Daily Average        (Date, variable) float64 9kB 54.4 ... 66.1
    Record Low Daily Average Year   (Date, variable) int64 9kB 2001 ... 2010
    Average High                    (Date, variable) float64 9kB 75.49 ... 73.71
    ...                              ...
    Average Low                     (Date, variable) float64 9kB 68.64 ... 71.26
    Highest Low                     (Date, variable) float64 9kB 77.2 ... 79.3
    Highest Low Year                (Date, variable) int64 9kB 2016 ... 2021
    Record Low                      (Date, variable) float64 9kB 45.5 ... 64.4
    Record Low Year                 (Date, variable) int64 9kB 2001 ... 2010
    Years                           (Date, variable) int64 9kB 27 31 ... 31 27
Attributes: (12/14)
    datum:                         MHHW
    day_threshold:                 2
    dirname:                       virginiaKeyFl
    hr_threshold:                  4
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    ...                            ...
    Air Temperature units:         °F
    Water Level units:             ft
    Water Temperature units:       °F
    Air Temperature data range:    ('1994-01-29', '2025-05-30')
    Water Level data range:        ('1994-01-29', '2025-05-29')
    Water Temperature data range:  ('1994-01-29', '2025-05-30')

xarray.Dataset

• Dimensions:
  • Date: 366
  • variable: 3
• Coordinates: (2)
  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)

  • Date
    (Date)
    object
    '01-Jan' '02-Jan' ... '31-Dec'

    array(['01-Jan', '02-Jan', '03-Jan', ..., '29-Dec', '30-Dec', '31-Dec'],
          dtype=object)

• Data variables: (16)
  • Daily Average
    (Date, variable)
    float64
    72.06 -1.108 72.78 ... -1.052 72.48

    array([[72.06481481, -1.10846774, 72.78392857],
           [72.12222222, -1.13256452, 73.23214286],
           [70.36296296, -1.11587097, 73.35      ],
           ...,
           [70.63076923, -0.97168333, 72.45      ],
           [69.73653846, -1.03998387, 72.47777778],
           [70.84038462, -1.05217742, 72.48333333]])

  • Record High Daily Average
    (Date, variable)
    float64
    78.7 -0.6695 80.4 ... -0.6525 80.5

    array([[78.7   , -0.6695, 80.4   ],
           [77.8   , -0.6775, 80.25  ],
           [78.    , -0.3575, 80.7   ],
           ...,
           [79.45  , -0.39  , 82.6   ],
           [79.5   , -0.44  , 81.95  ],
           [79.2   , -0.6525, 80.5   ]])

  • Record High Daily Average Year
    (Date, variable)
    int64
    2016 2022 2022 ... 2015 2022 2021

    array([[2016, 2022, 2022],
           [2022, 2022, 2022],
           [2015, 2018, 2017],
           ...,
           [2015, 1994, 2016],
           [2015, 1994, 2016],
           [2015, 2022, 2021]])

  • Record Low Daily Average
    (Date, variable)
    float64
    54.4 -1.776 66.45 ... -1.577 66.1

    array([[54.4   , -1.7765, 66.45  ],
           [56.65  , -1.578 , 66.85  ],
           [51.7   , -1.7915, 67.3   ],
           ...,
           [58.7   , -1.6585, 66.55  ],
           [54.75  , -1.537 , 66.4   ],
           [49.9   , -1.5765, 66.1   ]])

  • Record Low Daily Average Year
    (Date, variable)
    int64
    2001 1998 2011 ... 2000 1997 2010

    array([[2001, 1998, 2011],
           [2010, 2011, 2011],
           [2012, 2011, 2011],
           ...,
           [2009, 2008, 1995],
           [2000, 1996, 2010],
           [2000, 1997, 2010]])

  • Average High
    (Date, variable)
    float64
    75.49 0.04435 ... 0.07687 73.71

    array([[7.54925926e+01, 4.43548387e-02, 7.40178571e+01],
           [7.53962963e+01, 3.09677419e-03, 7.44607143e+01],
           [7.44370370e+01, 1.67419355e-02, 7.46000000e+01],
           ...,
           [7.44307692e+01, 1.53700000e-01, 7.38444444e+01],
           [7.33230769e+01, 9.42903226e-02, 7.37962963e+01],
           [7.44500000e+01, 7.68709677e-02, 7.37074074e+01]])

  • Lowest High
    (Date, variable)
    float64
    63.3 -0.646 67.8 ... -0.607 67.8

    array([[63.3  , -0.646, 67.8  ],
           [64.2  , -0.643, 68.   ],
           [57.   , -0.778, 68.9  ],
           ...,
           [67.1  , -0.669, 69.4  ],
           [59.5  , -0.61 , 68.4  ],
           [55.6  , -0.607, 67.8  ]])

  • Lowest High Year
    (Date, variable)
    int64
    2001 2004 2011 ... 2000 1996 2010

    array([[2001, 2004, 2011],
           [2010, 2012, 2011],
           [2012, 2012, 2011],
           ...,
           [2010, 2008, 2003],
           [2000, 1996, 2010],
           [2000, 1996, 2010]])

  • Record High
    (Date, variable)
    float64
    80.8 0.81 81.1 ... 80.1 0.725 81.7

    array([[80.8  ,  0.81 , 81.1  ],
           [79.7  ,  0.902, 81.   ],
           [79.2  ,  1.198, 81.5  ],
           ...,
           [80.6  ,  0.906, 83.3  ],
           [80.6  ,  0.915, 83.1  ],
           [80.1  ,  0.725, 81.7  ]])

  • Record High Year
    (Date, variable)
    int64
    2025 2022 2022 ... 2015 1994 2021

    array([[2025, 2022, 2022],
           [2016, 2018, 2022],
           [2019, 2018, 2017],
           ...,
           [2015, 1994, 2016],
           [2013, 1994, 2016],
           [2015, 1994, 2021]])

  • Average Low
    (Date, variable)
    float64
    68.64 -2.261 71.55 ... -2.181 71.26

    array([[68.63703704, -2.26129032, 71.55      ],
           [68.84814815, -2.26822581, 72.00357143],
           [66.28888889, -2.24848387, 72.1       ],
           ...,
           [66.83076923, -2.09706667, 71.05555556],
           [66.15      , -2.17425806, 71.15925926],
           [67.23076923, -2.18122581, 71.25925926]])

  • Highest Low
    (Date, variable)
    float64
    77.2 -1.791 79.7 ... -1.677 79.3

    array([[77.2  , -1.791, 79.7  ],
           [76.3  , -1.516, 79.5  ],
           [77.   , -1.339, 79.9  ],
           ...,
           [78.3  , -1.594, 81.9  ],
           [78.4  , -1.519, 80.8  ],
           [78.3  , -1.677, 79.3  ]])

  • Highest Low Year
    (Date, variable)
    int64
    2016 2024 2022 ... 2015 2019 2021

    array([[2016, 2024, 2022],
           [2017, 2024, 2022],
           [2015, 2024, 2017],
           ...,
           [2015, 2019, 2016],
           [2015, 2019, 2016],
           [2015, 2019, 2021]])

  • Record Low
    (Date, variable)
    float64
    45.5 -3.159 63.9 ... -2.822 64.4

    array([[45.5  , -3.159, 63.9  ],
           [49.1  , -2.835, 64.4  ],
           [46.4  , -3.035, 65.5  ],
           ...,
           [50.1  , -2.648, 62.2  ],
           [50.   , -2.766, 64.4  ],
           [44.2  , -2.822, 64.4  ]])

  • Record Low Year
    (Date, variable)
    int64
    2001 1998 2001 ... 2000 1997 2010

    array([[2001, 1998, 2001],
           [2010, 2006, 2001],
           [2012, 2011, 2001],
           ...,
           [2009, 2005, 2010],
           [2000, 2005, 2010],
           [2000, 1997, 2010]])

  • Years
    (Date, variable)
    int64
    27 31 28 27 31 ... 31 27 26 31 27

    array([[27, 31, 28],
           [27, 31, 28],
           [27, 31, 28],
           ...,
           [26, 30, 27],
           [26, 31, 27],
           [26, 31, 27]])

• Indexes: (2)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Level', 'Water Temperature'], dtype='object', name='variable'))

  • Date
    PandasIndex

    PandasIndex(Index(['01-Jan', '02-Jan', '03-Jan', '04-Jan', '05-Jan', '06-Jan', '07-Jan',
           '08-Jan', '09-Jan', '10-Jan',
           ...
           '22-Dec', '23-Dec', '24-Dec', '25-Dec', '26-Dec', '27-Dec', '28-Dec',
           '29-Dec', '30-Dec', '31-Dec'],
          dtype='object', name='Date', length=366))

• Attributes: (14)

  datum :

  MHHW

  day_threshold :

  2

  dirname :

  virginiaKeyFl

  hr_threshold :

  4

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  °F

  Water Level units :

  ft

  Water Temperature units :

  °F

  Air Temperature data range :

  ('1994-01-29', '2025-05-30')

  Water Level data range :

  ('1994-01-29', '2025-05-29')

  Water Temperature data range :

  ('1994-01-29', '2025-05-30')

monthly_records

<xarray.Dataset> Size: 5kB
Dimensions:                           (Month: 12, variable: 3)
Coordinates:
  * variable                          (variable) object 24B 'Air Temperature'...
  * Month                             (Month) object 96B 'Jan' 'Feb' ... 'Dec'
Data variables: (12/16)
    Monthly Average                   (Month, variable) float64 288B 68.66 .....
    Record High Monthly Average       (Month, variable) float64 288B 72.57 .....
    Record High Monthly Average Year  (Month, variable) int64 288B 2013 ... 2016
    Record Low Monthly Average        (Month, variable) float64 288B 63.04 .....
    Record Low Monthly Average Year   (Month, variable) int64 288B 2001 ... 2010
    Average High                      (Month, variable) float64 288B 76.05 .....
    ...                                ...
    Average Low                       (Month, variable) float64 288B 55.55 .....
    Highest Low                       (Month, variable) float64 288B 63.5 ......
    Highest Low Year                  (Month, variable) int64 288B 2013 ... 2016
    Record Low                        (Month, variable) float64 288B 48.3 ......
    Record Low Year                   (Month, variable) int64 288B 1997 ... 2010
    Years                             (Month, variable) int64 288B 23 31 ... 25
Attributes: (12/14)
    datum:                         MHHW
    day_threshold:                 2
    dirname:                       virginiaKeyFl
    hr_threshold:                  4
    stationid:                     8723214
    stationname:                   Virginia Key, FL
    ...                            ...
    Air Temperature units:         °F
    Water Level units:             ft
    Water Temperature units:       °F
    Air Temperature data range:    ('1994-02-01', '2025-05-30')
    Water Level data range:        ('1994-02-01', '2025-04-30')
    Water Temperature data range:  ('1994-02-01', '2025-05-30')

xarray.Dataset

• Dimensions:
  • Month: 12
  • variable: 3
• Coordinates: (2)
  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)

  • Month
    (Month)
    object
    'Jan' 'Feb' 'Mar' ... 'Nov' 'Dec'

    array(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct',
           'Nov', 'Dec'], dtype=object)

• Data variables: (16)
  • Monthly Average
    (Month, variable)
    float64
    68.66 -1.124 ... -0.9139 74.04

    array([[68.65629476, -1.12401237, 71.62517419],
           [70.89272269, -1.18149129, 73.14671022],
           [72.27276989, -1.07706229, 75.15274624],
           [75.63306437, -1.02213989, 78.47779399],
           [78.72849056, -0.95087643, 82.13263705],
           [81.53800571, -0.98853085, 85.23606117],
           [82.90998167, -1.02524941, 87.13986958],
           [83.29827764, -0.88312497, 87.34393222],
           [82.09382299, -0.5381523 , 85.7543954 ],
           [79.63475582, -0.36203893, 82.04959733],
           [75.04279933, -0.52922935, 77.38016782],
           [71.4254462 , -0.91394324, 74.03937634]])

  • Record High Monthly Average
    (Month, variable)
    float64
    72.57 -0.5275 ... -0.2977 82.55

    array([[ 7.25709677e+01, -5.27451613e-01,  7.88870968e+01],
           [ 7.49000000e+01, -6.95672414e-01,  7.63017857e+01],
           [ 7.76166667e+01, -5.01596774e-01,  7.99854839e+01],
           [ 7.93833333e+01, -5.01650000e-01,  8.33933333e+01],
           [ 8.19306452e+01, -5.12403226e-01,  8.48129032e+01],
           [ 8.36133333e+01, -5.02616667e-01,  8.76333333e+01],
           [ 8.49967742e+01, -6.88129032e-01,  8.94919355e+01],
           [ 8.59064516e+01, -4.23822581e-01,  9.01096774e+01],
           [ 8.37400000e+01, -2.12833333e-02,  8.94866667e+01],
           [ 8.12387097e+01,  1.75935484e-01,  8.59064516e+01],
           [ 7.86233333e+01,  9.05500000e-02,  8.14666667e+01],
           [ 7.68709677e+01, -2.97661290e-01,  8.25467742e+01]])

  • Record High Monthly Average Year
    (Month, variable)
    int64
    2013 2022 2017 ... 2015 2022 2016

    array([[2013, 2022, 2017],
           [2018, 2024, 2021],
           [2003, 2019, 2003],
           [2020, 2023, 2020],
           [2024, 2022, 2024],
           [2010, 2023, 2010],
           [2023, 2019, 2023],
           [2022, 2019, 2021],
           [2024, 2019, 2021],
           [2020, 2023, 2021],
           [2015, 2023, 2016],
           [2015, 2022, 2016]])

  • Record Low Monthly Average
    (Month, variable)
    float64
    63.04 -1.559 67.51 ... -1.34 68.2

    array([[63.03870968, -1.55930645, 67.50967742],
           [65.46551724, -1.64225   , 67.73928571],
           [66.07903226, -1.47172581, 69.22096774],
           [72.81333333, -1.4852    , 75.08833333],
           [76.97413793, -1.32648276, 78.64032258],
           [79.84      , -1.4131    , 83.25689655],
           [80.98064516, -1.42170968, 84.51451613],
           [81.82096774, -1.27498387, 85.43548387],
           [80.57166667, -1.05141667, 82.73333333],
           [77.5516129 , -0.88608065, 79.56290323],
           [71.37333333, -1.11588333, 74.59      ],
           [62.09666667, -1.34004839, 68.2016129 ]])

  • Record Low Monthly Average Year
    (Month, variable)
    int64
    2001 2002 2001 ... 2010 2008 2010

    array([[2001, 2002, 2001],
           [1996, 1995, 2005],
           [2010, 1996, 2010],
           [2004, 1994, 2004],
           [2007, 1996, 2001],
           [2014, 1996, 2002],
           [2013, 1994, 2013],
           [1994, 1996, 1995],
           [2001, 1996, 2004],
           [2000, 1998, 2000],
           [2012, 1998, 2012],
           [2010, 2008, 2010]])

  • Average High
    (Month, variable)
    float64
    76.05 -0.7102 ... -0.4773 77.61

    array([[ 7.60500000e+01, -7.10209677e-01,  7.54460000e+01],
           [ 7.64854167e+01, -7.48650000e-01,  7.70181818e+01],
           [ 7.84900000e+01, -6.95586207e-01,  7.92420000e+01],
           [ 8.07400000e+01, -6.88650000e-01,  8.18076923e+01],
           [ 8.24807692e+01, -6.04620690e-01,  8.52923077e+01],
           [ 8.47547619e+01, -6.88844828e-01,  8.79041667e+01],
           [ 8.57903846e+01, -6.96016667e-01,  8.91807692e+01],
           [ 8.58458333e+01, -4.56645161e-01,  8.94750000e+01],
           [ 8.51660000e+01, -1.05483871e-02,  8.80340000e+01],
           [ 8.38125000e+01,  1.30224138e-01,  8.56300000e+01],
           [ 7.98416667e+01,  1.90666667e-02,  8.06300000e+01],
           [ 7.74700000e+01, -4.77350000e-01,  7.76060000e+01]])

  • Lowest High
    (Month, variable)
    float64
    72.95 -1.329 70.5 ... -1.028 72.95

    array([[72.95  , -1.329 , 70.5   ],
           [74.2   , -1.414 , 73.7   ],
           [74.15  , -1.148 , 73.3   ],
           [77.3   , -1.186 , 78.45  ],
           [79.35  , -1.1025, 82.5   ],
           [82.75  , -1.1155, 85.3   ],
           [84.2   , -1.15  , 86.45  ],
           [84.05  , -0.989 , 87.25  ],
           [83.9   , -0.668 , 85.2   ],
           [80.95  , -0.5265, 82.7   ],
           [76.9   , -0.62  , 76.75  ],
           [72.5   , -1.0285, 72.95  ]])

  • Lowest High Year
    (Month, variable)
    int64
    2011 2009 2011 ... 2010 2008 2003

    array([[2011, 2009, 2011],
           [2000, 1995, 2016],
           [2010, 2008, 2010],
           [2004, 2001, 2010],
           [2007, 2004, 2018],
           [2014, 2003, 1996],
           [2012, 1994, 2013],
           [2003, 2001, 2000],
           [2000, 1996, 1997],
           [2010, 1998, 2004],
           [2012, 1995, 2001],
           [2010, 2008, 2003]])

  • Record High
    (Month, variable)
    float64
    78.05 0.015 81.7 ... 0.481 84.55

    array([[ 7.8050e+01,  1.5000e-02,  8.1700e+01],
           [ 7.8550e+01, -1.5250e-01,  8.1500e+01],
           [ 8.2850e+01, -5.6000e-02,  8.3200e+01],
           [ 8.5800e+01, -8.0000e-03,  8.5750e+01],
           [ 8.7250e+01,  4.3150e-01,  8.8600e+01],
           [ 8.7650e+01, -4.1000e-02,  9.0400e+01],
           [ 8.8700e+01, -1.9200e-01,  9.2000e+01],
           [ 8.8500e+01,  1.4500e-02,  9.2200e+01],
           [ 8.6700e+01,  1.8965e+00,  9.1150e+01],
           [ 8.6750e+01,  7.2000e-01,  8.9050e+01],
           [ 8.2050e+01,  8.7900e-01,  8.5000e+01],
           [ 7.9650e+01,  4.8100e-01,  8.4550e+01]])

  • Record High Year
    (Month, variable)
    int64
    2022 2020 2017 ... 1994 2023 2016

    array([[2022, 2020, 2017],
           [2021, 2024, 2021],
           [2003, 2019, 2021],
           [2020, 2023, 2020],
           [2024, 2022, 2024],
           [2009, 2023, 2010],
           [2018, 2019, 2021],
           [2022, 2014, 2021],
           [2021, 2017, 2021],
           [2023, 2017, 2016],
           [2020, 2022, 2020],
           [1994, 2023, 2016]])

  • Average Low
    (Month, variable)
    float64
    55.55 -1.507 67.71 ... -1.298 70.24

    array([[55.55      , -1.5068871 , 67.71      ],
           [59.3625    , -1.5467    , 69.18863636],
           [63.334     , -1.44827586, 71.098     ],
           [68.398     , -1.3832    , 74.68461538],
           [73.95192308, -1.26675862, 78.20192308],
           [77.63571429, -1.29075862, 82.23958333],
           [79.01923077, -1.36201667, 84.58076923],
           [79.51041667, -1.25251613, 84.41346154],
           [78.336     , -0.97132258, 83.102     ],
           [72.88333333, -0.82972414, 77.872     ],
           [65.97291667, -0.99663333, 74.172     ],
           [59.406     , -1.29768333, 70.244     ]])

  • Highest Low
    (Month, variable)
    float64
    63.5 -0.797 75.9 ... -0.6525 79.65

    array([[63.5   , -0.797 , 75.9   ],
           [70.    , -1.127 , 73.4   ],
           [72.    , -0.955 , 75.4   ],
           [72.6   , -0.991 , 81.05  ],
           [77.1   , -0.7855, 80.35  ],
           [80.75  , -0.807 , 85.2   ],
           [82.3   , -1.091 , 87.15  ],
           [83.55  , -0.761 , 87.25  ],
           [81.6   , -0.5235, 87.1   ],
           [77.8   , -0.102 , 83.4   ],
           [74.45  , -0.2755, 80.05  ],
           [70.5   , -0.6525, 79.65  ]])

  • Highest Low Year
    (Month, variable)
    int64
    2013 2022 2017 ... 2015 2022 2016

    array([[2013, 2022, 2017],
           [2018, 2019, 2023],
           [1997, 2019, 1997],
           [2015, 2024, 2020],
           [2003, 2024, 1994],
           [2004, 2024, 2004],
           [2022, 2019, 2023],
           [2022, 2019, 2021],
           [2024, 2008, 2021],
           [2020, 2023, 2021],
           [2020, 2022, 2016],
           [2015, 2022, 2016]])

  • Record Low
    (Month, variable)
    float64
    48.3 -2.016 63.95 ... -1.711 63.7

    array([[48.3   , -2.016 , 63.95  ],
           [47.9   , -2.0225, 63.15  ],
           [55.1   , -2.0045, 64.85  ],
           [61.2   , -1.9   , 69.4   ],
           [67.95  , -1.6075, 74.1   ],
           [75.1   , -1.831 , 79.95  ],
           [76.1   , -1.816 , 80.95  ],
           [77.    , -1.6095, 81.05  ],
           [74.3   , -1.3995, 79.5   ],
           [64.6   , -1.5915, 73.2   ],
           [57.35  , -1.583 , 70.8   ],
           [48.75  , -1.711 , 63.7   ]])

  • Record Low Year
    (Month, variable)
    int64
    1997 2001 2003 ... 2010 2004 2010

    array([[1997, 2001, 2003],
           [1996, 2008, 2005],
           [1996, 1996, 2010],
           [2009, 1994, 2003],
           [1999, 2011, 2001],
           [1995, 2006, 2014],
           [2013, 1994, 2013],
           [1994, 1996, 2012],
           [2001, 2004, 2004],
           [2005, 2010, 2005],
           [2006, 1998, 2000],
           [2010, 2004, 2010]])

  • Years
    (Month, variable)
    int64
    23 31 25 24 30 ... 30 25 25 30 25

    array([[23, 31, 25],
           [24, 30, 22],
           [25, 29, 25],
           [25, 30, 26],
           [26, 29, 26],
           [21, 29, 24],
           [26, 30, 26],
           [24, 31, 26],
           [25, 31, 25],
           [24, 29, 25],
           [24, 30, 25],
           [25, 30, 25]])

• Indexes: (2)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Level', 'Water Temperature'], dtype='object', name='variable'))

  • Month
    PandasIndex

    PandasIndex(Index(['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct',
           'Nov', 'Dec'],
          dtype='object', name='Month'))

• Attributes: (14)

  datum :

  MHHW

  day_threshold :

  2

  dirname :

  virginiaKeyFl

  hr_threshold :

  4

  stationid :

  8723214

  stationname :

  Virginia Key, FL

  tz :

  lst

  unit_system :

  english

  Air Temperature units :

  °F

  Water Level units :

  ft

  Water Temperature units :

  °F

  Air Temperature data range :

  ('1994-02-01', '2025-05-30')

  Water Level data range :

  ('1994-02-01', '2025-04-30')

  Water Temperature data range :

  ('1994-02-01', '2025-05-30')

monthly_records.coords['variable']

<xarray.DataArray 'variable' (variable: 3)> Size: 24B
array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)
Coordinates:
  * variable  (variable) object 24B 'Air Temperature' ... 'Water Temperature'

xarray.DataArray

'variable'

• variable: 3

• 'Air Temperature' 'Water Level' 'Water Temperature'

  array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)

• Coordinates: (1)
  • variable
    (variable)
    object
    'Air Temperature' ... 'Water Tem...

    array(['Air Temperature', 'Water Level', 'Water Temperature'], dtype=object)

• Indexes: (1)
  • variable
    PandasIndex

    PandasIndex(Index(['Air Temperature', 'Water Level', 'Water Temperature'], dtype='object', name='variable'))

• Attributes: (0)

We can still choose one environmental variable at a time, but now we get all of the records and corresponding years:

monthly_records.sel(variable='Air Temperature').to_dataframe().drop('variable', axis=1)

	Monthly Average	Record High Monthly Average	Record High Monthly Average Year	Record Low Monthly Average	Record Low Monthly Average Year	Average High	Lowest High	Lowest High Year	Record High	Record High Year	Average Low	Highest Low	Highest Low Year	Record Low	Record Low Year	Years
Month
Jan	68.656295	72.570968	2013	63.038710	2001	76.050000	72.95	2011	78.05	2022	55.550000	63.50	2013	48.30	1997	23
Feb	70.892723	74.900000	2018	65.465517	1996	76.485417	74.20	2000	78.55	2021	59.362500	70.00	2018	47.90	1996	24
Mar	72.272770	77.616667	2003	66.079032	2010	78.490000	74.15	2010	82.85	2003	63.334000	72.00	1997	55.10	1996	25
Apr	75.633064	79.383333	2020	72.813333	2004	80.740000	77.30	2004	85.80	2020	68.398000	72.60	2015	61.20	2009	25
May	78.728491	81.930645	2024	76.974138	2007	82.480769	79.35	2007	87.25	2024	73.951923	77.10	2003	67.95	1999	26
Jun	81.538006	83.613333	2010	79.840000	2014	84.754762	82.75	2014	87.65	2009	77.635714	80.75	2004	75.10	1995	21
Jul	82.909982	84.996774	2023	80.980645	2013	85.790385	84.20	2012	88.70	2018	79.019231	82.30	2022	76.10	2013	26
Aug	83.298278	85.906452	2022	81.820968	1994	85.845833	84.05	2003	88.50	2022	79.510417	83.55	2022	77.00	1994	24
Sep	82.093823	83.740000	2024	80.571667	2001	85.166000	83.90	2000	86.70	2021	78.336000	81.60	2024	74.30	2001	25
Oct	79.634756	81.238710	2020	77.551613	2000	83.812500	80.95	2010	86.75	2023	72.883333	77.80	2020	64.60	2005	24
Nov	75.042799	78.623333	2015	71.373333	2012	79.841667	76.90	2012	82.05	2020	65.972917	74.45	2020	57.35	2006	24
Dec	71.425446	76.870968	2015	62.096667	2010	77.470000	72.50	2010	79.65	1994	59.406000	70.50	2015	48.75	2010	25

Finally, write these to file for safe keeping.

daily_records.to_netcdf(os.path.join(outdir, 'statistics-daily.nc'), mode='w')
monthly_records.to_netcdf(os.path.join(outdir, 'statistics-monthly.nc'), mode='w')

We will plot these results in Part 3.

---