Document Type : Research Paper
Water management in conveyance and distribution canals is inherently challenging due to nonlinear hydraulic behavior, significant time delays between control actions and system responses, disturbances caused by variable and often sudden farmer off-takes, and uncertainties in hydraulic parameters. Accurate simulation of these systems is a prerequisite for designing intelligent controllers capable of maintaining stable water levels and ensuring fair water allocation. Two main categories of hydraulic simulators exist: fully nonlinear models based on the Saint-Venant equations, which provide high accuracy but require substantial computational resources, and simplified linear models, such as state-space representations, which offer faster computations at the cost of physical fidelity. Discrete reinforcement learning (RL) algorithms, particularly SARSA (State-Action-Reward-State-Action), have emerged as powerful tools for learning optimal control policies through interaction with an environment. However, a critical research gap persists: no prior study has quantitatively evaluated how the choice of hydraulic simulator (nonlinear vs. linear) affects the learning process, convergence speed, and final control policy quality of discrete RL algorithms. This study addresses this gap by providing the first systematic comparison of RL performance integrated with both simulator types.
The primary purpose of this research is to quantitatively compare the performance of discrete reinforcement learning (SARSA) when integrated with two distinct hydraulic simulators: (1) a fully nonlinear model based on the Saint-Venant equations (implemented in the AICSS software), and (2) a linear state-space model identified from input-output data of the nonlinear simulator. The comparison focuses on three key aspects: (a) prediction accuracy (error indices such as IAE and MAE), (b) control performance (efficiency and adequacy indices), and (c) learning speed (convergence rate). The study aims to answer the fundamental question: does the computational simplicity and higher speed of linear models justify their accuracy loss in discrete RL-based control of irrigation canals?
The study was conducted on the ASCE (American Society of Civil Engineers) standardized Test Canal No. 2, which consists of eight consecutive pools, eight vertical sliding gates (with a pump at the final pool), and eight lateral off-takes. This configuration represents a typical irrigation canal with cascaded hydraulic interactions.
The nonlinear hydraulic behavior was simulated using the AICSS (Agricultural Irrigation Conveyance System Simulator) software, which solves the full one-dimensional Saint-Venant equations (continuity and momentum) to model both steady and unsteady flow conditions.
A linear state-space model was derived using a data-driven system identification approach. First, the nonlinear model was excited by applying step changes of +5%, +10%, and +20% to each gate position relative to its reference value. The resulting input (gate positions) and output (water depths) data were recorded. Second, this incremental data was imported into MATLAB's System Identification Toolbox to estimate the state-space matrices with a sampling time of 36 seconds. Validation was performed using Pseudo-Random Binary Sequence (PRBS) signals to ensure generalizability and avoid overfitting.
The SARSA algorithm was integrated with both simulators. The state vector was defined as Si = [Pli, QTi], where Pli is the reach number and QTi is the off-take discharge. Actions corresponded to gate position adjustments. The reward function was designed to penalize deviations from target water depths. Hyperparameters were systematically tuned: initial temperature of 0.0007 for the Softmax action selection, initial learning rate of 0.3 with a decay factor of 1.067, discount factor of 0.9, and 500 learning episodes with 8 independent runs.
Quantitative evaluation used Maximum Absolute Error (MAE), Integral Absolute Error (IAE), Efficiency and Adequacy indices, classified into "Good," "Moderate," and "Poor" categories based on established thresholds.
The results clearly demonstrate the superior performance of the nonlinear simulator across all evaluation metrics.
The nonlinear model achieved remarkably low errors, with MAE and IAE limited to 12.3% and 8.9%, respectively, across all eight reaches. In contrast, the linear model exhibited an exponential error increase from upstream to downstream. In Reach 8, the linear model's MAE reached 59% and IAE reached 51%, both classified as "Poor." The nonlinear model produced errors smaller than the linear model in downstream reaches.
The nonlinear model achieved minimum efficiency of 0.962 and minimum adequacy of 0.972, both in the "Good" category, indicating precise water delivery without significant surplus or deficit. The linear model, while maintaining adequacy (adequacy up to 1.00), suffered a dramatic efficiency drop to 0.497 in Reach 8 ("Poor"), indicating substantial water wastage to compensate for modeling errors. A deep "gap" between efficiency and adequacy was observed for the linear model but not for the nonlinear model.
The linear model converged faster due to its simpler computational structure. However, this speed advantage did not translate into better control policies; instead, the linear model learned skewed policies that favored upstream off-takes while over-delivering to downstream ones.
Averaging Q-tables from 8 independent runs reduced operational phase errors significantly, lowering MAE from 21% (learning phase) to 10.7% (operational phase) for the nonlinear model.
This study provides definitive quantitative evidence that the choice of hydraulic simulator fundamentally determines the success of discrete reinforcement learning for irrigation canal control. The nonlinear Saint-Venant model consistently delivers high accuracy, stable performance, fair water distribution, and balanced efficiency-adequacy trade-offs across all canal reaches. The linear state-space model, despite its faster learning speed and acceptable performance within a ±20% linearization range, suffers from inherent limitations: it cannot simulate key nonlinear phenomena such as flow-dependent energy losses, wave propagation delays, and flow interactions between adjacent pools. These limitations become critically pronounced in downstream reaches, where cumulative errors lead to inefficient control policies, excessive water wastage, and unfair distribution. The findings refute the notion that linear models' computational simplicity can compensate for their accuracy loss in RL-based control. For sustainable, efficient, and equitable water resource management in conveyance and distribution canals, nonlinear simulators are better choice. Future work should focus on developing adaptive, hybrid models that combine physical constraints with data-driven learning to maintain accuracy across wider operating ranges while preserving computational efficiency.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing—original draft preparation, writing—review and editing, visualization, supervision, project administration, funding acquisition; K.Sh. All authors have read and agreed to the published version of the manuscript.
Statement: During the preparation of this work the author(s) used [Deepseek] in order to [wring]. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.
In this section, please provide details regarding where data supporting reported results can be Data available on request from the authors.
The authors avoided data fabrication, falsification, and plagiarism, and any form of misconduct.
The authors would like to thank anonymous referees for their constructive comments.
The authors declare no conflict of interest.
