Solar-Induced Chlorophyll Fluorescence (SIF) can provide key information about the state of photosynthesis and offers the prospect of defining remote sensing-based estimation of Gross Primary Production (GPP). There is strong theoretical support for the link between SIF and GPP and this relationship has been empirically demonstrated using ground-based, airborne, and satellite-based SIF observations, as well as modeling. However, most evaluations have been based on monthly and annual scales, yet the GPP:SIF relations can be strongly influenced by both vegetation structure and physiology. At the monthly timescales, the structural response often dominates but short-term physiological variations can strongly impact the GPP:SIF relations. Here, we test how well SIF can predict the inter-daily variation of GPP during the growing season and under stress conditions, while taking into account the local effect of sites and abiotic conditions. We compare the accuracy of GPP predictions from SIF at different timescales (half-hourly, daily, and weekly), while evaluating effect of adding environmental variables to the relationship. We utilize observations for years 2018–2019 at 31 mid-latitudes, forested, eddy covariance (EC) flux sites in North America and Europe and use TROPOMI satellite data for SIF. Our results show that SIF is a good predictor of GPP, when accounting for inter-site variation, probably due to differences in canopy structure. Seasonally averaged leaf area index, fraction of absorbed photosynthetically active radiation (fPAR) and canopy conductance provide a predictor to the site-level effect. We show that fPAR is the main factor driving errors in the linear model at high temporal resolution. Adding water stress indicators, namely canopy conductance, to a multi-linear SIF-based GPP model provides the best improvement in the model precision at the three considered timescales, showing the importance of accounting for water stress in GPP predictions, independent of the SIF signal. SIF is a promising predictor for GPP among other remote sensing variables, but more focus should be placed on including canopy structure, and water stress effects in the relationship, especially when considering intra-seasonal, and inter- and intra-daily resolutions.