Stray radiation produced by ultra-high dose-rates (UHDR) proton pencil beams is characterized using ASIC-chip semiconductor pixel detectors. A proton pencil beam with an energy of 220 MeV was utilized to deliver dose rates (DR) ranging from conventional radiotherapy DRs up to 270 Gy/s. A MiniPIX Timepix3 detector equipped with a silicon sensor and integrated readout electronics was used. The chip-sensor assembly and chipboard on water-equivalent backing were detached and immersed in the water-phantom. The deposited energy, particle flux, DR, and the linear energy transfer (LET(Si)) spectra were measured in the silicon sensor at different positions both laterally, at different depths, and behind the Bragg peak. At low-intensity beams, the detector is operated in the event-by-event data-driven mode for high-resolution spectral tracking of individual particles. This technique provides precise energy loss response and LET(Si) spectra with radiation field composition resolving power. At higher beam intensities a rescaling of LET(Si) can be performed as the distribution of the LET(Si) spectra exhibits the same characteristics regardless of the delivered DR. The integrated deposited energy and the absorbed dose can be thus measured in a wide range. A linear response of measured absorbed dose was obtained by gradually increasing the delivered DR to reach UHDR beams. Particle tracking of scattered radiation in data-driven mode could be performed at DRs up to 0.27 Gy/s. In integrated mode, the saturation limits were not reached at the measured out-of-field locations up to the delivered DR of over 270 Gy/s. A good agreement was found between measured and simulated absorbed doses.