Stimuli-responsive microgels can be used as stabilizers for emulsions. However, the details of structure and the viscoelastic property of the microgel-laden interface are still not well-known. We synthesized fluorescently labeled microgels and used confocal microscopy to observe their arrangement at the water/oil interface. The microgels aggregated spontaneously at the interface, and the aggregated structure reorganized due to thermal motion. The structure of the interfacial layer formed by microgels depended on the microgel concentration at the interface. We suggest that the structure was controlled by the aggregation and adsorption of microgels at the interface. The interparticle separation between microgels at the interface decreased over time, implying a slow aging process of the microgels at the interface. Magnetic beads were introduced at the interface and used to trigger deformation of the microgel layer. Under compression and shear the microgels in the aggregated structure rearranged, leading to plastic deformation, and some elastic responses were also observed.