RPKI repositories degraded performance
Postmortem

ARIN completed a scheduled, simulated failure of the ARIN RPKI repository today at 1500 EST to assess failure response under production conditions.

At 13:05 EST (UTC-5) , ARIN began restricting RRDP access to the sites that serve the ARIN Resource Public Key Infrastructure (RPKI) Hosted and RPS (hybrid) repositories. At 14:15, all access was administratively blocked, and the repositories was made unavailable. At 15:05, access to the repositories was restored. Full redundancy of services was confirmed at 15:18.

This decision to conduct the test in ARIN’s production environment was carefully considered. Testing outside of production would not have provided an accurate representation of performance and preparedness for repository failure conditions. Therefore, ARIN chose to simulate this scenario in a live environment to best evaluate real-world impacts and response.

We appreciate the patience and cooperation of the RPKI community throughout this exercise, and conduct this testing is part of our commitment to high-quality RPKI services.

Regards,

Brad Gorman
Senior Product Owner, Routing Security
American Registry for Internet Numbers (ARIN)

Posted Nov 13, 2024 - 16:54 EST

Resolved
This incident has been resolved.
Posted Nov 13, 2024 - 15:20 EST
Update
We are experiencing an outage of RPKI repositories.
Posted Nov 13, 2024 - 14:27 EST
Investigating
We are experiencing degraded performance on RPKI repositories for RRDP/rsync
Posted Nov 13, 2024 - 14:19 EST
This incident affected: Registry Services (RPKI RRDP Repository, RPKI Rsync Repository).