Python Infrastructure
All Systems Operational
python.org ? Operational
python.org - CDN ? Operational
python.org - Backends ? Operational
PyPI Operational
pypi.org - CDN ? Operational
pypi.org - Backends ? Operational
pypi.org - Email ? Operational
files.pythonhosted.org - Files ? Operational
files.pythonhosted.org - Redirects ? Operational
files.pythonhosted.org - Redirects Backends ? Operational
PyPI Hosting Platforms Operational
AWS elasticache-us-east-2 Operational
AWS elb-us-east-2 Operational
AWS ec2-us-east-2 Operational
AWS rds-us-east-2 Operational
Google Cloud Platform Google Cloud Storage Operational
docs.python.org Operational
docs.python.org - Backends ? Operational
docs.python.org - CDN Operational
bugs.python.org Operational
wiki.python.org Operational
mail.python.org Operational
Message Handling Services Operational
Mailing Lists and Archives - Mailman Operational
Mailing Lists and Archives - Mailman 3 Operational
psfmember.org Operational
us.pycon.org Operational
PyPy Operational
pypy.org Operational
speed.pypy.org Operational
Content Delivery Network ? Operational
Fastly Asia/Pacific (HK) Operational
Fastly US East (IAD) Operational
Fastly US East (MIA) Operational
Fastly US Central (DEN) Operational
Fastly US Central (DFW) Operational
Fastly US West (SEA) Operational
Fastly US West (SJC) Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Fastly Europe (LCY) Operational
Fastly Europe (LHR) Operational
Fastly Asia/Pacific (SYD) Operational
Fastly Asia/Pacific (NZ) Operational
Fastly Brisbane (BNE) Operational
Fastly Dubai (FJR) Operational
Fastly Melbourne (MEL) Operational
Fastly Osaka (ITM) Operational
Fastly Perth (PER) Operational
Fastly Tokyo (HND) Operational
Fastly Tokyo (TYO) Operational
Fastly Wellington (WLG) Operational
Fastly Dublin (DUB) Operational
Fastly Copenhagen (CPH) Operational
Fastly Frankfurt (HHN) Operational
Fastly Helsinki (HEL) Operational
Fastly London (LON) Operational
Fastly Madrid (MAD) Operational
Fastly Manchester (MAN) Operational
Fastly Milan (MXP) Operational
Fastly Oslo (OSL) Operational
Fastly Buenos Aires (EZE) Operational
Fastly Bogota (BOG) Operational
Fastly Curitiba (CWB) Operational
Fastly Rio de Janeiro (GIG) Operational
Fastly Santiago (SCL) Operational
Fastly Johannesburg (JNB) Operational
Fastly Cape Town (CPT) Operational
Fastly Vancouver (YVR) Operational
Fastly Toronto (YYZ) Operational
Fastly St. Louis (STL) Operational
Fastly Palo Alto (PAO) Operational
Fastly Newark (EWR) Operational
Fastly New York (LGA) Operational
Fastly Montreal (YUL) Operational
Fastly Minneapolis (STP) Operational
Fastly Minneapolis (MSP) Operational
Fastly Los Angeles (BUR) Operational
Fastly Kansas City (MCI) Operational
Fastly Houston (IAH) Operational
Fastly Dallas (DAL) Operational
Fastly Columbus (CMH) Operational
Fastly Chicago (CHI) Operational
Fastly Boston (BOS) Operational
Fastly Atlanta (PDK) Operational
Fastly Atlanta (FTY) Operational
Fastly Ashburn (WDC) Operational
Fastly New Delhi (DEL) Operational
Fastly Mumbai (BOM) Operational
Fastly Chennai (MAA) Operational
Fastly Vienna (VIE) Operational
Fastly Stockholm (BMA) Operational
Fastly Paris (CDG) Operational
Fastly Purging Operational
Fastly Tokyo (NRT) Operational
Fastly Singapore (QPG) Operational
Fastly Seoul (ICN) Operational
Fastly Manila (MNL) Operational
Fastly Kuala Lumpur (KUL) Operational
Fastly Dubai (DXB) Operational
Fastly Christchurch (CHC) Operational
Fastly Adelaide (ADL) Operational
Fastly Ghana (ACC) Operational
Fastly Sāo Paulo (CGH) Operational
Fastly Lima (LIM) Operational
Fastly Fortaleza (FOR) Operational
Fastly Seattle (BFI) Operational
Fastly Portland (PDX) Operational
Fastly Phoenix (PHX) Operational
Fastly Los Angeles (LGB - LAX/SNA) Operational
Fastly Honolulu (HNL) Operational
Fastly Gainesville (GNV) Operational
Fastly Detroit (DTW) Operational
Fastly Columbus (LCK) Operational
Fastly Calgary (YYC) Operational
Fastly Atlanta (ATL) Operational
Fastly Sydney (SYD) Operational
Fastly Palo Alto (PAO) Operational
Fastly Amsterdam (AMS) Operational
Fastly Kolkata (CCU) Operational
Fastly Hyderabad (HYD) Operational
Fastly Palermo (PMO) Operational
Fastly Sofia (SOF) Operational
Fastly Rome (FCO) Operational
Fastly Munich (MUC) Operational
Fastly Milan (LIN) Operational
Fastly Marseille (MRS) Operational
Fastly Lisbon (LIS) Operational
Fastly Brussels (BRU) Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
PyPI CDN Edge Errors ?
Fetching
PyPI Files CDN Edge Errors ?
Fetching
PyPI CDN Miss Times
Fetching
PyPI Files CDN Miss Times
Fetching
Past Incidents
Aug 4, 2022

No incidents reported today.

Aug 3, 2022

No incidents reported.

Aug 2, 2022

No incidents reported.

Aug 1, 2022

No incidents reported.

Jul 31, 2022

No incidents reported.

Jul 30, 2022

No incidents reported.

Jul 29, 2022
Resolved - Our CDN hit rate and latencies have recovered, this event is now resolved.
Jul 29, 19:01 UTC
Update - Error rates have returned to baseline, and latencies continue to return to normal across our services. We will continue to monitor as the database upgrades complete and caches re-fill.
Jul 29, 16:30 UTC
Update - Our backends have begun to stabilize as the caches are refilled. During the peak of the backend traffic from our CDN we recognized a few bottlenecks in our database layer and have provisioned additional storage IOPs in order to reduce impact of surges in the future.
Jul 29, 16:03 UTC
Update - Purge All has been issued and we are monitoring as the backends attempt to meet the request volume. We anticipate that the service will continue to stabilize and reach steady state over the next 30-90 minutes.
Jul 29, 15:30 UTC
Monitoring - In order to resolve an issue in our CDN cache caused by a misconfiguration of Surrogate-Keys, PyPI will need to issue a "Purge All" of our CDN to clear out lingering cached objects that are not accessible to purge individually.

This purge is very likely to impact performance when accessing PyPI for some time.

In order to avoid purges like this in the future, our backends have been configured with Surrogate-Keys that will at least allow us to purge specific endpoint types rather than the entire service.

Jul 29, 15:19 UTC
Jul 28, 2022
Postmortem - Read details
Jul 28, 19:10 UTC
Resolved - PyPI experienced an outage in its backends from 18:46-18:53 UTC on 2022-07-28. This outage stemmed from a cascading failure in the backend infrastructure that disrupted the management layer that deploys and secures PyPI's application servers. It is now resolved.
Jul 28, 19:00 UTC
Jul 27, 2022

No incidents reported.

Jul 26, 2022

No incidents reported.

Jul 25, 2022

No incidents reported.

Jul 24, 2022

No incidents reported.

Jul 23, 2022

No incidents reported.

Jul 22, 2022

No incidents reported.

Jul 21, 2022

No incidents reported.