Just after 1:30pm on October 7th, Pacific Gas and Electric (PG&E) stated that it may cut power to upwards of 600,000 customers in California. Just 24-hours later, PG&E updated the number of potentially affected customers to 800,000 and the panic began.
As PG&E did a massive DDoS on its customers, it's customers did the same to PG&E's website.
Besides launching the Mercury News app and listening in to KGO Newstalk 810, Silicon Valley residents looked to the pge.com website for details... or at least they tried to. Days went by with no access to the site - kind of important if you want to know the status of multi-day power outage, eh?
Let's look at the DNS resolution to PGE's website to see what we can learn.
Trace File #1: DNSPGEresolution.pcapng
We learn at this point that the PGE webserver is located at 107.162.156.137. Everyone going to pge.com hits the same IP address.
[I removed the HTTP redirection to the https://www.pge.com page. It was a standard HTTP 302 Found response pointing us to https://www.pge.com.]
Trace File #2: TLS2GPGE_fails.pcapng
PGE's shutting down the power? Let's shut them down!
This was a very legitimate Distributed Denial of Service attack on the PG&E website.
In the trace file, you will notice the Initial Round Trip Time was approximately 35 ms. It could be worse.
The client makes a TCP connection to the server (TCP stream 0) and then successfully makes a TLS connection to the server. So far, everything seems like it's going great!
Suddenly, the PG&E server sends a TCP reset to the client to shut down the connection. How rude! Our browser (Firefox) immediately tries to reconnect to the server (TCP stream 1) only to go through the same process and get that slap in the face shutting down the TCP connection.
WANT TO LEARN TO ANALYZE TRAFFIC LIKE THIS? We offer on-demand, online and instructor-led courses on Wireshark and TCP/IP communications! Check out the links under "Training" on the menu for more information and sign up for our biweekly newsletter to know when future blogs, events, or freebies are announced. Every new sign up also gets five free Wireshark labs!
My browser, Firefox, tried 10 times and, although I could make TCP and TLS connections to the PG&E server, the TCP connections were shut down by the PG&E server each time.
This is what happened when my browser attempted to reach the server a single time. Upon termination of the connections, the browser immediately reconnected with the server. Firefox did this 10 times. I never changed Firefox settings, so this must be a default behavior.
Multiply this by the population trying to get information on the outage area and length of time and you can imagine how the PG&E server got overloaded.
This would be considered a legitimate Distributed Denial of Service.
There's one more bit of interesting information about PG&E's server.
Later in the evening on October 10th, I captured another connection attempt to the PG&E server. This time, I used a dns contains "pge" || ip.addr==107.0.0.0/8 display filter and saved the (1) DNS queries/responses, (2) the HTTP connection and redirection, and (3) the slooooow, but finally successful load of the PG&E site over an encrypted connection.
Trace File: TLS2PGE_dogslow.pcapng
Check out the Server field in frame 12 of this trace file.
The Server field indicates the response is coming from "F5aas." That stands for "F5-as-a-service." Hmmm... So, it appears that PG&E is using F5 which should provide "availability, reliability, scalability, and performance." F5 has a number of configurations possible to protect hosts. Without capturing on the other side of the F5 host, it's tough to figure out what the configuration issues may be. No matter what the configuration is, it's not working.
Note: Imagine if this configuration was used to provide information needed by the entire country. This DDoS was regional with folks in California hitting the server.
We decided to do a whois query for the 107.162.156.137 address. Here's the result.
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2019, American Registry for Internet Numbers, Ltd.
#
NetRange: 107.162.0.0 - 107.162.255.255
CIDR: 107.162.0.0/16
NetName: DEFENSE-NET
NetHandle: NET-107-162-0-0-1
Parent: NET107 (NET-107-0-0-0-0)
NetType: Direct Assignment
OriginAS: AS55002
Organization: Defense.Net, Inc (DEFEN-12)
RegDate: 2013-12-19
Updated: 2013-12-19
Comment: For more information about Defense.Net please visit our web site: www.defense.net
Ref: https://rdap.arin.net/registry/ip/107.162.0.0
OrgName: Defense.Net, Inc
OrgId: DEFEN-12
Address: 801 5th Ave
City: Seattle
StateProv: WA
PostalCode: 98104
Country: US
RegDate: 2013-02-04
Updated: 2019-09-13
Comment: F5 Silverline
Ref: https://rdap.arin.net/registry/entity/DEFEN-12
OrgAbuseHandle: NETOP225-ARIN
OrgAbuseName: netops
OrgAbusePhone: +1-866-329-4253
OrgAbuseEmail: email@defense.net
OrgAbuseRef: https://rdap.arin.net/registry/entity/NETOP225-ARIN
OrgNOCHandle: NETOP225-ARIN
OrgNOCName: netops
OrgNOCPhone: +1-866-329-4253
OrgNOCEmail: email@defense.net
OrgNOCRef: https://rdap.arin.net/registry/entity/NETOP225-ARIN
OrgTechHandle: NETOP225-ARIN
OrgTechName: netops
OrgTechPhone: +1-866-329-4253
OrgTechEmail: email@defense.net
OrgTechRef: https://rdap.arin.net/registry/entity/NETOP225-ARIN
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2019, American Registry for Internet Numbers, Ltd.
#
Well, hello again F5! [Would love to hear the inside story from the F5 folks working with PG&E. Contact me!]
In TLS2PGE_dogslow.pcapng, we were not stopped in the TCP connection process (they didn't see the intense number of SYN packets and start sending TCP resets). We were not prevented from establishing TLS connections. We were stopped after sending our
application request.
It's obvious that PG&E needs to update their power infrastructure to avoid major outages like this. It also seems that they need to take another look at their communications system. Didn't PG&E realize that such announcement of such a huge planned outage would cause a flood of information requests to their server?
They may be getting security services through their F5 solutions, but I don't see any indication that they are getting properly-configured load balancing services.
Take some time and look through the trace files provided in relationship to this analysis. They are truly fascinating.
Follow Up
On my morning drive to make a "Network Forensics with Wireshark" presentation at the Silicon Valley HTCIA chapter meeting at Netflix HQ in Los Gatos, California, I caught the local news updates on PG&E. PG&E was apologizing for the server outage. They stated that they had yet another website up for people to access -- a bit late since the last of the power shutdowns had occurred. If you had power, you were ok, if you didn't, you were shut down.
At the end of my presentation (where I showed the trace files referenced in this blog), I decided to do a live capture of a browsing session to pge.com. The page loaded quickly, so we all knew something had changed.
Here's the trace file of today's browsing session, TLS2PGE_loadok.pcapng.
Here are some of the differences seen in today's browsing session to pge.com.
DNS A record resolution returns 2 IP addresses.
Second DNS A record resolution altered the IP address order (round robin).
HTTP request received a 301 Moved Permanently response, not a 301 Moved response.
Additional DNS A record queries resolve to a Canonical name including "akadns.net" - see https://developer.akamai.com/api/web_performance/global_traffic_management/v1.html for more information on Global Traffic Management.
Multiple connections were required - our client connected to both of the IP addresses to load the page.
What a difference a couple of days make, eh?