Black Hat Asia 2023 NOC: Connecting Singapore


On this weblog concerning the design, deployment and automation of the Black Hat Asia community, now we have the next sections:

  • Designing the Black Hat Community
  • AP (Entry Level) Placement Planning, by Uros Mihajlovic
  • Safety Middle Investigations, by Uros Mihajlovic
  • Meraki and ThousandEyes, by Uros Mihajlovic
  • Meraki Dashboards, by Steven Fan
  • Meraki Alerting, by Connor Loughlin
  • Meraki Programs Supervisor, by Paul Fidler
  • Constructing Instruments for Black Hat Workers, by Ryan MacLennan
  • A Higher Solution to Design Coaching SSIDs/VLANs, by Paul Fidler

Cisco is honored to be a Accomplice of the Black Hat NOC (Community Operations Middle), and was the Official Community Tools, Cellular Gadget Administration, Malware Evaluation, and DNS (Area Identify Service) Supplier of Black Hat Asia 2023.

This was Cisco’s seventh 12 months as a NOC accomplice for Black Hat Asia and the second time constructing the community. Beneath are our fellow NOC companions offering {hardware}, contributing to construct and safe the community for our joint buyer: Black Hat.

Designing the Black Hat Community

We used the experiences of Black Hat Asia 2022, Black Hat USA 2022 and Black Hat Europe 2022 to plan the community topology design and tools, with Black Hat, and the NOC companions.

It was a workforce effort to construct an enterprise degree community in 2 ½ days. We recognize the onerous work of the 12 Cisco Meraki and Cisco Safe engineers on web site (plus 4 nearly supporting engineers) to construct, function and safe the community; and nice NOC management and collaborative Companions.

Constructing this community is a problem. On one hand, we should enable actual malware on the Black Hat community for coaching, demonstrations, and briefing periods. On the opposite, we have to shield the attendees from assault inside the community from their fellow attendees and stop unhealthy actors from utilizing the community to assault the Web.

It’s a vital stability to make sure everybody has a secure expertise, whereas nonetheless having the ability to study from actual world malware, vulnerabilities and malicious web sites.

Along with the weekly conferences with Black Hat and the opposite companions, the Cisco Meraki engineering workforce additionally mentioned the challenges in a Webex area, with different engineers who labored on previous Black Hat occasions.

The mission:

  • Deploy 63 (11 spares) Meraki entry factors to supply Wi-Fi to 10 coaching programs, dozens of briefings, keynotes, and the Enterprise Corridor
  • Deploy 63 ten-foot (three meter) tripods and brackets supplied to Black Hat by Cisco Meraki world occasions

Division of labor is important to cut back errors and keep laser targeted on safety scope. Uros ensured each AP and Change was tracked, and the MAC addresses had been supplied to Palo Alto Networks for DCHP assignments. Stephen and Connor spent two days within the server room with the NOC companions, guaranteeing each change was working and configured accurately.

AP Placement Planning, by Uros Mihajlovic

Within the weeks earlier than deployment, Jeffry Handal targeted on planning and making a digital Wi-Fi web site survey. A number of necessities and restrictions needed to be considered. The report was based mostly on the Marina Bay Sands flooring plan and the area allocation necessities from Black Hat. Happily, we had extra APs obtainable to us than required.

Beneath is the Sign Power plan for the 4th flooring of the convention centre on the 5 GHz band.

Utilizing the expertise of Black Hat Asia 2022, discussing the necessities of Black Hat and dealing with the Marina Bay Sands IT, we finalized the AP deployment plan previous to arrival. We additionally grouped entry factors per room, so we might accurately deploy them in related areas. This additionally allowed Marina Bay Sands IT workforce to precisely lay out crucial cabling for the entry factors.

Earlier than the APs had been even on-line, we configured any crucial settings within the Meraki dashboard. This concerned wi-fi radio profiles, SSID configuration, site visitors shaping guidelines, and many others. Along with normal Black Hat SSID for all attendees, we additionally had particular SSIDs that ought to broadcast solely in particular areas. Utilizing Cisco Meraki’s SSID availability characteristic, we might tag entry factors accordingly to their location, which allowed us to broadcast applicable SSIDs.

Because the APs had been pre-staged and added to the Meraki dashboard, together with their location on the ground maps, the principle work was putting and cabling them bodily. Because of good planning, we might begin deploying the 63 APs as quickly because the convention area was obtainable, with solely a small variety of adjustments to optimize the deployment on-site. With a serving to hand from our Cisco Safety colleagues, we swiftly deployed tripods across the venue. As you possibly can see from the picture beneath, this was additionally an ideal workforce bonding expertise.

Throughout operations, the ground plans within the Meraki Dashboard had been a visible assist to simply spot an issue and navigate the workforce on the bottom to the precise spot, if one thing needed to be adjusted.

Because the sponsors and attendees crammed every area, within the Meraki dashboard, we had been in a position to see in real-time the variety of purchasers linked to every AP, presently and over the time of the convention. This enabled fast response if challenges had been recognized, or APs might be redeployed to different zones. Beneath is the Marina Bay Sands Degree 4. We might drill into any AP, as wanted.

Meraki’s built-in Location Analytics helped us visualize bodily area utilization. We might see the variety of attendees who handed by way of the lined space of the convention, with out them even connecting to the community. This gave us insights into customer footfall traits, corresponding to areas of curiosity, most visited cubicles, lecture rooms, or periods. For instance, beneath you possibly can see the twond day of coaching, with busy lecture rooms, whereas Enterprise Corridor in setup. You may additionally discover lengthy dwell instances nearer to the world overlooking the bay.

The Location Heatmap was displayed dwell exterior the NOC. Beneath you possibly can see the 9am Opening Keynote on 11 Might, earlier than the Enterprise Corridor opened.

Bodily safety can be an necessary side of cybersecurity. We have to know the way units transfer in area, know the place useful belongings are situated, and monitor their security. Christian Clasen takes this obtainable knowledge to a brand new degree in Half 2 of the weblog: Correlating Meraki Scanning Knowledge with Umbrella DNS Safety Occasions.

Meraki wi-fi community allowed us to supply a constant and distinctive expertise to occasion guests and employees. Every day, on common greater than 500 purchasers linked to the wi-fi community.

Safety Middle Investigations, by Uros Mihajlovic

Throughout our time within the NOC, we had the possibility to work with different vendor engineers and a few use instances that got here up led to attention-grabbing collaborations. We actively seemed for violations of the Black Hat Code of Conduct. Examples are utilizing the community as a platform to assault the Web, attacking others on the community and/or disrupting the community.

These alerts had been seen within the Safety & SD-WAN -> Safety Middle -> MX Occasions. Search for Half 2 of this weblog to find out about this investigation and response: Script Kiddie will get a Timeout, by Ben Greenbaum and Shawn Coulter

We had been in a position to simply establish the consumer’s approximate location based mostly on the entry level they had been linked to. Shopper location allowed us to establish the place the consumer was in a bodily location.

If the conduct continued and we would have liked to dam wi-fi purchasers, we might simply achieve this by attaching a gaggle coverage by way of the Meraki Dashboard, together with a quarantine VLAN and a splash web page. As well as, we might use a script that may be triggered by way of the interfaces of the opposite safety merchandise to use the identical group coverage through the Meraki APIs (Software Programming Interfaces). This integration was simply one of many many collaboration bits that we labored on.

Meraki and ThousandEyes, by Uros Mihajlovic

On the convention, an necessary gross sales software, used for partaking with occasion prospects, was having points connecting to the server. The gross sales workforce reached out to the NOC leaders to report the appliance slowness, which they suspected may be on account of our community.

Utilizing Meraki Wi-fi Well being, we might simply examine consumer efficiency and wi-fi expertise. Observing the total stack map from the consumer perspective additionally confirmed that upstream switching infrastructure just isn’t reporting any efficiency or latency points.

This allowed us to higher perceive the standing of our community. If any of those units within the consumer path had been reporting a problem, we might have simply remoted the difficulty to that gadget and troubleshoot. Contemplating the whole lot was reporting wonderful community well being, the subsequent step was to verify efficiency knowledge in additional element. After inspecting the efficiency knowledge, we might quicky and successfully decide that situation in not on account of our community.

Ruling out the community, now we might concentrate on the subsequent step of the troubleshooting course of: to exhibit the difficulty just isn’t on account of our community. The easiest way to do that is by having proof to indicate the place the difficulty is occurring. First, we needed to establish the server vacation spot the place the appliance was being hosted. Wanting on the Meraki software analytics, we might see that software is reaching out to a particular area. Subsequent, utilizing Cisco ThousandEyes cloud brokers, along with endpoint agent put in on our laptops, we configured scheduled artificial checks that may probe the appliance area. This instantly confirmed that constant latency from our host gadget to the server was round 200ms, with frequent spikes as much as 600ms (about half a second). Moreover, ThousandEyes helped us visualize the site visitors path for the app area. Utilizing this, we observed that area is hosted in AWS (Amazon Internet Providers) in Dublin, with site visitors path going by way of Paris. Every hop added latency, which was inflicting the reported points.

It is a notable instance of how Cisco instruments come collectively to cut back Imply-Time-To-Decision (MTTR). Meraki community well being supplied us with visibility of belongings we personal (e.g., wi-fi and switching community), whereas ThousandEyes supplied insights into belongings, we do not need management over (e.g., service and software suppliers). Subsequently, this supplied us with a holistic view of dependencies, permitting us to pinpoint the precise supply of the difficulty.

Meraki Dashboard, by Steven Fan

The Meraki dashboard provide a complete and user-friendly interface for observing the well being of the community. This contains all the suite of options supplied by Meraki, amongst which the Entry Factors (APs) and Switches are integral elements. These dashboards provided wonderful knowledge visualization capabilities, permitting customers to shortly comprehend and work together with the system’s standing. The flexibility to combination knowledge meant that we might collect and show data from a number of sources, giving us a holistic view of the community’s efficiency. Moreover, the dashboards enabled us to delve into the main points of any change, AP, or consumer swiftly, making troubleshooting and efficiency evaluation sooner and extra environment friendly.

All through the distinct levels of the convention, the Meraki dashboards had been invaluable. Within the three days main as much as the convention, through the setup section, we might monitor the community’s standing in real-time, guaranteeing that every one components had been functioning accurately and that any points might be addressed promptly. This was essential in guaranteeing a clean and dependable community setup.

Through the first two days of the convention, which had been devoted to targeted and intense coaching, the Meraki dashboards allowed us to maintain an in depth eye on community utilization and efficiency. We might see how the community was dealing with the elevated demand and made any crucial changes to make sure a secure and sturdy service.

Lastly, as we transitioned to the briefings and Enterprise Corridor levels of the convention, we might visualize the community site visitors. This visualization was essential in understanding how the community was getting used, figuring out any potential bottlenecks or points, and guaranteeing that every one attendees might entry and use the community companies successfully.

The brand new Abstract Report operate within the Meraki system served as a useful instrument for offering high-level statistics related to the community’s operation. This report contained an summary of an important metrics and knowledge, enabling us to shortly perceive the community’s efficiency.

One of many noteworthy options of this report was its automated emailing operate. Each morning, the system would ship this report on to our workforce’s inbox. This meant that we might begin every day with a direct understanding of the community’s standing, with no need to manually collect and analyze the information ourselves.

Along with saving time, this automated report additionally helped us keep proactive. If there have been any vital adjustments within the community’s efficiency, we might be alerted instantly by way of the report, permitting us to swiftly reply and handle any potential points. This was significantly helpful for executive-level employees who wanted a fast, complete overview of the community’s well being with out getting too concerned within the technical particulars.

Because the particular person with core duties for the change configuration and uptime, the Meraki dashboard made it fairly easy to shortly change the community topology, in line with the wants of the Black Hat buyer. In abstract, the Meraki dashboards had been a robust instrument in managing and optimizing our community all through the convention.

Meraki Alerting, by Connor Loughlin

Meraki Dashboard permits for alerting through Syslog, SNMP and Webhooks. For Black Hat, we utilized Webhooks to submit quite a lot of alerts to again Slack and Cisco Webex; this implies we will bounce to motion ought to there be a change in community connectivity or if sure thresholds (corresponding to consumer unhealthy roaming) with out having to observe Dashboard all day.

Configuration for that is straightforward; taking solely two steps to get this arrange. Firstly, configure the incoming webhook in your chosen platform after which paste the Webhook URL into Dashboard.

We enabled alerts for change & APs going offline, change port occasion adjustments, Dashboard configuration adjustments, and wi-fi consumer connectivity occasions.

Wi-Fi Roaming Timeline

A brand new addition to Dashboard is Shopper Roaming Timeline and Analytics. It supplies community directors an ideal troubleshooting instrument for when customers complain about dropped calls or lowered throughput sometimes precipitated poor roaming expertise. The brand new timeline reveals how a tool roams between APs and whether or not they skilled a profitable, suboptimal roam, unhealthy roam, ping-pong (when a tool continually bounces between APs), or the dreaded disconnect.

On this instance, I used to be strolling across the Enterprise Corridor with my iPhone in my pocket. You may see a lot of the roams had been optimum and fortunately my connectivity was not impacted. This degree of visibility helps community directors acquire useful perception about purchasers roam round their community, probably highlighting AP placement or density points. (This additionally reveals that correct planning and utilizing predictive web site surveys paid off.)

Wi-Fi Air Marshal

Through the first day of coaching, within the Meraki dashboard Air Marshal, we noticed packet flood assaults in opposition to we had been in a position to adapt and stay resilient.

We additionally noticed an AP spoofing. We shortly recognized the situation of the assault on the Foyer exterior the Enterprise Corridor. Ought to the assaults proceed, bodily safety had the data to intervene. We additionally had the flexibility to trace the MAC addresses all through the venue, as mentioned in Christian Clasen’s part partially two.

Meraki Programs Supervisor, by Paul Fidler

Provisioning of units

As we did in Las Vegas and London in 2022, a few of the iOS units needed to be restored once more. Utilizing the blueprint helped almost about time taken, however, once more, the limiting issue was the sheer period of time taken to obtain the 6GB file (which, when utilizing Apple Configurator, doesn’t like community interruptions). Studying level: guarantee all pictures are downloaded forward of time.

To obtain the iOS and restore, add the cell config and put together the 28 units, between two of us, took 2.5 hours. Clearly, there was some disruption because of the community nonetheless being constructed, which contributed to this time, however, even so, this was nonetheless a substantial variety of hours of toil. We now have fed again to the Black Hat administration workforce how leveraging Apple’s Automated Gadget Enrollment might actually simplify this job. There’s a safety profit with utilizing this as effectively: If somebody wipes a tool both on objective or accidentally, when the gadget subsequent connects to the web, it can mechanically re-enroll into Meraki Programs Supervisor, stopping the consumer from establishing the gadget with out administration. Supervision (A course of that Apple requires to show that you just bodily have the gadget) can be utilized, which leads to extra MDM profiles being obtainable to be despatched all the way down to the gadget, corresponding to Safe Endpoint / Readability, the flexibility to put in purposes silently, and issues like Dwelling Display structure and Lock Display messages, all of that are used at Black Hat.

Search logic

We now have traditionally left alone as soon as enrolled units within the dashboard, to save lots of time for future periods, by not having to rename / re-tag units. Nonetheless, over time, this has resulted within the progress of stale units in dashboard. It might have been clever to have purged stale units earlier than we bought right here, however that didn’t occur. So, as units had been briefly turned on then off, the information in dashboard was not simply used to find out stale vs non stale. So, the enrollment date was used to tag units with a brand new tag (Black HatAsias2023). Nonetheless, dashboard doesn’t mean you can present units that are NOT tagged with one thing. Fortunately, there are some rudimentary logic search capabilities to leverage.

For instance:

Give me units which have the leadretrieval tag however NOT the leadretrievalspecial tag

(tag:”leadretrieval” NOT tag:”leadretrievalspecial”))

Gadget Identification

Renaming of units: iOS units for session scanning, lead retrieval and registration have an asset barcode on the again of them which is how they are usually referenced by Swapcard. Because the units are in instances, it’s painful for the registration employees to seek out the asset quantity within the occasion of a problem, of position reassignment for that gadget (from session scanning to guide retrieval, for instance). So, what we do is twofold:

  1. The very first thing that we do is take the packing listing of asset quantity, serial quantity and run a script that makes use of the Meraki API to rename every gadget within the Programs Supervisor Dashboard
  2. The following factor now we have is a coverage in Programs Supervisor that units the textual content on the backside of the Dwelling Display while locked, so customers can see immediately which gadget it’s, with out having to take the case off / log in to the gadget, and open Settings > Basic > About

Clearly, utilizing the serial quantity to establish units on the Lock Display has safety implications.

The perils of third-party libraries and monitoring

In direction of the beginning of registration, Umbrella picked up just a few occasions pointing to and some different blocked domains. An investigation was launched. Preliminary pondering was that the appliance used to verify attendees in had used some third-party libraries (that is in all probability true to the units reaching out to a authentic app improvement web site). Nonetheless, after speaking to the SwapCard employees, it was decided that, on the time of gadget setup, the units go to an authentication web page, which is only a internet web page. This internet web page incorporates just a few monitoring capabilities, corresponding to Google Tag Supervisor which incorporates We blocked these monitoring domains in Umbrella, to higher safe Black Hat.

Shopper Vs MDM Administration

Many of the data we get again from a tool is by leveraging Apple MDM instructions. This contains put in apps, certs and profiles, for instance, but in addition data corresponding to normal gadget data. Nonetheless, there may be some data that just isn’t obtainable through MDM. This contains:

  • Location
  • Jailbreak detection
  • SSID

The rationale that the final is related is that the Registration app on the iPads has its personal VLAN that runs throughout the Black Hat community to a handful of servers that course of that data, preserving issues secure and safe. Nonetheless, these servers are NOT accessible exterior of this VLAN. I used to be trying by way of the standing of the managed units and observed a few iPads had been NOT linked to the right SSIDs. A fast chat to the registration employees highlights that after they had been handed out to Expo Corridor employees, the SSIDs for the iPads and iPhones weren’t up and operating, in order that they had been joined to the attendee Wi-Fi!

Visibility is King!

However it does spotlight an issue with Apple Administration, particularly on cell: If that app is NOT operating, then we don’t get that data. It turns into stale. So, I’m researching methods to make sure that, ought to a consumer / admin kill the SM app, it may be remotely spawned by sending a consumer a push notification.

Constructing Instruments for Black Hat Workers, by Ryan MacLennan

After deploying all of the iOS units for the Black Hat employees to make use of through the convention, we determined there wanted to be a method for them to see the battery degree of the units whereas they’re in Kiosk mode. Kiosk mode makes the chosen software use full display screen mode and can’t be exited. This mode occurs to cover the battery degree and different standing symbols which can be on the high of the gadget. This has precipitated points prior to now the place the employee may have their gadget die in the midst of lead era or checking in an attendee.

We will see the battery ranges of the units within the Meraki Dashboard; nonetheless, permitting entry to the Meraki Dashboard to anybody not managing the community just isn’t one thing we need to do. Because of this we created an online software utilizing NodeJs, Specific, Meraki APIs and ReactJs to permit the employees to view the battery ranges of the units. The appliance is containerized and deployed so the employees can simply get to the appliance and instantly see the bottom battery degree units.

The above picture reveals the interface of what the employees see and when the appliance will carry out its subsequent replace to refresh the gadget listing. If they should discover a particular gadget, they simply search by the fields proven or by the meta knowledge saved, however not proven for every gadget.

A Higher Solution to Design Coaching SSIDs/VLANs, by Paul Fidler

Deploying a community like Black Hat takes a variety of work, and repetitive configuration. A lot of this has been lined in earlier blogs. Nonetheless, to make issues simpler for this occasion, as a substitute of the 60+ coaching SSIDs we had in Black Hat USA 2022, the Meraki workforce mentioned the advantages of shifting to iPSKs with Black Hat NOC Management, which accepted the plan for Black Hat Europe 2022 and once more for Asia 2023.

For context, as a substitute of getting a single pre shared key for an SSID, iPSK performance permits you to have 1000+. Every of those iPSKs could be assigned its personal group coverage / VLAN. So, we created a script:

  • That consumed networkID, SSID, Coaching title, iPSK and VLAN from a CSV
  • Created a gaggle coverage for that VLAN with the title of the coaching
  • Created an iPSK for the given SSID that referred to the coaching title

This solely entails 5 API calls:

  • For a given community title, get the community ID
  • Get Group Insurance policies
  • If the group coverage exists, use that, else create a gaggle coverage, retaining the group coverage ID
  • Get the SSIDs (to get the ID of the SSID)
  • Create an iPSK for the given SSID ID

The majority of the script is error dealing with (The SSID or community doesn’t exist, for instance) and logic!

The consequence was one SSID for all of coaching: BHTraining, and every classroom had their very own password. This lowered the coaching SSIDs from over a dozen and helped clear the airwaves.

Take a look at Half 2:




Thanks to the Cisco NOC workforce:

  • Meraki Community: Steven Fan, Connor Loughlin, Uros Mihajlovic and Jeffrey Chua; with digital assist by Evan Basta and Jeffry Handal
  • Meraki Programs Supervisor: Paul Fidler and Connor Loughlin
  • Cisco Safe: Christian Clasen, Alex Calaoagan, Ben Greenbaum, Ryan Maclennan, Shaun Coulter and Aditya Raghavan; with digital assist by Ian Redden and Adi Sankar

Additionally, to our NOC companions: NetWitness (particularly David Glover, Iain Davidson and Alessandro Zatti), PNOCalo Alto Networks (particularly James Holland), Corelight (particularly Dustin Lee), Arista, MyRepublic and all the Black Hat / Informa Tech employees (particularly Grifter ‘Neil Wyler,’ Bart Stump, Steve Fink, James Pope, Mike Spicer, Jess Stafford and Steve Oldenbourg).

About Black Hat

For 25 years, Black Hat has supplied attendees with the very newest in data safety analysis, improvement, and traits. These high-profile world occasions and trainings are pushed by the wants of the safety neighborhood, striving to deliver collectively the most effective minds within the business. Black Hat evokes professionals in any respect profession ranges, encouraging progress and collaboration amongst academia, world-class researchers, and leaders in the private and non-private sectors. Black Hat Briefings and Trainings are held yearly in the US, Europe and USA. Extra data is offered at: Black Black Hat is dropped at you by Informa Tech.


We’d love to listen to what you assume. Ask a Query, Remark Beneath, and Keep Linked with Cisco Safe on social!

Cisco Safe Social Channels