6 – Defending Your Factory Floor from Cybersecurity Attacks - Part 3 with Jeff Smith of Dynics

The manufacturing world seems behind when it comes to defending their factory floor before a cyberattack happens. 

Join host John S. Rinaldi as he and guest Jeff Smith, CTO at Dynics, wrap up their discussion on what manufacturers should be using to secure their factory floor and IT/OT networks, including:

• The ICS-Defender from Dynics, a product made by control engineers FOR control engineers and the platforms it’s available on
• Deep Packet Inspection (DPI)
• What deny-by-default means and why your cybersecurity system needs it

Connect with John:
Email | LinkedIn | Newsletter | YouTube | Blog | Books

For all things industrial automation, visit rtautomation.com. 

Transcript

John Rinaldi: 0:00

[ Intro music] Hello everybody, this is John Rinaldi with another network connectivity podcast. Today we're in part three of our three part series about cyber security and protecting the manufacturing floor. And again, I'd like to welcome Jeff Smith, the CTO of Dynics.

Jeff Smith: 0:24

Good morning, John. Thanks for having me on again.

John Rinaldi: 0:26

Well, today we're going, we've talked in the first two parts. We talked about the kinds of threats that are out there. We've talked about some of the different ways that people are going about trying to solve that problem. Today. We're going to talk about a particular product that Dynics has called the ICS Defender. And so you're the main architect of that product. So tell me, why did, why did you build the ICS Defender?

Jeff Smith: 0:52

So, um, you know, one of my former roles, I was an end-user and, um, it's interesting that we went cold turkey, Ethernet IP, and the manufacturing space in about 2006. So to say we were on the bleeding edge of that is somewhat of an understatement. Um, and so, you know, we spent a few years building out the architecture, designing and getting it into play. And then in about 2010, we kind of turned her eyes towards security and remote access, uh, ahead of the curve again, I think. Um, so go ahead.

John Rinaldi: 1:27

Yeah, that certainly was, that was, that was before anybody had any inkling that, that security on the manufacturing floor was a problem?

Jeff Smith: 1:34

Yeah, yeah, it really was. So, you know, I, I work, I work a lot with several of the major vendors, um, of networking equipment, um, for OT and, you know, I looked at the product offerings. I had, I had a list of things that we wanted to accomplish, you know, I, I'm a big believer in pick your top five and go after the top five, then pick five more. Um, and I couldn't, I simply couldn't find what I wanted in, um, without excessive costs, excessive complexity, multiple pieces of hardware out on the plant floor. Um, so working with one of the local companies here in, um, Detroit, we started to, um, really flesh out what it is today, the ICS Defender. Um, so it went into production in 2012, uh, and it was really designed to meet the needs of OT folks versus, um, you know, something that's designed for the enterprise and then let's make it 24 fold, put din rail Mount on it and call it industrial.

John Rinaldi: 2:33

Well, you know, that's, that's what I've always said is that there's so many people that they look at that, Hey, we've got a cyber security product. Why don't we just make it, sell it to the manufacturing floor? And that's exactly what you do. It tends to be 24 volts and din rail. Now it's for manufacturing.

Jeff Smith: 2:48

Yeah. I actually had one of our suppliers at the time, send me their quote unquote. Well, not at the time. This was a couple of years later. They sent me their first swag at a, um, uh, an industrial perimeter firewall type of device. And it was, it had a hockey puck, like a laptop for power and 120 volt AC. It had four little feet that you set on a cabinet. Um, so I looked at it for awhile and because I'm not the kind to let that thing go. I took it out to one of our machine builders and opened up a double door panel and I duct taped it to the back plate. And then I ran right in the middle of the back plate. There was nothing else in there. And then I ran a big orange extension cord to it off the wall for 120 volts, took a picture of it and said, when you make this industrial, let's try it again.

John Rinaldi: 3:35

So of all the, you know, when you started looking around for a cyber security solution, what are the three or four things that you really couldn't get from the solutions that were available, that, that you put into Defender?

Jeff Smith: 3:46

So, uh, initially it was, we wanted remote access. So we wanted to do, um, you know, simple remote access and in a lot of the products at the time, the only type of VPN that they supported was, uh, IP sec versus SSL. So I really wanted something that, you know, our corporate standard at the time was SSL. And I thought, well, it's a lot less complex to set up than IP sec VPN. Um, I'm not doing site to sites with these things. So what if it's good enough for my IT group? Why isn't it good enough for my platform?

John Rinaldi: 4:18

Okay. All right. I'm going to stop you right there. So there's a lot of people that aren't going to know what the hell you just said and what is it, what is IP sec and SSL?

Jeff Smith: 4:26

So those are type of virtual, private networking, um, that the two different methods or approaches to virtual private networking, um, IP sec is one that's been around for a long time. And while it's very, very secure, it's also very complex to set up and maintain. Um, SSL VPNs is another type of a virtual private network. So if you work from home ever and you remote into your company and you access the company network, chances are, that's an SSL VPN. Um, so I, I wanted to be able to remotely support equipment, uh, that was kind of the first goal or monitor equipment. And then, you know, I'm a big believer in people don't know what they don't know. So it's, it's kind of hard to know you do your best job at trying to second guess what you need in the future, but in the areas of cybersecurity on the plant floor, um, that, that was, it was really such a, such a new territory that, you know, nobody really knew what they didn't know. Um, and so six months later I was looking for things like deep packet inspection because I had somebody flashed the incorrect, um, flashed the wrong PLC in a production line, which of course takes the production line down. Um, and so I was looking for something to use, to help protect us even from accidental issues. So going back to my vendor at the time, I said, well, all right, so this is what I'm using for VPN, even though it doesn't do what I want exactly. How do I add deep packet inspection to that? So I can really, I can make a PLC read only if I want to. And they said, well, that product do that. You need this product. And I said, well, okay, then does the new product do the VPN stuff? And the deep packet inspection? And of course the answer is no now I'm, um, yeah. So now I'm looking at buying, you know,$25,000 worth of hardware and software to solve both of those problems. And I'm like, you know, there's gotta be a better way. I don't need that many points of failure in my network. I don't need all this learning curve. I don't need, you know, in manufacturing having, um, part numbers in stores is a big deal. I don't want to have 12 different part numbers for this stuff in stores. I want one right now.

John Rinaldi: 6:32

What, uh, what is tell, tell, uh, tell everybody what is DPI, what is that deep packet inspection mean?

Jeff Smith: 6:38

So deep packet inspection is the ability to take, for example, an industrial protocol like Ethernet IP, and really tear it down to the bits and bytes level of the packet and understand if it is what it's supposed to be. Um, so with the ICS Defender, we have the ability to learn that traffic. And then once we've learned that traffic, that's the only thing that gets through. So even ICS Defenders that have been out there for years, that, that don't have to get any sort of, uh, definitions update because they do everything at the protocol level. Uh, they'll still block malware that we see today, even though they may not have, you know, they might be running the original version of firmware, for example. Um, so it's a very powerful way to control the traffic in and out of, um, a particular network or an area of a network.

John Rinaldi: 7:27

Let's, let's go a little deeper on that. So if, uh, if there's, uh, so it's watching for particular messages and allowing particular messages coming through from the outside world into the manufacturing systems, is that right?

Jeff Smith: 7:42

Uh, yeah. It's um, if, if you look at, um, and I'll pick on Ethernet IP again, Ethernet IP uses this, this, these elements of the messages that are called assembly instances, instances, and attributes, and, you know, certain attributes mean certain things. So for E is one, you know, one type of a message, which might be a CIP reader, a CIP, right? So we have the ability to take a packet apart and say, okay, here's a read, here's a read, here's a write. We don't let the write happen. And we throw the write away. So it's not allowed to write, or I can specify, for example, um, I want no writes to happen unless they're going to create a trend for troubleshooting because a trend can't affect the PLC solve of logic. So I'm going to make the thing completely read only except they can create a trend. Um, another great example is I will let you clear the faults on a PLC and go from program mode to run mode, but I won't let you go for run mode to program mode. So we have very, very finite control of, um, what actions, for example, a studio 5,000 might try take with a Rockwell PLC.

John Rinaldi: 8:56

So if someone compromises a PC, a windows PC, um, the IT side and, uh, that PC has access to, uh, to the PLCs on the manufacturing network, does can DPI stop them from, from doing things?

Jeff Smith: 9:14

Yep. You, you can. Um, for example, if you have RS links running on, uh, uh, uh, a computer or a PC, and it's connected to a network with PLCs, uh, unless you choose to allow it using the deep packet inspection engine, you really can't even see links, RS links on that PC wouldn't even be able to see those PLCs. As far as links was concerned, they didn't exist.

John Rinaldi: 9:37

Wow. Now, how does that different from the other? Yep. Everybody's got firewalls on the manufacturing floor. I mean, how is this different from the firewalls that are currently being used?

Jeff Smith: 9:48

Um, most firewalls or traditional firewalls, the best level or the best best depth you can get to is either is port based. So it's a, for example, Ethernet IP uses port 44818 in TCP. Um, so most cases you can say, okay, I'm going to allow traffic TCP traffic on port 44818 only. So that's great. You can eliminate a lot of potential traffic, um, unless somebody writes malware that happens to use port 44818, and TCP then, and then it'll get through. Um, and more importantly, if you allow everything on Ethernet, IP to come through, then that means I can flash the firmware. I can change the IP address. I can stop the controller. I can modify the logic when maybe I shouldn't. So simply being able to, to allow or disallow a particular protocol by a port, um, is what most traditional firewalls do. Having the ability to do deep packet inspection on that traffic means you can go, you can go several layers deeper than just the type of traffic and very be very specific on the, the, uh, the commands or the, the, you know, the request for the reads or the writes or whatnot that that traffic or protocol is attempting.

John Rinaldi: 11:05

Now. Now this seems to fly in the face of a lot of what I see people buying today and putting the manufacturer devices with dual nec, where they've got one Nick that goes to the IT system and, you know, say a power monitor or a drive that's monitoring the energy data. So it sends the energy data on that side. And it talks to the PLC on that with the other nec this, that, that situation you can't it's hard to protect when you have a lot of devices like that, this is it?

Jeff Smith: 11:37

Uh, it is because you, you know, you exponentially increase the attack surface. It's, it's interesting. I work with an organization called simony and, uh, one of the, one of the conversations, or one of the suggestions that I've heard recently is, is to develop. And this has been a proponent of this for years is to develop a secure data hub for the plant floor, whereby there's a common API on one application interface on one side, um, and then making that, so that everything on the plant floor kind of goes through that single point. It's it's, it's not an analyst, an analyst, it's a new word for me today. It's um, analogous analogist. Yes. Thank you. Before 10, I'm sorry. It's a, it's kind of like capping a bottle. Um, so if you go out and punch, you know, 57 holes in something, it's, it's pretty hard to figure out which hole the something's leaking through, but if there's only one hole and everything has to ingress and egress through that individual location, it's very easy to understand where the problem is and if necessary cap it.

John Rinaldi: 12:44

The, uh, that's very, that's very clear. I think that, uh, that you have to have that single point of access to the manufacturing network. And I think that people miss that, and part of that is because on the manufacturing floor, we only, we know every message that should be coming and going there. So I think that's the thing that most devices that are cyber secure really don't take advantage of is that fact that we know what we're doing, what we have there. It isn't like IT security.

Jeff Smith: 13:16

Yeah, that's very true. So it's a, you know, I've said for years that IT organizations and IT security needs to allow everything, but what is known or suspected to be bad in the OT space, we should allow nothing, but we know what to be explicitly good. Um, we have that luxury, whereas in the IT space, because it's such a dynamic environment, they don't, that's another good reason that products developed for the IT space don't necessarily fit well into the OT or industrial space.

John Rinaldi: 13:46

What are the kinds of threats that, uh, that people are unaware of with the, with the VPNs that they're using today to, to access their machines? I mean, most of them are, uh, call out to the cloud, the, uh, the user who wants to talk calls out to the cloud and in the cloud, they make this connection. And, and now they're, uh, they're in those systems secure. And do, should you use an ICS defender to protect that?

Jeff Smith: 14:14

Um, you certainly could th the issue there's, there's kind of two issues, and I've got a lot of customers who have run into this. Um, I always relate a story about a customer who put that type of security appliance in for his end user. And it was in Washington. And by the time he got off the plane, um, back home here in Michigan, uh, internet service was out, or this AMA, this, this cloud service providers, uh, portal was out on AWS for a couple hours, so they couldn't support the equipment. Um, so you're, you're really going through a point in somebody else's computer and you're, you know, all of your data, all of your information, everything do goes through that point and then back down. Um, so the, the security on that side is critical. Um, the second piece is it's, it's a single point. So if that's not available, you have no way to get to or support, um, whatever it is you're trying to support. Whereas if you're doing more of a point to point support, um, you know, there's, there's many routes across the internet, let's say.

John Rinaldi: 15:17

And, and taking that a little twist on that. Now there's a lot of people working from home now, still, even though the pandemic is essentially over, are they sick? They're sick, they're dialing it. Well, the highly done that shows you how old I am, they're coming in from their, uh, through the internet, from their houses and using a VPN. And is that, what's the security risk there with the typical VPNs that people are using for that? And could they use an ICS Defender in a situation like that? So they have one at their phone. Should they have one at the plant? Should they have both?

Jeff Smith: 15:54

Um, it, it boils down to what their needs are. So, for example, at Dynics, we use all of our own products. So we have multiple ICS Defenders. Um, we have a large enterprise grade ICS Defender, that's, you know, server room in our, in our world headquarters, for example, um, and all of the remote teams use that for support. So, uh, our connectivity, so they connect with just a client on their machine, into our system. Um, I have other, some municipalities, for example, wastewater, and they will have a ICS Defender, um, at their facility. And then all of their remote substations or remote users have ICS Defenders and they connect ICS Defender to ACS Defender. Um, we also have, uh, one of our technical support folks works remote from, um, Houston, Texas. We actually put an IP phone on his desk and put an ICS Defender net behind it, and using that we were able to give him straight up typical IP phone connectivity from his desk to our office, just as though he's his phone was sitting on our network here at the office.

John Rinaldi: 17:03

Wow. That's pretty cool. Um, are people at risk though, the ones that are just using straight home VPN to their manufacturing system with products like, uh, Iwan and other things?

Jeff Smith: 17:18

There's always a risk, um, multiple factors of authentication is important. Um, you know, being able to support the various network types. I know, for example, a lot of, some of those products you just mentioned, don't support multiple virtual networks, um, which if you're doing is you're following proper network segregation practices could be using something called a V LAN, and they simply don't support that. Um, they're, they're really designed for one off or two off people that want every now and then access there. Um, long-term the costs and the complexity of them is more than people realize.

John Rinaldi: 17:55

What, um, what are the principles that, uh, behind ICS Defenders is something called deny by default? Can you explain what deny by default means and why it's important?

Jeff Smith: 18:07

Yup. Um, so denied by default is something that all of our Dynics security products follow, um, and I'll, I'll explain what it is. And then I'll kind of explain, or maybe give an example of why it is denied by default means that if you pull out of the box and plug it in, um, nothing talks to anything, unless you explicitly require it to or explicitly configure it. Um, the reason for that is obvious. Um, if you look at a lot of our customers have used other products that are not denied by default. So, you know, even perimeter firewalls from some of the other major companies, um, when you plug it in, they've been so concerned about, um, you know, so concerned about let's not stop production, let's not stop production. And their default behavior is to allow all traffic. Well, if you look at how that goes on a plant floor, let's, let's just walk through that scenario for a moment. Um, I've been machine down. The operator says none of the buttons on my HMI are working. I've got faults on the HMI nothing's happening. Um, for whatever reason that the tradesman decides it's this little box. So they pull the box out, get a new one from stores, with the same part number, and they plug it in. If it's denied by default, then you don't compromise your security because you have to restore a configuration to it. That is valid. If it's not denied by default, if it's allowed by default, what, what's the, what's the indication to these folks on the plant floor that something is working in it, they're all done and they can walk away. Well, the HMI works, the lights work and the machine will go into production. So now you've got a maintenance person who says, okay, I replaced this little box and everything is working now I'm done. So they walk away and now you've got over time. You're opening up more and more and more holes because essentially that, that firewall or that appliance, that's not denied by default is just a big gateway for traffic. Um, I've had customers actually consider going to something, um, other than a firewall that, that, that explicitly only allows a one way traffic, because for that very reason, they ended up auditing their networks and found hundreds of these out of the box configurations that were allowed traffic by default.

John Rinaldi: 20:20

Are most firewalls the you buy off the shelf. I don't know. Cisco firewalls. Other ones you find on your manufacturing floor are, most of them will allow by, you know, just allow everything?

Jeff Smith: 20:31

Yes many of them are. Yes, unfortunately.

John Rinaldi: 20:34

Oh, wow. That's a huge risk then for people who really don't understand how to protect, how to protect a manufacturing floor, just allow all traffic through means that if you can, all sorts of malware or anybody.

Jeff Smith: 20:49

Yup. And that's the same problem. You get all these folks that are using, um, just NAT appliances. Um, what they don't understand is that there is NATS are typically a typical NAT is garbage in garbage out. You're literally bridging together to network. And unless there's some sort of enforcement on that traffic, um, whatever goes in one side comes out the other, uh, and the, you also get a concept called bleed, which means certain types of traffic will bleed from the bottom of that NAT, that bottom network on that NAT to the top network. Um, and in the ICS Defender, we've done a couple of things to, um, combat that issue is number one. Um, the only things that are allowed to talk are the exact devices identified in the NAT, um, out that port and as well as any traffic coming in can only be directed at those. And then we also allow you to run that deep packet inspection engine across that NAT so that you can literally learn what you want to have happen. And that's all that goes through.

John Rinaldi: 21:48

The typical device that we have today, if there's a NAT, anybody in there. So that, that address is out there on the IT side for say a PLC, anybody who's compromised, the IT network. And God knows that there's a lot of compromising of IT networks going on. Can use that NAT to do anything you want. Is that true?

Jeff Smith: 22:07

Yes.

John Rinaldi: 22:09

Wow. That's, that's a huge hole when you're using NAT and it's not, and you're not protecting it with something like the ICS Defender.

Jeff Smith: 22:16

Yes. it sure is.

John Rinaldi: 22:18

Um, how do you tell me about, uh, profitable let's uh, you know, what kind of platforms are, have you made available for ICS Defender? Tell me about the licensing. Give me some of the details about what, what users should be thinking about.

Jeff Smith: 22:33

Okay. Um, so with, with ICS Defender, it's, um, it's a bit of figure out what hardware fits your, your particular needs. So we have, um, we have very small din mount, 24 volt. We have a IP 65, we've got, um, several flavors of 19 inch rack mount. Uh, and it will also run completely virtualized mobile in say a VMware vSphere, um, KVM, most of the major virtualization platforms. So I actually have customers who use it on the cloud, their cloud business it's on the cloud side. So anybody that connects to their cloud service actually goes through a virtualized ICS Defender, and they don't even realize it.

John Rinaldi: 23:16

Wow.

Jeff Smith: 23:17

Yup. And then within the, so you kind of pick your poison in terms of the platform that makes sense. Um, and then you decide what license level makes sense. So one of the things that's always bugged me and I'll kind of harken back to that conversation. I made about having to have three and four appliances when I first went down the security road. Um, it always bugs me that once I go to all the trouble of putting something in stores and I go to all the trouble of teaching it and having my people across the globe, understand how work with it and use it is if I want to add more features or do more things, I have to get different hardware and maybe it configures completely differently. So I've got that whole learning curve. I've got all the validations of redo. So with ICS Defender, we designed a licensing model. So you can get an ICS Defender. It is nothing more than a very basic simple NAT using something we call a simple NAT wizard, um, and you can create and use it for NATS. We have number of customers that do, but then if you decide later, you want to add deep packet inspection. Well, then you can add that it's a software license upgrade. So you just apply a new license. And then all of a sudden that capability becomes available. So you could buy a$750, whatever it might be. I'm just making numbers up. You could buy a$750 version and later upgrade it to the$4,000 version and all the features that come with it. And those aren't the actual prices, but it gives you, it gives you, it gives you the ability to scale the product as you grow in your understanding of what your needs are. Because if, again, if you don't know what you don't know what I buy today, I don't want to be stuck with in a year if it's not what is going to work for me in a year.

John Rinaldi: 24:53

Well, that's great. And, uh, we've come to just about the end of our conversation today. And I think we did a, uh, a nice kind of shallow, I won't say a deep dive. You have to really get on the, on a, on a network and show the ICS Defender and operations. And certainly anybody out there who's interested in doing that can contact, uh, Real Time Automation, where we'd be glad to help you, help you with that, set that up. Um, tell me, um, one, one last thing I, you know, as I looked through the kinds of things that we didn't cover, what about, um, uh, vendors who coming into the plant to do things, how do they, how do you protect yourself from their having malware on their laptops? Or are they doing looking around and stealing some information or compromising your, your manufacturing system? How does ICS Defender help you in that case?

Jeff Smith: 25:47

Well, there's a couple of avenues to take, and it depends on, you know, um, how much security is dependent solely upon what risk someone is willing to live with. So you need more security if you're willing to live with less risk. Um, but by and large, we have a technology in the ICS Defender called captive portal, and that allows us to control, uh, access to the network parts of the network equipment on the network by user, by voucher, um, by certificate. So we can have a multiple, multiple levels of authentication, um, and, and allow you to control that type of access.

John Rinaldi: 26:26

Great. Well, I think that's, uh, uh, that's been a good overview and I want to thank you very much, Jeff, for your time today. And it's always a pleasure speaking with you. Uh, anybody out there that wants to know more about ICS Defender would like to see a demo, like to get one in their hands for testing. We can certainly arrange that if you go to rtautomation.com and look at the products tab under security, you'll find, uh, explanations of some of, some of the feature lists of, of what the different licenses have. And, uh, I really encourage you to do that because I think it's an outstanding product. So thanks very much, Jeff. You have a great day.

: 27:02

Thanks, John. I always appreciate it. You have an awesome day, too.[ end music].