Industrial cybersecurity breaches continue to gain traction every year, yet the manufacturing world still seems behind when it comes to defending their factory floor before an attack happens.
Join host John S. Rinaldi as he and guest Jeff Smith, CTO at Dynics, continue their discussion on solutions that manufacturers need to be doing to secure their factory floors and IT/OT networks, including:
Connect with John:
Email | LinkedIn | Newsletter | YouTube | Blog | Books
For all things industrial automation, visit rtautomation.com.
Industrial cybersecurity breaches continue to gain traction every year, yet the manufacturing world still seems behind when it comes to defending their factory floor before an attack happens.
Join host John S. Rinaldi as he and guest Jeff Smith, CTO at Dynics, continue their discussion on solutions that manufacturers need to be doing to secure their factory floors and IT/OT networks, including:
Connect with John:
Email | LinkedIn | Newsletter | YouTube | Blog | Books
For all things industrial automation, visit rtautomation.com.
[inaudible]
Speaker 2:Welcome to the network connectivity podcast today. Once again, we are proud and happy and enthused to have Jeff Smith CTO of Dynex with us. Again, I'm going to talk about cybersecurity. Good morning, Jeff.
Speaker 3:Good morning, John. Good to be here.
Speaker 2:Good. So today we're going to talk about in our last, well, our last podcast, we talked about a lot of the problems and the kinds of threats that there are to the manufacturing floor today. We're going to talk about solutions and, but, but the thing is we're, we're, uh, we're talking here in may of 2021 and the thing that's on everybody's mind and it's in the wall street journal on CNN, ABC news, the guardian, everywhere you go, they're all talking about the colonial pipeline pack. It's huge news. So, Jeff, what, what's your opinion on this? What really happened? What kind of conclusions can we draw from that?
Speaker 3:Well, so there's, there's still a little bit, quite a bit. That's not, not known. What we do know is that the attack was on the enterprise system, not the OT networks, um, but given the interconnectivity between enterprise and in the OT space, um, colonial pipelines felt that it was a precautionary measure to shut down OT operations. And it probably was, um, they, yeah, it was, it was plain and simple. It was a ransomware attack. So they, they were after money. Uh, I I've heard different numbers bantered about, obviously until colonial comes forward with, you know, all the finite details of the investigations that have taken place or are probably ongoing it's it's, you know, it won't be known, but I, I heard$5 million, which seems kind of light to me. Um, but you know, it's, it's a case of these, these attackers. Um, and for all intents and purposes, the belief is that it was the dark side, um, that caused, or that, that perpetrated the attack. And you know, that that's an organization that just looks for weak, weak, uh, infrastructure, things that are connected, um, people using older technologies, um, and you know, they, they, they actually go through I'm sure and evaluate which targets are most likely to be penetrated in which targets are most likely to pay out based on their impact. So, um, it's, it just brings to the forefront again, um, what, what what's necessary and the steps that are taken. I mean, there was a significant executive order, um, shortly after the attack. And some folks say that it's in response to that, but, uh, I agree with Eric buyers who states that, you know, the it's an 18 page executive order on cybersecurity. Um, that's normally those things are only a couple of pages long. And is Eric pointed out it's um, it's unlikely that from the weekend to a Monday, Tuesday that the, the U S government put together an 18 page executive, right?
Speaker 2:Yeah, I would, I would guess stop. Tell us more about what, what re what does it, what does ransomware means? I think that the general public, and probably a lot of, uh, even people in manufacturing only have a vague idea. So what exactly did they do? Uh, what does, what did they, what did they, did they encrypt their servers? What is, what does ransomware typically mean?
Speaker 3:Well, so it's, it's a great question. And there's a couple of things. There's a number of things that constitute is ransomware. Uh, it could be, uh, simply locking out access to protect critical parts of a system so that, um, you know, the operators can't get to them. Um, can't do things they need to run equipment, particularly in the OT space. It could be, um, encrypting, as you mentioned, information and encrypting it in such a way that it, it's not usable unless it's decrypted. Um, so there's, you know, there's lack of creativity is not a problem. They, uh, they come up with lots of ways and essentially they hold your system ransom. I mean, it is what it sounds like it's a, it's a piece of malicious software or malicious action that, that says, okay, I'm taking over this system and I'm going to own it until you pay me a chunk of money to give it back.
Speaker 2:So they could do things like change a password on a, on an entry to some, some, some, some system. And then now it's not available.
Speaker 3:Yup. Yup. And, you know, in the OT space, um, one of the things I used to talk about a lot was when you build equipment, um, you know, centralized servers are great, but it's, you know, in centralized locations for data are great. And there are certainly places where that all makes sense, but keeping your, your process data distributed, keeping your information in, you know, when you build a piece of equipment, one of the things we used to do building equipment was we made sure that throughout the entire process, um, critical IP to our, our process, wasn't all in one place. So at best you could compromise just a piece of that data or a piece of that equipment. You couldn't compromise the entire system. And that was done on purpose to completely, um, distribute our IP across the system so that people couldn't just grab it, you know, take over one piece of the system and have our entire process and all of our, our, you know, how we built equipment, how we built axles, for example,
Speaker 2:How confident are you in the ability of, of it people and the people who are responsible for it, security to counter these threats and seems to me that the hackers are spending all day long, 24 hours a day, thinking of ways to attack these systems and the defenders are, or it's only a part of their job, seems like they're always going to be behind the eight ball. Is that true?
Speaker 3:Uh, yeah. It's, um, it's, it's the nature of the beast, I suppose. Um, it's a, it's a combination of a couple of things you mentioned that, you know, they sit around thinking of great new ways to compromise systems. Um, and not only that, if you look at the, um, you know, it, and OT has been talked about for ages forever today, um, and you know, a lot of that convergence has been ongoing now for years. And if you look at the systems that are out there, um, and I saw this in manufacturing, um, across multiple systems, not just a data type systems or network systems, you know, even if they're delivered at the point of, you know, where they start into the manufacturing and the OT process, um, over time, they, they, they degrade the security degrades and it's, it's not maintained like it is in the it space. Equipment is not refreshed. Technologies are not refreshed. So you have seven year old and 10 year old manufacturing systems out there that, that are, um, you know, when they were put in and they might've been reasonably secure in the day, although probably not. Um, and that whole thing is just broken down through misconfiguration, you know, new devices being added to it. Um, lots of extra connectivity requirements. Um, Greg was, did a really good article and, and referenced a lot of that same stuff. Um, so it's, uh, it's a, it's a problem. What
Speaker 2:About, uh, you mentioned about keeping your data and you'd be your IP and your data distributed in different systems. What about the cloud? Is it, would it be reasonable for somebody to infer from your comments that, well, I'll just entrust my data up in the cloud. I'll give it to Amazon and Microsoft and Google, and those guys know how to protect data. So I, so I'll be fine. That is that reasonable, you know,
Speaker 3:It's, uh, it's an interesting and philosophical conversation. Um, on one hand, the argument can be made that, you know, those guys are constantly keeping their security up-to-date. So they're, they're, they're much more, um, much more likely to have latest security technologies in place monitoring systems in place being able to respond to events, um, which is, which is probably very true. But on the other hand, it's a single point of attack where, you know, if I take down, if you take, if you watch the past couple of years, we've had incidences where AWS went down, for example, in anybody's cloud hosted services that were part of that system, uh, were compromised in terms of either loss of use or, you know, I haven't heard of any big breaches on, on those platforms. Um, it'll be interesting to see how this, uh, executive order changes a level of transparency that some of these big providers are willing to put out there
Speaker 2:About those cloud platforms is the ability for people who work there to be compromised either. Uh, you know, some kind of, you know, you are bribed to do to things or to provide information or be threatened with, Hey, we're good. We found out that, that you have, you're having an affair, you're you you're, you've done this, you've done that. And we're going to expose you, uh, if you don't, if you don't give us the information or do this, I think that that's really probably the biggest threat for Amazon and Microsoft is having some internal security.
Speaker 3:Uh it's so it's entirely possible. I mean, it's it, you somewhat lose the security by obscurity. Um, if, if I keep my, you know, the other side of that same argument is if I keep my data all on prem, um, somebody has to come to me specifically. Um, if I, if I can get into AWS or those systems, which I'm not suggesting it's by any stretch, easy, um, it's still much closer to a single point of failure, which in the manufacturing space is something we've avoided for years. Um, and the other, the other concern is, um, w you know, you read a lot about let's monitor the networks. Well, if you're monitoring the network and you're not taking proactive action based on what you're learning or understanding, um, then once I'm in, um, you, you might be able to find out I am in and do something about it later, but if I'm in for a couple of hours before you're aware of it, um, or if I'm in for five minutes and I've planned this attack properly, um, I've already done the damage I need to do. So finding out about me after the fact, isn't that big of a, uh, Hey, look, what happened, because you're probably gonna hear from me anyway, when I ask you for a chunk of money, right? So if you didn't know, I was there, you will now,
Speaker 2:Well, well, most of these attacks, especially on the it side, the attackers are generally in four hours, days, weeks, right? They're not, they don't just, uh, get in and now do immediately do something they don't take. They do reconnaissance. I mean, the, these people that we're dealing with now are not a kid sitting in his mom's basement, eating Twinkies. We're talking about people that are pretty professional, that are well financed that look at this as a job. And some of these are nascent nation states back. So we're E we're not getting people that are, these things are compromised long before you know, that they're compromised. Correct.
Speaker 3:Uh, in some cases, it certainly is. I'm not sure in the case of ransomware, um, if the systems are compromised for any length of time, because obviously the, the attackers are looking for a payday. Um, so once, once they're in, if they've, you know, typically I think they probably do a lot of research upfront and understand what kind of systems somebody is using and, you know, do a little probing. Um, but once I think they get in, they do their best to take advantage of that situation. Um, those who are just putting malware out there for the sake of, you know, compromising systems with neces, not necessarily a plan for monetary reward, I think that's where you're right. I think those guys are getting in, you know, those are the really well-funded guys who are, um, aren't understanding for a buck. They get all the money they want from their government or whomever. And, you know, they, they probably set up a lab and have your cut your equipment in there. I mean, look at some of the sh square D Schneider electric things that have happened. There is no way they didn't do that without having that equipment upfront to test and play with. Um, so yeah, I suspect those guys are in the systems for quite a while.
Speaker 2:Well, so if you're a manufacturer, here's a profound question. You're responsible for your control system and your manufacturing floor. Do you have to assume now that your it system is compromised? Is that a reasonable assumption people should make
Speaker 3:It is. Um, I always, when I, when I speak on the topic, I talk about that point where you, you connect those two systems, or it should be multiple points where those two systems connect. So there's no single point of ingress or egress from enterprise to OT. Um, my, one of my comments was I considered their network untrusted and I expected them to consider mine. I'm trusted. Um, in everything I do is with that idea in mind that their network is untrusted.
Speaker 2:I think in some places, people, it, people are going to tell you that you're paranoid, we're secure. We, you know, we spend invest a lot of money and time and security. We're fine. You're being paranoid, Jeff, to answer that,
Speaker 3:Um, I could probably show about 27 cases in the last couple of years, that would have been significant, that would prove them incorrect.
Speaker 2:Well, that's, that's probably true. I think the evidence is certainly certainly shows now that, uh, a lot of it systems have been inactive. I think that it's, it's generally acknowledged that the attackers are ahead of the defenders, as we said before. So let's turn our attention more toward the manufacturing floor. And so what are the, what are the various kinds of different ways that people are, are, are architecting solutions to protecting the manufacturing floor? And let's talk about the advantages and disadvantages of each. So the first one on my list, and you may have more, the first one on my list is these monitoring systems. It seems like there's a number of companies that have these systems that they want to put in infrastructure or software. That's going to look at all of the, all of the messages going around, and it got claim that got special AI algorithms. And they're going to be able to tell you that, that there's something wrong going on with what's the advantages and disadvantages of systems like that.
Speaker 3:Well, the advantage is that you can do some really interesting things with data over time. Uh, AI systems require input to make decisions. So in many cases, putting that type of system on the network for an hour or two hours or three hours is isn't going to do anything for you, but it looks at, you know, lots of data points, uh, over time. And it starts to pick out patterns and starts to look at, you know, um, things that would consider typical. Um, there's, there's a definite advantage in being able to see anomalous behavior, um, that the downside to that is on the OT space. Um, you know, things change a lot until they don't. Um, so, you know, bringing equipment in that's in startup mode for a few months, are you happy year? Um, it's not unexpected to have lots of different things happening on that network. And then once the equipment is up and running in production, um, then it becomes a little more interesting. But so from the day you go into production is really when your data becomes starts becoming useful, because everything that happened up to that point is, is useful, but not as useful. Um, the other, the other problem with those systems, particularly in regard to, uh, existing OT systems, is they're designed to look at traffic flows across the network. Um, so, you know, there's a lot of requirement to span port, which is, you know, take all of the traffic from 15 ports on a switch and pump it out a single port, um, many OT grade switches, even those that are considered managed switches can't handle that level of traffic when you pump all that traffic out, a single port, if that switch uses a central AC versus a per port ASIC, which is the kind of the ASIC is the, the chip that controls the traffic flow. Um, it's, it's, it's going to choke on that traffic and not be able to perform. So, you know, that's a problem having to actually put taps, which is just what it sounds like you're tapping into a network to feed data back up. Um, that's multiple points of failure, potentially. It's a lot of risk because you're affecting a known system. Um, and, and probably the, the, in my mind, the most problematic is, um, if you give an attacker or if an event occurs, it doesn't take long to become a problem if it's planned. So if you're looking at monitoring traffic, um, and your system can't take proactive response to it, meaning other, all it does is let somebody know or throw an alert up on a dashboard, or maybe send out, uh, an email, um, or generate an end of day report. Then there's potentially a lot of time that attacker has to be on the network. And if, if they've already done the damage, when you find out about it, um, great, that gives you the opportunity to mitigate it, but the goal would be to prevent them from doing it in the first place.
Speaker 2:Well, you, you know, I always compare that to you wake up in the morning and the neighbor comes over and says, Hey, Jeff, I saw somebody breaking in your garage last night. I just wanted to let you know. Yeah. Well, why didn't you let me know last night, come over and stop. You know, it doesn't do any good to know that eight hours later. Yeah.
Speaker 3:So monitoring is definitely important, but if, if it's only half of the equation, if you can't monitor and take action or be proactive, um, then it leaves a lot larger attack surface.
Speaker 2:Yeah. So the other, the other big solution that being proposed, and I hear that there's a lot of momentum behind it is a device security where you're going to embed a security chip inside every single device on the manufacturing floor. And it's going to encrypt all the IO traffic it's going to do authentication and authorization and all of those, all of those things. And that's the way the future. And that's going to be, that's going to be our salvation on the manufacturing floor. And of course the biggest one of those is social security that, that Allen Bradley is, is try and ODBA are pushing. What's the, what's the advantages and disadvantages of device security solutions. Jeff.
Speaker 3:So that's a great question. And, you know, it's, it's, um, it's a risk versus reward versus effort to manage scenario. Um, in, in theory, if every device was in, did an encrypted communication back to a, you know, a PLC denial modules, um, that'd be great. It would have to be the same across every vendor, every device. So any of these, you know, so if I'm using Siemens, Rockwell, you know, sip, profi, mod bus, um, what's the likelihood that it's going to all work the same, not, um, more importantly, if, if you're doing that type of thing, it's probably some type of certificate based exchange. Um, and I, I can't imagine, and this is a lot of my colleagues in the field will tell you the same thing. Cause we've talked about this for awhile. Um, trying to get some of our OT staff who are very capable in many ways, but are not necessarily network or security savvy at two o'clock in the morning. Um, I can't ever remember having an electrician who would have been skilled or trained enough to do a certificate exchange between an IO module and a PLC to get that encrypted communication going. Um, that that's kind of a big problem. So, you know, it's gonna cost a tremendous amount of money to have staff on hand who can do that type of thing. Um, and then of course, like I said, if it's, if it's Rockwell, they do it one way with sip security. If it Siemens, they do it another way. If it's, you know, mod bus and Schneider a different way. Right,
Speaker 2:Right. And there's lots of plants. Of course there are no, they typically use Allen Bradley, but they buy a machine that shows up with Siemens controllers on it. So now are you going to support and man, and try to manage two different kinds of device security systems? Yeah. I don't see it. And the other thing, and of course the big momentum here, there's a lot of people selling chips. Intel has a security solution. I heard that Amazon has a security chip. And I mean, if I was in the chip business, certainly I would want to have my children, every device in the world. There's a lot of money there in, in these systems. And like you said, the big, the big problems I see is that the operational impact of trying to manage these, oh, this device security is one problem. The other problem is we've got, we've got so much old stuff. If you have 200 devices in your sub-debt and 199 of them are protected with sip security and the other one isn't have you achieved anything?
Speaker 3:Well, that's a great point. I mean, we've got, we've got a customer we're working with right now that is potentially going to go try to retrofit security into 29 plants. Um, and you know, we're talking about four or 5,000 edge devices endpoints on all of them per plant. So, you know, 250,000 end points, um, is going to replace all those. Um, nobody is going to take on the risk of this type of certificate base. And, uh, frankly, um, you know, I used to talk about security. People try to eat, eat an elephant. You can't do it one bite at a time. You can't do the whole thing at once. Um, there's so many other things that people need to worry about before they worry about encrypting IO. I mean, so it's a, it's an IO module that, that drives a couple of discreet IO points and maybe has a read only webpage on it. What, what's the, what's the risk of that being compromised? And more importantly, what's the risk? What, what's the end effect of that being compromised? Probably now, if I'm in a nuclear facility, my level of risk tolerance is completely different than if I'm in a traditional OT, manufacturing, um, environment. It's, it's, it's all about what, what risk I'm willing to live with. And in that instance, it's just tremendous overkill.
Speaker 2:Well, and another thing that I see too, is I'm thinking if I'm a manufacturer, I can buy a valve block that has no security for X dollars, and I can buy a bell block that has device security. It's going to cost me 1.5 X or 1.2, five X. Well, Jesus, what do I do? I really need that. Am I going to my, going to spend extra money on every device on a new line to do that and increase the cost of my machine by 50%? I mean, it's like buying an insurance who it's, it's a really hard sell to sell that insurance. And, you know, we want to, everybody wants to buy security, but nobody wants to pay for it. Nobody really wants to manage it either. So device security just seems to me incredibly problematic and I can't see that working. And I've been saying that, and it's got me into a little bit of trouble with ODB and Rockwell, but that's life. Let's talk. Let's talk about another, another way to do this is perimeter security. Um, tell, tell us what, what is, what, what is, what do we mean by perimeter security and what are the advantages and disadvantages of that?
Speaker 3:So perimeter security is essentially, um, I, I refer to it as capping the bottle. So you're, you're, you're taking those points, those perimeters of your network, where you touch other systems, um, and you're putting in place something that will, uh, protect the traffic coming in and out, excuse me, allow traffic to go only a single direction. Um, so it's, it's, it's a way to encapsulate the network within a security perimeter, if you will. Um, you know, if you look at things like we've done in the past with robots and motion perimeters, it's the same concept, right? You're, you're, you're effectively trying to create a secure bubble around a particular piece of your network.
Speaker 2:What are the advantages of, of that? And some of the disadvantages of doing stuff like that. I mean, people don't want to do that. You know, you're getting a device right now say a drive that has one RJ, 45, that's going to talk either another profile and another, another port that's gonna that's for it to access, to get the energy data out of the drive. I mean, how do you, how do you do perimeter security when device manufacturers are building products like that?
Speaker 3:Well, yeah, so a dual homing, anything is a bad idea from a security perspective, meaning I have one connection that goes to this network and one connection. It goes to another network. Um, particularly if you're dual homing a PC, um, because with, with very little effort, I can bridge both of those networks through that PC or device. So, um, most security folks will tell you any kind of dual home is a bad plan. Um, so, you know, in, in, and I w you haven't really brought this up yet, but if, if I were to do, if I were, if someone were to tell me, what's my optimal solution for a plant floor on, on existing equipment, I would say go with a solid perimeter security solution. Um, it's going to be the least effective, least expensive, most effective if it's the right tool and the right product. Um, and it's going to, it's going to take you 80% of the, where you, where you think you should be security wise. And that first 80% is great. I mean, the last 20%, every couple of percent will cost you with the first 80% cost in terms of, um, implementation and management. So, you know, you have to really determine what your level of tolerance for risk is. Um, and then, then there's another technology that's up and coming, which is called software defined networking. Um, and if you, if you take solid perimeter security and you do software defined networking within that perimeter, um, that's really just about as secure as I can see something getting these days.
Speaker 2:I think we're going to have to, we're going to have another talk at some point in the future about SDN. Um, what about, what do you with the vendors who are coming in and they want to tune up the robot or check out the welding system, or, you know, upgrade stuff to do they get access to the, to the network. Are they going to come in through the, through this standard perimeter between how do you do that? How do you do that?
Speaker 3:It should definitely be managed through the perimeter, but too often, people are allowed to walk up, plug their laptop into, you know, a convenience port on the front of the panel. And it just puts them on the local network where they can get to everything and see everything, um, in it's less likely with, you know, PLC based systems, but it's more likely with systems that have any kind of PCs on it that that's going to be a problem. Um, simply because anything, any malware that might be on their, on their system now can go out and poke around those PCs and potentially affect them. And I've seen that many times, um, as well as, uh, if you put a, you know, if it's a PC and it's already on the plant floor and you have a supplier community pops as a USB key into that PC, um, that's another ingress point, uh, to the network.
Speaker 2:Well, if you've got a PC on the factory floor and it's running Microsoft, some version of Microsoft windows, can you do, how do you do security up based? So if you've got your perimeter security set up so that okay, that PC needs to communicate from this source to this destination once a day, and then it allows that, but now that now that we've got a windows update or are you just freeze that freeze that PC in place and not do updates, how do you manage those PCs on, uh, in the, in the, in a control system?
Speaker 3:So that's, that's a great, that's a great question. Um, there's no, there's not an easy answer to that. I will tell you what I've done in the past. That's proven very successful, um, using a tool that basically wait-lists the PO the PC is a great way to go. Um, it prevents things from being, you know, you might get some problematic piece of software into Ram, but as soon as that PC is rebooted, it's gone because it can't write itself to the disc can infect the executable. It can't do all the things that it wants to do. Um, but even more. So the other side of that coin is I don't put PCs directly on controls networks. Um, I prefer single home. So if my PC has to connect to the enterprise and it has to connect to the, uh, OT network or the, the controls network, if you will, um, I put them on the it network or the enterprise network because the enterprise network are designed to handle things like patching and OSTP updates, things of that nature and virus definition, updates. And then if you're using the right type of security appliance, those PCs can all connect back into that OT network from the enterprise using a secure tunnel. Um, and then it solves a number of problems and gives them, gives you a very specific control of what those PCs are allowed to access on the controls network. Whereas if you just drop them on the controls network and plug the cable in, they can get to anything anywhere.
Speaker 2:So you're really, you're really much more secure by putting them on the, on the it network. They can get, they can get their standard security updates, and then you can just control what they can do through the perimeter security device. Yep. Great idea. I liked that a lot. I, I hadn't, I hadn't considered that before, so, well, uh, we're coming to the end of the session here. Uh, next, our next call, Jeff, we're going to talk about the IC, the perimeter security device that, uh, you guys at Dynex have built the ICS defender, which I have been, uh, I'm incredibly enthusiastic about. I think it's got incredible capabilities, and if you follow the, the, uh, uh, link that's associated with this podcast, you can find out more about the ICS defender. And we're going to talk more about that in depth and, uh, on our next call. So thanks very much, Jeff. This has been a great call. I enjoyed it. I learned, I learned a couple of things that I didn't know as I always do when I talk to you, I want you to have a great day. Thanks,
Speaker 3:John. And you too. Appreciate you letting me participate.[inaudible].