John S. Rinaldi's Ode to Automation
John S. Rinaldi's Ode to Automation
4 – Defending Your Factory Floor from Cybersecurity Attacks - Part 1 with Jeff Smith of Dynics
Industrial cybersecurity breaches continue to gain traction every year, yet the manufacturing world still seems behind when it comes to defending their factory floor before an attack happens.
Join host John S. Rinaldi as he and guest Jeff Smith, CTO at Dynics, discuss what manufacturers need to be doing to secure their factory floors, including:
- Justifying the expense to small manufacturers to invest in their security infrastructure
- IoT concerns for security, especially when pushing data to the Cloud
- Why the ICS-Defender is the best security device for the plant floor
Connect with John:
Email | LinkedIn | Newsletter | YouTube | Blog | Books
For all things industrial automation, visit rtautomation.com.
[ Intro music] Hello, this is John Rinaldi, and welcome to another network connectivity podcast. Today, we're going to talk about manufacturing, security, something that's been on my mind a lot. And I think it's on the line of a lot of people in the, in manufacturing, a lot of stuff in the news all the time about different companies being attacked. There's a lot of what I think is misinformation and today we've got a guest that can help us clear all that up. Welcome Jeff.
Jeff Smith:Yeah. Good morning. Thanks John.
John Rinaldi:So Jeff is a, uh, uh, as the CTO for Dynics security company in Ann Arbor, Michigan, and has a significant history in this area. So Jeff why don't you just kinda introduce yourself and tell us a little bit about who you are and what you do.
Jeff Smith:Sure. Thanks. Uh, as John mentioned, I'm the CTO with Dynics. We have a suite of security products for the OT space. Um, I've been in the automation and controls and security space for probably 25 or 30 years now, um, prior to coming to Dynics, I was a VP of a cloud services company that did a lot of data collection with the floor. Um, and then prior to that, I was with an, uh, with a major tier one automotive as a senior manager of, uh, controls technology globally. And that's kind of where I started down the road of, um, OT security back in'06, we went hardcore EtherNet/IP on the plant floor. Um, so to say we were on the bleeding edge of the technology might be an understatement. Um, it became very apparent to us within the first couple of years that, uh, we needed to start paying attention to security. So I think we were ahead of the curve a little bit on that, and we had our security plan in place probably by 2011, 2012 and it's still in place today.
John Rinaldi:That was, uh, that was pretty early. I don't think anybody was paying attention much attention to security back then, at least on the manufacturing floor, certainly on the IT side.
Jeff Smith:No, no, they really weren't. And it's been interesting to watch the progression of what people find important, you know, kind of from 2009 to 2012, took us two to three years to really put together the plan that we had, um, working with a lot of our major vendors from the PLC and switching side of the business and ended up, we really couldn't get the solution we wanted because not only were customers not thinking about it, uh, the manufacturers weren't really thinking about it or that they, you know, people building the PLCs. So we ended up designing our own solution. Um, and actually that solution has stayed with me and is now part of the Dynics defender platform only significantly more advanced than it was back in the day.
John Rinaldi:Tell me, let's start with kind of an overview level. So if you're a manufacturer, say mid-size manufacturer a couple hundred people, you know, one, one manufacturing plant, a couple of two, three lines, you don't have a big place. How worried should you be? What's the situation?
Jeff Smith:Um, it's typically you'll hear a lot of people say it's not a matter of if it's a matter of when, um, I think everyone's risk is different. Interestingly, most folks think that for example, the large automotives or the big companies are going to be the targets when in reality, it, it tends to be those middle tier suppliers. Um, if I can take down a supplier, it's a critical manufacturer for a Ford F-150, for example, um, not only is that company willing to pay me to get their production capability back, probably their end customer is going to be hurt so badly that they will do the same. So it's companies like that that have typically less security than anybody else. And they're actually at higher risk. I mean, am I going to go up against the Ford or a GM if I'm trying to do something nefarious, probably not really. Um, could I go up against a tier one automotive or, you know, a tiered supplier and that size, or even a tier two who feeds a critical product to one of those larger companies? Absolutely. So at the end of the day, I get what I want.
John Rinaldi:So where did, how did these attacks come in? How did, uh, well, what kind of attacks we talking about? And we hear the terms worms, we hear malware ransomware. What, what kind of stuff, what do you see as the primary threats and what does, how does this stuff work?
Jeff Smith:Well, it it's really, um, that's a great question and it's really, you know, they, they tend to, um, extol the, the problems of, you know, malware and worms. Um, and there's, there's lots of ways that can happen. Fishing is a huge problem, uh, pH phishing, where they'll send in a malware link or a link and people click on it. So, you know, fortunately that typically has to come through the IT infrastructure. So that's less of a problem, um, you know, dropping USB keys and random places and allowing them to be plugged in, in your plant is a huge problem. Uh, and just, uh, sometimes it's a lack of understanding of how to deal with that. I can give you an example of a customer who had a particular piece of malware infecting a bunch of their PCs that were running steeplechase. To solve the problem, they downloaded the patch, put it on a USB, he plugged it into that PC loaded the patch to that PC then unplugged the USB key and went to 30 other PCs. And any of those 30 PCs that weren't infected were now, because when they plugged it into that PC, initially not following proper procedure, uh, it infected the USB key. Wasn't a read only key. Like it should've been, you can buy keys that you have a physical switch that makes them read only. Um, and so, you know, just lack of knowledge if it allows us to propagate as well, so...
John Rinaldi:Would you recommend that you bring up the USB, would you recommend that manufacturers just not have any USB on their manufacturing floor at all? Do you think you have, do we need to go that far?
Jeff Smith:I don't think we need to go that far. I just think we need to make sure that people understand some process around that. And then, and we're careful about how we do it. Um, you know, th the difference between that solving the problem and causing more problem was simply lack of knowledge on the part of the person in the plant.
John Rinaldi:And when we're talking about these kinds of plants, as you know, and I know it's usually, there's just one, one guy, he's the guy that takes care of anything. If the drive goes down, they call him. If the PLC's got a problem, they call him. If the, if the printer in the office doesn't work, they call him, he doesn't have to, you don't really have time for this stuff. So that's, that's I think is the big problem is that he's not focusing on it because he's so damn busy.
Jeff Smith:Yeah. And that's because a lot of these companies don't realize that that really needs to be an area that they have some dedicated resource or someone who is significantly able to help them do that. Now whether they have somebody on staff or they bring in a consultant or a services group, um, to at least do some of that, uh, I think in most, you know, most don't in even problems with, uh, you know, one of the most common problems in the manufacturing space is, um, errors by actual associates of the company on the plant floor, doing things that without any ill intent at all ends up being a problem that, you know, a good cybersecurity policy or practice could have prevented.
John Rinaldi:Misconfiguring a managed switch, for example, that kind of stuff. Yep. It does. Everything still works. You just have, you've got a security hole. You don't know you created well.
Jeff Smith:Yeah. And I mean, a great example is denied by default. Um, with our products, we specifically make everything denied by default while many of our competitors or other companies will make their products, you know, allow by default. And the problem is if your machine breaks, the scenario is the maintenance guy goes out, replaces the, whatever it is that broke. Um, if it happens to be the security appliance, they put it in. If it's allow all, they think they're done and happy because the machine starts working the, uh, agent working. But what they've actually done is put a completely unconfigured device and opened a huge hole. I had customers who wanted to go with data diode simply because they, they didn't get that you could put a denied by default device in there, and then it only works when you configure it.
John Rinaldi:Right. Now, going back to something you said a few minutes ago about spending money to have someone dedicated to, to, to be looking at the security problem. I think what we're talking about there though is, is insurance and insurance is a really hard sell because you just know ROI from something that didn't happen. So how do you, how does this customer, how does a small manufacturer justify that expense?
Jeff Smith:Uh, they, they probably can't justify it, you know, from a dollars and cents perspective until they have a problem. But if you look at the, if you put value on the public appearance, that, that, or the, the, the dent it puts in your public appearance, if all of a sudden all of their information, data or production capability is exposed, or if they lose the ability to build for awhile and they look at other companies who have had that same issue, it's not a very difficult extra to put a dollar amount on that. I mean, they insure their businesses. They insure their buildings. They insure, you know, they have for a lot of reasons. And so, while I'm not suggesting that, um, they need to run out and buy cybersecurity insurance, they need to consider that as a part of the cost of doing business. And, you know, when you've got nation states like China and the Soviet union and other countries who are actively looking for ways to compromise US company security, um, that's, it's a matter of time. It's not a matter of if it's simply a matter of time.
John Rinaldi:So this is just a, it's just an ongoing business expense that you have to accept, just like you got to pay for the electricity, you really need to pay for the cyber security?
Jeff Smith:Yeah. I, I can't tell you how many times a customer has come back to me and said, um, we paid for every single ICS defender we ever bought last week. Um, because they were hit by some particular piece of malware, which took down their ability to manufacture every place they didn't have one.
John Rinaldi:Um, so, so how do those attacks at least attacks mostly coming in through the IT system? Like you talked about before somebody in the back office clicks on a, on a, on a email that looks like it's coming from UPS, uh, and then they, and all of a sudden now their credentials are compromised. And then somehow the attacker finds a way into the manufacturing system from IT. Is that the typical route?
Jeff Smith:Um, it it's a common route, uh, in, in, you know, believe it or not, John, there are still companies who put their PLCs and various control systems on the internet so they can work on them from home. So they literally expose their control systems. Um, it all you'd have to do is put up a honeypot for about a week. Um, you know, just something that exposed a Modbus based PLC or an EtherNet/IP based PLC to the internet, and then log how many attempts to compromise it. It would be probably in the hundreds or thousands.
John Rinaldi:Explain the term honeypot for people who may not have heard that term.
Jeff Smith:So a honeypot is, um, it's a system that you expose to the internet purposefully and you have, you know, protections around it. So nothing nefarious can actually happen. And what you do is you, you watch what people try to do. You watch the people trying to get in, you watch them, and you look at the different ways and you can monitor the different ways that they try to compromise the system. So it's basically carrot and you've dangled it out into the internet. And then you kind of sit back and see who does what and how effective they can be.
John Rinaldi:Wow. That's pretty, that's pretty interesting. So who's behind this stuff. You, you talked about nation states, so it's, it's not the hacker, it's not the kid in the, in his mother's basement. Uh, that's trying to do this stuff. Are you saying it's actually very sophistic that these attackers are very sophisticated people now?
Jeff Smith:Yeah, it's, uh, they are, and, you know, 10 years ago, five years ago, there was still a problem to a degree, but it wasn't, um, it wasn't to the degree it is now, and it wasn't nearly as sophisticated as it is now. Um, you always have to worry about everybody. Fortunately, the kid at home, um, probably doesn't have any ill intent other than he wants to know if he can get in, um, other folks that are actually, you know, take a look at some of the PLC problems that we've had in the last couple of years, compromises, where they actually were able to reverse engineer how the firmware interacted with the runtime operating system, and then they c ould flip bits in the runtime operating system that affected the firmware, which then allowed them to flash the firmware. U m, that's not somebody at home, that's not a kid sitting at home. That's someone who has that equipment in that lab has set it up and is actively working to get around it.
John Rinaldi:Um, that's, well-funded, that's, well-funded, that's, that's people who have incredible amount of expertise said that that's the China's and North Korea and Iran that do that kind of stuff. Right?
Jeff Smith:It certainly is, and it's even, you know, it's government funding groups, it's, it's other types of organizations. Um, that's not to say that, you know, another big issue isn't just errors or mistakes or issues that happen internally. I mean, that's, that's the most obvious probably the most frequent, or at least it was where people make mistakes inside that cause problems, and you know, so it's, it's not that you're blocking a, you know, an attempted attack. It's simply somebody made a mistake and proper cybersecurity or an attempt to cybersecurity on the plant floor would have prevented it. I was gonna say another thing that, that I talk about with people a lot is, um, you know, a few years ago after the, the, uh, attacks on Washington and the airplanes and, um, September 11th, they started deploying cockpit doors and airplanes that can't be breached. Um, and so that's a hardware solution to that problem. And what I commonly tell controls folks, or I ask them when I'm speaking, I'll ask a room full of people. How many of you had reviewed your electrical drawings and prints with an eye towards security? Have you ever considered that, you know, with everything in the control system is so heavily software based now, and now they're even talking about running controls from the cloud. Um, have you ever given consideration to the fact that if I have a simple, uh, pressure sensor, which ties to a shutdown physically and electrically, it's a very inexpensive solution, but it doesn't matter what the software tries to do. So if someone saw compromises your communications and your software, if you have a physical, um, preventative device, there in the plant floor, because it's going to stop them, you have to have physical proximity to that device to overwrite it. And to do that in conjunction with a modification to software communications is, is, is virtually never impossible, but very, very difficult to compromise. So control systems and OT. We have, uh, we have a cockpit door if you will, that the IT space doesn't have, they, they can't do that. It seems in the OT space. We've forgotten that we can't.
John Rinaldi:I talk about that a lot. Absolutely. The big difference, big advantage we have with manufacturers, we know all the traffic that should be happening every minute, minute by minute. And the IT side, they have no idea, you know, somebody walking in, starts a new application, adds a new computer. Everything's changing second by second, but we don't have it doesn't happen on the manufacturing side.
Jeff Smith:No, you know, I, I was, I say this something similar, you know, the, IT space has to protect against what they suspect to be bad while letting everything through in the OT space. We shouldn't let anything through except that, which we know to be good.
John Rinaldi:Yeah, exactly. And that that's a key to the success and something we'll, we'll be talking about in a future conversation. What is it, what are the devices on the manufacturing floor that we should be worrying about? Is it isn't the PLS? Is it the controllers, the PLCs? Is it the end devices, valves and, and, uh, drives and other and linear actuators? Or is it the actual infrastructure stuff like switches and things and routers, or is it all of it? What should manufacturers be really be focused on?
Jeff Smith:Uh, that's going to depend upon, um, what someone's appetite for risk is, um, uh, the easy answer is everything, but, you know, w when you look at mitigating the risk, it's important to look at what you have on the plant floor and where you believe that risk to be. Um, I always ask folks how much cyber or how much security is enough. Um, and I get a lot of interesting answers in my answer to that question. After thinking about it for a lot of years is you, you need whatever security you need to mitigate the risk you're not willing to live with. So identifying that risk is key. You know, when people first started going cyber security on the OT space, you know, after we, after we did, but they were talking about, well, we need to lock out all web browsers on IO modules because that's, you know, that's a security risk. Okay. Is it really, um, if you have a read only what a web server that someone can use on an IO module to troubleshoot it is there is the risk of that read only web server being compromised to that one IO module with a couple of IO points is what's, what's the likely outcome of that being compromised. Is there an outcome that would be detrimental that couldn't be resolved quickly? And what is the cost of not having that diagnostics tool, um, available to maintenance and troubleshooting people to keep production going?
John Rinaldi:There you go. And that's exactly what you lose when you do things like encrypting messages and the manufacturing control system is now you lose the ability to wire sharp stuff and try to understand what's going wrong and what happened because you can't see anything then. So, you know, there's a, yes, you encrypt, it makes it much more difficult for an attacker to understand what's going on, but it also makes you very, your job much more difficult to try to troubleshoot that control system.
Jeff Smith:Yep. And, um, you know, the, it also makes it, I mean, if you look at the maintenance guys that we've had, I've, I've worked with some various sharp maintenance guys. The problem is those maintenance folks have 57 different things every night that they're working on in troubleshooting. So now you introduce an IO module to the network where they have to arrange a certificate exchange between that IO module and the controller before the controller will talk to that IO module. And that happens to work, you know, one way on product A and it works completely differently on product B and product C um, their ability to keep that system up and running, um, becomes extremely difficult in the level of skillset that you have to imbue into that person who was just out there troubleshooting drives the day before. Um, and the cost is tremendous.
John Rinaldi:But you've touched on it. And that's a huge problem now is getting those people with those kinds of skill sets that understand what a linear actuator is and how a valve works and also understand networking and also understand IT stuff and security. I mean, where do you find these kinds of people?
Jeff Smith:Yeah. They're there they're few and far between, or they're, they're, they're burst by fire.
John Rinaldi:Right. Yes. They, they have, they have to have a lot of experience to do that. What does that now, what about the big concern here is that IOT? So how do you, you know, people are at want to add a lot more sensors. They want to add a lot more data collection and grab a lot more data is that's increasing the security risk, isn't it?
Jeff Smith:Uh, significantly, uh, even more so if they try to push that sensor data to a cloud.
John Rinaldi:Right. How does that, you know, where does, what's the, what's the real risk there? What, what, what, uh, how does that increase the, the, what do you call it? The, a threat spacer, I think it's called, right?
Jeff Smith:Well, think, think about it this way. Take a paper plate and punch 67 holes in the paper plate, and then pour a couple of handful of sands over it and see how much of that sand gets through, um, take that same paper plate, punch one hole in the middle for the same amount of sand on it and see how much it gets through. Um, it's, that's exactly what they're doing. They're punching holes in their security and their infrastructure. So they they're exponentially increasing the attack surface on their enterprise, um, products that do that need to use a proxy. So internally on their local networks, they all talk back to a proxy sitting up in a server room, and then the server room proxy talks to the cloud. So you have one place to look at one place to worry about one place to secure. Um, and if there's a problem, it's easier to cap a bottle, um, than it is to run around and try to cap 67 bottles.
John Rinaldi:Well, this is one of the, the discussions I have a lot with, with people is where should this traffic go? It should IOT traffic beyond the controls network, or should IOT, traffic have its own access to the, to the internet and the cloud and such. And, um, where do you stand in, in, in that kind of area in that business?
Jeff Smith:Um, I, I am not in favor of IOT traffic having direct access to the internet. Um, great, great example of that is a few years ago, and I'll, I'll be pretty vague with the details. Um, one of the casinos in Vegas bought a large aquarium and the aquarium was monitored with some IOT type sensors back to the cloud service company that provided the aquarium monitoring. They also had a secondary connection to that system from the local casinos and someone hacked that cloud service, rode that sensor connection back down to the aquarium, went from the aquarium and jumped into the casino system and made off with a lot of money. Wow. Um, so, um, I am not a fan of a separate network like that. I think it's, um, it's, again, it's, it's keep all the traffic inside and only allow one point of ingress or egress.
John Rinaldi:Well, now we have, because of the pandemic, we have a lot more people working at home and a lot of these people need to access manufacturing systems and they're using VPN VPNs to do that. How secure is that?
Jeff Smith:Um, done properly? I mean, we've been using VPNs for a lot of years. Um, you know, there's SSL, VPN is probably the most common, um, and done properly its secure. Um, it's an encrypted tunnel. Um, so depending on the level of encryption you're using, um, it's, it can be very, very secure if you take stuff default out of the box as is hopefully whoever designed the solution you're using, um, used something that wasn't, you know, that wasn't compromised seven years ago already, uh, and actually, um, put some thought into what their default setting should be and should look like, but, you know, by and large it's, it should be fine.
John Rinaldi:But there isn't a real risk that Joe sitting at home on his home desktop or his own laptop and attackers said go to Joe's house breakthrough, that security to steal his credentials and now they can use those credentials to get into the manufacturing system, isn't that a pretty big risk?
Jeff Smith:It is a risk. And that goes back to, um, uh, really you shouldn't be allowing personal computers on your enterprise network. I always recommend that. Um, one of the checks that your VPN does, if it's a corporate VPN, for example, is to make sure that your laptop, that you're connecting with or PC that you're connecting with, not only does it have a certificate and everything else that has to go along with that, it has to be joined to the domain. So if that PC is not joined to the domain, it can't connect.
John Rinaldi:All right. So for the people that are, that don't have an IT background, explain what joined to the domain means?
Jeff Smith:So most computers, if you're running Microsoft windows have, um, it it's the default quote unquote work group, and it's whatever every computer belongs to kind of out of the box, uh, in, in most enterprise, in most companies, they have, um, something they call a domain, which allows them to control and encapsulate access. Um, and you have to actually tell your windows PC or configure it in such a way that, that domain, that server at the company side recognizes that laptop or that PC as a part of its domain. It has a membership there. So think of it as, um, I think I'm thinking about like this it's, it's your membership to the health club. When you walk into the health club, uh, they want to see your membership card before they'll even let you in the door. The same thing is true. If you connect your PC through a VPN and you try to log in to the, to, um, your corporate domain or you're using your corporate credentials, um, not only does it credentials have to be correct, you have to have the appropriate certificate on your machine, and it has to be joined to that domain. And the only way we allow here at Dynics, for example, the only way you can join a PC to the domain is the PC has to be on premise here and physically hard-lined in that's the only way we allow a PC to be joined. So any PC that goes to somebody's home laptop wise, for example, while they're working from home, we know was on premise, was joined by someone with the proper authority, has the appropriate endpoint protection on it. Um, and has everything is as secure as we can possibly make it.
John Rinaldi:So they, so you have to, at some point, bring that you're, you're using a home laptop to access the Dynics system. You have to, you have to, you have to bring it into the, into the, into the walls of the building and get it joined to the domain before you can actually take it home and use it at home. Is that true?
Jeff Smith:Right, but we don't generally allow you to use your quote unquote, home computer on our network.
John Rinaldi:Oh, okay, so you provide a company laptop essentially?
Jeff Smith:Yeah, we provide a laptop.
John Rinaldi:All right. And that's the only laptop that they can use to, to access the notes that the access?
Jeff Smith:If following proper procedures, yes.
John Rinaldi:So that's, that's, so that's double security. If you're just using a regular VPN and you're using your home laptop, that's a much higher risk than what you're describing there?
Jeff Smith:Yes.
John Rinaldi:Very interesting.
Jeff Smith:Another interesting thing. And I'm going to plug a company that we work with, um, that we've done a lot with, even with some RPEs with US cyber command is a company called centripetal networks. Um, they sit on the ISP side of an enterprise network. So the inbound side of an enterprise network. And in real time, they can look at the traffic coming in and out of your organization with no degradation in performance. And they're constantly looking at that traffic and comparing it against 80 or 90 sites worth of registered malware signatures. And, um, they it's called a rule gate and it's there on the northbound side of the enterprise. We live on the southbound side, which makes for a good partnership. But when you're looking at an org, you're protecting your organization, that's, that's think of it as deep packet inspection for your ISP.
John Rinaldi:Very cool. I don't think I knew about them. Yeah. Um, well, we're, we're, we're reaching the end of the podcast. Um, wanted to thank you very much. I thought this was really interesting time went fast or had a great discussion. So we're going to, we're going to continue our discussion with Jeff in a future podcast. And before I leave, I just wanted to talk a little bit about the, the products that Dynics sells is ICS Defender. Real Time Automation is now a reseller for that product because we looked at the market, saw that our customers really need a security solution. And Dynics has ICS Defender has an incredibly rich set of important features. It's built by control engineers for control engineers, Jeff designed it, and he's really noticed he was, he was out there. He was one of the people out on the manufacturing floor with the problem. And it's proven technology, as Jeff said, the, uh, you know, I don't know, we didn't talk much about it, but they were tested cyber security tests by the US cyber command. And they were one of the few vendors that were selected that actually passed the test. So, uh, that was pretty wonderful. Um, pretty awesome. So, uh, we'll be talking again and thank you very much for listening and look, talk to you on the next podcast. Have a great day.[ End music].