skoop.dev

One (but I’m sure far from the only) thing where business and technology clash is the topic of security. A lot of the projects that we do for clients are based on Joomla! (previously Mambo) technology. Aside from building the site or application, we also offer a SLA for managing the software and, for instance, installing security updates. Some clients choose to do this, others don’t. And so you’re left with insecure software on your servers.

Two weeks ago, we were hacked. Unfortunately, due to the logging policy of the company we rent our servers from, we could not check the access logs of the moment we got hacked to see how they got in, but we suspect they got in through a security vulnerability in one of those old components of a client that does not have a SLA. However, even clients with a SLA were affected. Why? Because every time you upload a file (something that in Joomla! is common even for component installation), the files are stored on the server with the owner ‘www-data’, the Apache user. And so if a hacker gets in through a vulnerable PHP script, which is executed by Apache, it can write to every single file writable for the Apache user.

As we found out about the hack, we immediately started to clean the mess and figure out what had happened. The latter turned out to be impossible, but the former was of course possible and one hell of a job. It turned out that the “hacker” was really a spammer, who was earning money by getting impression on a site called Free20, by adding his referral ID to a url in an iframe that he appended to all writable PHP, HTML, Javascript and other similar textfiles. Result: Most sites were giving errors and showing tons of iframes instead of the site it should be showing.

Luckily, as we found out, the spammer used only two or three different iframe strings, so pretty quickly we were able to write a script that grep-ed for the “free20” string, and then went through all affected files to replace the used strings by an empty string. All in all though, that was a pretty useless Sunday evening to spend of course.

But here we thread on dangerous territory. Of course, from a technical standpoint, every time a security patch comes out, this should be installed on all sites that are vulnerable that you manage. But this is where the business comes in. They of course want the client to pay for the time that we spend on this patching. And this is understandable: Time spent on installing security patches is time not spent on projects that are bringing in money. But by not installing it, two expensive senior developers spend their Sunday cleaning up things. And in this case, I immediately also set out to do a full week of assessing other dangers to the server integrity. So that’s expensive time that could’ve otherwise been spent on those paying projects as well. Also, even sites of clients with a SLA were affected. And it’s hard to sell to those clients that even though they have a SLA and their sites are kept secure, they got affected by a hack.

So at this moment, the server is clean and we have a clear view on what could be seen as a threat to our security. We are in the process of getting a different server setup and clear procedures on the deployment of sites. Yet still the problem remains: How to handle the SLA vs non-SLA clients. Our solution at this time is simply to have one server dedicated to SLA clients. All projects running on this server can be deemed secure, and the changes of hackers getting into this server are very slim, unless we have unknown vulnerabilities in our code. The other server will contain sites of clients that do not have a SLA. We clearly notify them of their risks, and that they will have to pay if ever the server gets hacked and they want us to clean it up. That way, the client will know of the risk and willingly take it, and in the case of a hack on this server, we will be able to get paid for cleaning it up.

If there is anyone here with similar experiences or with other solutions to this problem, I’d gladly hear of them.