This article was written in collaboration with Dutch Digital Agencies.
Wednesday, October 21, was an all-hands-on-deck day for digital agencies and hosting providers. Two critical vulnerabilities in PHP required far-reaching decisions to be made in a short period of time. Websites and applications were taken offline as a precaution, while teams worked late into the night to secure systems.
For Shock Media and Friday, these were exceptional hours. Shock Media is one of the larger managed hosting providers in the Netherlands. Friday is a digital agency with more than 120 specialists in strategy, development, data, AI and marketing, working for major clients in sectors including healthcare and automotive.
Justin aan de Stegge from Shock Media and Frank Klein Koerkamp, CTO at Friday, look back on the events, the difficult decisions and their collaboration under pressure. They also discuss the challenges surrounding security, responsibilities and technological dependency.
A day full of far-reaching decisions
At Shock Media, an incident report was opened at eight o’clock on Wednesday morning. The Incident Response Team gathered and immediately implemented security measures. Through Active Response, Shock Media’s security solution, attempts to exploit the vulnerabilities were actively monitored and blocked. Over the course of the afternoon, the number of attempts increased dramatically. At the same time, Shock Media kept customers informed via the status page and email.
Friday also activated an incident team. Frank coordinated the response and consulted with management about the measures to be taken. During the morning, Friday decided to actively advise customers to temporarily take their websites and applications offline.
“We looked at which systems our customers were running and what data was being processed in them. We gave particularly strong advice to customers handling sensitive data to take measures. But the final decision remained with the customer. We did not take anything offline without permission,” Frank explains.
Shock Media eventually decided to temporarily disable PHP to eliminate the risk of exploitation. As soon as the official security updates became available, they were rolled out and the environments could safely be made available again.
Security versus continuity
For digital agencies, a security incident is not just a technical issue. It directly affects their customers’ business operations. An online store that temporarily goes offline loses revenue. Critical business applications can bring entire processes to a standstill. At the same time, the risk of staying online can be much greater.
This made the decision to shut down systems as a precaution a difficult one. “As a hosting provider, we cannot always foresee the consequences of a website or application going offline,” Justin explains. “It could be an online store, but also a municipal website or a healthcare portal. That’s why it’s important for customers to understand the risks they face and make their own assessment.”
For some customers, going completely offline was not an option. Friday therefore looked for solutions that would allow essential business processes to continue running while publicly accessible components were restricted. For example, employees at one hotel customer were able to retain access to the systems they needed for daily operations.
The incident also highlighted how important it is to know in advance which technologies you use and which customers could be affected by a vulnerability. Friday uses dependency tracking for this purpose, which keeps track of the software versions running in customer environments. “You shouldn’t wait until an incident occurs to find out which systems you’re running,” Frank says. “We were able to quickly determine which customers were affected and which measures were necessary.”
What if a customer postpones updates for years?
Another issue that came to light during the incident was the maintenance of existing systems. Shock Media has a strict update policy, but sees customers sometimes postponing essential updates for years. For example, because of the costs involved or because outdated systems are difficult to modify.
According to Justin, this becomes a problem when a critical vulnerability is discovered and no security update is available for the software version being used. “You cannot expect a situation like that to be resolved at the last minute without consequences.”
Friday recognises this dilemma as well. At the start of a project, the company discusses which data will be processed, what security is required and how the platform can be configured securely. Through periodic health checks, customers subsequently gain insight into software versions, security and performance, among other things.
Nevertheless, customers sometimes consciously choose not to implement certain updates. According to Frank, it is important to explicitly document these decisions and the associated risks. “If a customer decides not to implement certain measures, we make clear where our responsibility ends. You want to avoid discussions about this only arising when something goes wrong.”
Friday is also currently working towards ISO 27001 certification. This helps the organisation further formalise its processes, risks and responsibilities. According to Frank, the incident demonstrated the value of this preparation, but also provided new insights. For example, a standard maintenance page, which itself runs on PHP, turned out not to be a suitable solution during a PHP incident. Communication and incident procedures are therefore also being evaluated.
Who is responsible for what?
A key part of the collaboration between Shock Media and Friday is the clear division of responsibilities. At Shock Media, this primarily covers the infrastructure, from the data centre through to the middleware. Responsibility for the application lies with the agency that developed it. At Friday, the division is different in certain areas because the company carries out some management tasks itself.
“We have clearly established who is responsible for what,” Justin explains. “That’s important in our day-to-day collaboration, but even more so during an incident.”
Friday and Shock Media have also made this division technically transparent. As a result, it was immediately clear during the PHP incident who needed to take which measures. The teams were able to quickly exchange information and coordinate decisions through a shared Slack channel.
According to Frank, this is precisely where the value of a good partnership lies. “We don’t see Shock Media as an ordinary supplier, but as a genuine partner. During an incident like this, you realise what that means. You have to be able to rely on each other and know that everyone takes responsibility.”
Not every digital agency has the same technical expertise or capacity in-house. The collaboration between Friday and Shock Media demonstrates how important it is to organise specialist knowledge effectively and make agreements about management, security and incidents in advance.
European digital sovereignty and the influence of AI
In addition to the security of digital systems, there is also the question of how dependent organisations want to be on technology companies outside Europe. As a result, European digital sovereignty is receiving increasing attention. Organisations are taking a more critical look at where their data is stored, who has access to it and which alternatives are available.
According to Justin, this is particularly relevant for government organisations. They want to reduce their dependence on non-European technology, but this is not always straightforward in practice. Many existing platforms offer functionalities for which no equivalent European alternative is immediately available. This requires a trade-off between control, functionality, performance and costs.
Shock Media notices that customers are increasingly making conscious choices in favour of hosting in the Netherlands or Europe. In addition to data location, availability, performance and legal requirements also play a role.
For digital agencies, this means that choosing a technology partner involves more than just functionality and price. The dependencies you create are becoming increasingly important too. Can you easily switch to another provider? Where is data processed? And how much control does the customer retain over their own systems?
According to Frank, the rise of AI makes these questions even more relevant. AI helps organisations work faster and resolve security issues, but it can also be used to identify vulnerabilities. Furthermore, using external AI applications introduces new risks when it comes to protecting customer data.
At Friday, the principle of ‘Own your intelligence’ is therefore central. “You need to know what information you give to AI and where it ends up. That intelligence belongs to you or your customer. You need to remain in control of it yourself.” Among other things, Friday uses AI gateways to regulate which data is sent to external AI services. Some of its AI applications run on infrastructure at Shock Media. This allows Friday to maintain control over the data and retain the ability to switch between technology providers.
Shock Media also makes conscious choices in this area. “We want to make AI work for us as much as possible, but we need to remain critical about which applications we use and how we handle them. Especially when customers entrust us with their data.”
A partnership that proves its value
During the PHP incident, the value of the close collaboration between Shock Media and Friday became clear. Both organisations were able to act quickly, continuously coordinate and build on each other’s expertise. Because responsibilities had been clearly established in advance and the teams could communicate directly, they were able to act quickly and carefully under considerable pressure.
For both organisations, it was an intensive day, but also a confirmation of their preparation and collaboration. At the same time, Frank and Justin emphasise that security requires continuous attention. New vulnerabilities keep emerging, technology is changing and dependence on digital systems continues to grow. According to Frank, it is primarily about having insight and being able to act quickly. “Make sure you know what systems you’re running, what data is being processed in them and who is responsible for what. And involve your customers in that.”
Justin also emphasises the importance of trust. “Customers entrust us with an important part of their business continuity. You have to live up to that trust every day, especially when the pressure is high.” Frank aptly summarises the partnership: “Your partnership is only truly put to the test in situations like these. That’s when you have to stand together. And I think we did that very well.”
