Internet service providers and telecom companies often face the need to filter traffic — for example, to block resources that conflict with regulatory requirements. Existing solutions are quite expensive and therefore not suitable for everyone. Moreover, they can cause disruptions in data transmission if traffic volume suddenly spikes.
Ilia Masandilov, Head of the IT Department at the ISP IP-Home, developed his own traffic filtering solution that is easy to install and requires significantly fewer resources. In this article, he shares the details of the solution and the key stages of its implementation.
Why Internet Traffic Filtering Is Needed
Traffic filtering is the process of analyzing network packets and blocking unwanted requests or resources. This is done to comply with legal or corporate requirements, protect corporate networks, or restrict access to specific types of content.
Filtering can be applied both at both the level of an individual company and across entire regions. For example, a company may restrict employee access to social networks, while entire countries may prohibit extremist materials or the distribution of illegal substances.
The responsibility for filtering usually falls on ISPs, who rely on specialized software or hardware solutions — primarily Deep Packet Inspection (DPI) systems. These systems process all user traffic, filtering out unwanted websites or pages.
However, DPI solutions have drawbacks: they are costly and resource-intensive. For smaller providers, deploying DPI systems is prohibitively expensive, often reducing profit margins to nearly zero. Additionally, as the subscriber base and traffic volume grow, providers are forced to purchase more DPI equipment. Since these systems must process all traffic, they also create a single point of failure.
Nfq_filter — A New Approach to Traffic Filtering
The weak point of traditional filtering solutions is that they pass absolutely all traffic through themselves, requiring massive amounts of resources. When our telecom company faced the need for traffic filtering, I realized that the total traffic volume was far greater than the portion that actually needed filtering. The most efficient solution would be to cut out the unnecessary data.
I came up with a way to configure the network so that only the traffic requiring filtering would be redirected to the server. This led to the creation of nfq_filter — a solution that combined BGP peering configuration between network devices with filtering software. At that time, no such solution existed.
Initially, I deployed nfq_filter within my company. Later, I published it in an open repository and shared it on provider forums. The solution was well received, and soon hundreds of ISPs adopted it — ultimately reaching millions of users. Commercial services later emerged, based on the same approach.
How the Solution Works
ISPs use the BGP dynamic routing protocol, which allows subscriber nodes (BNG servers) to connect to routers, such as the Cisco 7600.
Here’s the core idea of my approach:
- The server parses IP addresses of resources that need to be filtered and compiles lists. Not all IP addresses need to be blocked entirely — sometimes it is enough to restrict specific URLs.
- The system sends these IP address lists to BNG servers as individual /32 routes.
- BGP prioritizes the most specific route. Thus, if a device announces a single IP address to a BNG server, traffic to that address is routed through it.
- As a result, traffic to the filtered IP addresses is routed through my system, where the addresses are blocked.
For HTTP websites, the system can block individual pages without touching the rest of the site. For HTTPS, while the specific page cannot be seen, the domain can be identified via the SNI header, allowing blocking at the domain level.
To prevent user connections from hanging, the software sends back a fake TCP Reset, properly terminating the session. For HTTP pages, users are redirected to a special notification page about the block.
Later, we extended this mechanism so that subscribers who forgot to pay for services were redirected to a payment reminder page.
Building the Solution
When our company faced the need to implement traffic filtering, the required hardware and software came with a massive price tag. I decided there had to be another way. Within a few days, I set up a test environment on a standalone server to test my idea.
First, I verified the concept by routing specific IP addresses to the server for filtering. Then I gradually increased the traffic load until I was confident the system worked.
Afterward, I developed the second part of the solution, which automatically parsed resource lists to be blocked, determined their IP addresses, and announced them to our BGP. The BGP, in turn, directed traffic from these IP addresses to the filtering server.
I continued experimenting with the filter, redirecting traffic and refining the code. Overall, the entire project — from concept to production — took about a month.
Advantages of Nfq_filter and Similar Solutions
At that time, our traffic volume was about 40 Gbps. To filter that amount of traffic with traditional solutions, we would have needed to purchase a large amount of expensive equipment — for example, Cisco DPI systems capable of handling only 10 Gbps each, with strict limits on the number of filtering rules.
As traffic grew, the required resources would only increase, and eventually, even extensive deployments would not be enough. In contrast, nfq_filter consumes far fewer resources and is much easier to scale.
Nfq_filter-based solutions are also far more fault-tolerant. ISP networks include subscriber and provider nodes, with backup nodes in place to ensure traffic reroutes in case of outages. This ensures redundancy.
But with DPI systems, all traffic flows through them, creating a single point of failure. If the DPI fails, the entire connection collapses. To prevent this, filtering equipment must be installed on every redundant link, effectively doubling or tripling the hardware requirements — or sacrificing redundancy altogether.
Another issue with many DPI systems is the limited number of filtering rules they can handle. If the list of blocked IP addresses grows too large, they simply cannot keep up. Nfq_filter has no such limitations and scales much more easily.
As a result, nfq_filter and similar solutions can be used to filter any traffic volume — even if it grows geometrically. Resource usage does not increase proportionally, since nfq_filter only processes unwanted requests rather than passing all traffic through itself.
