Parallelizing Work With Python’s Multiprocessing

In this blog post, we will look at how to solve a simple problem with a Python program, and then we will try to speed it up by using Python’s multiprocessing module.

The main goal of this post is to illustrate how a program can be made much faster by parallelizing work through multiple processes, as opposed to running the whole workload through a single process.

Problem introduction – TCP port scanning

The problem we will be trying to solve is known as TCP port scanning. The problem consists of finding open TCP ports in a given IP address. Such a process could be used by network administrators to identify potential risks in their networks and by attackers to attempt to gain control over exposed systems.

TCP ports are represented in 16 bits, so we have a maximum of 65535 ports per IP address. Port 0 is reserved and cannot be used, so we will focus on the range 1 to 65535.

Given a hostname such as www.google.com, a start-port, and an end-port our program will have to find the IP address of the given hostname and then print all open TCP ports in the given range.

In order to do this, we will have to make our program iterate through all ports from start-port to end-port and on each step attempt to establish a connection through the current port. If the connection can be successfully established we know the port is open, then we will print a message to let the user know.

A simple single-process solution

Let’s try to create a simple Python function using the socket module. This function takes an IP address and port number as inputs. It returns True if the port is open, and False otherwise.

The function is very simple. First, it wraps its calls inside a try/except block. It then tries to create a connection to the specified address and port. If this connection is successful it will immediately close it and return True, letting us know the port is open. If any problem occurs and the connection cannot be established it will return False, letting us know the port is closed.

The value of timeout=1 is needed to allow our program some time (1 second in this case) to establish the connection. If after 1 second our program can’t establish a connection we will assume the port is closed.

Now let’s wrap our function inside a complete program by reading some command-line arguments and printing appropriate messages. We will use argparse for argument parsing and time to measure execution time.

This is how our program works. When executed it reads the –hostname, –start-port and –end-port arguments. If a port range is not specified it will default to all ports, 1 to 65535. It then creates a variable called start_time to store the current timestamp in seconds, executes the scan_host function, and finally prints the elapsed time in seconds.

The scan_host function first translates the given hostname to an IP address, then iterates through all ports in the specified range and calls our initial is_port_open function for each port. If it finds an open port then it prints a message.

Let’s name our program port_scanner.py and save it.

Scanning 500 ports

Time to do some tests! Let’s see how long it takes to scan through 500 ports.

So our program works just fine and we were able to find two open ports. However, scanning 500 ports took 500 seconds. This is something we could have predicted given our 1-second timeout per connection attempt.

Given this, if we wanted to scan through all 65535 available ports, our program could take 65535 seconds to complete, or a little over 18 hours.

If we don’t have all day to portscan a single host, one thing is clear: our program must run faster.

One thing which comes to mind would be lowering our timeout value, but this could compromise accuracy. TCP connections need some time to establish, and not giving our program enough time could result in wrongly assuming some ports are closed when in reality could simply take some more time to accept a connection.

A better approach to speed up our program would be to try and connect to multiple ports at once, instead of trying a single port at a time. Fortunately, we can achieve this by parallelizing our workload across multiple processes. This is when multiprocessing comes to the rescue.

Speeding things up with Python’s multiprocessing

Python’s multiprocessing module provides a set of classes that allow to spawn subprocesses from a program’s main process. We will look at how we can use the Process class to speed up our port scanning program.

First, let’s modify our scan_host function to take a new workers argument and spawn a set of processes to divide the workload.

Let’s look at the different parts of our new function.

The new argument workers indicate how many subprocesses we want to launch. So given start_port and end_port we can calculate the total number of ports to scan and then divide this number by the amount of workers we will be launching:

At this point, we can iterate through our port range and compute the start_port and end_port of each one of our workers.

So to illustrate this with an example, if we give our program the following inputs:

Then the workers would be set up with the following arguments:

Now in each iteration, we can create a new instance of the Process class to spawn a new subprocess with the given arguments. We will then start the process and store it in our processes list.

When our workers launch they will call the function provided as the target argument of the Process constructor. In this case the function is scan_address.** We will look at this function later.

Finally, we will call Process.join on each process to wait until they all finish.

Now, let’s put all pieces together into a new program.

Let’s name our new program port_scanner_parallel.py and save it.

Scanning 500 ports again

Now that we have what should be a much faster port scanner, let’s try scanning 500 ports again. This time we will launch 10 parallel workers.

As we can see, with 10 parallel workers we just gained a 10x improvement in execution time!

Last time, with a single process, it took 500 seconds to scan through 500 ports. Now with 10 parallel subprocesses, it takes only 50 seconds.

Scanning all ports of a host

Now that we have such a fast port scanner we can push things to the limit. Let’s try scanning all 65535 ports of a host with 100 parallel workers.

We were able to scan all ports of this host in just 131 seconds and have found four open ports.

Conclusion

We have looked at how to solve the TCP port scanning problem in Python. We initially looked at a simple single-process solution and then learned how to speed it up by using Python’s multiprocessing module.

We have learned how dividing the workload between a set of parallel workers can offer massive improvements in execution time.

Many computing problems can be parallelized like this, and now that you know how to use multiprocessing you have added a valuable tool to your toolbox. It is now up to you to apply it wisely.

Happy coding!

Join 2000+ Founders and Developers crushing their businesses and careers with monthly advice. You can also follow us on LinkedIn , Twitter & Instagram!