Workers pool in go lang!
Introduction
In Go language, the term worker pool refers to a collection of goroutines that are used to execute tasks concurrently. The worker pool pattern is a powerful tool for managing concurrent tasks in Go programs. It allows for efficient resource management and enables programs to handle large volumes of tasks simultaneously. In this article, we will discuss the worker pool pattern in Go and how it can be used to execute tasks concurrently.
What is a Worker Pool?
A worker pool is a collection of goroutines that are used to execute tasks concurrently. In a worker pool, a fixed number of goroutines are created and assigned to process incoming tasks. As new tasks are added to the pool, they are distributed among the available goroutines. Once a goroutine has finished executing a task, it becomes available to process another task.
Benefits of Worker Pools
There are several benefits of using a worker pool in Go programs. Some of these include:
Improved performance: Worker pools enable concurrent execution of tasks, which can significantly improve program performance.
Efficient resource management: By limiting the number of goroutines created, worker pools prevent the creation of too many resources and thus prevent resource exhaustion.
Improved task prioritization: With a worker pool, tasks can be prioritized based on their importance or urgency, which can improve program responsiveness.
Example of Worker Pool in Go
Let's look at an example of a worker pool in Go that can process a series of tasks concurrently.
package main
import (
"fmt"
"sync"
"time"
)
type Job struct {
id int
randomno int
}
type Result struct {
job Job
sumofdigits int
}
var jobs = make(chan Job, 10)
var results = make(chan Result, 10)
func digits(number int) int {
sum := 0
no := number
for no != 0 {
digit := no % 10
sum += digit
no /= 10
}
time.Sleep(2 * time.Second)
return sum
}
func worker(wg *sync.WaitGroup) {
for job := range jobs {
output := Result{job, digits(job.randomno)}
results <- output
}
wg.Done()
}
func createWorkerPool(noOfWorkers int) {
var wg sync.WaitGroup
for i := 0; i < noOfWorkers; i++ {
wg.Add(1)
go worker(&wg)
}
wg.Wait()
close(results)
}
func allocate(noOfJobs int) {
for i := 0; i < noOfJobs; i++ {
randomno := i
job := Job{i, randomno}
jobs <- job
}
close(jobs)
}
func result(done chan bool) {
for result := range results {
fmt.Printf("Job id %d, input random no %d , sum of digits %d\n", result.job.id, result.job.randomno, result.sumofdigits)
}
done <- true
}
func main() {
startTime := time.Now()
noOfJobs := 50
go allocate(noOfJobs)
done := make(chan bool)
go result(done)
noOfWorkers := 10
createWorkerPool(noOfWorkers)
<-done
endTime := time.Now()
diff := endTime.Sub(startTime)
fmt.Println("total time taken ", diff.Seconds(), "seconds")
}
In this example, we define a Job struct that represents a task to be executed by the worker pool. We also define a Result struct that contains the output of a task.
Creating a worker pool Step by Step:
Creating a worker pool in Go involves creating a channel to receive tasks and a group of workers to execute those tasks. The basic steps to create a worker pool are as follows:
Step 1: Create a task type:
The first step in creating a worker pool is to define the type of task that will be executed by the workers. In Go, a task can be represented as a function that takes some input and produces some output. Here is an example of a simple task type:
type Task func(input int) (output int, err error)
Step 2: Create a channel to receive tasks:
The next step is to create a channel that will receive the tasks to be executed by the workers. The channel should have a buffer size that is large enough to hold all the tasks that will be submitted to the pool. Here is an example of creating a channel with a buffer size of 100:
taskCh := make(chan Task, 100)
Step 3: Create a group of workers:
The third step is to create a group of workers that will execute the tasks submitted to the channel. In Go, a worker can be represented as a goroutine that continuously reads tasks from the channel and executes them. Here is an example of creating a worker function:
func worker(id int, tasks <-chan Task, results chan<- int) {
for task := range tasks {
output, err := task(id)
if err == nil {
results <- output
}
}
}
The worker function takes an ID to identify the worker, a channel to receive tasks, and a channel to send results. It reads tasks from the task channel, executes them, and sends the results to the result channel.
Step 4: Start the workers:
The final step is to start the workers by creating a goroutine for each worker and passing them the task channel and result channel. Here is an example of starting three workers:
numWorkers := 3
results := make(chan int, 100)
for i := 0; i < numWorkers; i++ {
go worker(i, taskCh, results)
}
Using the worker pool:
Once the worker pool is created, it can be used to execute tasks concurrently. Here is an example of how to use the worker pool to execute a large number of tasks:
for i := 0; i < 1000; i++ {
taskCh <- func(id int) (int, error) {
// Execute the task
return id * id, nil
}
}
close(taskCh)
for i := 0; i < 1000; i++ {
result := <-results
fmt.Println(result)
}
This code submits 1000 tasks to the task channel and then waits for the results to be sent to the result channel. Each task is represented as a function that calculates the square of the input.
Conclusion:
In conclusion, creating a worker pool in Go language is an effective way to execute a large number of tasks concurrently. It involves creating a channel to receive tasks and a group of workers to execute those tasks. By using a worker pool, you can significantly improve the performance of your concurrent programs.
I hope this helps, you!!
More such articles:
https://www.youtube.com/channel/UCiTaHm1AYqMS4F4L9zyO7qA
\==========================**=========================
If this article adds any value to you then please clap and comment.
Let’s connect on Stackoverflow, LinkedIn, & Twitter.