Scribd
Upload
12 views8 pages

Go Programming Language Tutorial (Part 7)

A Go Programming Language Tutorial (Part 7)

Uploaded by

eowug
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views8 pages

Go Programming Language Tutorial (Part 7)

A Go Programming Language Tutorial (Part 7)

Uploaded by

eowug
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 8

Go Programming Language Tutorial (Part 7)

This tutorial emphasizes building high-performance data pipelines, mastering advanced concurrency
concepts, and creating custom tools for developers.

1. Building Data Processing Pipelines


Data pipelines handle large-scale data processing, transforming inputs into meaningful outputs.

Example: Log Processing Pipeline


go
Copy code
package main

import (
"bufio"
"fmt"
"os"
"strings"
)

// Stage 1: Read logs


func readLogs(filePath string) (<-chan string, error) {
out := make(chan string)
file, err := os.Open(filePath)
if err != nil {
return nil, err
}

go func() {
scanner := bufio.NewScanner(file)
for scanner.Scan() {
out <- scanner.Text()
}
close(out)
file.Close()
}()
return out, nil
}

// Stage 2: Filter logs


func filterLogs(input <-chan string, keyword string) <-chan string {
out := make(chan string)
go func() {
for line := range input {
if strings.Contains(line, keyword) {
out <- line
}
}
close(out)
}()
return out
}

// Stage 3: Write filtered logs


func writeLogs(input <-chan string, outputPath string) error {
file, err := os.Create(outputPath)
if err != nil {
return err
}
defer file.Close()

for line := range input {


_, err := file.WriteString(line + "\n")
if err != nil {
return err
}
}
return nil
}

func main() {
inputPath := "logs.txt"
outputPath := "filtered_logs.txt"
keyword := "ERROR"

logStream, err := readLogs(inputPath)


if err != nil {
fmt.Println("Error reading logs:", err)
return
}

filteredLogs := filterLogs(logStream, keyword)

if err := writeLogs(filteredLogs, outputPath); err != nil {


fmt.Println("Error writing logs:", err)
} else {
fmt.Println("Filtered logs written to", outputPath)
}
}

2. Advanced Concurrency
Pipeline Pattern
The pipeline pattern processes data in stages using Goroutines and channels.

Example: Integer Pipeline


go
Copy code
package main

import "fmt"

func generate(nums ...int) <-chan int {


out := make(chan int)
go func() {
for _, n := range nums {
out <- n
}
close(out)
}()
return out
}

func square(input <-chan int) <-chan int {


out := make(chan int)
go func() {
for n := range input {
out <- n * n
}
close(out)
}()
return out
}

func main() {
numbers := generate(1, 2, 3, 4, 5)
squared := square(numbers)

for result := range squared {


fmt.Println(result)
}
}

Fan-Out, Fan-In
Distribute work across multiple Goroutines (Fan-Out) and aggregate results (Fan-In).

Example: Fan-Out, Fan-In


go
Copy code
package main

import (
"fmt"
"sync"
)

func worker(id int, jobs <-chan int, results chan<- int) {


for job := range jobs {
fmt.Printf("Worker %d processing job %d\n", id, job)
results <- job * 2
}
}

func main() {
jobs := make(chan int, 5)
results := make(chan int, 5)

var wg sync.WaitGroup

// Fan-Out: Start workers


for i := 1; i <= 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
worker(id, jobs, results)
}(i)
}

// Send jobs
for j := 1; j <= 5; j++ {
jobs <- j
}
close(jobs)

// Wait for workers to finish


go func() {
wg.Wait()
close(results)
}()

// Collect results
for result := range results {
fmt.Println("Result:", result)
}
}

3. Testing Distributed Systems


Distributed systems require robust testing strategies to ensure resilience and scalability.

Fault Injection
Simulate failures to test system behavior.

Example: Timeout Simulation


go
Copy code
package main

import (
"errors"
"fmt"
"time"
)

func unreliableService() (string, error) {


time.Sleep(3 * time.Second) // Simulate delay
return "", errors.New("service timeout")
}

func main() {
ch := make(chan string, 1)

go func() {
result, err := unreliableService()
if err != nil {
ch <- err.Error()
} else {
ch <- result
}
}()

select {
case res := <-ch:
fmt.Println("Response:", res)
case <-time.After(2 * time.Second):
fmt.Println("Error: Service timeout")
}
}

Chaos Testing
Tools like Chaos Monkey and LitmusChaos introduce random failures. Implement fault injection
locally for development.

4. Custom Developer Tools with Go


Go is excellent for building CLI tools and developer utilities.

Example: Simple CLI Tool


Using flag for CLI Arguments
go
Copy code
package main

import (
"flag"
"fmt"
)

func main() {
name := flag.String("name", "World", "The name to greet")
times := flag.Int("times", 1, "Number of times to greet")
flag.Parse()

for i := 0; i < *times; i++ {


fmt.Printf("Hello, %s!\n", *name)
}
}

Run it:
bash
Copy code
go run main.go -name="Go Developer" -times=3
Example: Git-Like CLI
Using Subcommands
go
Copy code
package main

import (
"flag"
"fmt"
"os"
)

func initCommand() {
fmt.Println("Initializing project...")
}

func buildCommand() {
fmt.Println("Building project...")
}

func main() {
initCmd := flag.NewFlagSet("init", flag.ExitOnError)
buildCmd := flag.NewFlagSet("build", flag.ExitOnError)

if len(os.Args) < 2 {
fmt.Println("expected 'init' or 'build' subcommands")
os.Exit(1)
}

switch os.Args[1] {
case "init":
initCmd.Parse(os.Args[2:])
initCommand()
case "build":
buildCmd.Parse(os.Args[2:])
buildCommand()
default:
fmt.Println("Unknown command")
os.Exit(1)
}
}

Run it:
bash
Copy code
go run main.go init
go run main.go build
5. Optimizing Performance in Go
Memory Profiling
Use pprof to identify memory bottlenecks.

Example: Profiling Memory Usage


go
Copy code
package main

import (
"net/http"
_ "net/http/pprof"
)

func main() {
go func() {
http.ListenAndServe(":6060", nil)
}()

// Simulate workload
data := make([]int, 0)
for i := 0; i < 1e6; i++ {
data = append(data, i)
}
}

Run the profiler:


bash
Copy code
go tool pprof http://localhost:6060/debug/pprof/heap

Optimizing Goroutines
Minimize Goroutine leaks by ensuring channels are closed and Goroutines terminate correctly.

6. Further Exploration
1. Learn Advanced Data Structures:
• Explore custom implementations of trees, tries, and graphs in Go.
2. Integrate Machine Learning:
• Use libraries like gorgonia to perform ML tasks in Go.
3. Build Streaming Applications:
• Use tools like Apache Kafka for real-time data processing.
This tutorial introduces advanced techniques for data pipelines, concurrency, and testing distributed
systems, along with building powerful developer tools. Mastering these concepts will prepare you to
handle complex, high-performance Go applications. Happy coding!

You might also like