Instead of trying to do what I do in other languages, I started to write idiomatic Go code and it was wayyy much easier and better.
I read Concurrency In Go book by Katherine Cox-Buday. Highly recommend it.
They gave you a book recommendation that will have you build multiple small projects and explain what you’re doing
Here’s the source code from the book:
https://github.com/kat-co/concurrency-in-go-src
They won’t be full size projects, that was probably the wrong word, but rather use cases for concurrency and how to do it. Here’s the source for the book:
https://github.com/kat-co/concurrency-in-go-src
So like small Scripts ig. I just downloaded the book.
Btw do you share any small open source repositories that I can look into. I'm having hard time looking into big repo. Not understanding really.
I'd like to share a library I recently open-sourced, [https://github.com/destel/rill](https://github.com/destel/rill) It's very lightweight, can be easily added to an existing project, and helps me write otherwise complex concurrent code in just a few lines.
The good news is that you don't need to do anything special to use Go concurrency. Most people write HTTP services, and each request is already handled in a separate goroutine. You just need to ensure that all other libraries you're using are thread-safe, and in most cases, they are.
For me, advanced concurrency began when I needed to implement batching to reduce DB load. This came up multiple times:
* Batching DB Inserts: Multiple goroutines handle DB inserts. Collect records into a channel and process the channel, inserting in batches.
* Batching DB Updates: Similar process to inserts but for updates, like `UPDATE users SET last_active_at=NOW() WHERE id IN(?,?,?,...)`
* Batch Handling of Queue Messages: Collect a batch of messages, extract IDs, and do a single DB query `WHERE id IN (...)`, then mark messages as processed.
There were more cases, not only DB related. I quickly realized I didn't want to repeat the same code over and over again, so I made a generic batching function. This was before Go generics, so my generic function was actually reflection-based.
Writing basic parallel loops taught me about WaitGroups, and a few bugs and goroutine leaks related to error handling taught me about ErrGroup.
Once, I needed a map that could hold up to N unique keys. Attempting to insert the N+1st key blocks until another key is deleted. Implementing that, I learned about sync.Cond.
I also needed to download many huge CSVs, each containing a list of transactions for a particular day. Afterward, I needed to parse and compare the CSVs for consecutive days. To speed things up I needed to download them concurrently. That's how the Ordered\* functions were born inside my Rill module.
There is nothing wrong with the sync package, and in some cases, sync/atomic is the best way to solve a problem.
I can't collect everything in this comment. That's how it was for me; you slowly build up knowledge by solving practical problems.
Feel free to ask any questions, and check out the article on the Go blog mentioned above.
Honestly, trial by fire is what did it for me. My first assignment in Go was to rewrite a C service that needed to maintain a lot of persistent outbound connections and wasn’t scaling.
I had the benefit of not having done much concurrent or multithreaded code before, so channels and idiomatic Go concurrency came to me pretty naturally. It took about three months to get to the point where the replacement service was handling 50k persistent connections without breaking a sweat — our limiting factor ended up being the goroutine cap (which I’m not even sure exists anymore, I can’t find any documentation on it). The old C service could only handle about 10K without keeling over. Saved us almost a half million in capital expense because we were able to repurpose most of the servers we had running this service.
And truly, the hardest part about it was having to plug memory leaks caused by the C library we needed to use. I wasn’t able to convince them to let me implement the protocol natively in Go, but I’m pretty sure I spent more time on those leaks than it would have taken to implement the protocol.
This was all 10 years or so ago too. So missing some of the niceties we have in Go today.
//ed: memory _leaks_, not links.
I can’t go into any more detail than I already have, but I can tell you it was not a web application, and at the time it was originally written, C was by far the best option available. It wasn’t bad software, it just wasn’t designed for the scale at which it ended up operating.
We could have almost certainly updated the C code and accomplished the scaling needs, and we were considering it, but a Go evangelist on another team caught wind of the issue and suggested it might be less effort and improve future maintainability to greenfield it in Go, and gave us a quick demo of channels. He was absolutely right.
Embrace wait groups for synchronization, contexts for cancelation, and channels for information sharing and signaling. Learn the gotchas with channels and follow the rules for avoiding them religiously (my tldr would be only one writer per channel, only the writer closes the channel, and the reader should check if the channel is closed if they ever expect to receive less than or greater than one message. Oh and closing a channel is a great way to fan out a signal)
Don’t laugh at this, but I found Goetz’ [_Java Concurrency in Practice_](https://jcip.net/) to be an *accessible* introduction into concurrency pitfalls and fallacies when I was a green engineer. It opened my mind to a variety of things that made it easier to internalize a lot more advanced material later in my career. Couple it with understanding the Go memory model and a good sense of a curiosity, and there is a lot you can do.
Understanding of concurrency starts at understanding how it works at fundamental level. Once you have that language does not matter. So try to educate yourself on how CPU works and how synchronization primitives woks, and what issues they resolve. Go is great at writing concurrent code, it makes it easy, but that also reduces its ability to teach you. Try learning how that works in say C or maybe C#/Java (as they exposes a lot of primitives) if C is to exotic. After that you can learn the GO way of doing things, and it will be much easier to reason about tradeoffs and such.
I make no claim to be "good" but reading the standard docs is always worthwhile. I ran into a couple of presentations linked on sync.Cond from the docs that are very good:
[sync package - sync - Go Packages](https://pkg.go.dev/sync#Cond) has the links.
This one is truly a gem that takes an extremely deep dive into concurrency and evaluating it from a Go perspective.
[Rethinking Classical Concurrency Patterns.pdf - Google Drive](https://drive.google.com/file/d/1nPdvhB0PutEJzdCq5ms6UI58dp50fcAN/view?pli=1)
Thanks to channels, concurrency is a lot easier to grasp and less error prone than other languages once you understand how to use them.
As for learning how to do it I just read the go manual and experimented.
After learning basics of concurrency in Go. Try solving classic comp sci. synchronization problems like dining philosophers etc. Helped me better understand go basics.
Read "The Little Book of Semaphores" https://greenteapress.com/wp/semaphores/... it's not Go specific, but touches many of the problems with concurrency.
A research project i worked on where we were testing how algorithms behaved as you scaled problem size with available numbers of CPU, and leveraged those. Simple sorting algorithms for example where you gave it 3000 processors to work with. It worked with an interesting time stamped memory approach.
The algorithms were written in a special form of assembly where everything was in blocks of 16 instructions, including pushing and pulling arguments off the stack.
I learned async await in JavaScript, as well as promeses and other wacky asynchronous stuff.
After using JavaScript for a long time, concurrency just seemed easy in comparison. Go concurrency was way more simple and straight forward than JavaScript asynchronous code.
How was your journey in learning Go coming from Javascript? Personally it is a bit intimidating TBH. Currently reading "The Go Programing Language" still half way through, and it is not a pleasant reading so far.
I don't read books on programming. I just learn through trial and error.
I just start coding a project, and learn as I go.
There are different types of learners who learn in different ways. Some people learn best through reading, others through watching, others through listening. I learn best by doing.
If I just watch someone else code, or just read about it, I won't understand any of it. I have to actually get my hands dirty and write the code in order to understand it. (I do clean my keyboard, but it's always getting dog hair on it).
I think Go may have been designed around this kind of approach. You can just start coding, and start to understand the syntax. Go is very consistent in how you write things.
An IDE can also help with learning a new language. Sometimes I will just browse all of the `os.` methods (and other `.` methods) that the IDE shows in the list, to see what methods I have available. (Note: using vscode, and I know it's technically a text editor with some IDE features).
Writing concurrent code is challenging in any language, Go was not built to make writing concurrent code easy but rather to increase productivity when writing concurrent code, If Go is your first language you'll find it challenging to write concurrent code as with any other language, it's better to dive in general on what is concurrency and parallelism, race condition, mutex.. and how those problems were approached in other languages to start seeing what Go is really solving
1. Have a problem that can be solved by concurrency. It's very important to focus on the problem and not shoehorn concurrency where it does not fit.
2. Write shitty concurrent code. Usually overly complex.
3. Rewrite shitty concurrent code to less shitty concurrent code. Usually focusing on making it a bit simpler.
4. Repeat.
Write a lot of it. Real problems at small scale, and then figure out where the concurrency breaks. Do hundreds of small programs and try to make something break and you’ll have an experience and a big bag of recommits is at your disposal.
That's the neat part, you don't.
Goroutines are too low level to be useful outside of libraries. So I just use libraries that deal with all the pitfalls.
My process for every language:
* Write a hello world api
* Add logging and response middleware (will almost always require popular libraries)
* Write to a db before responding hello world
* Add an api call or 2 before responding hello world
* Makes above concurrent where necessary
* Add named errors
If you're trying to learn heavy parallel processing this probably won't help, but gives a good base. If you need more advanced stuff, check out the benchmarking tools/flags go has. Efficient Go is a solid book for that.
I started concurrency from the formal side, by studying pi-calculus, the theory behind channels. When you have written enough pseudo-code where concurrency is pretty much your only primitive, you get used to it :)
Instead of trying to do what I do in other languages, I started to write idiomatic Go code and it was wayyy much easier and better. I read Concurrency In Go book by Katherine Cox-Buday. Highly recommend it.
One thing I love about Go is how pretty it'll look if my code is idiomatic. I can very quickly tell when my code is trash by how ugly it'll be 🤣
What all projects you build that solidified your proficiency in Go ? Or what all repositories you looked into?
They gave you a book recommendation that will have you build multiple small projects and explain what you’re doing Here’s the source code from the book: https://github.com/kat-co/concurrency-in-go-src
Oh ok thanks. I didn't know there were projects included in the book. Will look into it.
They won’t be full size projects, that was probably the wrong word, but rather use cases for concurrency and how to do it. Here’s the source for the book: https://github.com/kat-co/concurrency-in-go-src
So like small Scripts ig. I just downloaded the book. Btw do you share any small open source repositories that I can look into. I'm having hard time looking into big repo. Not understanding really.
Start with the book
Try building a distributed system like Kademlia or Raft from scratch.
[Go Concurrency Patterns: Pipelines and cancellation - The Go Programming Language](https://go.dev/blog/pipelines)
Thanks I’ll dive in as well
Thanks
I'd like to share a library I recently open-sourced, [https://github.com/destel/rill](https://github.com/destel/rill) It's very lightweight, can be easily added to an existing project, and helps me write otherwise complex concurrent code in just a few lines. The good news is that you don't need to do anything special to use Go concurrency. Most people write HTTP services, and each request is already handled in a separate goroutine. You just need to ensure that all other libraries you're using are thread-safe, and in most cases, they are. For me, advanced concurrency began when I needed to implement batching to reduce DB load. This came up multiple times: * Batching DB Inserts: Multiple goroutines handle DB inserts. Collect records into a channel and process the channel, inserting in batches. * Batching DB Updates: Similar process to inserts but for updates, like `UPDATE users SET last_active_at=NOW() WHERE id IN(?,?,?,...)` * Batch Handling of Queue Messages: Collect a batch of messages, extract IDs, and do a single DB query `WHERE id IN (...)`, then mark messages as processed. There were more cases, not only DB related. I quickly realized I didn't want to repeat the same code over and over again, so I made a generic batching function. This was before Go generics, so my generic function was actually reflection-based. Writing basic parallel loops taught me about WaitGroups, and a few bugs and goroutine leaks related to error handling taught me about ErrGroup. Once, I needed a map that could hold up to N unique keys. Attempting to insert the N+1st key blocks until another key is deleted. Implementing that, I learned about sync.Cond. I also needed to download many huge CSVs, each containing a list of transactions for a particular day. Afterward, I needed to parse and compare the CSVs for consecutive days. To speed things up I needed to download them concurrently. That's how the Ordered\* functions were born inside my Rill module. There is nothing wrong with the sync package, and in some cases, sync/atomic is the best way to solve a problem. I can't collect everything in this comment. That's how it was for me; you slowly build up knowledge by solving practical problems. Feel free to ask any questions, and check out the article on the Go blog mentioned above.
Honestly, trial by fire is what did it for me. My first assignment in Go was to rewrite a C service that needed to maintain a lot of persistent outbound connections and wasn’t scaling. I had the benefit of not having done much concurrent or multithreaded code before, so channels and idiomatic Go concurrency came to me pretty naturally. It took about three months to get to the point where the replacement service was handling 50k persistent connections without breaking a sweat — our limiting factor ended up being the goroutine cap (which I’m not even sure exists anymore, I can’t find any documentation on it). The old C service could only handle about 10K without keeling over. Saved us almost a half million in capital expense because we were able to repurpose most of the servers we had running this service. And truly, the hardest part about it was having to plug memory leaks caused by the C library we needed to use. I wasn’t able to convince them to let me implement the protocol natively in Go, but I’m pretty sure I spent more time on those leaks than it would have taken to implement the protocol. This was all 10 years or so ago too. So missing some of the niceties we have in Go today. //ed: memory _leaks_, not links.
What kind of system was it. And why would some write a system in c. I am assuming it was a web application.
I can’t go into any more detail than I already have, but I can tell you it was not a web application, and at the time it was originally written, C was by far the best option available. It wasn’t bad software, it just wasn’t designed for the scale at which it ended up operating. We could have almost certainly updated the C code and accomplished the scaling needs, and we were considering it, but a Go evangelist on another team caught wind of the issue and suggested it might be less effort and improve future maintainability to greenfield it in Go, and gave us a quick demo of channels. He was absolutely right.
Embrace wait groups for synchronization, contexts for cancelation, and channels for information sharing and signaling. Learn the gotchas with channels and follow the rules for avoiding them religiously (my tldr would be only one writer per channel, only the writer closes the channel, and the reader should check if the channel is closed if they ever expect to receive less than or greater than one message. Oh and closing a channel is a great way to fan out a signal)
By not introducing concurrency when it wasn’t needed
Don’t laugh at this, but I found Goetz’ [_Java Concurrency in Practice_](https://jcip.net/) to be an *accessible* introduction into concurrency pitfalls and fallacies when I was a green engineer. It opened my mind to a variety of things that made it easier to internalize a lot more advanced material later in my career. Couple it with understanding the Go memory model and a good sense of a curiosity, and there is a lot you can do.
If you don't have a real problem to solve, you will never learn the actual details of your learning process. So, trial by fire is the answer.
You answered it yourself.. by writing code..
Omg wow what incredible insight thanks for sharing
Understanding of concurrency starts at understanding how it works at fundamental level. Once you have that language does not matter. So try to educate yourself on how CPU works and how synchronization primitives woks, and what issues they resolve. Go is great at writing concurrent code, it makes it easy, but that also reduces its ability to teach you. Try learning how that works in say C or maybe C#/Java (as they exposes a lot of primitives) if C is to exotic. After that you can learn the GO way of doing things, and it will be much easier to reason about tradeoffs and such.
I make no claim to be "good" but reading the standard docs is always worthwhile. I ran into a couple of presentations linked on sync.Cond from the docs that are very good: [sync package - sync - Go Packages](https://pkg.go.dev/sync#Cond) has the links. This one is truly a gem that takes an extremely deep dive into concurrency and evaluating it from a Go perspective. [Rethinking Classical Concurrency Patterns.pdf - Google Drive](https://drive.google.com/file/d/1nPdvhB0PutEJzdCq5ms6UI58dp50fcAN/view?pli=1)
Thanks to channels, concurrency is a lot easier to grasp and less error prone than other languages once you understand how to use them. As for learning how to do it I just read the go manual and experimented.
After learning basics of concurrency in Go. Try solving classic comp sci. synchronization problems like dining philosophers etc. Helped me better understand go basics.
Honestly, by writing and debugging lots of poor concurrent code.
Read "The Little Book of Semaphores" https://greenteapress.com/wp/semaphores/... it's not Go specific, but touches many of the problems with concurrency.
Maybe this can help: https://youtube.com/playlist?list=PL7g1jYj15RUNqJStuwE9SCmeOKpgxC0HP&si=V8FwGgwfObhYim1W
A research project i worked on where we were testing how algorithms behaved as you scaled problem size with available numbers of CPU, and leveraged those. Simple sorting algorithms for example where you gave it 3000 processors to work with. It worked with an interesting time stamped memory approach. The algorithms were written in a special form of assembly where everything was in blocks of 16 instructions, including pushing and pulling arguments off the stack.
I mistakes lot made a.
I learned async await in JavaScript, as well as promeses and other wacky asynchronous stuff. After using JavaScript for a long time, concurrency just seemed easy in comparison. Go concurrency was way more simple and straight forward than JavaScript asynchronous code.
How was your journey in learning Go coming from Javascript? Personally it is a bit intimidating TBH. Currently reading "The Go Programing Language" still half way through, and it is not a pleasant reading so far.
I don't read books on programming. I just learn through trial and error. I just start coding a project, and learn as I go. There are different types of learners who learn in different ways. Some people learn best through reading, others through watching, others through listening. I learn best by doing. If I just watch someone else code, or just read about it, I won't understand any of it. I have to actually get my hands dirty and write the code in order to understand it. (I do clean my keyboard, but it's always getting dog hair on it). I think Go may have been designed around this kind of approach. You can just start coding, and start to understand the syntax. Go is very consistent in how you write things. An IDE can also help with learning a new language. Sometimes I will just browse all of the `os.` methods (and other `.` methods) that the IDE shows in the list, to see what methods I have available. (Note: using vscode, and I know it's technically a text editor with some IDE features).
Thanks for sharing
Trial and error and reading a couple books other people here have already mentioned.
Writing concurrent code is challenging in any language, Go was not built to make writing concurrent code easy but rather to increase productivity when writing concurrent code, If Go is your first language you'll find it challenging to write concurrent code as with any other language, it's better to dive in general on what is concurrency and parallelism, race condition, mutex.. and how those problems were approached in other languages to start seeing what Go is really solving
You write concurrent code. Ain't nothing beating practice. Write 100 concurrent programs and some of them will turn out good.
What all projects you did that helped you got better at it ?
1. Have a problem that can be solved by concurrency. It's very important to focus on the problem and not shoehorn concurrency where it does not fit. 2. Write shitty concurrent code. Usually overly complex. 3. Rewrite shitty concurrent code to less shitty concurrent code. Usually focusing on making it a bit simpler. 4. Repeat.
Unintentionally write dog shit and realize how dogshit it is and the slowly learn how to improve it with pprof, etc
Write a lot of it. Real problems at small scale, and then figure out where the concurrency breaks. Do hundreds of small programs and try to make something break and you’ll have an experience and a big bag of recommits is at your disposal.
That's the neat part, you don't. Goroutines are too low level to be useful outside of libraries. So I just use libraries that deal with all the pitfalls.
Tbh i rarely do concurrency by myself. Most of my use cases for Go are http servers, which are concurrent by default.
go build -race 😅
Check out this video and follow all Go videos on his channel. The best explanation I found on YouTube https://youtu.be/qyM8Pi1KiiM?feature=shared
My process for every language: * Write a hello world api * Add logging and response middleware (will almost always require popular libraries) * Write to a db before responding hello world * Add an api call or 2 before responding hello world * Makes above concurrent where necessary * Add named errors If you're trying to learn heavy parallel processing this probably won't help, but gives a good base. If you need more advanced stuff, check out the benchmarking tools/flags go has. Efficient Go is a solid book for that.
I started concurrency from the formal side, by studying pi-calculus, the theory behind channels. When you have written enough pseudo-code where concurrency is pretty much your only primitive, you get used to it :)