WYSIWYG. Convention over configuration. Productivity > Performance Know how much your code costs. No hidden overloads, overhead, etc. Type is key. Integrity first in all code Key Terms: Data-oriented design, mechanical sympathy (constrain to arch), zero-value
Go doesn't have casting, it has conversion. Have to be explicit, no "implicit casting". Even if two types have identical values, assignment/conversion must be explicit:
type key int32
var i int32
var k key
i = k //doesn't work
zero-value: All allocated memory is set to zero value (var declaration)
7 bytes of value, but 8 bytes allocated. Alignment is added to the bool (1 extra byte)
| bool | pad (1byte) | int16 | float32 |
int16 needs to be on a 2-byte memory addr (0, 2, 4, 6, etc). So pad 1.
If int16 was int32, we would need to pad 3 bytes (0, 4, 8, etc). This makes order important, because padding is wasteful.
Only matters if it represents pure data. Are we storing lots of these? Worry. If only making a handful, tradeoff for readability (e.g. grouping like values together)
Tip: Always order highest -> smallest
// example represents a type with different fields.
type example struct {
flag bool
counter int16
pi float32
}
becomes
// example represents a type with different fields.
type example struct {
pi float32
counter int16
flag bool
}
Go has no constructors, just literals
// Declare a variable of type example and init using
// a struct literal.
e2 := example{
flag: true,
counter: 10,
pi: 3.141592,
}
You can declare and initialize in one go (haha), an "anonymous struct" type:
e := struct {
pi float32
counter int16
flag bool
}{
flag: true,
counter: 10,
pi: 3.141592,
}
Casting applies as before to NAMED structs.
type bill struct {
pi float32
counter int16
flag bool
}
type lisa struct {
pi float32
counter int16
flag bool
}
var b bill
var l lisa
b = l // doesn't work
b = bill(l) // DOES work
Note, bill must have the exact same layout as lisa. Can assign an anonymous (unnamed) struct:
b = e // works
Because e isn't a named type, it is a "schematic" of a struct, which matches bill
Used for un/marashalling (reflection package)
type bill struct {
flag bool `json:"f"`
}
Note, this affects conversion:
type lisa struct {
flag bool
}
var l lisa
var b bill
b = l // doesn't work, because of tag
This is intended to be "fixed" in 1.8. So we can tag the same data differently across multiple structs, allowing us to marshal data into multiple formats quickly.
Pointers are for sharing. No need to share, no need for pointer.
Standard memory layout: stack, heap.
Stack memory: single goroutine/function (not shared) Heap memory: multiple goroutines/functions (shared). Where "allocations" are made
Misc facts: Each thread has 1 MB stack memory by default. Goroutines start with 2 KB.
Stack memory is allocated when goroutine begins. A stack frame is created (determined at compile time, max memory needed) on every entry into a function. Same as usual, stacks grow down.
Standard ref/deref:
func main() {
count := 10 //allocated in 'main' stack frame
// Display the "value of" and the "address of" count.
println("Before:", count, &count)
}
If you have a pointer to a struct, the . operator transparently dereferences field values:
type Foo struct {
bar int
}
func main() {
f := Foo{ bar: 10 }
ptr := &f
//these do the same thing
f.bar = 11
ptr.bar = 11
}
All information is passed BY VALUE in go. NO by reference:
func increment(inc int) {
// Increment the "value of" inc.
inc++
println("Inc: ", inc, &inc)
}
func main() {
count := 10
increment(count) //pass count by value
println(count) //value not changed
}
Can pass an address (pointer) by value:
func increment(inc *int) { //*int is it's own type
// Increment the value that the "pointer points to". (de-referencing)
*inc++
println("Inc ", *inc, inc)
}
NOTE, you cannot do pointer arithmatic. Cannot inr/dcr pointer addresses. int* must point to a valid int in memory.
Reduce allocations where possible. Example of variable escaping to stack.
type user struct {
name string
name string
}
fucn main() {
u1 := stayOnStack() //main receives it's own copy of value
u2 := escapeToHeap()
}
// u does not escape stack
func stayOnStack() user {
u := user {
name: "Bill",
email: "[email protected]",
}
return u
}
// u escapes to heap
func escapeToHeap() *user {
u := user{
name: "Bill",
email: "[email protected]"
}
return &u
}
Escape analysis is performed at compile-time to determine if a value will continue to be referenced after a stack frame is removed. In escapeToHeap the address of the allocated user is passed to main, so it must be allocated to the heap. This happens when you "share up" the stack (call a function that allocates and returns an address to a value). "sharing down" the stack never needs to allocate to heap.
Building with go build -gcflags -m gives more in-depth feedback on compiler decisions, including heap allocations, inlining, etc.
Building with go build -gcflags -S shows assembly output. Plan9? assembly?
Every function as a preamble which declares how much stack space it needs. Used to check if stack is large enough for function call, and grows stack accordingly.
Go uses continguous stacks so that all pointers can be updated relative to each other during a grow.
An example showing the stack growing. If we increase const size, a larger int array is allocated in each stack frame, forcing the stack to grow. The output will show the address of s changing when we have to grow the stack:
// Number of elements to grow each stack frame.
// Run with 10 and then with 1024
const size = 10
// main is the entry point for the application.
func main() {
s := "HELLO"
stackCopy(&s, 0, [size]int{})
}
// stackCopy recursively runs increasing the size
// of the stack.
func stackCopy(s *string, c int, a [size]int) {
println(c, s, *s)
c++
if c == 10 {
return
}
stackCopy(s, c, a)
}
Output w/ size == 10:
0 0x10327f80 HELLO
1 0x10327f80 HELLO
2 0x10327f80 HELLO
3 0x10327f80 HELLO
4 0x10327f80 HELLO
5 0x10327f80 HELLO
6 0x10327f80 HELLO
7 0x10327f80 HELLO
8 0x10327f80 HELLO
9 0x10327f80 HELLO
Output w/ size == 1024:
0 0x10347f78 HELLO
1 0x1034ff70 HELLO //stack grew
2 0x1034ff70 HELLO
3 0x1034ff70 HELLO
4 0x1034ff70 HELLO
5 0x1035ff68 HELLO //stack grew
6 0x1035ff68 HELLO
7 0x1035ff68 HELLO
8 0x1035ff68 HELLO
9 0x1035ff68 HELLO
GC has change a lot across different releases. This is how 1.6 works.
Goal: Reduce the pressure on the garbage collector.
GC has one knob:
WB == write barrier. While active, all memory writes pass through barrier (so that GC stays informed during sweep). These writes are marked as "uncertain" or "grey". STW == stop the world. No memory writes.
Steps for GC. Mostly standard mark-and-sweep, but with a third color added (grey) so that we don't have to STW the world for an extended time: 1. Off: GC is disabled, pointer writes are just direct memory writes: *slot = ptr 2. Stack scan: WB starts. Briefly STW to collect ptrs from globals & goroutine stacks. 3. Mark: Mark objects (turn from white to black) and follow ptrs until ptr queue is empty. 4. Mark termination: STW starts. Rescan global & changed stacks, finish marking (for grey objects), shrink stacks. 5. Sweep: WB/STW off. Reclaim unmarked objects (white objects) as needed. Adjust GC pacing for next cycle 6. Off: turn off until next cycle.
One of Jean Paul's most favorite things. Go has a novel approach.
Constants in Go is a value that only exists at compile time. Only numeric types (int, bool, float, etc), can be constants. The minimum precision for a constant is 256 bits, geared for representing high-precision values.
Can declare a constant of a type and a kind. If it has a type, it has to live by that type's rules (precision, etc). If it has a kind, it can be implicitly converted to a type at compile time:
// Untyped constants recieve a 'kind'
const ui = 12345 // kind: integer
const uf = 3.141592 // kind: floating-point
// Typed constants use the constant type system, but precision is restricted
// based on declared type
const ti int = 12345 // type: int
const tf float64 = 3.141592 // type: float64
const bigInt = 2384234729587293475029347509238745092834 // allowed
//var bigInt int64 = 98729387420973094872304918273094172384 // compiler error, overflows
// this statement fails at compile time. fmt.Println needs to represent bigInt
// as an integer-type, but overflows
fmt.Println(bigInt)
Constants in declarations:
// Variable answer will be of type float64.
var answer = 3 * 0.333 // KindFloat(3) * Kind Float(0.333), but >=256-bit precision
// Constant third will be of kind floating-point
const third = 1 / 3.0 // KindFloat(1) / KindFloat(3.0)
// Constant zero will be of kind integer.
const zero = 1 / 3 // KindInt(1) / KindInt(3)
// Const arithmetic between type and untyped constatnts. Must have like types
const one int8 = 1
const two = 2 * one // int8(2) * int8(1)
Use type names ONLY IF you need a new representation of information. For example:
type duration int64
Duration is not an int64 value. It is it's own type that holds 64 bits of integer-like data.
var d duration
d := duration(1000)
nanosecond := int64(10)
d = nanosecond // compiler error, d isn't an int64, it's a duration
An example of constants from the time package:
type Duration int64
const (
Nanosecond Duration = 1
Mircosecond = 1000 * Nanosecond
...
)
// Add returns the time t+d. Function only accepts explicit Duration types, not integers
func (t Time) Add(d Duration) Time
// fiveSeconds is a typed constant of type Duration.
const fiveSeconds = 5 * time.Second // time.Duration(5) * time.Duration(1000000000)
now := time.Now()
// Subtract 5 nanoseconds from now time?
lessFiveNanoseconds := now.Add(-5) // -5 is interpreted as a Duration at compile time
// Subtract 5 seconds using a declared constant
lessFiveSeconds := now.Add(-fiveSeconds)
minusFive := int64(-5)
lessFiveNanoseconds = now.Add(minusFive) // FAILS! Compiler error, minusFive is not of type Duration
Some notes on scope in Go.
func main() {
var u *user
// u, err are attached to the if statement scope
if u, err := retrieveUser("sally"); err != nil {
fmt.Println(err)
return
}
// u is a zero-pointer
fmt.Printf("%+v\n", *u)
// INSTEAD
u, err := retrieveUser("sally")
if err != nil {
fmt.Println(err, u)
return
}
// Display the user profile
fmt.Printf("%+v\n", *u)
}
If returning multiple values, return the zero value, not a variable, if you are going to ignore a return value. Example:
// retrieveUser retrieves the user document for the specified
// user and returns a pointer to a user type value.
func retrieveUser(name string) (*user, error) {
// Make a call to get the user in a json response.
r, err := getUser(name)
if err != nil {
return nil, err //don't care about the user, return nil
}
// Unmarshal the json document into a value of
// the user struct type.
var u user
err = json.Unmarshal([]byte(r), &u)
return &u, err
}
If you don't understand the data you are working with, you don't understand the problem you are trying to solve.
Data transformation is the heart of solving problems. If your data is changing, the problem you are solving is changing.
Uncertainty about the data is not a license to guess, but a directive to STOP and learn more.
Coupling data together and writing code that produces predictable access patterns to the data will be the most performant.
Changing data layouts can yield more significant performance improvements than changing just the algorithms.
If performance matters, you must have mechanical sympathy for how the hardware and operating system work. Write code that has predictable access patterns to memory.
Ex: Access to main memory can be as large as 107 cycles. Every page miss incurs a huge cost. Object-oriented design inherently creates linked lists.
A contiguous block of memory. Commonly used for iteration, which is good for prediction. Most important data structure from a hardware perspective.
Arrays are well-defined at compile time (built-in type). Size must be fixed.
var strings [3]string //array of strings with 3 elements
strings[0] = "Apple"
strings[1] = "Orange"
strings[2] = "Plum"
// in one line
numbers := [4]int{1, 2, 3, 4}
// let the compiler determine the length
numbers2 := [...]int{1, 2, 3, 4, 5}
// shorthand to repeat a value in initialization
numbers3 := [...]int{4:0, 4:1, 4:0} //4 0s, then 4 1s, then 4 0s
An array of a certain size is considered it's own type:
var five [5]int
four:= [4]int{10, 20, 30, 40}
five = four // compiler error, cannot assign [4]int to [5]int
This allows arrays to be allocated in the stack (they can be added to a well-defined stack frame)
Slices are a part of Go's reference types: slices, maps, interface, channels, functions. Only pointers and reference types can be nil.
"The most important data structure in Go". Allows us to work with arrays in a "productive way".
Used with slices, channels, and maps. "Makes" the header value for that type, initializes the header value and the backing structure. Making a slice:
slice := make([]string, 5)
// then works like an array
slice[0] = "Apple"
slice[1] = "Orange"
slice[2] = "Plum"
// cannot access an index beyond the slice's length
slice[4] = "Ehhhh, runtime error"
fmt.Println(slice)
A three-word data structure. Pointer to backing array (contiguous memory), length of the array, and capacity of the array (always >= length).
DON'T TAKE THE REFERENCE OF A SLICE. Just copy the value of the slice header, for goodness sakes.
DON'T MAKE A SLICE OF POINTERS. You want data in contiguous blocks.
Recap, most important to hardware (optimal caching).
Most important for productivity, while keeping the predictable access of arrays.
// Create a slice with a length of 5 elements and a capacity of 8.
slice := make([]string, 5, 8)
slice[0] = "Apple"
slice[1] = "Orange"
slice[2] = "Banana"
slice[3] = "Grape"
slice[4] = "Plum"
fmt.Printf("Length[%d] Capacity[%d]\n", len(slice), cap(slice))
var data []string // a 'nil' slice of strings
data := []string{} // an empty slice of strings
Can only access elements up to "length", and there are "capacity" elements allocated in the backing array.
Prefer 'nil' slices on returns, unless you need to represent the slice as an empty list (for un/marshalling).
Slices can be appended to:
data = append(data, "another element")
Appends "another element" to the slice of strings. If len < cap, increase len, and add the element. If len == cap, grow the slice (copy the backing array into a larger backing array), then add the new element. The go runtime determines how much to increase the size of the backing array.
Rule of thumb, don't create a slice of pointers. Slices hold data or values.
Can create slices that use the same backing data:
// Create a slice with a length of 5 elements and a capacity of 8.
slice1 := make([]string, 5, 8)
slice1[0] = "Apple"
slice1[1] = "Orange"
slice1[2] = "Banana"
slice1[3] = "Grape"
slice1[4] = "Plum"
// Take a slice of slice1. We want just indexes 2 and 3.
// Parameters are [starting_index : (starting_index + length)]
slice2 := slice1[2:4]
// Can make a slice specifying a capacity
slice2 := slice1[2:4:4] //len 2, cap 2
slice2[0] = "CHANGED" // change value in backing array
slice2 = append(slice2, "OVERWRITE") // append to slice2 overwrites the next element past it's length (slice1[5] in this example)
Slice2 uses the same backing array as slice1. slice2[0] == slice1[2]. The capacity of slice2 is cap(slice1) - starting_index(slice2). If either slice has to grow, the other slice header will not reflect the changes. A new backing array is created for the growing slice, the old backing array remains for the other slice.
A 'nil' slice is still valid:
var myNil []int
myNil = append(myNil, 1) // works fine, myNil is a valid slice
All strings are valid UTF8 sequences, stored in bytes. If we iterate over a string:
// Declare a string with both chinese and english characters.
s := "世界 means world"
// Iterate over each character in the string.
for i := range s {
fmt.Printf("Index: %d\n", i)
}
Output:
Index: 0
Index: 3
Index: 6
Index: 7
Index: 8
Index: 9
Index: 10
Index: 11
Index: 12
Index: 13
Index: 14
Index: 15
Index: 16
Index: 17
Each rune may be 1-4 bytes.
Simple key-value structure.
// user defines a user in the program.
type user struct {
name string
surname string
}
// Declare and make a map that stores values
// of type user with a key of type string.
users := make(map[string]user)
// Add key/value pairs to the map.
users["Roy"] = user{"Rob", "Roy"}
users["Ford"] = user{"Henry", "Ford"}
users["Mouse"] = user{"Mickey", "Mouse"}
users["Jackson"] = user{"Michael", "Jackson"}
// Iterate over the map.
for key, value := range users {
fmt.Println(key, value)
}
NOTE: When you "range" over a map, the key are returned in a random order.
You can initialize your map directly:
// Declare and initialize the map with values.
users := map[string]user{
"Roy": {"Rob", "Roy"},
"Ford": {"Henry", "Ford"},
"Mouse": {"Mickey", "Mouse"},
"Jackson": {"Michael", "Jackson"},
}
Any value type is acceptable, but you cannot use any type for the key, it must be hashable:
type users []user
// Declare and make a map uses a slice of users as the key.
u := make(map[users]int)
// compiler error: invalid map key type users
You cannot define your own hasing function.
E.g. how to deal with change in your data. Need to build thin layers of abstraction so you can react to change without large changes to your code.
A function is called a 'method' when it is declared with a receiver. A receiver attaches behavior to types. In this example, we implement a method with a user receiver:
// user defines a user in the program.
type user struct {
name string
email string
}
// notify implements a method with a value receiver.
func (u user) notify() {
fmt.Printf("Sending User Email To %s<%s>\n",
u.name,
u.email)
}
There are two types of receivers: value receivers and pointer receivers. The previous example was a value receiver. A value receiver receives a copy of the calling structure, a pointer receiver shares the calling data structure.
// changeEmail implements a method with a pointer receiver.
func (u *user) changeEmail(email string) {
u.email = email
}
CONSISTENCY RULES THE DAY. Only use one type of receiver for any type.
Invoking a method:
// Values of type user can be used to call methods
// declared with a value receiver.
bill := user{"Bill", "[email protected]"}
bill.notify()
// Pointers of type user can also be used to call methods
// declared with a value receiver.
lisa := &user{"Lisa", "[email protected]"}
lisa.notify()
// Values of type user can be used to call methods
// declared with a pointer receiver.
bill.changeEmail("[email protected]")
bill.notify()
// Pointers of type user can be used to call methods
// declared with a pointer receiver.
lisa.changeEmail("[email protected]")
lisa.notify()
This demonstrates, if you mix value/pointer receivers, that the calling value is referenced/dereferenced accordingly when a method is invoked.
You can make methods on arbitrary types:
type duration int64
const (
nanosecond duration = 1
microsecond = 1000 * nanosecond
millisecond = 1000 * microsecond
second = 1000 * millisecond
minute = 60 * second
hour = 60 * minute
)
// setHours sets the specified number of hours.
func (d *duration) setHours(h float64) {
*d = duration(h) * hour
}
// hours returns the duration as a floating point number of hours.
func (d duration) hours() float64 {
hour := d / hour
nsec := d % hour
return float64(hour) + float64(nsec)*(1e-9/60/60)
}
myDuration := 1000 * hour
_ := myDuration.hours()
Each method has a function pointer and a data pointer. For pointer-receivers, the data pointer points to a pointer. For value-receivers, the data pointer points to a copy of the calling structure. Example of how this matters:
// Declare a function variable for the method bound to the d variable.
// The function variable will get its own copy of d because the method
// is using a value receiver.
f1 := d.displayName
// Call the method via the variable.
f1()
// Change the value of d.
d.name = "Lisa"
// Call the method via the variable. We don't see the change.
f1()
// =========================================================================
fmt.Println("\nCall Pointer Receiver Method with Variable:")
// Declare a function variable for the method bound to the d variable.
// The function variable will get the address of d because the method
// is using a pointer receiver.
f2 := d.setAge
// Call the method via the variable.
f2(45)
// Change the value of d.
d.name = "Joan"
// Call the method via the variable. We see the change.
f2(45)
Recommended layout: type -> factory functions -> methods.
Interfaces with the concept of composition, gives us the ability to create thin layers of abstractions. Provides polymorphism. Interfaces are reference types, e.g. there is some header information, and has a 'nil' zero value. It is a two-word structure. The first word is a pointer into the "ITable" which stores the type of the concrete-type and the method pointer of that type that implements the interface, the second word points to the concrete-type value that implements the interface (either a value or pointer).
Interfaces only declare behavior (no state). To implement an interface, a type must just define all of the intefaces methods.
// reader is an interface that defines the act of reading data.
type reader interface {
read(b []byte) (int, error)
}
// file defines a system file.
type file struct {
name string
}
// read implements the reader interface for a file.
func (file) read(b []byte) (int, error) {
s := "<rss><channel><title>Going Go Programming</title></channel></rss>"
copy(b, []byte(s))
return len(s), nil
}
You can use interfaces to create polymorphic functions:
// retrieve can read any device and process the data.
func retrieve(r reader) error {
data := make([]byte, 50)
fmt.Println(len(data))
len, err := r.read(data)
if err != nil {
return err
}
fmt.Println(string(data[:len]))
return nil
}
retrieve accepts any value/pointer of a concrete type that implements the reader interface.
// read implements the reader interface for a network connection.
func (pipe) read(b []byte) (int, error) {
s := `{name: "bill", title: "developer"}`
return copy(b, []byte(s)), nil
}
func main() {
// Create two values one of type file and one of type pipe.
f := file{"data.json"}
p := pipe{"cfg_service"}
// Call the retrieve funcion for each concrete type.
retrieve(f)
retrieve(p)
}
A regular naming convention for single-method intefaces, append 'er' or 'or' after the method name, ex read method -> reader inteface, write method -> writer inteface, select method -> selector interface.
Dictates which methods belong to a value type, and which methods belong to a pointer type. These dem rules:
1. For values of type T, ONLY methods of value receivers belong to the type
2. For pointers of type T, methods with value AND pointer receivers belong to the type
Why? Integrity. You can't always guarantee that you can get the address of a value that implements an interface.
// duration is a named type with a base type of int.
type duration int
// notify implements the notifier interface.
func (d *duration) notify() {
fmt.Println("Sending Notification in", *d)
}
func main() {
duration(42).notify()
// ./example3.go:18: cannot call pointer method on duration(42)
// ./example3.go:18: cannot take the address of duration(42)
}
In this case, *duration implements notify, but duration(42) does not have an address (it's a constant), so it does not.
Embedding is quasi-inheritance. You can add an inner-type to types, promoting it's state and methods to the outer-type:
// user defines a user in the program.
type user struct {
name string
email string
}
// notify implements a method that can be called via
// a pointer of type user.
func (u *user) notify() {
fmt.Printf("Sending user email To %s<%s>\n",
u.name,
u.email)
}
// admin represents an admin user with privileges.
type admin struct {
user // Embedded Type
level string
}
func main() {
// Create an admin user.
ad := admin{
user: user{
name: "john smith",
email: "[email protected]",
},
level: "super",
}
// We can access the inner type's method directly.
ad.user.notify()
// The inner type's method is promoted.
ad.notify()
}
How to write APIs in Go. How to organize your source code.
Every package is a reusable library. Each library provides one piece and only one piece.
The biggest problem that every team has when starting with Go. You need 4 packages (as a start): 1. Log: Where do I send logs? 2. Config: What happens when configuration changes? How is it deployed? 3. Trace: How to trace a request through the system. 4. Metrics: Assess the health of the system
Every repo is a project. If the project is building binaries, it has three top-level folders: - vendor: All of the packages that are being used, but not owned by this project. It must OWN (not lease) all of the source code. You should only have to download one repo. Use go-vendor/godep to keep packages up to date. Don't like glide because you lease. - cmd: Has a subfolder for every product we are building. All the information is here for building the package/binary - internal: Packages that can only be used internal to the project. The compiler specifically denies importing any packages under an "internal" directory. Can only be imported by code within the project itself.
Have a 'kit' project ardenlabs example. Common tools that are used across many projects. Packages here need to have the highest level of decoupling. "The only thing you are allowed to import is the standard library"
Rules of thumb: no capital letters or underscores in folder names. Every package has one source code file that is named after it.
Identifiers in a packaged are either exported or unexported.
If the first letter of any identifier is a capital letter, it is exported and can be viewed outside of its package.
If the first letter of any identifier is a lowercase letter, it is unexported and cannot be viewed from outside of its package.
Rules of thumb: if you are returning a type out of a package, make sure it is exported. In struct fields, seperate exported, and unexported structs.
An import is a physical location on disk, relative to your GOPATH. This includes both GOPATH/src/ AND your vendor folder.
It is idiomatic to separate std library imports from everything else. Some like std library, internal packages, and external vendored packages.
Must save your go files as UTF8.
Only two aliases in Go (nothing else can be aliased). Rune == int32, byte == uint8.
Questions:
- Don't get casting vs conversion. Where do we lose integrity? What is the cost of
c := int32(10)? Extra allocation? - Padding. If you but the bool at the end, doesn't the next struct still have to be padded? In between structs, you still have to pad the bool
- Doesn't 32/64 bit integer (based on architecture) mess with integrity? How do we know the cost of a line of code? Account for all architectures?