Practical Golang: Using websockets

Introduction

This is the first post in the practical Golang series. Posts in it are meant to provide short and informative introductions to various topics.

This one is a about websockets, which are an awesome and easy way to provide communication between your web app and server.

Here we will use the gorilla websocket library, but you could also use a few others.

We will create two basic apps which should cover most day to day usage:
1. A client subscribing to a server to get continues information.
2. A client-ping server-pong app.

Dependencies

Make sure to go get:

go get github.com/gorilla/websocket

What’s the theory?

  1. The client connects to your server using his web browser.
  2. He gets back the website.
  3. He connects to your server through his websocket client using javascript.
  4. Your server accepts it using a standard http handler.
  5. You create a websocket connection from the http connection.
  6. You communicate with the client.
  7. The connection gets closed by one of the sides.

Creating the subscription app

Preparations

Next to our main go file we will need a html folder to place our html file in.

Let’s name it creatively, like index.html

Now the contents:

<!DOCTYPE HTML>
<html>
<head>

    <script type="text/javascript">
         function myWebsocketStart()
         {
               var ws = new WebSocket("ws://localhost:3000/websocket");

               ws.onmessage = function (evt)
               {
                  var myTextArea = document.getElementById("textarea1");
                  myTextArea.value = myTextArea.value + "\n" + evt.data
               };

               ws.onclose = function()
               {
                  var myTextArea = document.getElementById("textarea1");
                  myTextArea.value = myTextArea.value + "\n" + "Connection closed";
               };

         }

    </script>

</head>
<body>
<button onclick="javascript:myWebsocketStart()">Subscribe</button>
<textarea id="textarea1">MyTextArea</textarea>
</body>
</html>

I’ll just go over it quickly as the main subject here is the go code.

We create a button and a textarea, after the user clicks the button he connects to our websocket. Whenever he receives a message, or the connection gets closed, we append the info to our textarea.

We will also need our, so creatively named, main.go file, with the basic structure and file server written:

package main

import (
    "github.com/gorilla/websocket"
    "net/http"
    "os"
    "fmt"
    "io/ioutil"
    "time"
    "encoding/json"
)

func main() {
    indexFile, err := os.Open("html/index.html")
    if err != nil {
        fmt.Println(err)
    }
    index, err := ioutil.ReadAll(indexFile)
    if err != nil {
        fmt.Println(err)
    }
    http.HandleFunc("/websocket", func(w http.ResponseWriter, r *http.Request) {
    })
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, string(index))
    })
    http.ListenAndServe(":3000", nil)
}

Awesome, now let’s create the websocket part.

Writing the websocket code

Little bit of planning

Our server will create a Person object containing a name and age in seconds. Every two seconds it will send the client the current state of the person.

The meat

First we’ll need to define our Person type:

type Person struct {
    Name string
    Age  int
}

We’ll also need to create an upgraded variable, in which we define our read and write buffer sizes.

var upgrader = websocket.Upgrader{
    ReadBufferSize: 1024,
    WriteBufferSize: 1024,
}

Now, how do we create the websocket connection? Pretty easily in fact:

http.HandleFunc("/websocket", func(w http.ResponseWriter, r *http.Request) {
  conn, err := upgrader.Upgrade(w, r, nil)
  if err != nil {
    fmt.Println(err)
    return
  }
  fmt.Println("Client subscribed")
}

That’s all we need to have a client. Now let’s create Bill, our person, right after we get the client subscribed:

fmt.Println("Client subscribed")

myPerson := Person{
  Name: "Bill",
  Age:  0,
}

Now we need the main websocket handling code, which we will wrap into an endless for loop, which we get out of only if the channel closes or Bill gets 40 seconds old.

for {
  time.Sleep(2 * time.Second)
  if myPerson.Age < 40 {
    myJson, err := json.Marshal(myPerson)
    if err != nil {
      fmt.Println(err)
      return
    }
    err = conn.WriteMessage(websocket.TextMessage, myJson)
    if err != nil {
      fmt.Println(err)
      break
    }
    myPerson.Age += 2
  } else {
    conn.Close()
    break
  }
}
fmt.Println("Client unsubscribed")

We send the messages using conn.WriteMessage in which we specify the message type, can be binary or text, and the content. If Bill is 40 years old or more, we break out of the loop. So far so good, but what if we want bidirectional communication?

Creating the ping-pong app

Preparations

As before, we will need a html folder for our html file with the creative name index.html

And here’s the code:

<!DOCTYPE HTML>
<html>
<head>

    <script type="text/javascript">
         function myWebsocketStart()
         {
               var ws = new WebSocket("ws://localhost:3000/websocket");

               ws.onopen = function()
               {
                  // Web Socket is connected, send data using send()
                  ws.send("ping");
                  var myTextArea = document.getElementById("textarea1");
                  myTextArea.value = myTextArea.value + "\n" + "First message sent";
               };

               ws.onmessage = function (evt)
               {
                  var myTextArea = document.getElementById("textarea1");
                  myTextArea.value = myTextArea.value + "\n" + evt.data
                  if(evt.data == "pong") {
                    setTimeout(function(){ws.send("ping");}, 2000);
                  }
               };

               ws.onclose = function()
               {
                  var myTextArea = document.getElementById("textarea1");
                  myTextArea.value = myTextArea.value + "\n" + "Connection closed";
               };

         }

    </script>

</head>
<body>
<button onclick="javascript:myWebsocketStart()">Start websocket!</button>
<textarea id="textarea1">MyTextArea</textarea>
</body>
</html>

The only differences are, that when we open the connection, we send a “ping” message and notify our user about it. Now, whenever we get back a “pong” message, we append it to our textarea and after 2 seconds we answer with a “ping” message again.

We will again need the basic go file structure with the upgrader defined already, and the connection created:

package main

import (
    "github.com/gorilla/websocket"
    "net/http"
    "os"
    "fmt"
    "io/ioutil"
    "time"
)

var upgrader = websocket.Upgrader{
    ReadBufferSize: 1024,
    WriteBufferSize: 1024,
}

func main() {
    indexFile, err := os.Open("html/index.html")
    if err != nil {
        fmt.Println(err)
    }
    index, err := ioutil.ReadAll(indexFile)
    if err != nil {
        fmt.Println(err)
    }
    http.HandleFunc("/websocket", func(w http.ResponseWriter, r *http.Request) {
        conn, err := upgrader.Upgrade(w, r, nil)
        if err != nil {
            fmt.Println(err)
            return
        }
    })
    http.HandleFunc("/", func(w http.ResponseWriter, r * http.Request) {
        fmt.Fprintf(w, string(index))
    })
    http.ListenAndServe(":3000", nil)
}

Writing the websocket code

Ok, so now, whenever we get a “ping” message, we wait 2 seconds and answer with a “pong” message. If we get anything else, we just close the connection.

conn, err := upgrader.Upgrade(w, r, nil)
if err != nil {
  fmt.Println(err)
  return
}
for {
  msgType, msg, err := conn.ReadMessage()
  if err != nil {
    fmt.Println(err)
    return
  }
  if string(msg) == "ping" {
    fmt.Println("ping")
    time.Sleep(2 * time.Second)
    err = conn.WriteMessage(msgType, []byte("pong"))
    if err != nil {
      fmt.Println(err)
      return
    }
  } else {
    conn.Close()
    fmt.Println(string(msg))
    return
  }
}

Using the ReadMessage function on our connection we get the type, content, and maybe error. We check the message and answer.

Conclusion

That’s actually all, have fun with it and build something great!

Getting started with Apache Spark and Scala on Windows

Introduction

This tutorial is intended for people who really need to run Apache Spark on windows. Usually it would be better to run it in a Linux VM or on Docker.

There are a few things that cause problems with Spark on windows. But using this way of installation I managed to minimize the impact.

Getting the needed files

First, download the spark-1.6.0-bin-hadoop2.6.tgz file from the Apache Spark Downloads website. You should also install a program to open it, for example 7zip.

Next, download hadoop2.6.0 for Windows

This is all we will need.

Putting everything together

Extract both files to their own folders.

Open the environment variables settings. Add your Spark bin folder to your PATH variable and create a new SPARK_HOME environment variable and set it to your Apache Spark base folder.

Next, create another environment variable named HADOOP_HOME and set it to your hadoop base folder.

Preparing the configuration files

Open the conf folder in your Apache Spark base location. There should be a file called log4j.properties.template, change the name to log4j.properties.

For the sake of convenience you can open the file and change the value of rootCategory from INFO to WARN.

Launching it

After trying different ways, this one seems to be the least error-prone:

To launch the master open the command line and type in:

spark-class.cmd org.apache.spark.deploy.master.Master

Next, open your browser, and navigate to: http://localhost:8080/

There you should see the address of your spark master, like spark://192.168.1.27:7077

Open another command line, and start one worker using:

spark-class.cmd org.apache.spark.deploy.worker.Worker masteraddress

That’s it, spark is running.

To run the spark shell, use:

spark-shell –master masteraddress

To submit a packaged application jar, use:

spark-submit –class mainclassname –master masteraddress packagedjarlocation

Have fun with it!

Getting started with OAuth2 in Go

Introduction

Authentication is usually a crucial part in any web app. You could always roll your own authentication mechanics if you wanted, however, this creates an additional barrier between the user and your web app: Registration.

That’s why OAuth, and earlier OAuth2, was created. It makes it much more convenient to log in to your app, because the user can log in with one of the many accounts he already has.

What we’ll cover in this tutorial

We will set up a web app with OAuth2 provided by Google. For this we’ll need to:
1. Create a web app in Google and get our ClientID and a ClientSecret.
2. Put those into accessible and fairly safe places in our system.
3. Plan the structure of our web app.
4. Make sure we have the needed dependencies.
5. Understand how OAuth2 works.
6. Write the application logic.

Let’s begin.

Creating a project in Google and getting the client ID and secret

First, go to the Google Cloud Platform and create a new project. Later open the left menu, and open the API Manager. There, search for the Google+ API and enable it.

Next, open the credentials submenu, and choose Create credentials -> OAuth client ID. The application type should be Web application and give it a custom name if you want. In “Authorized JavaScript origins” put in the address of the site you’ll be login in from. I will use http://localhost:3000. Then, in the field Authorized redirect URLs put in the address of the site, to which the user will be redirected after logging in. I’ll use http://localhost:3000/GoogleCallback.

Now the client ID and client secret should be displayed for you. Write them down somewhere safe. Remember that the client secret has to stay secret for the entire lifetime of your app.

Safely storing the client ID and secret

There are many ways to safely store the client ID and secret. In production you should make sure that the client secret remains secret.

In this tutorial we won’t cover this. Instead, we will store those variables as system environment variables. Now:
* Create an environment variable called googlekey holding your client ID.
* Create an environment variable called googlesecret holding your client secret.

Planning the structure

In this tutorial we’ll write code in one file. In production you would want to split this into multiple files.

Let’s start with a basic go web app structure:

package main

import (
  "fmt"
  "net/http"
)

func main() {
  fmt.Println(http.ListenAndServe(":3000", nil))
}

Now we’ll set up a simple site:

const htmlIndex = `<html><body>
<a href="/GoogleLogin">Log in with Google</a>
</body></html>
`

We will also need:
* The home page, where we will click the login button from.
* The page handling redirection to the google service.
* The callback page handling the information we get from Google.

So let’s set up the base structure for that:

func main() {
    http.HandleFunc("/", handleMain)
    http.HandleFunc("/GoogleLogin", handleGoogleLogin)
    http.HandleFunc("/GoogleCallback", handleGoogleCallback)
    fmt.Println(http.ListenAndServe(":3000", nil))
}

func handleMain(w http.ResponseWriter, r *http.Request) {
}

func handleGoogleLogin(w http.ResponseWriter, r *http.Request) {
}

func handleGoogleCallback(w http.ResponseWriter, r *http.Request) {
}

Dependencies

You will need to

go get golang.org/x/oauth2

if you don’t have it already.

Understanding OAuth2

To really integrate OAuth2 into our web application it’s good to understand how it works.
That’s the flow of OAuth2:
1. The user opens the website and clicks the login button.
2. The user gets redirected to the google login handler page. This page generates a random state string by which it will identify the user, and constructs a google login link using it. The user then gets redirected to that page.
3. The user logs in and gets back a code and the random string we gave him. He gets redirected back to our page, using a POST request to give us the code and state string.
4. We verify if it’s the same state string. If it is then we use the code to ask google for a short-lived access token. We can save the code for future use to get another token later.
5. We use the token to initiate actions regarding the user account.

Writing the application logic

Before starting remember to import the golang.org/x/oauth2 package.
To begin with, let’s write the home page handler:

func handleMain(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintf(w, htmlIndex)
}

Next we need to create a variable we’ll use for storing data and communicating with Google and the random state variable:

var (
    googleOauthConfig = &oauth2.Config{
        RedirectURL:    "http://localhost:3000/GoogleCallback",
        ClientID:     os.Getenv("googlekey"),
        ClientSecret: os.Getenv("googlesecret"),
        Scopes:       []string{"https://www.googleapis.com/auth/userinfo.profile",
            "https://www.googleapis.com/auth/userinfo.email"},
        Endpoint:     google.Endpoint,
    }
// Some random string, random for each request
    oauthStateString = "random"
)

The Scopes variable defines the amount of access we get over the users account.

Note that the oauthStateString should be randomly generated on a per user basis.

Handling communication with Google

This is the code that creates a login link and redirects the user to it:

func handleGoogleLogin(w http.ResponseWriter, r *http.Request) {
    url := googleOauthConfig.AuthCodeURL(oauthStateString)
    http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

We use the googleOauthConfig variable to create a login link using the random state variable, and later redirect the user to it.


Now we need the logic that get’s the code after the user logs in and checks if the state variable matches:

func handleGoogleCallback(w http.ResponseWriter, r *http.Request) {
    state := r.FormValue("state")
    if state != oauthStateString {
        fmt.Printf("invalid oauth state, expected '%s', got '%s'\n", oauthStateString, state)
        http.Redirect(w, r, "/", http.StatusTemporaryRedirect)
        return
    }

    code := r.FormValue("code")
    token, err := googleOauthConfig.Exchange(oauth2.NoContext, code)
    if err != nil {
        fmt.Println("Code exchange failed with '%s'\n", err)
        http.Redirect(w, r, "/", http.StatusTemporaryRedirect)
        return
    }

    response, err := http.Get("https://www.googleapis.com/oauth2/v2/userinfo?access_token=" + token.AccessToken)

    defer response.Body.Close()
    contents, err := ioutil.ReadAll(response.Body)
    fmt.Fprintf(w, "Content: %s\n", contents)
}

First we check the state variable, and notify the user if it doesn’t match. If it matches we get the code and communicate with google using the Exchange function. We have no context so we use NoContext.

Later, if we successfully get the token we make a request to google passing the token with it and get the users userinfo. We print the response to our user.

Conclusion

That’s all we have to do to integrate OAuth2 into our Golang application. I hope that I helped someone with this problems as I really couldn’t find beginner-suited, detailed resources about OAuth2 in Go.

Now go and build something amazing!

Update: if you plan to integrate OAuth2 into the Authentication of your app, make sure to read this too: http://oauth.net/articles/authentication/
Thanks to tbroyer for providing the link.