jonson

Jonson

A library allowing you to expose API endpoints using JSON-RPC 2.0.

You will be able to expose functions either using:

• a http endpoint per rpc
• a single http endpoint serving all calls
• websocket

In order to do so, Jonson consists of:

• a server which exposes either the http endpoint(s) and/or a websocket connection
• a factory which allows you to provide functionality to your API endpoints
• parameter validation (coming soon)
• error message encryption/decryption to hide sensitive information from the client

Project structure

Jonson thinks in systems. A system is a set of things that, as a whole, form emergence. Systems also tend to interact with other systems. As a result, we would be talking of a system of systems.

Let's assume an auth service (a system by itself). An auth system consists of authorization and authentication (system of systems). The ideal folder structure for a Jonson project, following the systemic approach, would look something like this:

/<project-name>
  /cmd
    /server
      main.go
  /internal
    /systems
      /authorization
        authorization.go
      /authentication
        authentication.go 
/go.mod

Remote procedure calls

When following the systemic approach, we can now start implementing our remote procedure calls. Let's follow the example of an auth service. The authentication endpoint might need functions like register, login and logout. Within the autentication/authentication.go folder, we can now set up our remote procedure calls.

The remote procedure calls will be generated by the server (explained later). In order to expose the endpoints properly, we need to follow a naming scheme: <MethodName>V<version>.

A remote procedure call accepts parameters (optional) and returns a result (optional) or an error.

To detect parameters which need to be marshaled/unmarshaled during the request, add a jonson.Params interface within your parameters which you will be sending.

In order to validate parameters, make the RegisterV1Params implement jonson.ValidatedParams interface. By doing so, before each function call Jonson will make sure that the JonsonValidate() function will be called. In case any errors have been added to the v *Validator, Jonson won't execute the given function.

// Authentication is our authentication system
type Authentication struct {
}

func NewAuthentication() *Authentication {
  return &Authentication{}
}

type RegisterV1Params {
  jonson.Params
  Username string `json:"username"`
  Password string `json:"password"`
}

func(r *RegisterV1Params) JonsonValidate(v *jonson.Validator){
  if len(r.Username) > 20 || len(r.Username) < 5{
    v.Path("username").Code(10000).Message("insufficient length")
  }
  if len(r.Password)  < 8{
    v.Path("password").Code(10001).Message("insufficient length")
  }
}

type RegisterV1Result struct {
  Uuid string `json:"uuid"`
}

// RegisterV1 allows us to register a new account
func (a *Authentication) RegisterV1(ctx *jonson.Context, params *RegisterV1Params) (*RegisterV1Result, error) {
  if (len(params.Username) <= 5){
    return nil, jonson.ErrInvalidParams
  }
  // put your register logic here
  return &RegisterV1Result {
    Uuid: "27fd79d0-e776-41c4-809a-3d1865b4f729",
  }, nil
}

type LoginV1Params struct {
  Username string `json:"username"`
  Password string `json:"password"`
}

// LoginV1 allows an account to log in
func (a *Authentication) LoginV1(ctx *jonson.Context, params *LoginV1Params) error {
  // put your login logic here
  return nil
}

// LoginV1 allows an account to log in
func (a *Authentication) LogoutV1(ctx *jonson.Context) error {
  // put your logout logic here
  return nil
}

For more complicated parameters and their validation, you can also provide validators on nested structs, such as:

type Profile struct {
  Name string
  Address *Address `json:"address,omitempty"`
}

func(p *Profile) JonsonValidate(v *jonson.Validator){
  if len(a.Name) < 2{
    v.Path("name").Message("name insufficient")
  }
  if (p.Address != nil){
    v.Path("address").Validate(p.Address)
  }
}

type Address struct {
  Street string `json:"street"`
  Zip string `json:"zip"`
}

func(a *Address) JonsonValidate(v *jonson.Validator){
  if len(a.Street) < 2{
    v.Path("street").Message("street insufficient")
  }
  if len(a.zip) < 2{
    v.Path("zip").Message("zip insufficient")
  }
}

The validator allows you to optionally set Debug(msg string) and Code(code int) to the error. In case code is not available, jonson.ErrInvalidParams' code will be used. The debug message will be encrypted and added to the error details using jonson.Secret.

Factory

Let's assume, the account wants to have access to a database or the current time. We could provide the database to the Authentication system itself (by passing a parameter to the constructor and keeping a reference within the Authentication struct) or we start diving into the possibility of using a factory. A factory allows us to define certain infrastructure or functional components during startup and provide those functional components at runtime.

Going back to the "auth service" example, let's see how to add a component that provides database access and one that provides the curent time. First, we would create a new folder internal/infra which will contain all files that implement our infrastructure setup. We can now create a new InfrastructureProvider:

type InfrastructureProvider struct {
  db *sql.Db
  newTime func() time.Time
}

func NewInfrastructureProvider(db *sql.Db, newTime func() time.Time) *InfrastructureProvider {
  return &InfrastructureProvider{
    db: db,
    newTime: newTime,
  }
}

// @generate
type DB struct {
  *sql.DB
}

func (i *InfrastructureProvider) NewDB(ctx *jonson.Context) *DB {
  return &DB{
    DB: i.db,
  }
}

// @generate
type Time struct {
  time.Time
}

func (i *InfrastructureProvider) NewTime(ctx *jonson.Context) *Time {
  return &Time{
    Time: i.newTime()
  }
}

In order for the providers to work, Jonson needs you to follow a specific naming scheme: the functions providing a type need to start with the keyword "New" followed by the type the provider instantiates, such as: NewTime returning *Time.

NOTE: your providers need to return either a pointer to a struct or an interface.

You might have noticed the // @generate tag: these are used to mark the types that we want to be able to 'inject' and use in our systems through the use of a Require<type> function that will be generated by the script jonson-generate. Since we tagged Time and DB in the example above, jonson-generate will create two functions for us:

func RequireTime(ctx *jonson.Context) *Time {
  // ...
}

func RequireDB(ctx *jonson.Context) *DB {
  // ...
}

To register the providers in the factory, use factory.RegisterProvider passing the pointer to the InfrastructureProvider. For details, check out the section "Putting it all together";

In case our provider is really simple, we can also use a single function:

// @generate
type ServiceName struct {
  Name string
}

func ProvideServiceName(ctx *jonson.Context) *ServiceName {
  return &ServiceName {
    Name: "auth",
  }
}

To register a simple provider function, use factory.RegisterProviderFunc passing the pointer to the InfrastructureProvider. Again, for details, check out the section "Putting it all together";

Once the types are provided by the factory, you can access them in your remote procedure calls:

// LoginV1 allows an account to log in
func (a *Authentication) LoginV1(ctx *jonson.Context, params *LoginV1Params) error {
  // the factory provides the database and we can now access it here in the code
  db := infra.RequireDB(ctx)

  // put your logic here
  return nil
}

The generated types will be instantiated once per API call and then stored within the context. In case a provider becomes invalid (e.g. we were storing a session provider and the account logged out), we can call the context.Invalidate method passing the type which we need to invalidate. The context allows us to also store new values on the fly (e.g. the user logged in and we want to provide a session) by calling context.StoreValue. NOTE: as a security feature, context.StoreValue will panic in case a provided value already exists;

In case you're calling a remote procedure from within a remote procedure, a new context will be created. However, some contexts you will want to share between those calls, such as time, http request/responses and more. For those contexts that are shareable between contexts, Jonson allows you to specify a provided type as jonson.Shareable:

// @generate
type Time struct {
  jonson.Shareable
  time.Time
}

Time will now be passed between contexts.

Some context values want to be finalized. Jonson allows you to specify a Finalize(err[]error) method on your provided types. In case a finalize method is found, it will be called after the remote procedure call within the context has been completed. You can e.g. clean up certain open connections within Finalize().

type Time struct {
  jonson.Shareable
  time.Time
}

func (t *Time) Finalize(err []error)error {
  t.Time = nil
  return nil
}

The Factory allows for specifying a Logger which will be used to output certain debug logging information. Per default a no-op-logger will be used which won't output any logging information. In case you woul like to inspect certain information from jonson, provide a logger:

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
factory := jonson.NewFactory(logger)

The logger will be provided to the underlying remote procedure calls using the factory by mounting a logger provider. Use jonson.RequireLogger(ctx) to get access to the logger.

Method handler

The method handler parses all remote procedure calls from registered systems using reflection and exposes methods to call those remote procedure calls. To register a system with the method handler use the function methodHandler.RegisterSystem().

For each call, the method handler will also make sure that the factory's providers will be provided to the called functions.

Besides those functions provided by the factory, the method handler will provide a few infrastructure related providers, such as:

func RequireHttpRequest(ctx *jonson.Context) *http.Request{}
func RequireHttpResponseWriter(ctx *jonson.Context) http.ResponseWriter{}
func RequireWSClient(ctx *jonson.Context) *jonson.WSClient{}
func RequireRpcMeta(ctx *jonson.Context) *jonson.RpcMeta{}
func RequireSecret(ctx *jonson.Context) jonson.Secret{}

The method handler will be passed to the exposing technology during startup, such as:

• websocket
• http
• a combination of the above

In most cases, you shouldn't need to use the method handler. Check out "Putting it all together" to see the method handler in action.

The method handler allows for setting optional (nil) options. You can specify the missing parameter validation level:

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
handler := jonson.NewMethodHandler(factory, secret, &jonson.MethodHandlerOptions{
  MissingValidationLevel: jonson.MissingValidationLevelError,
})

By setting a different value (MissingValidationLevelIgnore, MissingValidationLevelInfo, MissingValidationLevelWarn, MissingValidationLevelError, MissingValidationLevelFatal), you can modify the method handler's startup behaviour. In case of MissingValidationLevelIgnore, the validation on rpc params will be ignored. In case of MissingValidationLevelFatal, the application will panic during startup. All other states will log to the logger according to their level (info, warn, error).

Server

The server implements the standard http.Handler interface. You can either use the server.ListenAndServe() method directly or alternatively write your own server which can use the http.Handler interface provided by the server.

NewServer() accepts multiple Handlers which can be one of:

• rpc over http (using a single endpoint): jonson.HttpRpcHandler
• rpc over websocket: jonson.WebsocketHandler
• a single http endpoint per rpc: jonson.HttpMethodHandler
• default http handlers which use the http.Request and http.ResponseWriter functionality: jonson.HttpRegexpHandler

During startup, you can decide which endpoints you want to provide.

RPC over HTTP

The NewHttpRpcHandler will handle all registered remote procedure calls within a single endpoint which can be defined by the software developer.

The exposed http endpoint will only accept POST requests.

RPC over HTTP: one endpoint per method

The NewHttpMethodHandler will expose each remote procedure call as its own endpoint. By default, none of the endpoints will check for the correct http method. In case you want to enforce the usage of GET or POST, use jonson.HttpGet and jonson.HttpPost as parameters within your remote procedure call:

// LoginV1 allows an account to log in
func (a *Authentication) LoginV1(ctx *jonson.Context, _ jonson.HttpPost, params *LoginV1Params) error {
  // the factory provides the database and we can now access it here in the code
  db := infra.RequireDB(ctx)

  // put your logic here
  return nil
}

Now, the endpoint will only accept http calls using POST. In case the endpoint is called using a single endpoint for rpc or websocket, the required jonson.HttpPost has no effect.

Secret

In order to encrypt/decrypt server errors that should not be exposed to the client, jonson.Secret allows you to implement either your own encryption/decryption or use the built-in one with jonson.NewAESSecret(). For the AES secret, consider a key with 16, 24 or 32 bytes in length. In case the key does not have any of the above mentioned lengths, your program will panic.

For debugging purposes, you might want to use the jonson.NewDebugSecret() that will not encrypt/decrypt but simply pass the error to the rpc response.

Putting it all together

In our main, we can now spin up our remote procedure calls:

func main(){
  // in order to encrypt/decrypt our messages, we need a secret.
  secret := jonson.NewDebugSecret()

  // connect to mysql
  db := sql.MustConnect("")

  // let's initialize our providers first
  factory := jonson.NewFactory()

  // register a provider defining multiple provider instantiation methods
  factory.RegisterProvider(infrastructure.NewInfrastructureProvider(db, func(){
    return time.Now()
  }))

  // register a simple provider function
  factory.RegisterProviderFunc(infrastructure.ProvideServiceName)

  // let's instantiate our systems
  authentication := authentication.NewAuthentication()
  authorization := authorization.NewAuthorization()

  // let's expose the system's remote procedure calls to the method handler
  methodHandler := jonson.NewMethodHandler(factory, secret, nil)

  // let's register our systems with the method handler
  methodHandler.RegisterSystem(authentication)
  methodHandler.RegisterSystem(authorization)

  // right now, our systems are parsed by the method handler but not yet exposed.

  // the rpc handler will serve all remote procedure calls from the method handler
  // once calling the /rpc http endpoint
  rpcHandler := jonson.NewHttpRpcHandler(methodHandler, "/rpc")

  // the http method handler will expose all remote procedure calls
  // as their own endpoint, such as:
  // /authentication/login.v1
  // /authentication/logout.v1
  // ...
  httpHandler := jonson.NewHttpMethodHandler(methodHandler)

  // the ws handler will handle all incoming requests using websocket on the
  // http endpoint /ws
  wsHandler := jonson.NewWebsocketHandler(methodHandler, "/ws", jonson.NewWebsocketOptions())

  // the regexp handler allows us to define
  // regular expressions which will be handled
  // using the default http.Request and http.ResponseWriter.
  regexHandler := jonson.NewHttpRegexpHandler(methodHandler)
  regexpHandler.RegisterRegexp("/health", func(ctx *jonson.Context, w http.ResponseWriter, r *http.Request, parts []string){
    w.Write("UP")
  })


  // create a new server and handle all the technologies previously defined.
  server := jonson.NewServer(
    rpcHandler,
    httpHandler,
    wsHandler,
    regexpHandler,
  );

  // last step: let's listen and serve ;-)
  server.ListenAndServe(":8080")
}

NOTE: the server will ask each registered handler (rpc, ws, ...) whether they are eligible to serve a given endpoint in the order they were passed. The first one that returns "true", wins. In case your application is mostly used with websocket connections, it might be a good idea to pass the wsHandler as the first argument when calling jonson.NewServer().

Exposed paths

The methods a client will try to call can be exposed with different technologies as mentioned above (websocket, http rpc or http methods).

In case you are using http methods, the paths exposed will look like this: //.v. The system- and method names will be converted to kebab-case: account.GetProfileV1 will result in account/get-profile.v1. The params you send (body) needs to match the json specification of your rpc's params. The result will be returned within the body as json following your rpc's return value's json schema.

For successful remote procedure calls, the http status code will be 200. For errors during the call, the http status code will be in the 4xx and 5xx range - depending on the error that occured. The response body will contain the json rpc error as per specification.

In case you are using rpc over websocket or http, your methods will look the same. However, you will have to wrap the request in the jsonRpc request object.

{
  "jsonrpc": 2.0,
  "id": 1,
  "method": "<systemName>/<methodName>.v<version>",
  "params": {},
}

The response will reflect the id sent in the request. Each request should use its unique id per client to map the request to the response on the client's side.

{
  "jsonrpc": 2.0,
  "id": 1,
  "result": {}
}

In case of an error response, the client will receive no result but an error in the response.

{
  "jsonrpc": 2.0,
  "id": 1,
  "error": {
    "code": -32000,
    "message": "Internal server error"
  }
}

Error handling

Jonson predefines a few jsonRpc default errors which are described in the spec. You can either clone those and add your own data by calling e.g. jonson.ErrInvalidParams.CloneWithData(yourData) or define your own errors by using jonson.Error. A jsonRpc error consists of a message, a code and optional data. For further details on error messages, have a look at: jsonRpc error object

Advanced factory features

In most cases, you will use the providers using their generated RequireXXX functions, such as:

// LoginV1 allows an account to log in
func (a *Authentication) LoginV1(ctx *jonson.Context, params *LoginV1Params) error {
  // the factory provides the database and we can now access it here in the code
  db := infra.RequireDB(ctx)
  // put your logic here
  return nil
}

Additionally, Jonson allows you to use any provided type in your remote procedure call's parameters. In case the parameter is not providable and not of type jonson.Context or a remote procedure call jonson.Params, the function will not be called.

You can for example directly define the db as a parameter in your function and access it within your logic.

// LoginV1 allows an account to log in
func (a *Authentication) LoginV1(ctx *jonson.Context, db *infra.DB, params *LoginV1Params) error{
  // put your login logic here
  return nil
}

This feature comes in very handy in case you want to check whether an account is authenticated or not.

type AuthenticationProvider struct {

}

// @generate
type Private struct {}

func (a *AuthenticationProvider) NewPrivate(ctx *jonson.Context) *Private{
  req := jonson.RequireHttpRequest(ctx)
  sessionId := req.Cookie("sessionId")
  if (sessionId == ""){
    panic(jonson.ErrUnauthenticated)
  }
  // more logic here

  return &Private{}
}

Within your endpoint, you can now use Private as a safeguard. In case the calling user does not possess a valid session, the provider will panic and the function will never be callable.

type MeV1Result struct {
  Name string
}

// MeV1 returns my profile
func (a *Authentication) MeV1(ctx *jonson.Context, private *Private) (*MeV1Result, error) {
  // By now, we know that the user does possess a valid session.
  // We cano now safely proceed with the function's flow
  return &MeV1Result {
    Name: "Silvio"
  }, nil
}

Time provider

Since it's used in basically all applications, jonson comes with a pre-defined time provider.

The time provider allows you to also mock a timing instance during your tests (see: testing).

For production purposes, you will potentially want to use a real time provided to your remote procedure calls. Use jonson.RealTime to provide a real timestamp.

timeProvider := jonson.NewTimeProvider(func()jonson.Time{
  return jonson.NewRealTime()
})

Auth provider

Most applications need some sort of authentication. You can use the jonson.AuthProvider to create an authentication provider.

NewAuthProvider requires you to pass an auth client which implements IsAuthenticated and IsAuthorized.

IsAuthenticated: the account is logged in; IsAuthorized: the account is logged in and has access rights to the called route.

You will probably implement the client similar to the example below:

type AuthClient struct {
}

var _ jonson.AuthClient = (&AuthClient{})

func(a *AuthClient) IsAuthenticated(ctx *jonson.Context)(*string, error){
  req := jonson.RequireHttpRequest(ctx)
  cookie, err := req.Cookie("session")
  if err != nil{
    // missing session cookie
    return nil, nil
  }
  value := string(cookie.Value)
  // look up the session in your database or remote system
  var accountUuid string
  err := db.Get(&accountUuid, `...`)
  if err != nil {
     // db connection error?
    return nil, err
  }
  return &accountUuid, nil
}

func(a *AuthClient) IsAuthorized(ctx *jonson.Context)(*string, error){
  req := jonson.RequireHttpRequest(ctx)
  // we need the meta from the request to check whether
  // the account is able to call the underlying method
  meta := jonson.ReuqireRpcMeta(ctx)
  cookie, err := req.Cookie("session")
  if err != nil{
    // missing session cookie
    return nil, nil
  }
  value := string(cookie.Value)
  // look up the session in your database or remote system
  // _and_ make sure the account can access the current method
  var accountUuid string
  var canAccess bool
  canAccess, err := db.Get(&accountUuid, `...`, meta.Method)
  if err != nil {
     // db connection error?
    return nil, err
  }
  if (!canAccess){
    return nil, nil
  }
  return &accountUuid, nil
}

For nested in-process-calls of methods (e.g. method A calls method B using generated remote procedure calls), a new context is being forked. The new context makes sure to only copy values from context A to context B that have been explicitly marked as shareable. Let's assume method A is private an method B is private: caller Alice can access method A but cannot access method B; Since method A now tries to call method B, we must make sure to not provide jonson.Private to the context forked for the call towards method B; In case we would make private shareable, Alice (since she obtained access to method A) would implicitly gain access to method B. This could call a potential security risk.

Public, however, can be shared between forked contexts: a logged in user will remain authenticated (logged in) across contexts.

Testing

Jonson provides a package github.com/doejon/jonson/jonsontest which allows you to quickly spin up a test context boundary. Within your test contexts, you will be able to call any API endpoint.

factory := jonson.NewFactory()
factory.RegisterProvider(NewAuthenticationProvider())
secret := jonson.NewDebugSecret()
methodHandler := jonson.NewMethodHandler(factory, secret, nil)
methodHandler.RegisterSystem(NewAccount())

t.Run("gets profile", func(t *testing.T) {
  contextBoundary := jonsontest.NewContextBoundary(t, factory, methodHandler)
  var p *GetProfileV1Result
  contextBoundary.MustRun(func(ctx *jonson.Context) (err error) {
    p, err = GetProfileV1(ctx, &GetProfileV1Params{
      Uuid: testUuid,
    })
    return err
  })
  if p.Name != "Silvio" {
    t.Fatalf("expected name to equal Silvio, got: %s", p.Name)
  }
})

The test context boundary is pre-equipped with functions to provide a http.Request and http.ResponseWriters by using contextBoundary.WithHttpSource(). In case needed, you can also specify your RpcMeta by using contextBoundary.WithRpcMeta().

In case you want to mock a time during testing, use jonsontest.NewFrozenTime() or jonsontest.NewReferenceTime():

factory := jonson.NewFactory()
factory.RegisterProvider(NewAuthenticationProvider())

frozenTime := jonsontest.NewFrozenTime()

factory.RegisterProvider(jonson.NewTimeProvider(func(){
  return frozenTime
}))

secret := jonson.NewDebugSecret()
methodHandler := jonson.NewMethodHandler(factory, secret, nil)
methodHandler.RegisterSystem(NewAccount())

t.Run("gets profile", func(t *testing.T) {
  // do something
  frozenTime.Add(time.Second * 10)
  // do something 10 seconds later
})

Testing auth

For projects relying on jonson.Private and jonson.Public for authorization and authentication, you can use jonsontest.AuthClientMock to mock callers towards your remote procedure calls.

fac := jonson.NewFactory()

// create a new auth client mock and pass the mock
// towards the auth provider
mock := jonsontest.NewAuthClientMock()
fac.RegisterProvider(jonson.NewAuthProvider(mock))

mtd := jonson.NewMethodHandler(fac, jonson.NewDebugSecret(), nil)
mtd.RegisterSystem(&System{})

// create a new account (super user in this case) which has access
// to everything
accSuperUser := mock.NewAccount("e6dd1e60-8969-4f08-a854-80a29b69d7f3").Authorized()

t.Run("accSuperUser can access set and get", func(t *testing.T) {
  // provide the super user to the context boundary - the account will now be the calling account
  // of your tests
  NewContextBoundary(t, fac, mtd, accSuperUser.Provide).MustRun(func(ctx *jonson.Context) error {
    // call your generated remote procedure call methods
    return GetV1(ctx)
  })
  NewContextBoundary(t, fac, mtd, accSuperUser.Provide).MustRun(func(ctx *jonson.Context) error {
    return SetV1(ctx)
  })
})

Feel free to create as many test accounts as necessary. The test account allows you to specify the behavior of the created account:

// generate an account that is neither authenticated nor authorized
acc1 := mock.NewAccount("e6dd1e60-8969-4f08-a854-80a29b69d7f3")

// generate an authenticated account (logged in)
acc2 := mock.NewAccount("e6dd1e60-8969-4f08-a854-80a29b69d7f3").Authenticated()

// generate an account that has access to everything
acc3 := mock.NewAccount("e6dd1e60-8969-4f08-a854-80a29b69d7f3").Authorized()

// generate an account that has access to specific methods only
acc4 := mock.NewAccount("e6dd1e60-8969-4f08-a854-80a29b69d7f3").Authorized(&jonsontest.RpcMethod{
  RpcHttpMethod: jonson.RpcHttpMethodPost,
  method: "/user/get.v1"
})

Authorized accounts are also authenticated (logged in). No need

Code generation

To create types for internal remote procedure calls (in between systems) as well as to generate the RequireProvider() functions, use the jonson generator.

To generate types in a system (or provider), add the following line to one of your system's files.

package example

//go:generate go run github.com/doejon/jonson/cmd/generate

For projects forking jonson, you can provide your own jonson import as a flag during code generation:

//go:generate go run github.com/doejon/jonson/cmd/generate -jonson=github.com/doejon/jonson

Using go generate ./..., you should now see two new files being created within your system containing providers and remote procedure calls: jonson.procedure-calls.gen.go and jonson.providers.gen.go.

The procedure calls file contains all remote procedure calls specified within the current system. These helper methods allow us to call another system's procedure without doing an http round trip.

In order to trigger code generation, tag the types that should be requirable with // @generate.

Current generation limitations: The generator currently only works with the default method name used within jonson.

Dedication

The whole idea for this library was born after a long iteration period with dear friends. It is heavily influenced by one of the best mentors of my (professional) life.