Unexpected security footguns in Go's parsers

ORMs love to put people and projects in the situation where it's much easier for data types in your program to match exactly with the tables in your database, but reality often requires them to be decoupled instead. That "create user" request contains "isAdmin" smells like someone mapped that exact request to a table, and just because the table has "isAdmin", the data type now also has it.

What? It's about parsers in Go's standard library.

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end.

Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult.

There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.

1. As shown in the article, exposing the internal model to APIs directly is a security footgun. I've seen sites get hacked because the developer serialized internal objects without any oversight just like in the article and accidentally exposed secrets.

2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI).

In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems.

All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs.

Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't

> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs.

Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.

I don't know about Go, but in Java and .NET world this is trivially solvable with libraries like MapStruct. If you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC):

  record Profile(int id, String login, boolean isAdmin) {}

  interface UserMapper {
    // arrays are just one example, plain models and plenty
    // of other data structures are supported
    Profile[] usersToProfiles(User[] user);

    // other mappers...
  }

  class UserController {
    //
    @GET("/profiles")
    Profile[] getUserProfiles() {
      var users = userRepo.getUsers();
      return userMapper.usersToProfiles(users);
    }
  }

It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read.

https://mapstruct.org

Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.

> also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway.

Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea.

Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.

Covered in the article along with a potential snafu - adding anything else (`-,omitempty`) turns it into a key "-", not the "ignore this field" indicator.

Go's simplifications often introduce complexities elsewhere, however, as this article demonstrates with the complexities of correctness of a stringly-typed DSL.

There's no free lunch here, and the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.

> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )

Go actually ties visibility to casing, instead of using separate annotations. And it will not serialise private fields, only public.

Python has no concept of visibility at all, conventionally you should not access attributes prefixed with `_` but it won't stop you.

> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )

I think calling it a convention is misleading.

In Python, you can access an _field just by writing obj._field. It's not enforced, only a note to the user that they shouldn't do that.

But in Go, obj.field is a compiler error. Fields that start with a lowercase letter really are private, and this is enforced.

So I think it's better to think of it as true private fields, just with a... unique syntax.

Not just weird, it’s dangerous to do this - to easy to miss validation as fields are added over time. Better to explicitly validate all fields IMO.

> struct tags greatly reduce the boilerplate code required to map fields to fields.

Are you somehow under the impression that Go is unique in having a terse way to map fields to fields?

> It’s really quite novel once you understand it.

It's the opposite of novel, putting ad-hoc annotations in unstructured contexts is what people used to do before java 5.

It's not very novel. There's far better ways of solving this than allowing a random string to be embedded as aux information to a struct field. Examples: F# type providers, or OCamls PPX system for extending the language in a well defined way. Macro rewriting systems also allow for better safety in this area.

This allows you to derive a safe parser from the structural data, and you can make said parser be really strict. See e.g., Wuffs or Langsec for examples of approaches here.