Third Attempt: Length-prefixed Response
However, we still get the same ResourceExhausted error. Let’s double check that API Gateway is properly sending back a binary protobuf response.
To debug more, we can set:
export GODEBUG=http2debug=2
This will give us output about what is going back and forth over the wire for the HTTP/2 requests.
http2: Framer 0xc0002c8380: wrote SETTINGS len=0
http2: Framer 0xc0002c8380: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215
http2: Framer 0xc0002c8380: read WINDOW_UPDATE len=4 (conn) incr=2147418112
http2: Framer 0xc0002c8380: wrote SETTINGS flags=ACK len=0
http2: Framer 0xc0002c8380: wrote HEADERS flags=END_HEADERS stream=1 len=84
http2: Framer 0xc0002c8380: wrote DATA flags=END_STREAM stream=1 len=12 data="\x00\x00\x00\x00\a\n\x05Hello"
http2: Framer 0xc0002c8380: read SETTINGS flags=ACK len=0
http2: Framer 0xc0002c8380: read HEADERS flags=END_HEADERS stream=1 len=313
http2: Framer 0xc0002c8380: read DATA stream=1 len=15 data="\n\rHello, world."
http2: Framer 0xc0002c8380: read DATA flags=END_STREAM stream=1 len=0 data=""
err making request: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (222848364 vs. 4194304)
http2: Framer 0xc0002c8380: wrote WINDOW_UPDATE len=4 (conn) incr=15
HTTP/2 Response From API Gateway
We see that as our request goes up, it writes a DATA frame with the content “\x00\x00\x00\x00\a\n\x05Hello”. However, what we get back is “\n\rHello, world.”. What are all those \x00 values in the request? This turns out to be a special serialization format that gRPC uses called “Length Prefixed Messages”. On the wire, this looks like:
See the gRPC HTTP/2 protocol mapping for more detail.
Here’s a quick and dirty implementation of the prefix construction with an updated handler.
const (
payloadLen = 1
sizeLen = 4
headerLen = payloadLen + sizeLen
)
func msgHeader(data []byte) (hdr []byte, payload []byte) {
hdr = make([]byte, headerLen)
hdr[0] = byte(uint8(0))
// Write length of payload into buf
binary.BigEndian.PutUint32(hdr[payloadLen:], uint32(len(data)))
return hdr, data
}
func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
message := &api.AliveResponse{Message: "Hello, world."}
b, err := proto.Marshal(message)
if err != nil {
return events.APIGatewayProxyResponse{
StatusCode: 500,
}, err
}
hdr, data := msgHeader(b)
hdr = append(hdr, data...)
return events.APIGatewayProxyResponse{
Body: base64.StdEncoding.EncodeToString(hdr),
Headers: map[string]string{
"Content-Type": "application/grpc+proto",
"grpc-status": "0",
},
IsBase64Encoded: true,
StatusCode: 200,
}, nil
}