Who’s who in eth2: Raul Jordan from Prysmatic Labs

Welcome to Who’s who in eth2, presented by Elias Simos, Protocol Specialist at Coinbase Cloud. In this series, Elias interviews key contributors to the development and growth of Ethereum and eth2. Here we discuss exploring their involvement in eth2, visions for the future, and perspectives on what eth2 means for the world.

The series includes interviews with notable builders, researchers, infrastructure experts, and leaders from the eth2 ecosystem, along with principal members of eth2’s four client teams: Prysmatic Labs, Sigma Prime, Nimbus, and Teku.

In this first post, Elias interviews Raul Jordan, Co-lead Ethereum Developer at Prysmatic Labs. Raul shares with us, among other things, how the team first met, their open forum ethos and how it has been paying off for the business and industry, and what the future might hold. 

Q: Why don't we start with a little bit of background on yourself and your involvement with eth2?

My name is Raul Jordan. I'm one of the maintainers of Prysmatic Labs. I'm originally from Honduras and first came to the U.S., where I currently live, to study CompSci at Harvard.

Raul Jordan, Prysmatic Labs

I’ve always been fascinated by what you can do with a computer. The fact that you can use a keyboard and just type in magic words and all of a sudden create something that people on the other side of the world can use was pretty mind-blowing to me as a student.

During my time as a student, I was introduced to Ethereum. I met a few people from the community, for example the teams at Augur and Aragon, and was immediately intrigued.

Toward the end of 2017, after the whole hype of the bull run, conversations around the future of Ethereum were really ramping up. At the time, Vitalik [Buterin] put out a lot of content on the topic, as well as offering grants for work related to Ethereum scaling.

Back then, I was still an “outsider” wanting to break into the ecosystem, and scaling seemed like one of the most interesting problems to apply myself to. But at the time it was not obvious who was working on such problems. The only information available was something like one FAQ page and a bunch of scattered blog posts.

I'm somebody who likes to take problems into my own hands. I really think that if you work with the right people, you can accomplish anything. So I just gave it a shot. I put out a call on Reddit that went something like, “Hey, I’m a developer looking into Ethereum. Anyone want to collaborate with me on prototyping this sharding thing Vitalik talks about?”

Figure 1: Raul's first email to everybody who responded to the call-to-action reddit thread

And then I actually got a bunch of responses. The first one was from Preston Van Loon [who went on to become co-founder at Prysmatic Labs], saying he was interested. Next thing you know we are meeting up at his office in Google, where he was working full-time. We bounced ideas back and forth and settled on giving this a shot. Then suddenly we had around six or seven volunteers in the same boat.

Looking back, I think it was all about timing and being in the right place at the right time. We eventually managed to get a grant from the Ethereum Foundation to do this on a more incentivized basis. And that's how Prysmatic Labs came to be.

"I'm somebody who likes to take problems into my own hands. I really think that if you work with the right people, you can accomplish anything"

Q: Let’s dig a little deeper into the background of the team — how it all came together and the journey of Prysmatic Labs from inception to present day

Absolutely! I think if you want to pull off an organic, open-source initiative with people from around the world, you need to have people who are really motivated by a shared goal.

I started this effort with Preston while he was at Google. Two other people we had with us from day one — Terrence Tsao and Nishant Das — are still with us today; both are just stellar individuals. They were there from the beginning, always learning, curious, and wanting to improve.

Most of the early team were initially working two jobs. They would work as software engineers at their day jobs, then come back at five and work until midnight on this volunteer project. We then started getting a few more grants from the Foundation, and that's when things really took off.

I remember I was with Preston in an Ethereum meet-up when he told me: “The only reason I would leave Google is if I had a 10x opportunity.” And I think that, in terms of the opportunity that shipping eth2 would provide, it was more than enough for Preston and the other guys to jump ship and join full-time.

Q: What about the values that are core to Prysmatic Labs?

We want to showcase to people that we are grassroots. We are builders who like to bring good software engineering practices to the mix and be very community driven, which we have been from day one.

That's something that I noticed, personally, was lacking in the eth1 development space. People complained a lot about things happening behind closed doors. So we wanted this to be an open forum — and I think that drove the point home. I think by now we must have more than 10k people on our Discord server.

Figure 2: Subscribers on Prysmatic Labs' Discord server between November 202 and February 2021

"I think that our company culture is 'no nonsense.' We just want to build code that works and has been proven to work"

Q: How does that openness — having the community involved from early on in the development journey — impact the end product? Where do you think it really shines? Where do you think it introduces bottlenecks in the workflow that you wouldn't expect to have if the process was more controlled?

I think the openness is incredible — thinking about what it brings to the robustness of protocols. If you only have one implementation of the spec, in my view, you're 99% more likely to have serious bugs, meaning that you don't properly implement the specification. And you will never know, because there's no other implementation that could point you in the direction of production-level disagreements, such as blocks and state roots. Having multiple implementations is a big strength.

On the flipside, I would say wrangling interoperability between teams that are distributed all over the place can be challenging. Things improved as time went by, but initially — in part because of fragmented communication — we had to rewrite our code from scratch almost two or three times, because the specification changed under our feet.

And when there are disagreements, without a clear leader in the process, how do you resolve that conflict? You end up having this deadlock, and it feels like things are moving really slowly.

Overall, however, the relationship between all the teams has been really cordial. The Ethereum Foundation [EF] had a big role to play here, driving the research, and project managing the development at a high level. None of us would be here if we didn't have help from the likes of Danny Ryan and Protolambda from the Foundation. They are really the heroes of this.

At the end of the day, I think all the difficulties along the way have been for the better. We have a really stable mainnet so far.

Q: Regarding communication between the different teams that were implementing the specification, would you say it manifested as a hub-and-spokes flow, or more as a nexus where you had free cross-communication flow between the different implementers?

I would say it's probably more of a nexus approach. The EF is basically in charge of the research, but that doesn't mean they dictate it. The field was open from the beginning for broader members of the core implementer community to come in with feedback and ideas. Actually, I believe the top contributor outside of the core research team is Terrence from our team.

During the latter months of development, maybe six or seven months leading up to mainnet launch, everything was driven pretty much by client implementers, as the spec was frozen, meaning there were no major changes on the Phase 0 spec. In that time, all the teams were basically just communicating with each other, doing multi-client testing and such.

"If you only have one implementation of the spec, in my view, you're 99% more likely to have serious bugs, meaning that you don't properly implement the specification. And you will never know, because there's no other implementation that could point you in the direction of production-level disagreements such as blocks and state roots. Having multiple implementations is a big strength"

Q: Going back to the 10x opportunity, I have a sense that, based on the way you framed it, it's not only about 10x financial opportunity. What is it about eth2 that makes it a 10x opportunity in your mind?

What that means is that this really represents a breakthrough technology in computer science — and leaving a mark as one of the major implementations running the protocol is really what drives us to do what we do. The team has had incredible chemistry. We want to prove to people that we can ship this, that we can be a part of history.

Thinking further down the line, I think we can go in so many directions after the roadmap is complete. And I would love to do it with the current team that we have today.

Q: Let's now talk a little about the process of creating Prysm [the client]. What were some of the key design decisions that you made early on — thinking about software itself, as well as things outside of that?

The way we approached the problem from very early on was very hands-on. We wanted to take problems into our own hands and really build something useful.

When picking a programming language, often it comes down to personal preference or your past experience. But for us, we didn't have much experience in Go development [Prysm is built in Go]. But we wanted the software to be practical and made the intentional decision to build it in Go.

I think that our company culture is “no nonsense.” We just want to build code that works and has been proven to work.

So we picked Go as the programming language because I would say it's [been] the language of the cloud for the past four or five years. A significant portion of cloud services and large-scale applications that serve billions of users today are written in Go. And because of the large developer base that builds with Go there is a very long library of tools that we can draw from in the development process; that means we don’t need to go write an API or a json RPC from scratch.

As an example, in Prysm we use protocol buffers, which are a transport mechanism for API payloads. And we use GRPC, which is an RPC library, for implementing these RPC services at scale.

These things serve, quite literally, billions of users on a day-to-day basis. They run all of Uber, they run all of Google, and countless other companies too. So we know these things work. We don't feel we need to reinvent the wheel.

Q: What about non-technical decisions?

One of the biggest things that helped us get to where we are today was deciding to launch a single-client public testnet for people to try out very early on. We didn't even have a peer-to-peer networking specification at the time we launched the first Prysm testnet. And we were like, screw it! We're going to build our own, because people want to try this out. They wanted to see what staking was like. They wanted to test the limits of the software.

We also were very deliberate about building a lot of documentation, onboarding guides etc, so people could engage with the software. Having users trying it out and giving us continuous feedback really helped to push the envelope. It wasn't until a year later that we had a multi-client testnet.

The Prysmatic Labs team

Q: That's a really great segue to the community-building part of things. You mentioned that having strong written communication really helped push things forward. What other approaches did you leverage to engage with the community that you think drove results?

Writing is for sure appreciated by the community. All of us on our team are always answering stakers’ questions. People jump in the Discord server and they're like, “What does this mean? Can you guys explain it?” And somebody always pops in and answers.

Over time, we built up a strong culture of DevOps. For example, when the Medalla incident happened, we followed our standard practice of doing post-mortems. We  go through everything and try to understand what happened. 

We want to take out learnings, identify where we got lucky, and ask: “How can we make sure this never happens again?” We built that culture of post-mortems and we made them public. 

In fact, the Medalla Medium post was basically just us copy-pasting that post-mortem into a blog post. Thankfully, now the eth2 initiative has a dedicated response team — the Blue Team — where you have people not just from our team, but from other teams and the Foundation, who are able to respond to incidents if and when they happen.

Q: I say client diversity is key for long-term success. You say...

Totally, I would say the same. I think it's really nice to be in a position where a lot of people are using your software — but it also comes with a lot more responsibility. So if something were to go wrong with Prysm, for example, we would really depend on all of the other clients to step up and keep the network going. Diversity helps the software become a lot more resilient.

Even if a client has a small share of the entire set of operators, it's still critical that that project is maintained and that it's supported by the parties that need to support it. If there's a serious issue in the network, and those clients are still well-maintained, they can help remediate these problems and people can migrate those clients.

The relationship we have with all the other teams has improved a lot over time. We appreciate each other, and we appreciate the improvements that everyone is making to their software.

"The key to the success that Prysm has had so far is that we focused a lot on the average staker"

Q: How do you think eth2 can achieve an appropriate level of client diversity?

The key to the success that Prysm has had so far is that we focused a lot on the average staker. I will tell you, the average staker right now most likely runs Windows, is somewhat technical and as such is aware of the risks and what it means to stake, but is not a guru on the command line — doesn't have a Linux set up in their living room, so to speak. So the first thing I’d say is to really focus on the user.

Another one is really good documentation, good communication, making sure you respond to feature requests. And eventually, making the software much easier to use through point-and-click interfaces.

Q: Do you think there are any spec-level changes in the cards that could incentivize a broader distribution of clients?

So that question is a difficult one. I think there are a lot of ways that you can try, but the problem is that you game those metrics. It's really difficult to create an incentive for beacon nodes that cannot be gamed — that's something to keep in mind. Honestly, the best we can do really is to focus on improving the bottom line.

Q: Thinking about the future, what do you think the biggest challenges that lie ahead for eth2 are?

Now, we have a ticking time bomb on our hands, in the sense that eth2 is live. We still have to ship Phase 1, add sharding, etc. That, in and of itself, is a separate beast. And these things need to be done in parallel. So, how do we prioritize our time as a team to accomplish this?

So far, we’ve relied a lot on the wisdom of the Ethereum Foundation on the research side for what to do next. They've been incredibly insightful. And now we need to ship the next phases ASAP — ideally this year.

That's the main challenge I see ahead. We’re on a treadmill that is not going to stop. We need to get going and to implement this stuff.

Q: Further down the line, after it’s all said and done, what do you see Prysmatic Labs becoming?

I want Prysmatic Labs to be a highly regarded core-implementation team. We are a team that thrives on experimentation, with a core focus on improving the Layer 1. And we want to keep going, even once that roadmap is done. 

I think our team can succeed at any future challenge. It will be a dream come true for me if we can build something on top of eth2 in the future. So whether that is a really interesting smart contract-based application or tackling scalability, I'll be really thrilled to do that.

Q: Final question: what's your favorite detail/anecdote about another client?

I thought it was supercool when Nimbus posted a photo of themselves running a small connected network of Raspberry Pis or something. It’s really amazing how little resources the client consumes. Like, any sort of device can probably run Nimbus — that is really cool. I think those guys are pushing the envelope of what's possible on lower-end devices.

And like I said, if there's a team or client that remains well-maintained, and is aligned with the specification, and works on mainnet, we need it to be used by people, we need it to keep going and to be supported. Hopefully one day they can get it running on a smart fridge or something similar. I think those guys have the ability to do it.

--

Interview by Elias Simos