Hypercore in Rust
S01:E04

Hypercore in Rust

Episode description

In this episode we talk to Blake who’s working on a Rust implementation of the Hypercore protocol. Hypercore (formerly known as .dat and part of the dat ecosystem ) is one of the oldest P4P networks.

Blake presents two usecases for Hypercore as a protocol which have previously been under-explored but are highly promising—for RSS and for package-management—showcasing the versatility of P4P protocols and how there can be a larger ecosystem of P2P networks.

Also mentioned in the podcast is the Pear protocol, Mapeo. and the chat app Keet. Find Blake on Bluesky and Mastodon.

Download transcript (.srt)
0:00

is

0:26

Is it you specifically personally developing the rest implementation for Hypercore?

0:32

As of now, I think I'm the main, I'm basically the only developer doing it.

0:37

There's been other people in the past, and now it's all been handed over to me.

0:42

Damn.

0:43

Yeah.

0:44

That's a lot.

0:45

There's been waves of interest.

0:47

And I think, yeah, going back a little further around 2020, and then there was a lot of implementation

0:52

work.

0:53

And a few years ago, whenever I picked it up, one person was working on it, and then I

0:58

was like, I think I can push this over the finish line, and that's what I've been doing.

1:03

Wow.

1:04

How long has the finish line like two years?

1:07

I didn't really know what I was getting into.

1:10

And I hear you.

1:13

Yeah.

1:15

Welcome to today's episode of Solarcast.

1:24

And in this episode, we will be talking to Blake from the Hypercore protocol.

1:32

And specifically, Blake is working on, as you might have heard already in the intro, a

1:38

rust implementation of Hypercore.

1:42

And Hypercore is part of a big ecosystem.

1:46

And shout out to the previous podcast that was also touching on this, the DUT ecosystem

1:51

podcast.

1:53

And that is also another previous older name for the same protocol.

1:58

Specifically, Blake also talks us through some, for me at least, and I think for many,

2:05

new use cases for the Hypercore protocol.

2:09

And I learned a lot in this podcast, and I hope you find it as interesting as I did.

2:15

With no further ado, let's dive in.

2:21

Thank you so much for joining me here, Blake.

2:31

Thank you.

2:32

And you are in a very soon to be snowy New York.

2:36

You just told me.

2:37

Yeah.

2:38

I live the, I'm here in Ridgwood, Queens in New York City.

2:43

Cozy.

2:44

And we're here to talk about your project today, which now is all in the hands of you.

2:54

But historically has been carried by many different people.

2:59

I'm hearing from what you've shared.

3:02

And it is the rest implementation of Hypercore.

3:06

Yeah.

3:07

I picked it up a few years ago whenever I was in between jobs.

3:13

And at the time there were a few other people working on it.

3:17

And since then it becomes the main developer.

3:20

From having seen two different rest implementations started and not always finished of Scuttlebutt,

3:26

I've understood that it's quite a, like, it's quite a tremendous task that you've

3:33

picked up.

3:35

But before we dive into a little bit of what this means, some of the listeners might not

3:40

even know about Rust as a programming language.

3:43

For those who don't know, it's a programming language.

3:45

And it's a very particular programming language.

3:50

Before we dive into the first questions, do you just want to tell us like, what different

3:54

shades of Rust from other programming languages?

3:57

Yeah.

3:58

I love Rust.

4:00

I think like a thing that's talked about a lot is that it's a very secure programming

4:06

language.

4:07

It eliminates a whole class of like memory problems, which is good.

4:11

But I think a thing that is often overlooked that developers love is like how ergonomic

4:17

it is for software developers.

4:20

It's very modern.

4:21

And when they designed it, they thought about things like testing and building the documentation

4:27

and cross compilation and things like that, that have been kind of like bolted on to previous

4:31

programming languages.

4:33

So it has some really first class tooling that makes a lot of things super easy.

4:39

And yet it's a systems programming language.

4:42

So you could use it to write very low level things like for microcontrollers or writing

4:48

operating systems and stuff like that.

4:50

But it's also used for everything up to web programming and backend web services.

4:57

I'm amazed to hear you use the word easy in the same sentence as describing Rust because

5:04

from what I've understood, not everyone finds it very easy to get into.

5:08

Yeah, that's true.

5:10

It has sort of some new concepts that aren't familiar from other programming languages.

5:16

So getting into it can be a little difficult.

5:20

But with other things, it's the complexity that's there, I think is necessary.

5:27

And it highlights some things that could be bugs in other languages.

5:31

It definitely makes you think harder about how you use data.

5:35

But with that, it gives you a lot more guarantees about correctness, which I think makes things

5:43

easier in a certain way because there's fewer bugs and problems.

5:47

So there's a steep learning curve, but once you get there, it's smooth sailing.

5:53

Or easier sailing than a lot of other things.

5:57

Yeah, yeah.

5:59

I think it does require a lot more thoughtfulness in planning.

6:02

And for things like rapid, prototyping, it's not always the best thing.

6:07

But if you're building something foundational, I think it's a really good choice.

6:12

Yeah.

6:13

And that's actually what you're doing right now.

6:15

Yeah.

6:16

With your project on rebuilding Hypercore.

6:19

So not everyone who's listening knows what Hypercore is, but it is, as you said, quite

6:26

foundational.

6:29

And it's been around for a decade now.

6:32

Yeah.

6:33

Yeah, Hypercore is really cool if people aren't familiar with it.

6:38

Maybe they are familiar with the BitTorrent protocol, or maybe people have torrented things.

6:45

It's not so common anymore, but it used to be the way that everybody got their music

6:49

before Spotify if you didn't go to the store and buy it.

6:53

So BitTorrent is just a way for people on the internet to share static files with each

6:58

other and it peer to peer away, meaning like from one person's computer to another without

7:03

having a centralized server or data center in between you.

7:09

And Hypercore is different because instead of sharing a static file, you can share a

7:13

file that you can change.

7:15

So you can do things like share a directory that you could add files to or share a more

7:22

complicated data structure.

7:26

And those are the kind of things that we want to do with Hypercore and use those features

7:31

to build more rich peer to peer applications.

7:36

This is really also interesting to hear your take on Hypercore because I've encountered

7:42

it in so many different areas.

7:44

It's got a very rich history at this point.

7:49

And previously it was known as DUT protocol or DIT.

7:55

So from your perspective, why Hypercore?

8:02

Why does it feel important to you?

8:06

Or why does it feel foundational for you?

8:08

That's a good question, especially I know that there's some other peer to peer things

8:15

in this kind of space like IPFS.

8:18

And I've talked to some people that say things like why not IPFS?

8:23

And the way I think about it is IPFS is kind of like a, you could think of it as

8:29

kind of like a hash table where you have like key value storage.

8:35

But with IPFS, the things that you're storing aren't mutable.

8:42

What does mutable mean?

8:43

Oh, like it's difficult.

8:44

If you want to store data in IPFS by default, it's like not easy to change that data or mutate

8:50

it.

8:51

And with Hypercore, it's more like a list of things.

8:55

And in Hypercore, you can have a list that you can add stuff to.

8:59

So with IPFS, if you have a thing and you're like, you're trying to build an application

9:04

and you want to have a way to say, give me newer things, like give me new things that

9:08

have been added, that's not really built into the core concept of IPFS.

9:15

It's more like you have a key and you want to go get another thing.

9:19

And maybe that has other keys that you can go to link to other things.

9:23

So it's good for things like that.

9:25

But for peer-to-peer data that changes that you want to be able to get newer stuff, Hypercore

9:31

is good for things like that.

9:33

And for the like applications that I've been, that I was originally interested in whenever

9:38

I started playing around with Hypercore, which was building like an RSS, peer-to-peer RSS

9:43

protocol, it was a much more natural fit.

9:46

I'm super curious about this peer-to-peer RSS protocol thing.

9:51

Also, in part because like starting to make podcasts and engaging more with RSS feeds,

9:59

it's become very clear how essential they are in a more community approach towards content

10:09

creation online.

10:10

Yeah.

10:12

First of all, what are RSS feeds?

10:15

Why do they matter to you?

10:17

And then let's go into it.

10:19

Yeah.

10:20

RSS feeds are a, well, it's a web standard.

10:24

It stands for really simple syndication.

10:27

And I think it was standardized in the mid-2000s.

10:34

I know that Aaron Schwartz was part of like the standardization process, whatever he was

10:41

a child.

10:42

Rest in peace.

10:46

And basically it's a way to share a list of things that you add new things to.

10:52

Normally things like blog posts where you write a blog post and it goes at the top and

10:58

you want people to be able to like get updates from your blog or things like podcasts or

11:03

another good example.

11:04

Or anytime you want to share a feed of data, it's best used where you're just adding things

11:11

to it, but it is possible to change older entries in it.

11:15

I didn't actually know that it was possible to change older entries, but it makes sense

11:19

because I can upload new podcast episodes from my old ones.

11:24

Yeah.

11:25

Or how does it work?

11:26

I think it's handled by the different clients differently.

11:31

So they might not know that they need to fetch old things.

11:35

Yeah.

11:36

I think the way a lot of people do it is like if they have a podcast, maybe the file

11:41

that they share is like a link to a podcast and then they'll change with that link points

11:47

to or edit a file behind the link and things like that.

11:50

Is that similar then to how it would work in HyperCorp?

11:53

With RSS, usually you have a server that you're putting your RSS feed on and a bunch

12:00

of people with RSS clients would periodically check that server.

12:05

They'd have like a URL for your server and check it for updates.

12:08

With HyperCorp, you could have something that looks very similar to a user from a client

12:15

perspective.

12:16

However, there wouldn't be a server and the way that the RSS feed, this peer-to-peer

12:25

RSS feed would be addressed wouldn't be through a URL.

12:27

It would be through a public key, which a user wouldn't really have to care about necessarily.

12:34

Is it easy to care about it?

12:37

I care about it because it makes it functional, but I think it's also good.

12:41

I do care about that a normal user shouldn't have to care about the small differences because

12:47

it should be easy for people to use.

12:50

Yeah.

12:51

But yeah, from a publisher's perspective, I think you can have an inner.

12:56

That's something that is very similar to the same flow where you write a blog post and publish

13:02

it in updates here, RSS feed.

13:05

The only thing that might be different is for this centralized RSS thing, you have a server

13:12

that's running.

13:13

And with the peer-to-peer one, you want to make sure that your content is shared with

13:18

other peers.

13:19

And so if I were just to share my RSS feed off my computer, press publish and turn it

13:25

off.

13:26

Somebody might not actually ever read it.

13:28

You want to start giving that data out to the swarm of peers because once it's out there

13:34

and people are reading it, then you can turn off your computer and it would be fine because

13:38

other people have it and they're able to share it with each other.

13:41

Yeah.

13:42

So does it become like, I'm hearing into this now, but does it become more resilient then?

13:48

Yeah.

13:49

It becomes more resilient for things where the content is more in high demand.

13:56

And if you want to be certain that your peer-to-peer thing is always available, you can have a

14:04

server or you can have a seat box is what the equivalent would be in mid-tolerant land

14:10

where you have a server that's constantly receding your thing so you don't have to worry about

14:15

having your computer that you wrote your blog post on all the time.

14:21

Once something becomes popular, you don't really have to worry about that because there

14:27

will be different people reading it.

14:29

And as long as one of those computers is on, then they'll be able to share with each other.

14:33

Yeah, because each computer contains its own copy of the media.

14:37

But it's not magic.

14:41

Somebody still needs to be, it needs to be alive somewhere for things to be shared with

14:47

each other.

14:48

So you can't always just count on it to have somebody store your data somewhere for free.

14:54

So, okay, let's go back to the RSS comparison then.

15:00

When I'm publishing something or when someone is publishing something to RSS, they create

15:07

a copy on a server.

15:12

And that's the server that's hosting their RSS client.

15:14

Is that how it works?

15:16

Yeah.

15:17

From there on, is that shared to multiple other RSS servers?

15:22

It's served to other for regular RSS.

15:25

It would be, they would publish it to like an RSS server, which is just like a normal

15:29

web server.

15:30

And RSS is just like a XML format served from a web server.

15:35

Yeah.

15:36

And other RSS clients, like someone on their phone or their computer, whenever it automatically

15:41

updates their feeds, it would pull in your new content.

15:46

But then those clients don't automatically share them with other clients.

15:50

But with like a peer-to-peer based RSS protocol, those clients would share the data with other

15:56

clients.

15:57

And so the central server would become less important.

16:01

And those clients would be any user I presume.

16:05

Yeah.

16:06

Like any listener, anyone who wants to engage with the content would be in server terms,

16:12

a client.

16:13

Yeah.

16:14

And that's where we come back to the peer-to-peer architecture.

16:16

And especially for things where you're sharing a lot more data, such as a podcast.

16:23

I don't know, your server, you wouldn't have to pay for as much bandwidth on your server

16:28

if you're serving your podcast in a peer-to-peer way.

16:33

That would all be distributed between other peers.

16:36

Let's go back a bit into, because now we've darred into a little bit of the nitty-gritty

16:42

how it works.

16:44

How did you find Hypercore?

16:48

How did you end up working on Hypercore?

16:50

And how did you end up building a Rust implementation and why?

16:53

Like, what was that journey for you?

16:56

I think I first heard about the DAP project in the late teens, whenever I was living in

17:03

San Francisco.

17:05

I used to hang out at a place called Noisebridge there, which is like a hacker space that I

17:11

love.

17:12

And people were there talking about it.

17:14

Someone explained it to me and I thought it was cool.

17:17

And I had seen it around.

17:19

I know that it was affiliated with like some Scuttlebutt people who I saw around Oakland

17:26

and went to Mozilla's conference in 2019.

17:32

And there was like a workshop on how to use Beaker browser, which is like a web browser

17:37

that was based on Hypercore.

17:39

And I thought that was really cool.

17:41

I just must ask, did we meet there?

17:45

Possibly.

17:46

We can be, right?

17:47

Yeah, I was.

17:48

Because I was there as well.

17:49

Okay, that's cool.

17:51

Yeah, I went.

17:53

They, I had a privacy browser extension called Privacy Possum that used like heuristics for

18:00

blocking crackers.

18:02

And Mozilla invited a bunch of privacy oriented like web extension people to come to like

18:07

a little mini conference there because Google was changing the way their web extensions

18:13

were to prevent things like, well, ostensibly they said it's for security, but it also

18:19

had this nice effect for them where it made these extensions that blocked tracking harder

18:28

to do.

18:30

So we all went there for a meeting.

18:32

But yeah, we could have met there.

18:35

It's kind of like a funny one, right?

18:36

Because like Firefox is what 70% funded by Google, right?

18:42

Yeah, something like that.

18:45

Because Firefox generates a lot of traffic for Google.

18:48

And I think that's how they get a lot of their money.

18:50

But as of now, Firefox has not switched over to those web extension APIs.

18:56

And that's why like you can still use nice ad blockers on Firefox that'll stop you from

19:00

seeing YouTube ads and things like that.

19:04

But you can't do that on Chrome.

19:06

Go Firefox.

19:08

It's amazing that somehow, although we're all tied into this web of funding, we can keep

19:16

privacy alive in some spaces of the web.

19:21

And that's also something that Hypercore does, I'm assuming.

19:25

Yeah, definitely.

19:26

With peer to peer applications, like you are revealing your IP address to other peers

19:32

that you're connecting to, ever, it is like an encrypted connection between those other

19:38

peers.

19:39

So no one knows like what you're actually sending just by like watching your traffic

19:44

over the internet.

19:46

I think it's a different privacy story because then the back to the RSS thing, like this

19:55

RSS server knows everybody that is downloading the data, at least by their IP address.

20:02

And in the peer to peer story, it's like the people that you're sharing the data with

20:06

to.

20:07

So maybe the original, like one peer won't see everybody that's sharing the data necessarily.

20:16

You see your local environment where you're engaging.

20:20

But did you meet Paul then?

20:22

Because Paul was also, Paul, crazy, was also at the...

20:26

Yeah, I think he gave the, he gave the work.

20:30

And then you ended up in Beaker browser, which is it defunct by now?

20:36

Or is it still going?

20:38

Um, I don't know actually.

20:41

I don't know.

20:42

I haven't used it since then.

20:44

I thought it was really interesting, but I didn't have like an immediate use case for

20:50

it.

20:51

And that's like whenever I started working on Rust Hypercore, like that was one of, or

20:56

like that gets back kind of like into like why Hypercore Rust stuff and not Hypercore

21:02

JavaScript stuff, at least for me.

21:05

Which I don't know if I should talk about that.

21:08

Please do dive into it.

21:11

Okay.

21:12

Yeah.

21:13

Like I implemented this peer to peer RSS protocol in JavaScript, which the main implementation

21:20

of Hypercore is it in JavaScript.

21:24

It made like a little client for it and it's way to like a script for like ripping podcasts

21:30

off of an existing RSS feed and sharing them in a peer to peer way.

21:33

But I wanted this thing to, I wanted people to be able to use this like RSS protocol.

21:40

And I thought I would wait a good way to do that would be like approaching existing RSS

21:45

clients and giving them a library that they could use to integrate so that you could just

21:50

get in your podcast client and your RSS client be able to get like a peer to peer

21:55

feed of like podcasts or whatever RSS content.

21:59

However, you can't just give a JavaScript library to a application like the intent of

22:06

pod that's on my phone that's written in Java because of the way that JavaScript works

22:12

you can't just like necessarily embed it easily within another language.

22:17

But with Rust you can because Rust is like a low level library, you can compile it to

22:23

a C library, which is kind of like the lingua franca of all programming languages, every

22:29

programming language has a way to call a C library.

22:32

And so if I could make this abstract this peer to peer protocol into a library in Rust,

22:39

then I could give that to people to use to integrate it into things like a podcast client

22:45

or an RSS client.

22:47

And I think that's really important for like adoption and trying to meet people where they're

22:51

at with like the tools that they have is being able to integrate with them.

22:56

So yeah, and I think that the current JavaScript implementation is run by this organization

23:03

called WholePunch and they're great.

23:06

And I think that they have a reasonable approach to this problem too where they've built sort

23:14

of a runtime for the whole like hyper core ecosystem.

23:19

And it's really cool.

23:21

It's called pair and it's sort of like an application that you can build other hyper core

23:27

applications inside of and share like a pair URL and like run JavaScript hyper core applications

23:36

within it.

23:37

And you it also like distributes the applications in a peer to peer way and they get updated

23:42

automatically and if you're to peer way it's pretty cool.

23:45

And they have like a they have a chat app which is sort of like the premier application

23:50

on it called Keat like K-E-E-T.

23:56

And the runtime itself is called pair like the fruit.

23:59

I think they're ripping ripping off peer.

24:01

Yeah, it's very cute and sweet.

24:07

What's it called?

24:08

Not metaphor, but pun.

24:12

I had a few conversations one with Sarah Path a while ago and then back in the day I

24:18

met some of the people from hyper core or no from the ones developing pair right now.

24:24

And what's the name of their company again?

24:28

WholePunch.

24:29

Yes, WholePunch.

24:32

So basically as far as I've understood it, the dot slash hyper core ecosystem has had

24:38

like a lot of different stages of evolution of people of focuses.

24:47

And as you're mentioning like now it's like with the pair protocol and with whole punch

24:53

and also your efforts on turning hyper core into rust.

24:58

And so I'm wondering like if you look out on the landscape of hyper core slash dots, what

25:07

is like bubbling?

25:08

What is active?

25:09

What is the state of things?

25:12

That's interesting.

25:13

I think I have been in my own little area for a while.

25:18

I am on the Keat and I talked to some people in there about there's like a rust hyper core

25:24

channel where I post updates and there's occasionally some people that have been helping

25:30

recently.

25:33

But I think that that's a I'm not too tapped into what else is going on.

25:40

I have been somewhat single minded in my focus of building this project.

25:48

As you need to be.

25:50

Yeah.

25:51

And I think that I feel that I have like a very clear vision of like what I want to do.

26:00

And so I'm just like plowing through and doing that kind of thing where I think that

26:08

hyper core could be something very infrastructural for a lot of use cases that where we're not

26:16

using peer to peer today where we could.

26:20

I'm writing a blog post about this but recently the open source security foundation and like

26:28

a bunch of programming language foundations wrote a joint statement about how hosting

26:35

package registries like the things where whenever you're using a programming language the thing

26:41

that you're downloading your libraries that you're using from those are becoming so expensive

26:49

partially because the rise of like hectic AI and also like continuous integration systems

26:55

constantly downloading things from them hosting them's becoming really really expensive.

27:00

And there's also like like new compliance and stuff that they're having to deal with.

27:07

So their problem is like they are sharing a whole bunch of content with a whole bunch

27:13

of people.

27:14

And a lot of that like taking in the post I'm writing about like the rust package registry

27:23

the thing you download all of your libraries from is called crates.io.

27:29

And there's like billions of downloads for these things on there.

27:35

And a lot of them are downloaded by are if basically I'm saying that if there was like

27:41

a peer to peer solution for this this would dramatically reduce their costs.

27:47

And what in the open source security foundations proposal a lot of like the suggestions were

27:54

like we need to like partner more with commercial entities and have like like tiered access

28:02

and like ways to pay for these things for commercial entities which seems like a fix.

28:09

It would definitely like fix some things.

28:11

It also might bring up certain conflicts of interest.

28:17

These foundations are like nonprofits they're but not in like the super benevolent sense

28:25

necessarily they are sort of like also industry consortiums.

28:32

But I think we shouldn't just let them become totally like paid for by big companies.

28:45

Using hyper core to build a peer to peer package.

28:48

I think it's like a really realistic application that is a technical solution to this problem

28:56

that would like dramatically reduce costs and not and things like in rust they wouldn't

29:04

have to like take a bunch of money from Microsoft if we could do something like this.

29:11

I 100% agree and I think that's an amazing use case for a specifically hyper core as

29:19

well since it is like a collaborative thing for one large like database or one.

29:26

Yeah and in the way that software libraries are specifically like versioned they are like

29:33

sort of like RSS we'd like you add anything to each one and they're like linked together.

29:39

And this is a data structure that you could represent in hyper core like in a pretty straightforward way.

29:47

I think this is also really it touches on a fascinating topic because like if we look at

29:55

peer to peer architectures as a whole and there's there's this concept in scuttle what we used to

30:02

call it singularities.

30:10

So I realized that I said singularity here what I actually meant to say was single tons also the

30:17

next time I say it.

30:19

And it's like the DHT system and the distributed hash tables you have the singularity where if

30:34

you would create a dot or a hyper core network in one place and then a hyper core network in

30:39

another place they wouldn't necessarily be able to communicate to each other.

30:43

And that's very essential quality there's some researcher called Shapiro who

30:51

probably between the pronunciation there but who talked about this from the perspective of

30:57

calling it like the grassroots networks where it's like one part of the network pops up in one

31:03

place and then it meets another part of the network and then they can merge and communicate.

31:08

With DHTs that's not quite possible right so in some ways my own internal critique towards

31:15

DHT based protocols has been like that they're not suitable for communication networks per se

31:23

that are natural organic and ever flowing but as you're making like pointing out here there's

31:31

some key qualities of these network they're not qualities there's some key implementations

31:40

that they're fantastic for and I had never heard or even thought about hyper core being used for

31:49

RSS like structures or for package managers so it's it feels like a very fresh take on these

32:00

protocols or on hyper core specifically. Thank you maybe it's fresh because I have been

32:08

a little bit isolated in my focus I just haven't been talking so it's good we're talking about it.

32:15

It is good we're talking about it and also I mean as you mentioned like you're communicating

32:21

with other people on Keats like a lot of people have built a lot of different things on hyper core.

32:27

Yeah. One of those implementations which is as far as I know the largest of all networks

32:37

or infrastructures building on peer-to-peer offline first communication protocols and that's my

32:44

pale and my pale I haven't heard that. This is incredible to me and this also just goes to show

32:53

how distributed networking development is as distributed in practice of developing it as it is

33:06

in architecture. Yeah. Okay but if you haven't heard about it I'll send it to you.

33:11

It's developed by Awana Digital. Maybe they're back to being called digital democracy I don't know

33:18

they were kind of switching name for a while. Basically my pale runs over hyper core. Okay.

33:26

And I think it's the biggest active implementation of hyper core in the world. Okay, interesting.

33:32

What is it? What does it do? So my pale is super freaking fascinating.

33:41

It's basically... Oh it's like a mapping thing. Exactly. It's a mapping tool and it started

33:48

in Ecuador where I forgot the name of the indigenous tribe but there's an indigenous tribe or maybe

33:56

collaboration between multiple ones and that were mapping their territories and because oil

34:04

companies were trying to claim that the oil companies could come in and drill oil and ruin

34:09

the ecology. And then by using a hyper core they could go around and map the like geolocations

34:22

of where they were active to showcase them later in court that this was actually their territories.

34:31

Yeah. Yeah and that has spread. So now my paleo is used by like 400 different communities around

34:37

the world. Most of them indigenous. So yeah. But this is mind flowing to me.

34:48

Because like one of the things that I'm fascinating by in my studies is like how distributed organizing

34:55

happens. And it's so cool to find like to see see it in action that you're here developing something

35:07

that is use case. Like the use case could be for anything that's using hyper core because you're

35:12

basically making it easier to interface with hyper core. Right? Yeah. Yeah definitely. That's

35:19

one another reason I wanted to choose Rust. I'm interested. I'm going to check out their

35:23

implementation later. Yeah. And going back to writing in Rust, another aspect of like implementing

35:33

this. I've been using some of Mozilla's Mozilla has a tool that's basically because Rust can be

35:40

compiled down to like a C library. And almost every language can call C. They have a tool for building

35:47

taking your Rust code and generating libraries for other languages like Python and Kotlin and Swift,

35:53

which which just those three libraries it's like back and led programming, building apps for

35:59

iPhone and building apps for Android. And with a solid Rust implementation, we would have a good

36:06

like Android and iPhone implementation that could be used on phones. Having the same

36:12

implementation, I think that use use by these libraries prevents like fragmentation and other

36:18

ethics of like different implementations not being able to work with it. Yeah, definitely.

36:24

And it sounds like the work you're doing really just scales up the possibilities for hyper core.

36:29

And that kind of brings me to another question. Like, why do you think that there's not more

36:36

emphasis from other like, for example, whole punch on building hyper core in Rust?

36:43

Um, I have talked to Maffin Tasha a little bit. I know that they have their own priorities that

36:50

they're focusing on. And I the last I talked to them about is like they were working on like,

36:58

a system level implementation written C. And I think that has to do with a so something about

37:07

the way they're running stuff in the pair runtime. But I would like to talk with them. I'd be happy

37:16

to talk with them more about that. And I think part of the reason is like, I haven't been

37:24

marketing my work very much. And like, like, I think I haven't been like talking to many people

37:30

about it. And sometimes whenever I'm, you know, whatever I have some work to do on this project,

37:38

I'm like, I could either tell people about it or write about it or write more code. And I usually

37:44

end up writing more code, which isn't always the best solution.

37:49

I hear you so much on that one, not personally, because I'm not a programmer. But from working on

37:56

NGI, next generation internet, one of the things that was like the key aspects that made NGI functional

38:03

was that they tore down like the whole system of making it complicated to advocate for oneself.

38:11

And instead, the applications were super simple. So that people who were like you,

38:16

who wanted to focus on writing the code could do that without having to put in like a whole

38:21

month on making applications to apply for funding or something like that.

38:25

Yeah, that's cool.

38:27

I think so. And good on you for doing the work and good on you for being here talking to me about it.

38:33

Yeah.

38:35

Because both are important, because I think this, like as someone who's been in this ecosystem for a

38:41

long time, although I've never worked directly with Hypercore, like you've already opened quite

38:49

a few perspectives on the purpose of Hypercore as a protocol, one of the purposes,

38:56

there's probably multiple. But it combats my main internal critique. So I'm happy to hear.

39:05

Yeah. Thank you. Yeah. I want to, well, maybe I could maybe now is about the time to plug,

39:15

but I want to, if anyone is interested in this kind of thing, they should definitely reach out to me.

39:21

And you can do my GitHub is just github.com slash CalLix, or you could email me.

39:29

It's just email at callix.website. Yeah. And I'd be happy to work with people.

39:38

You also mentioned that there was like a group chat where you sometimes write updates and people

39:44

could join. How would people start joining that group chat if they wanted to?

39:50

Yeah, there is a group chat on Keat, which is the Hypercore chat application. And it is called

40:00

the channel is called Rust Hypercore. And I will have to share a link to it. Or I think that with

40:08

Keat, now you can search for different channel names. But it should be fairly straightforward

40:14

to find or if you reach out, I can connect people. So this process of using your own software to

40:20

communicate and using it actively as you're developing, it's commonly known as dogfooding,

40:26

right? So how's that going? Well, that

40:31

Keat is Keat is using the JavaScript implementation, not the Rust one, but it's

40:36

I, it is useful. And I should say that the Rust implementation is still somewhat nascent. It is

40:50

currently pinned to like a previous version of Hypercore. And there's a few aspects of it that

40:56

need to be completed before like practical usage can be there. Whenever I picked up the project a few

41:04

years ago, some key pieces were missing, like a pure discovery, which means like

41:13

if you want to download a Hypercore and find other people with it, the way that you actually do that.

41:18

And so for the past year, I was implementing the pure discovery part and then things like replication,

41:27

whenever you like the encryption and replication. So you find a peer, which is good, you need to

41:35

talk to them, then you need to like create an encrypted connection to them. And then you need

41:40

to chatter about what hypercores you have with each other and start replicating data between them.

41:47

And so those pieces are all basically done in isolation. And I'm at the point where I am

41:54

bringing them together and sort of like demos of demonstrating like getting a Hypercore,

42:01

discovering another peer on the network and replicating data with it are possible. But

42:08

right now the actual like API and some work needs to be done to make this actually useful, but it is

42:16

very close. Exciting. Okay, shout out. Anyone looking for a project and wanting to learn more about

42:26

Rust or wanting to learn more about Hypercore and think that they can contribute, reach out to Blake

42:34

Blake Griffith, I guess, because I just read your blog where your name says as well. So I'm

42:43

guessing it's out in the public. But so that's so if you look at like at a timeline, what do you

42:55

think is the how how much okay, theoretically, you had an infinite amount of if you had an infinite

43:04

amount of money, and it was you working on it only, when do you think this could be done?

43:12

Um, with one developer? Yes. So, so I guess the timeline is the time there's like several

43:19

different stages of a timeline, right? Yeah. Getting, um, publishing, having Hypercore that is just like

43:29

that there's two libraries in the JavaScript ecosystem called Hyper Swarm and Hypercore.

43:33

Yeah. Having just those two and having them be able to like replicate data with each other,

43:40

I think would happen in the next month, because that's like what I'm wiring together right now.

43:44

Oh my gosh. And then there's other, there's other aspects on top of it, right? And there's a

43:52

key value store library built on Hypercore called a Hyper B, like a B tree. And I've implemented that,

44:01

but the thing after I have those parts working, there's things like updating it to be at future

44:08

parity with the latest version of Hypercore. And there's also like things that I've like

44:12

skipped over through the implementation that I would love to like polish off, like congestion

44:18

control where whenever peers are chattering with each other, if for a real implementation,

44:25

they would need to like look, hey, I, my packets didn't send and I need to like control how much

44:30

I'm sending a lot of like small technical things like that that should get polished off. Um, but

44:36

then I also want to re implement this peer to peer RSS protocol on these things. And

44:45

the core data structure, I don't think is so complicated, but would really make it useful is

44:50

having more polished tools for like publishing on it and having like a library that's like

44:56

designed to be integrated with other tools in like an existing RSS client. And I think those

45:03

are a little more long term goals, like a year away, if this was my full time job. Yeah.

45:11

So all right, well, I'm cheers and crossing my fingers and holding my thumbs.

45:19

Holding my thumbs is the Swedish way of saying I wish you luck.

45:28

I'm Swedish.

45:34

But it that said, I think we're on a timeline on our own as well here for the podcast. So I

45:45

think we're about perfectly time to wrap. And is there anything else that you would like to

45:54

add, make a shout out for a anything this

46:03

Yeah, just just hit me up. I'd love to hear from people, even if you're not a developer, like

46:08

soon I'll be asking people to try things and dinner things. And yeah, try out, keep the chat

46:16

application. You could find the my Rust channel on there. And I think that's it.

46:22

Thank you so much for joining. And if you want to see more from Blake, you can check out Calyx.

46:29

website, which is your website. And you also got a blog. And I also have a GitHub sponsors page,

46:37

if anybody wants to chip me five bucks a month, I'd appreciate it. Yes. And then we can get to

46:44

Rust implementation of hyper core sooner. I hope so, because I'm convinced by these use cases. And

46:53

I'm really with like a peer to peer package manager, such as over hyper core. So thank you

47:02

so much for joining. Okay, thank you, Zell. Have a great day. Bye. Nice talking to you.

47:14

Solarcast