Over at the Nogripracing.com gone? thread, many are reliving the pain of losing access to mod files when the sites hosting them have to shutdown.
I've been considering an idea for a solution that utilizes bittorrent which, running a simple app on a home computer or hosted server, can provide potentially safer file hosting by distributing files across many stakeholders and decentralizing ownership so that if one goes down it's not such a big problem as if one whole website goes down.
By building a small swarm of bittorrent peers among those dedicated to the mods, each hosting all or part of a given mod file collection, we reduce the risk of complete loss by having the data duplicated across many nodes as opposed to concentrated in one location such as is the case right now with NoGrip, RD, EEC, and many other sites.
The question is, what kind of will exists in the community to move ahead with something like this? How many people would contribute by running a bittorrent client, on a home computer or server, and sharing mod files in the swarm?
Additionally, are there any others like me who, perhaps, have a personal server they could dedicate some resources from to achieve at least a couple of 24/7 nodes, since presumably home connections wouldn't be seeding 24/7. Could RD, EEC, or other existing sites contribute server resources?
It seems a real shame to lose accumulated years or resources on a site like NoGrip out of the blue, so I'm interested in gauging the community's interest and will to move on something like this.
Problems:
In the future, we'll be looking at the 'persistent web' where, by design, data loss like this won't happen, and peer-to-peer technology like bittorrent will underpin it. Until then, though, we do have the technology already available, with a little effort, and if the will is there to make it happen.
I've been considering an idea for a solution that utilizes bittorrent which, running a simple app on a home computer or hosted server, can provide potentially safer file hosting by distributing files across many stakeholders and decentralizing ownership so that if one goes down it's not such a big problem as if one whole website goes down.
By building a small swarm of bittorrent peers among those dedicated to the mods, each hosting all or part of a given mod file collection, we reduce the risk of complete loss by having the data duplicated across many nodes as opposed to concentrated in one location such as is the case right now with NoGrip, RD, EEC, and many other sites.
The question is, what kind of will exists in the community to move ahead with something like this? How many people would contribute by running a bittorrent client, on a home computer or server, and sharing mod files in the swarm?
Additionally, are there any others like me who, perhaps, have a personal server they could dedicate some resources from to achieve at least a couple of 24/7 nodes, since presumably home connections wouldn't be seeding 24/7. Could RD, EEC, or other existing sites contribute server resources?
It seems a real shame to lose accumulated years or resources on a site like NoGrip out of the blue, so I'm interested in gauging the community's interest and will to move on something like this.
Problems:
- Where are files uploaded to initially?
- Modders might want to seed their own files to begin with
- Or, certain nodes might provide upload capabilities
- How would other nodes sync the newest files and start seeding them?
- I'm not sure about an automated way to achieve this although bittorrent clients may allow for this in some way.
- Otherwise, a manual process of request-to-seed notifications (ie. email, tweet, forum thread, etc.) sent to nodes might be employed.
- Files are hosted but how to find them?
- We'd still need easily searchable catalogs like RD, EEC, etc. provide
- Too many files / too large for any one node to host
- Each node can decide what they want to host. For example, I might host only GTR2 mod files, another might host only AC files, etc.
- Each node can decide what sub-section of files to host. For example, one node might host only tracks and cars, while another might host patches and documents.
- Home users don't have the bandwidth
- Bittorrent clients can be bandwidth limited so the impact is minimal to you
- Bittorrent is designed to aggregate bandwidth from many nodes so that many small nodes together can provide enough bandwidth for fast downloads
- Home users computers aren't up 24/7
- Hopefully, having many nodes around the world, a few would presumably be running at any given hour of the day.
- Additionally, servers that are up 24/7 would be part of the swarm providing access at all times
- Ideally, sites like RD, EEC, and others would contribute as it would lower their costs to offload their file hosting downloads.
- Sites like RD wouldn't like taking away file hosting
- Actually, it would take away the 'bad' kind of traffic (raw downloads) they don't like, leaving them with only the 'good' kind of traffic (ad-filled pages) used to link the mod file downloads. They'd save on bandwidth which is a significant cost for sites like RD.
- Sites would still be used for the catalog and notifying of new versions. Discussions would still happen on the sites' forums.
In the future, we'll be looking at the 'persistent web' where, by design, data loss like this won't happen, and peer-to-peer technology like bittorrent will underpin it. Until then, though, we do have the technology already available, with a little effort, and if the will is there to make it happen.
