Compare commits

..

5 Commits

11 changed files with 3675 additions and 85 deletions

View File

@ -16,102 +16,30 @@ The process operates in 3 stages:
2. Saturation Stage - The number of new domains is lower than the amount of existing domains.
3. Rescanning Stage - Existing domains are periodically rescanned looking for new links.
## Contact
The project's maintainer and primary contact is Luke Harding \<<luke@lukeh990.io>\>. Feel free to email with any questions.
## MTI Worker Protocol (MTIWP) Specification
MTIWP is a TCP-based binary application layer protocol used to communicate between the workers and coordination server.
### Packet Makeup
The packet is made up of the following fields:
- Header
- Version (2 Bytes)
- Sender ID (6 Bytes)
- Timestamp (8 Bytes)
- Channel (2 Bytes)
- Method (1 Byte)
- Payload Length (2 Bytes)
- Version (2 Bytes)
- Worker ID (6 Bytes)
- Timestamp (8 Bytes)
- Method (1 Byte)
- Payload Length (2 Bytes)
- Payload (0-65535 Bytes)
**All fields are required to be in big endian format.**
#### Version
The version of the MTIWP protocol. Currently the only acceptable value is 0x0001
#### Sender ID
The Sender ID is a value to identify the sender. These values are to be presented in the same way a ethernet MAC address is supposed to be. (ie. XX:XX:XX:XX:XX:XX).
The server will always reserve the 00:00:00:00:00:00 sender ID. The server will designate IDs to new workers in a response with a provision method.
***NOTE: The sender ID is not a real MAC address. It is just formatted like one.***
#### Timestamp
The timestamp is based on the [TimestampSecondsWithFrac\<f64\>](https://docs.rs/serde_with/latest/serde_with/struct.TimestampSeconds.html) provided by the serde crate for Rust.
#### Channel
Due to the async communication nature of the server and worker. There needs to be a way for the receiver to distinguish the intended receiving task while maintaining a single TCP connection. The client must maintain one worker to handle all send/receive operations.
There are a couple of channels which are reserved:
- Channel 0x00 - Client/Server Initialization
- Channel 0x01 - Ping/Pong heartbeat cycle
All other channels are to be dynamically allocated. If the client is initiating a request the channel is randomly picked from the range 0x02 - 0x08f. Likewise, if the server is initiating the request, the channel must be picked from the range 0x90 - 0xff.
The initiator is responsible for maintaining a list of in use channels and freeing those not in use anymore.
#### Methods
Valid Methods:
- ACK (0x00)
- Ping (0x01)
- Pong (0x02)
- Hello (0x03)
- Provision (0x04)
- Index (0x05)
- Cancel (0x06)
- Summary (0x07)
- Error (0x08)
- Channel End (0x09)
- Goodbye (0x0A)
- Index (0x04)
- Cancel (0x05)
- Summary (0x06)
##### ACK Method
An ACK (ACKnowledgement) is a packet designed to indicate the previous request has been received and executed but no return data is given.
Further Detail Goes Here
##### Ping & Pong Methods
This packet is designed to be used to ensure the TCP connection between worker and server is always in use. The ping method is to only be used by a server. The server will send a ping approximately every 1 second the client is required to send a pong within 1 more second or the connection will be closed.
## Contact and License
This packet has no attached payload.
Project Contact: Luke Harding <luke@lukeh990.io>
##### Hello Method
The hello method is send by clients when first establishing a connection.
This packet has no attached payload.
##### Provision Method
##### Index Method
##### Cancel Method
##### Summary Method
##### Error Method
##### Channel End Method
##### Goodbye Method
#### Payload Length
The payload length is a 2 byte unsigned integer that is used to determine how much payload data is expected. If the receiver is unable to read enough bytes to satisfy the payload length then an error is returned to the sender.
### Initialization Process
The initialization process defines how the client establishes the connection with the server. The primary packet methods are: Hello and Provision.
After the TCP stream has been initialized, the first packet will come from the server as a Hello packet on channel 0.
After processing the message, the server has 2 options, it can either reply with a provision packet or an error. The provision request will contain the 6 bytes that make up the client's assigned sender ID.
The connection is now initialized. The heartbeat cycle will start and the client will wait for a command.
### Ping/Pong Heartbeat
WIP
Licensed Under a GPL v2 License

4
website/.vscode/extensions.json vendored Normal file
View File

@ -0,0 +1,4 @@
{
"recommendations": ["astro-build.astro-vscode"],
"unwantedRecommendations": []
}

11
website/.vscode/launch.json vendored Normal file
View File

@ -0,0 +1,11 @@
{
"version": "0.2.0",
"configurations": [
{
"command": "./node_modules/.bin/astro dev",
"name": "Development server",
"request": "launch",
"type": "node-terminal"
}
]
}

3
website/README.md Normal file
View File

@ -0,0 +1,3 @@
# Map the Internet Website
This website will be hosted [here](https://map-the-internet.lukeh990.io)

4
website/astro.config.mjs Normal file
View File

@ -0,0 +1,4 @@
import { defineConfig } from 'astro/config';
// https://astro.build/config
export default defineConfig({});

18
website/package.json Normal file
View File

@ -0,0 +1,18 @@
{
"name": "map-the-internet-website",
"license": "GPL-2.0-only",
"type": "module",
"version": "0.0.1",
"scripts": {
"dev": "astro dev",
"start": "astro dev",
"build": "astro check && astro build",
"preview": "astro preview",
"astro": "astro"
},
"dependencies": {
"astro": "^4.7.0",
"@astrojs/check": "^0.5.10",
"typescript": "^5.4.5"
}
}

View File

@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 128 128">
<path d="M50.4 78.5a75.1 75.1 0 0 0-28.5 6.9l24.2-65.7c.7-2 1.9-3.2 3.4-3.2h29c1.5 0 2.7 1.2 3.4 3.2l24.2 65.7s-11.6-7-28.5-7L67 45.5c-.4-1.7-1.6-2.8-2.9-2.8-1.3 0-2.5 1.1-2.9 2.7L50.4 78.5Zm-1.1 28.2Zm-4.2-20.2c-2 6.6-.6 15.8 4.2 20.2a17.5 17.5 0 0 1 .2-.7 5.5 5.5 0 0 1 5.7-4.5c2.8.1 4.3 1.5 4.7 4.7.2 1.1.2 2.3.2 3.5v.4c0 2.7.7 5.2 2.2 7.4a13 13 0 0 0 5.7 4.9v-.3l-.2-.3c-1.8-5.6-.5-9.5 4.4-12.8l1.5-1a73 73 0 0 0 3.2-2.2 16 16 0 0 0 6.8-11.4c.3-2 .1-4-.6-6l-.8.6-1.6 1a37 37 0 0 1-22.4 2.7c-5-.7-9.7-2-13.2-6.2Z" />
<style>
path { fill: #000; }
@media (prefers-color-scheme: dark) {
path { fill: #FFF; }
}
</style>
</svg>

After

Width:  |  Height:  |  Size: 749 B

2
website/src/env.d.ts vendored Normal file
View File

@ -0,0 +1,2 @@
/// <reference path="../.astro/types.d.ts" />
/// <reference types="astro/client" />

View File

@ -0,0 +1,140 @@
---
---
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width" />
<meta name="generator" content={Astro.generator} />
<title>Map the Internet | Home</title>
</head>
<body>
<header>
<h1>Map the Internet</h1>
<!-- <ul>
<li>
Go to the Map
</li>
<li>
About
</li>
</ul> -->
</header>
<main>
<p>
The Map the Internet Project is a crowdsourced map of the internet.<br>
The ultimate goal is to create a mesh representaion of the internet's topology.<br>
</p>
<p>
The project is currently in its early days.<br>
If you want to follow development take a look at my <a href="https://git.lukeh990.io/luke/map-the-internet">git repo</a>.
</p>
</main>
<footer>Copyright © Luke Harding 2024<br>Licensed under GPL v2 | <a href="https://git.lukeh990.io/luke/map-the-internet">Repo</a></footer>
</body>
</html>
<style scoped>
@import url('https://fonts.googleapis.com/css2?family=Red+Hat+Mono:ital,wght@0,300..700;1,300..700&display=swap');
html, body {
height: 100%;
margin: 0;
font-family: "Red Hat Mono", monospace;
font-optical-sizing: auto;
font-weight: 400;
font-style: normal;
background-color: #1b1e21;
color: #ffffff;
}
body {
display: flex;
flex-direction: column;
}
header {
display: flex;
justify-content: center;
align-items: center;
flex-direction: column;
text-align: center;
padding: 1rem;
}
header > ul {
list-style: none;
margin: 1rem 0;
padding: 0;
font-weight: 600;
}
header > ul > li {
display: inline-block;
margin: 0 1rem;
}
header > ul > li::after {
content: "";
background-color: #fff;
display: block;
position: relative;
right: 2.5%;
height: 1px;
width: 105%;
}
header > h1 {
margin: 0;
font-weight: 700;
}
header > h1::after {
content: "";
background-color: #fff;
display: block;
position: relative;
right: 5%;
height: 2px;
width: 110%;
}
main {
padding: 2rem;
flex: 1;
display: flex;
align-items: center;
flex-direction: column;
}
main > p {
max-width: 100ch;
text-align: center;
}
footer {
background-color: #0b0c0d;
text-align: center;
font-size: 80%;
padding: 0.5rem;
}
a {
color: #b3cfe3;
}
</style>

3
website/tsconfig.json Normal file
View File

@ -0,0 +1,3 @@
{
"extends": "astro/tsconfigs/strict"
}

3468
website/yarn.lock Normal file

File diff suppressed because it is too large Load Diff