Compare commits

..

No commits in common. "a4ede3e887164da0aa5eeffb8e860caabc5912b0" and "00861dc7178b988e17cb72650c0d6f6675448d41" have entirely different histories.

2 changed files with 2 additions and 38 deletions

View File

@ -1,39 +1,3 @@
# Map the Internet (MTI)
# map-the-internet
A project that is designed to scrape the internet to create a visualization of the internet.
## How does it work?
![A diagram depicting how the workers communicate with the server](static/diagram.png?)
The process of mapping the internet is fairly simple:
1. The server assigns a worker a domain/list of paths to index.
2. The worker goes through each page of respecting robots.txt.
3. The worker makes a list of all a tags which link to an external domain.
4. The list is sent to the server which compares each external domain to its database.
The process operates in 3 stages:
1. Initial Stage/Exponential Growth - The number of new domains is much greater than the existing domains.
2. Saturation Stage - The number of new domains is lower than the amount of existing domains.
3. Rescanning Stage - Existing domains are periodically rescanned looking for new links.
## MTI Worker Protocol (MTIWP) Specification
MTIWP is a TCP-based binary application layer protocol used to communicate between the workers and coordination server.
The packet is made up of the following fields:
- Version (2 Bytes)
- Worker ID (6 Bytes)
- Timestamp (8 Bytes)
- Method (1 Byte)
- Payload Length (2 Bytes)
- Payload (0-65535 Bytes)
Valid Methods:
- ACK (0x00)
- Ping (0x01)
- Pong (0x02)
- Hello (0x03)
- Index (0x04)
- Cancel (0x05)
- Summary (0x06)
Further Detail Goes Here
A project that is designed to scrape the internet to create a visualization of the internet.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB