In this tutorial we will learn how to use Rust to write a simple netcat client and server using the standard library only. A netcat client is like a Swiss Army knife for networking. It is similar to PuTTY and telnet. You can use it to connect to a server and send and receive data. We will create an app that can behave both as a client and server.
Hereβs a video of the app that we are going to build in action.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
You can find the finished source code for this tutorial here.
Letβs create a new project by running cargo create --bin rtelnet
. Then we will add the
following dependencies to our Cargo.toml
file.
# Command line argument parsing.
clap = { version = "4.4.13", features = ["derive"] }
# Pretty logging.
femme = { version = "2.2.1" }
log = { version = "0.4.20" }
# Colorization and ANSI escape sequence codes.
r3bl_tui = { version = "0.5.1" }
r3bl_ansi_color = { version = "0.6.9" }
This Rust app has a single binary, and depending on the command line arguments, it will behave
either as a client or server. We will use the clap
crate to parse the command line arguments.
We will configure clap
so that the following commands will work:
cargo run server
cargo run client
We want to allow the user to specify the following options and chose their own address and
port. If the user does not specify any options, we will use the default values. The
default value for --address
is 127.0.0.1
, and the default value for --port
is
3000
.
cargo run server --address 127.0.0.1 --port 8080
cargo run server --address 127.0.0.1
cargo run server --port 8080
cargo run client --address 127.0.0.1 --port 8080
cargo run client --address 127.0.0.1
cargo run client --port 8080
Letβs also add an option that we can use to disable log output to stdout. By default, we
will log to stdout. But if the user specifies the --log-disable
flag, then we disable
all log output.
Hereβs the clap configuration that gives us this behavior.
use clap::{Parser, Subcommand};
pub use defaults::*;
mod defaults {
use super::*;
pub const DEFAULT_PORT: u16 = 3000;
pub const DEFAULT_ADDRESS: &str = "127.0.0.1";
}
pub use clap_config::*;
mod clap_config {
use super::*;
#[derive(Parser, Debug)]
pub struct CLIArg {
/// IP Address to connect to or start a server on
#[clap(long, short, default_value = DEFAULT_ADDRESS, global = true)]
pub address: IpAddr,
/// TCP Port to connect to or start a server on
#[clap(long, short, default_value_t = DEFAULT_PORT, global = true)]
pub port: u16,
/// Logs to stdout by default, set this flag to disable it
#[clap(long, short = 'd', global = true)]
pub log_disable: bool,
/// The subcommand to run
#[clap(subcommand)]
pub subcommand: CLISubcommand,
}
#[derive(Subcommand, Debug)]
pub enum CLISubcommand {
/// Start a server on the given address and port
Server,
/// Connect to a server running on the given address and port
Client,
}
}
Letβs start with the simpler of the two, the client. We will use std::net::TcpStream
to
create a TCP socket client. We will need an IP address and port in order to make a TCP
connection. And to run the client we will need to run the following command:
cargo run client
Hereβs what the main function of our app looks like:
fn main() {
println!("Welcome to rtelnet");
let cli_arg = CLIArg::parse();
let address = cli_arg.address;
let port = cli_arg.port;
let socket_address = format!("{}:{}", address, port);
if !cli_arg.log_disable {
femme::start()
}
match match cli_arg.subcommand {
CLISubcommand::Server => start_server(socket_address),
CLISubcommand::Client => start_client(socket_address),
} {
Ok(_) => {
println!("Program exited successfully");
}
Err(error) => {
println!("Program exited with an error: {}", error);
}
}
}
The function that performs the client logic looks like this.
fn start_client(socket_address: String) -> IOResult<()> {
log::info!("Start client connection");
let tcp_stream = TcpStream::connect(socket_address)?;
let (mut reader, mut writer) = (BufReader::new(&tcp_stream), BufWriter::new(&tcp_stream));
// Client loop.
loop {
// Read user input.
let outgoing = {
let mut it = String::new();
let _ = stdin().read_line(&mut it)?;
it.as_bytes().to_vec()
};
// Tx user input to writer.
let _ = writer.write(&outgoing)?;
writer.flush()?;
// Rx response from reader.
let incoming = {
let mut it = vec![];
let _ = reader.read_until(b'\n', &mut it);
it
};
let display_msg = String::from_utf8_lossy(&incoming);
let display_msg = display_msg.trim();
let reset = SgrCode::Reset.to_string();
let display_msg = format!("{}{}", display_msg, reset);
println!("{}", display_msg);
// Print debug.
log::info!(
"-> Tx: '{}', size: {} bytes{}",
String::from_utf8_lossy(&outgoing).trim(),
outgoing.len(),
reset,
);
log::info!(
"<- Rx: '{}', size: {} bytes{}",
String::from_utf8_lossy(&incoming).trim(),
incoming.len(),
reset,
);
}
}
Here are a few things to note about the client code:
BufReader
and BufWriter
for the TcpStream
that we get from
TcpStream::connect()
. This is because we want to read and write data in chunks, and
not one byte at a time, for performance reasons, and to simplify the logic. These two
structs allow us to read and write data very easily in chunks that are delimited by new
lines (\n
).Ctrl+C
will the
client exit. The default behavior for Rust is to
exit the process when this happens. This drops the TCP connection causing the server to exit as
well.TcpStream
, but the
stdin()
stream. This behaves very similarly to the TcpStream
stream. We can read
data from it in chunks delimited by new lines (\n
). Once the user types a message and
presses enter that message, eg: "hi"
, and the new line are stored in the it
variable, eg: "hi\n"
. We then convert the String into a byte array, eg: [104, 105,
10]
, and then convert it into a Vec<u8>
. We then send it to the server. We must call
flush()
since BufWriter
buffers the data and does not send it to the server until we
call flush()
for IO performance reasons. It queueβs up the data and sends it in
chunks, instead of sending it one byte at a time.stdin()
as we
have already seen. The main thread blocks until there is some data that can be read from
the server. Or if the TCP connection errors out in any way (timeout or closed by various
means). If there is an error, then this function returns an error, and the main thread
exits. Note that the start_client()
function itself returns an IOResult
, which is
just a type alias for pub type IOResult<T> = std::io::Result<T>;
. The error handling
is quite simple. If there is an error, we print it out and exit the program.incoming
variable using
reader.read_until(b'\n', &mut it);)
. This is because we expect the server to send us
data that is terminated by a new line (\n
). So we read the data until we encounter a
new line. This is a blocking call, so the main thread blocks until there is some data
that can be read from the server. Note that the \n
is included in incoming
variable,
much like it is in stdin()
.
String::from_utf8_lossy(&incoming);
to convert this incoming:
Vec<u8>
into a String
. We call .trim()
on the String, so that the trailing \n
is removed.trim()
returns a &str
, so if you want to turn it into a String, you have
to run in through this expression format!("{}",
String::from_utf8_lossy(&incoming).trim())
function.loop
, after the incoming data has been read from the server,
we print it out to the terminal. Since the server will send us ANSI escape sequence
codes that colorize the text that we print to the terminal, we want to reset the color
after we print the text, so it does not pollute our stdout()
output stream. We use the
SgrCode::Reset
code to reset the
color
of the text that we print to the terminal.Now letβs create the server. We will use std::net::TcpListener
to create a TCP socket
server. We will need an IP address and port in order to make a TCP connection. To run the
server we will need to run the following command:
cargo run server
The server code is very similar to the client code. We need a server loop that runs
forever, and we need to first read (blocking until there is any data available) and then
write data in chunks delimited by new lines (\n
). When there is no data available to
read EOF
is reached on the reader (aka, input TCP stream) then we break out of this loop
and exit. When data comes in (delimited by \n
) we process it and send a response back to
the client. We process this data by applying a lolcat
effect
on it, so the client will get a very colorful version of whatever text message that they
sent to the server.
One more thing we will see when implementing the server is having to spawn multiple
threads to handle each incoming client connection. While the client is a single threaded
app, the server is a multi-threaded app. The client is only concerned w/ a single TCP
connection, but the server is concerned with multiple TCP connections, each connection
emanating from a different client process running the cargo run client
and creating a
new OS process. Fortunately Rust is built for fearless concurrency and parallelism from
the ground up.
Hereβs the main function of our server app:
pub fn start_server(socket_address: String) -> IOResult<()> {
let tcp_listener = TcpListener::bind(socket_address)?;
// Server connection accept loop.
loop {
log::info!("Waiting for a incoming connection...");
let (tcp_stream, ..) = tcp_listener.accept()?; // This is a blocking call.
// Spawn a new thread to handle this connection.
thread::spawn(|| match handle_connection(tcp_stream) {
Ok(_) => {
log::info!("Successfully closed connection to client...");
}
Err(_) => {
log::error!("Problem with client connection...");
}
});
}
}
Here are a few things to note about the server code:
IOResult
just like the client code. There are frequent calls to the ?
operator, which is shorthand for matching on the Result
and returning early if thereβs
an error. This is rudimentary error handling, and its good enough for this pedagogical
example. Note that even in this pedagogical example, we donβt use the unwrap()
method
which will induce a panic if thereβs an error. We always use the ?
operator, which
will return early if thereβs an error. It isnβt a good idea to get into the habit of
using unwrap()
outside of tests. These habits are hard to break once theyβre formed.
You can even add the following
#![warn(clippy::unwrap_in_result)]
in the top level module of your project to have the compiler warn you if you use
unwrap()
outside of tests. Hereβs an
example.TcpListener::bind(socket_address)?;
. This does not start a server
yet. It just reserves a port on the given address, assuming that it is available. If some other
process has already bound to that port, then this will return an error.TcpListener
instance, we can call accept()
on it to start listening
for incoming connections. This is a blocking call, so the main thread blocks until there
is an incoming connection. Once there is an incoming connection, we get a TcpStream
instance, which we can use to read and write data to the client. This is a blocking
call. Which means that the main thread wonβt be able to do anything else, like process
other incoming connections, while it is waiting here, for a connection to come in.thread::spawn()
to create a new thread and have it handle the
incoming connection. We spawn a new thread for each incoming connection. This is not a
scalable solution, but it is good enough for this
pedagogical example. We will learn about more scalable solutions in a the Write a
simple TCP chat server in Rust tutorial.Now, letβs look at the handle_connection()
function that is called by the spawned
thread. This is the function that handles the incoming connection from the client. And it
defines our βprotocolβ, along with the client code. We arenβt using any formalized
protocol like HTTP or SMTP. We are just sending bytes back and forth between the client
and server, and interpreting them in a certain way, which is our informal protocol. This
code is very similar to the client side code, including the loop
and the BufReader
and
BufWriter
structs. And even looking for EOF
to break out of the loop. Except that we
donβt block on stdin()
for input here.
fn handle_connection(tcp_stream: TcpStream) -> IOResult<()> {
log::info!("Start handle connection");
let reader = &mut BufReader::new(&tcp_stream);
let write = &mut BufWriter::new(&tcp_stream);
// Process client connection loop.
loop {
let mut incoming: Vec<u8> = vec![];
// Read from reader.
let num_bytes_read = reader.read_until(b'\n', &mut incoming)?;
// Check for EOF. The stream is closed.
if num_bytes_read == 0 {
break;
}
// Process.
let outgoing = process(&incoming);
// Write to writer.
write.write(&outgoing)?;
let _ = write.flush()?;
// Print debug.
log::info!("-> Rx(bytes) : {:?}", &incoming);
log::info!(
"-> Rx(string): '{}', size: {} bytes",
String::from_utf8_lossy(&incoming).trim(),
incoming.len(),
);
log::info!(
"<- Tx(string): '{}', size: {} bytes",
String::from_utf8_lossy(&outgoing).trim(),
outgoing.len()
);
}
log::info!("End handle connection - connection closed");
Ok(())
}
Finally, letβs look at the process()
function that takes the incoming bytes to the
outgoing bytes. This is where we add some fun and color and flair to our app. We colorize
the incoming bytes using a lolcat effect and send it back to the client.
use r3bl_tui::ColorWheel;
fn process(incoming: &Vec<u8>) -> Vec<u8> {
// Convert incoming to String, and remove any trailing whitespace (includes newline).
let incoming = String::from_utf8_lossy(incoming);
let incoming = incoming.trim();
// Prepare outgoing payload.
let outgoing = incoming.to_string();
// Colorize it w/ a gradient.
let outgoing = ColorWheel::lolcat_into_string(&outgoing);
// Generate outgoing response. Add newline to the end of output (so client can process it).
let outgoing = format!("{}\n", outgoing);
// Return outgoing payload.
outgoing.as_bytes().to_vec()
}
Now that you have a handle on the basics of writing a simple netcat client and server, you can read this tutorial to learn more about creating a more advanced TCP server that netcat, telnet, or PuTTY clients can connect to, in order to have multiple client apps chat with each other.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
In this tutorial we will build a simple chat server using Tokio. The server will be able to handle multiple clients, and each client will be able to send messages to the server, which will then broadcast the message to all other connected clients.
tokio::net::TcpListener
and tokio::net::TcpStream
to create a
TCP server that listens for incoming connections and handles them concurrently.tokio::sync::broadcast
to broadcast messages to all connected
clients.Read this tutorial to learn more about the basics of TCP client and server programming in Rust (without using Tokio).
Hereβs a video of the app that we are going to build in action.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
You can find the finished source code for this tutorial here.
ββCLIENT-1ββββββββ ββCLIENTβ2ββββββββ ββCLIENTβ3βββββββ
β β β β β β
βββββββββΌβββββββββ ββββββββΌββββββββββ βββββββΌββββββββββ
β β β
ββSERVERββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββΌβββββββββββββ
β β β β β
β β
β handle_client_task() handle_client_task() handle_client_task() β
β βββββββββββββββββββββ ββββββββββββββββββββββ βββββββββββββββββββββββ β
β β ββββββ ββββββ β β ββββββ ββββββ β β ββββββ ββββββ β β
β β β TX β β RX β β β β TX β β RX β β β β TX β β RX β β β
β β βββ¬βββ βββ²βββ β β βββ¬βββ βββ²βββ β β βββ¬βββ βββ²βββ β β
β β β β β β β β β β β β β β
β βββββββΌβββββββΌβββββββ βββββββΌβββββββΌββββββββ ββββββββΌβββββββΌββββββββ β
β β β β β β β β
β β β β β β β β
β βββββββΌβββββββ΄βββββββββββββββΌβββββββ΄ββββββββββββββββββΌβββββββ΄ββββββββ β
β β (TX, RX) = channel::broadcast() β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The server has a main
function that creates a tokio::net::TcpListener
and listens for
incoming connections. When a new connection is received, it spawns a new task to handle
the connection using tokio::spawn()
.
Using tokio::select!
, the task tries to do the following concurrently, and waits until
one of them completes:
When one task above completes, the other is dropped. Then the code path with the completed task executes. Then the code returns to the infinite loop, if it hasnβt returned already.
A client can be any TCP client, such as telnet
, nc
, or PuTTY.
Letβs create a new project by running cargo create --bin tcp-server-netcat-client
. Then
we will add the following dependencies to our Cargo.toml
file.
# tokio.
tokio = { version = "1.35.1", features = ["full"] }
# stdout logging.
femme = { version = "2.2.1" }
log = { version = "0.4.20" }
# r3bl_rs_utils_core - friendly name generator.
r3bl_rs_utils_core = { version = "0.9.12" }
r3bl_tui = { version = "0.5.1" }
We will implement the following algorithm for our server in our main function:
TcpListener
and bind to an address & port.TCPStream
.tokio::spawn()
to spawn a task to handle this client connection and its
TCPStream
.In the task that handles the connection:
BufReader
& BufWriter
from the TCPStream
. The reader and writer allow us to
read data from and write data to the client socket.tokio::select!
to concurrently:
recv()
):
BufWriter
to write the message).BufReader::read_line()
):
incoming
from reader.process(incoming)
and generate outgoing
. This colorizes the incoming
message with a lolcat effect to generate the outgoing
message.incoming
message to other connected clients (via the broadcast channel).You can find the finished source code for this tutorial here.
Hereβs the code for the main function, and some supporting type aliases and structs:
pub type IOResult<T> = std::io::Result<T>;
#[derive(Debug, Clone)]
pub struct MsgType {
pub socket_addr: SocketAddr,
pub payload: String,
pub from_id: String,
}
#[tokio::main]
pub async fn main() -> IOResult<()> {
let addr = "127.0.0.1:3000";
// Start logging.
femme::start();
// Create TCP listener.
let tcp_listener = TcpListener::bind(addr).await?;
log::info!("Server is ready to accept connections on {}", addr);
// Create channel shared among all clients that connect to the server loop.
let (tx, _) = broadcast::channel::<MsgType>(10);
// Server loop.
loop {
// Accept incoming socket connections.
let (tcp_stream, socket_addr) = tcp_listener.accept().await?;
let tx = tx.clone();
tokio::spawn(async move {
let result = handle_client_task(tcp_stream, tx, socket_addr).await;
match result {
Ok(_) => {
log::info!("handle_client_task() terminated gracefully")
}
Err(error) => log::error!("handle_client_task() encountered error: {}", error),
}
});
}
}
To run the server, you can run cargo run
. There are no command line arguments to pass or
parse.
Since tokio::spawn
sounds similar to thread::spawn
it might be easy to assume that
tokio::spawn
creates a new thread. This would go against the idea of even using tokio
(which is all about concurrency and non blocking IO), since handling one connection per
thread isnβt scalable, which is what we did in
this tutorial: Write a simple TCP chat server in Rust.
tokio::spawn
does not create a thread; it creates a Tokio task, which is a
co-operatively scheduled entity that Tokio knows how to schedule on the Tokio runtime (in
turn, the Tokio runtime can have as many worker threads as you want - from 1 upwards).
By using tokio::spawn
, you allow the Tokio runtime to switch to another task at points
in the task where it has a .await
, and only those points. Your alternative, if you donβt
want multiple tasks, is to use things like select!
, join!
and functions with select
or ` join` in their name to have concurrent I/O in a single task.
The point of spawning in Tokio is twofold:
select
to wait for one of many options, or join
to wait for all options
to finish).More information:
The handle_client_task
function is where all the magic happens.
Hereβs the code for the handle_client_task()
function:
async fn handle_client_task(
mut tcp_stream: TcpStream,
tx: Sender<MsgType>,
socket_addr: SocketAddr,
) -> IOResult<()> {
log::info!("Handle socket connection from client");
let id = friendly_random_id::generate_friendly_random_id();
let mut rx = tx.subscribe();
// Set up buf reader and writer.
let (reader, writer) = tcp_stream.split();
let mut reader = BufReader::new(reader);
let mut writer = BufWriter::new(writer);
// Send welcome message to client w/ ids.
let welcome_msg_for_client =
ColorWheel::lolcat_into_string(&format!("addr: {}, id: {}\n", socket_addr, id));
writer.write(welcome_msg_for_client.as_bytes()).await?;
writer.flush().await?;
let mut incoming = String::new();
loop {
let tx = tx.clone();
tokio::select! {
// Read from broadcast channel.
result = rx.recv() => {
read_from_broadcast_channel(result, socket_addr, &mut writer, &id).await?;
}
// Read from socket.
network_read_result = reader.read_line(&mut incoming) => {
let num_bytes_read: usize = network_read_result?;
// EOF check.
if num_bytes_read == 0 {
break;
}
handle_socket_read(num_bytes_read, &id, &incoming, &mut writer, tx, socket_addr).await?;
incoming.clear();
}
}
}
Ok(())
}
read_from_broadcast_channel()
does this work.handle_socket_read()
does this work.Whichever task completes first, the tokio::select!
block will go down that code path,
and drop the other task.
Hereβs the code for the read_from_broadcast_channel()
function:
async fn read_from_broadcast_channel(
result: Result<MsgType, RecvError>,
socket_addr: SocketAddr,
writer: &mut BufWriter<WriteHalf<'_>>,
id: &str,
) -> IOResult<()> {
match result {
Ok(it) => {
let msg: MsgType = it;
log::info!("[{}]: channel: {:?}", id, msg);
if msg.socket_addr != socket_addr {
writer.write(msg.payload.as_bytes()).await?;
writer.flush().await?;
}
}
Err(error) => {
log::error!("{:?}", error);
}
}
Ok(())
}
Hereβs the code for the handle_socket_read()
function:
async fn handle_socket_read(
num_bytes_read: usize,
id: &str,
incoming: &str,
writer: &mut BufWriter<WriteHalf<'_>>,
tx: Sender<MsgType>,
socket_addr: SocketAddr,
) -> IOResult<()> {
log::info!(
"[{}]: incoming: {}, size: {}",
id,
incoming.trim(),
num_bytes_read
);
// Process incoming -> outgoing.
let outgoing = process(&incoming);
// outgoing -> Writer.
writer.write(outgoing.as_bytes()).await?;
writer.flush().await?;
// Broadcast outgoing to the channel.
let _ = tx.send(MsgType {
socket_addr,
payload: incoming.to_string(),
from_id: id.to_string(),
});
log::info!(
"[{}]: outgoing: {}, size: {}",
id,
outgoing.trim(),
num_bytes_read
);
Ok(())
}
fn process(incoming: &str) -> String {
// Remove new line from incoming.
let incoming_trimmed = format!("{}", incoming.trim());
// Colorize it.
let outgoing = ColorWheel::lolcat_into_string(&incoming_trimmed);
// Add new line back to outgoing.
format!("{}\n", outgoing)
}
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
There are few things that generate as much fear and anxiety in developers as git
merge
conflicts. git
is very popular and very powerful, and it is a low level command line
tool. And it is not very user friendly. It is meant to be orchestrate-able and automated
using scripts and CI/CD tools, and build systems; it is extremely flexible. It is not
meant to be used in an interactive manner w/ a human user at the keyboard.
This just creates an opportunity for others to come along and craft user experiences on
top of git
that are more use case driven. And these UXes can come in the form or GUIs,
TUIs, or even conversational interfaces.
But thatβs not the focus of this article which is all about the CLI experience of inducing and resolving merge conflicts. So letβs get started.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
Letβs create a local repo from scratch and set things up so that we can predictably generate a merge conflict. Hereβs what we will do at a high level:
main
branch.main
branch and add some content to it.feature
branch based on the main
branch.feature
branch.main
branch with a change that is going to conflict w/ a change in feature
branch.feature
branch into the main
branch.Hereβs a script to get you started:
#!/usr/bin/env bash
# Create a local repo.
export TMP_REPO_DIR="~/Downloads/tmp/git-merge-conflict-demo"
if [ -d $TMP_REPO_DIR ]; then
echo "Folder exists, recreating $TMP_REPO_DIR"
rm -rf $TMP_REPO_DIR
mkdir -p $TMP_REPO_DIR
else
echo "Folder does not exist, creating $TMP_REPO_DIR"
mkdir -p $TMP_REPO_DIR
fi
cd $TMP_REPO_DIR
git init
git checkout -b main
# Create a file in the main branch and add some content to it.
# This is the "OG change".
echo -e "This is a new feature.\n## 3. Example 3" > file.txt
git add file.txt
git commit -m "Add myexample3"
# Create a develop branch based on the main branch.
git checkout -b develop main
# Person A comes along and changes this line w/ a plus in the develop branch.
echo -e "This is a new feature.\n## 3. Example 3+" > file.txt
git add file.txt
git commit -m "Fix typo w/ plus in develop branch"
# Person B comes along and change this line w/ a minus in the main branch.
# This is going to conflict with the change in the develop branch.
git checkout main
echo -e "This is a new feature.\n## 3. Example 3-" > file.txt
git add file.txt
git commit -m "Fix typo w/ minus in main branch"
# Merge (using rebase, so no extra commit) the develop branch into the main branch.
git rebase develop
This results in a merge conflict. And when you run git diff
it looks like this:
diff --cc file.txt
index 89da142,ef43c8f..0000000
--- a/file.txt
+++ b/file.txt
@@@ -1,2 -1,2 +1,8 @@@
This is a new feature.
++<<<<<<< HEAD
+## 3. Example 3+
++||||||| parent of 7c0f0e4 (Fix typo w/ - in main branch)
++## 3. Example 3
++=======
+ ## 3. Example 3-
++>>>>>>> 7c0f0e4 (Fix typo w/ - in main branch)
Letβs use some pictures to understand the story of how we got here. And how to resolve this.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
As developers we tend to spend a lot of time in the terminal. It is a great place to exercise precise control over our computers. And it is a great place to automate tasks. However, there are some rough edges to this experience. For example, even though the interaction metaphor w/ CLI apps is a conversation, we have to be very precise in the language we use in this conversation. Lots of trial and error, patience and resilience are required to make it work. And it does not have to be this way.
To use a racing analogy, terminals are like race cars. They are fast and powerful, and you can exercise direct and precise control over them. But if you get things wrong, there can be consequences. Porsche is a car company that wins endurance races, and a very long time ago, they decided to make a race car that was friendly to the ergonomics of their drivers.
The thinking was that if the driver is comfortable, they will perform better, and the 24 hour race
wins will be just a little bit closer within reach. Similarly, we can add some interactivity to our
CLI apps to make them more ergonomic to use. And we can do this without going full TUI. We can do
this in a way that is additive to the existing CLI experience. We can βtuifyβ our CLI apps that are
built using clap
.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
For more information on general Rust type system design (functional approach rather than object oriented), please take a look at this paper by Will Crichton demonstrating Typed Design Patterns with Rust.
Here are some great resources to learn more about good CLI design concepts. The Rust crate clap
is
used by a lot of Rust apps to implement this CLI. And in this tutorial we will take a look at how to
add some interactivity to these clap
CLI apps using the r3bl_tuify
crate.
Note that these resources are all about CLI and not TUI. There isnβt very much information out there about TUIs. It is a new and evolving space.
clap
docsclap
command and subcommand structure guidelinesThe CLI guidelines above do a great job of explaining how to create a good CLI experience. However they do not cover how to add interactivity to your CLI apps. Why would we want to do this? Letβs take a real example to illustrate the benefits of this next.
This example is a little βmetaβ. The r3bl_tuify
crate, that allows interactivity to be added to
clap
CLI apps, is available as a binary and library. The binary which can be used from the command
line (and uses clap
) uses the library to provide an interactive experience when certain arguments
arenβt provided on the command line.
The idea with the binary target is that you might want to quickly incorporate some interactivity
into your shell scripts without getting into the Rust library. In this case, you can use the rt
binary target to do that. This binary takes quite a few arguments as you might imagine. However, you
donβt have to supply all of them at the start.
So instead of typing this massive command at the start (where cargo run --
simply runs the binary
called rt
):
cat TODO.todo | cargo run -- select-from-list \
--selection-mode single \
--command-to-run-with-each-selection "echo %"
You can simply type the following shorter command and have the app prompt you for the rest of the information that it needs:
cat TODO.todo | cargo run -- select-from-list
Hereβs a video of this in action, where the app is prompting the user for two items:
selection-mode
andcommand-to-run-with-each-selection
interactively π:The r3bl_tuify
app itself uses clap
to parse the command line arguments. Hereβs an overview of
what that looks like (all of it using the derive
macro approach). Hereβs a link to the
main.rs::AppArgs
.
#[derive(Debug, Parser)]
#[command(bin_name = "rt")]
#[command(about = "Easily add lightweight TUI capabilities to any CLI apps using pipes", long_about = None)]
#[command(version)]
#[command(next_line_help = true)]
#[command(arg_required_else_help(true))]
pub struct AppArgs {
#[clap(subcommand)]
command: CLICommand,
#[clap(flatten)]
global_opts: GlobalOpts,
}
#[derive(Debug, Args)]
struct GlobalOpts {
/// Print debug output to log file (log.txt)
#[arg(long, short = 'l')]
enable_logging: bool,
/// Optional maximum height of the TUI (rows)
#[arg(value_name = "height", long, short = 'r')]
tui_height: Option<usize>,
/// Optional maximum width of the TUI (columns)
#[arg(value_name = "width", long, short = 'c')]
tui_width: Option<usize>,
}
#[derive(Debug, Subcommand)]
enum CLICommand {
/// Show TUI to allow you to select one or more options from a list, piped in via stdin π
SelectFromList {
/// Would you like to select one or more items?
#[arg(value_name = "mode", long, short = 's')]
selection_mode: Option<SelectionMode>,
/// Each selected item is passed to this command as `%` and executed in your shell.
/// For eg: "echo %". Please wrap the command in quotes π‘
#[arg(value_name = "command", long, short = 'c')]
command_to_run_with_each_selection: Option<String>,
},
}
The things to note are that the selection_mode
and command_to_run_with_each_selection
fields of
the CliCommand::SelectFromList
enum are optional. This is where the r3bl_tuify
crate comes in.
It will prompt the user for these two fields if they are not supplied on the command line.
You can add this programmatically using the library to your existing CLI apps.
The piping option using the binary is severely limited, so the library option is strongly recommended. The binary is more of a convenience for shell scripts only on Linux.
You can see how to use the library to perform this interactivity in
main.rs::show_tui()
.
Here are two examples of adding interactivity.
Hereβs an example of adding interactivity using a list selection component. This is useful
when the values that a field can take are known in advance. In this example, they are
since selection-mode
is a clap
EnumValue
that can only take one of the following
values: single
, or multiple
.
In this scenario, --selection-mode
is not passed in the command line. So it only
interactively prompts the user for this piece of information. Similarly, if the user does
not provide this information, the app exits and provides a help message.
cat TODO.todo | cargo run -- select-from-list --command-to-run-with-each-selection "echo %"
// Handle `selection-mode` is not passed in.
let selection_mode = if let Some(selection_mode) = maybe_selection_mode {
selection_mode
} else {
let possible_values_for_selection_mode =
get_possible_values_for_subcommand_and_option(
"select-from-list",
"selection-mode",
);
print_help_for_subcommand_and_option("select-from-list", "selection-mode").ok();
let user_selection = select_from_list(
"Choose selection-mode".to_string(),
possible_values_for_selection_mode,
max_height_row_count,
max_width_col_count,
SelectionMode::Single,
);
let it = if let Some(user_selection) = user_selection {
if let Some(it) = user_selection.first() {
println!("selection-mode: {}", it);
SelectionMode::from_str(it, true).unwrap_or(SelectionMode::Single)
} else {
print_help_for("select-from-list").ok();
return;
}
} else {
print_help_for("select-from-list").ok();
return;
};
it
};
Hereβs an example of adding interactivity using a text input field. This is useful when the values
that a field can take are not known in advance. The r3bl_tuify
crate uses the reedline
crate to
do this.
Fun fact:
reedline
is the text input field (line editor) that is used innushell
.
In this scenario, --command-to-run-with-each-selection
is not passed in the command
line. So it only interactively prompts the user for this piece of information. Similarly,
if the user does not provide this information, the app exits and provides a help message.
cat TODO.todo | cargo run -- select-from-list --selection-mode single
// Handle `command-to-run-with-each-selection` is not passed in.
let command_to_run_with_each_selection =
match maybe_command_to_run_with_each_selection {
Some(it) => it,
None => {
print_help_for_subcommand_and_option(
"select-from-list",
"command-to-run-with-each-selection",
)
.ok();
let mut line_editor = Reedline::create();
let prompt = DefaultPrompt {
left_prompt: DefaultPromptSegment::Basic(
"Enter command to run w/ each selection `%`: ".to_string(),
),
right_prompt: DefaultPromptSegment::Empty,
};
let sig = line_editor.read_line(&prompt);
match sig {
Ok(Signal::Success(buffer)) => {
if buffer.is_empty() {
print_help_for("select-from-list").ok();
return;
}
println!("Command to run w/ each selection: {}", buffer);
buffer
}
_ => {
print_help_for("select-from-list").ok();
return;
}
}
}
};
// Actually get input from the user.
let selected_items = {
let it = select_from_list(
"Select one line".to_string(),
lines,
max_height_row_count,
max_width_col_count,
selection_mode,
);
convert_user_input_into_vec_of_strings(it)
};
There are many more components that need to be added to make it easier to βtuifyβ lots of existing CLI experiences. Things like multi line editor component w/ syntax highlighting, scroll view, form input fields, and more. If you would like to contribute to this effort, it would be great to have your help.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
In this tutorial we will learn how to use just
by example to
manage project specific commands. just
is like make
, but it is written in Rust, and it works
with cargo
.
Before we get started, please take a look at the
just
project README.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
Letβs say you have a justfile
that looks like this, and it has a single recipe called
list
:
list:
ls -l
And you run it by typing just list
. It just turns around and runs this sh -c "ls -l"
.
Thatβs it. So on Windows, this doesnβt work, because sh
isnβt installed by default. So
you have to install cygwin. And then you have to install just
and then you have to
install sh
.
Alternatively, you can specify that you want to use powershell
instead by
adding this to the top of the justfile
: set shell := ["powershell.exe", "-c"]
. Or you
can just run this just --shell powershell.exe --shell-arg -c list
to run just
itself
at the command prompt.
You can also supply different shell interpreters like python
or node
. And you can even
provide shebang
lines like #!/usr/bin/env python
or #!/usr/bin/env node
at the top
of each recipe.
π In order for our just
file to work, we must first install the Rust toolchain and just
and
cargo-watch
:
rustup
by following the instructions
here.cargo-watch
using cargo install cargo-watch
.just
on your system using
cargo install just
. It is available for Linux, macOS, and Windows.
just
you can follow
these instructions.just
using cargo install just
or brew install just
you will not get shell
completions without doing one extra configuration step. So on Linux it is best to use
sudo apt install -y just
if you want them.For Rust projects, typically we will have a build, run, test project specific commands. Letβs start
with these simple ones first. The benefit of just
is that we can use it to run these commands on
any platform (Linux, Mac, Windows). And we donβt need to create OS or shell specific scripts to do
this π.
Letβs start by creating a justfile
in the root of our project. The justfile
is where we will
define our project specific commands. Here is what it looks like for the
r3bl_ansi_color
repo:
build:
cargo build
clean:
cargo clean
run:
cargo run --example main
These are pretty simple commands. The syntax is pretty simple. The first line is the command name. And the second line is the command to run. The command can be a single command or a series of commands.
Now in order to run this, we can just run just --list
in the root of our project. And it will show
us the list of commands that we can run.
$ just --list
Available recipes:
build
clean
run
Then to run a command, we can just run just <command_name>
. For example, to run the build
command, we can run just build
.
$ just build
This is pretty straightforward. You can just list all other other just commands inline. Hereβs an example.
all: clean build test clippy docs rustfmt
The all
command will run the other commands in the order theyβre written.
Currently our justfile
will run on Linux and macOS. To make it run on Windows, we can
run just
itself using powershell.exe
. Here is what it looks like:
just --shell powershell.exe --shell-arg -c build
Or we can add the line set shell := ["powershell.exe", "-c"]
to the top of the
justfile
.
Alternatively, we can use nu
shell instead of powershell.exe
since it is written in
Rust and available via cargo install nu
.
Letβs add a new command called all
to our justfile
. This will just turn around and run the
build
and clean
commands. Here is what it looks like:
all: build clean
Now, we can also use just
in CI / CD environments. For example, here is the rust.yml
file
for this repoβs Github Actions. It runs just all
in the build
step.
The one thing to note is that we are installing just
in the docker container before we
run the just
command. We do this by pulling in the prebuilt binary for Ubuntu as shown
here: curl --proto '=https' --tlsv1.2
-sSf https://just.systems/install.sh | bash -s -- --to DEST
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
# Install just before running it below.
- name: Install just
run: curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh | bash -s -- --to /usr/local/bin
# Simply run the `just all` command.
- name: all
run: just all
Using just all
is relatively straightforward way to run all our build steps (that would
run in a CI / CD environment) on our local computer w/out installing docker. While
ensuring that these same steps are carried out in the CI / CD environment.
We can also pass arguments into our commands. Letβs say that we have a command that we can use to run a single test. We can pass the name of the test into the command. Here is what it looks like:
watch-one-test test_name:
# More info on cargo test: https://doc.rust-lang.org/cargo/commands/cargo-test.html
# More info on cargo watch: https://github.com/watchexec/cargo-watch
cargo watch -x check -x 'test -- --test-threads=1 --nocapture {{test_name}}' -c -q
There are a few things to note here:
test_name
. If an argument is not passed in then just
will display an error and print a
message stating that an argument is required.justfile
. The `` enclose a
variable name.Now we can run this command by passing in the name of the test that we want to run. For example, if
we want to run the test_ansi_color
test, we can run just watch-one-test test_ansi_color
.
$ just watch-one-test bold
Hereβs an example of a justfile
that has a lot more commands for you to look at:
r3bl_ansi_color justfile.
The just
project README has lots of information on how to use
just
. It is best to have a specific thing you are looking for before you visit this page. Here are
some interesting links inside the README:
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
In this tutorial we will create a new Chrome Extension using Manifest V3 that allows us to create our own names for a URL or a set of URLs. This is useful when you have a set of URLs that you want to open at once, or you want to create a name for a URL that is hard to remember. Or if you just donβt want to use bookmarks. We will also save these shortlinks to the Chrome Sync key-value pair store. This extension will also allow the user to type commands when it is activated (in its popped up state). And we will use Typescript and React to build it.
Before we get started, here are some good references to take a look at:
π Please star and fork / clone the Shortlink repo π Install the Chrome Extension π οΈ
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
The first thing to is to create a new repo on GitHub using this template repo.
You can do this in two ways.
gh auth login
).
# More info: https://cli.github.com/manual/gh_repo_create
gh repo create shortlink --public --template r3bl-org/chrome-extension-typescript-react-template
# More info: https://cli.github.com/manual/gh_repo_clone
gh repo clone shortlink
At this point we have a shortlink
git repo on our local machine that is setup to build a Chrome
Extension. You can run the following command to build it.
npm install
npm run build
This will generate a dist/
directory that contains the Chrome Extension. You can load this into
Chrome by:
chrome:extensions
in the URL bar.dist/
directory. Your extension will be loaded
into Chrome.In our extension we will ask for the minimum of
permissions from the user.
This ensures that our extension doesnβt have access to anything more than it needs. All of this is
specified in the public/manifest.json
file. Hereβs an example of what this file might look like
when we are done building our extension.
{
"manifest_version": 3,
"name": "R3BL Shortlink",
"description": "Make go links",
"version": "2.0",
"icons": {
"16": "icon16.png",
"32": "icon32.png",
"48": "icon48.png",
"128": "icon128.png"
},
"action": {
"default_title": "Click to make go link for URL",
"default_popup": "popup.html",
"default_icon": {
"16": "icon16.png",
"32": "icon32.png",
"48": "icon48.png",
"128": "icon128.png"
}
},
"omnibox": {
"keyword": "go"
},
"background": {
"service_worker": "js/background.js"
},
"commands": {
"_execute_action": {
"suggested_key": {
"default": "Alt+L",
"mac": "Alt+L"
},
"description": "Make a go link for URL in address bar"
}
},
"permissions": ["activeTab", "tabs", "storage", "clipboardWrite"]
}
Now that we have our permissions sorted, we can start by adding functionality to the extension. When
we activate the extension by clicking its icon in the Chrome toolbar or by pressing the shortcut
Alt+l the popup.tsx
file will be run which itself is loaded by popup.html
.
You can learn more about activating the extension and the popup in the
chrome.browserAction
docs.
This popup.tsx
file will be the entry point for our extension. It is the main function in a node
program or the App
top level component in a React app. It sets up the UI and handles the user
input events (key presses).
This is what the UI looks like on Linux on my machine:
This is what the file looks like in the real Shortcut extension:
popup.tsx
. If you
go through this code, these are some of the things you will notice:
main()
function just sets up the main React component Popup
and mounts it to the DOM
(div
with id root
).useEffect()
hooks which ensures that when chrome.storage
changes, the global
state is updated and the component is re-rendered. Learn more about chrome.storage
in the API
reference here. Another hook
is responsible for painting the badge on the extension icon in the toolbar (when the
Shortlink[]
in the state changes).Popup
function component returns some JSX that is used to render the global state, which
are two things: Shortlink[]
and string
. The Shortlink[]
is used to render the list of
shortcuts and the string
is used to render the input field.handleOnChange()
and handleEnterKey()
function is where the user input that is typed is
interpreted into a command and then executed.There are some other files of note. Please take a look at their linked source code.
command.ts
: The
main logic for parsing a string
into a command is handled by this file. The
parseUserInputTextIntoCommand()
function does all the work of converting a given string
into
a Command
, and has a very Rust βvibeβ. Please check out how this works. It makes it very easy
to add or change commands in the future.storage.ts
:
This is where all the functions to manipulate the storage that syncs w/ Chrome accounts is
located. Functions that handle shortlinks to be deleted, or added, or updated can all be found
here. The Chrome storage API is async which is why the code in this file is written in the way
that it is.omnibox.ts
:
This file works w/ background.ts
to handle the omnibox functionality. The omnibox is the
address bar in Chrome. When
the user types go
and then a space, the omnibox will be activated and the user can type in a
shortcut. When the user presses Enter, the background.ts
file will be run and the
shortcut will be expanded to the full URL.Please read this guide on how to publish the
extension. You will have to get a developer account, and then upload the extension binaries. Thereβs
a make-distro-zip.sh
script provided in this repo that will create a zip file that you can upload
to the Chrome Web Store.
As part of publishing a version you have to provide justification for why you are requesting the permissions that you are. The fewer the permissions that you use, the better for the end user, and also for the review process to take less time.
π Please star and fork / clone the Shortlink repo π Install the Chrome Extension π οΈ
If you would like to get involved in an open source project and like Chrome extensions, please feel free to contribute to the Shortlink repo. There are a lot of small features that need to be added. And they can be a nice stepping stone into the world of open source contribution π.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
Ockam is a suite of programming libraries, command line tools, and managed cloud services to orchestrate end-to-end encryption, mutual authentication, key management, credential management, and authorization policy enforcement β all at massive scale. Ockamβs end-to-end secure channels guarantee authenticity, integrity, and confidentiality of all data-in-motion at the application layer.
π Please star and fork / clone the Ockam repo π.
One of the key features that makes this possible is Ockam Routing. Routing allows us to create secure channels over multi-hop, multi-protocol routes which can span various network topologies (servers behind NAT firewalls with no external ports open, etc) and transport protocols (TCP, UDP, WebSockets, BLE, etc).
In this blog post we will explore the Ockam Rust Library and see how routing works in Ockam. We will work with Rust code and look at some code examples that demonstrate the simple case, and more advanced use cases.
Before we get started, letβs quickly discuss the pitfalls of using existing approaches to securing communications within applications. Security is not something that most of us think about when we are building systems and are focused on getting things working and shipping.
Traditional secure communication implementations are typically tightly coupled with transport protocols in a way that all their security is limited to the length and duration of one underlying transport connection.
In other words using traditional secure communication implementations you may be opening the doors to losing trust in the data that your apps are working on. Here are some aspects of your apps that may be at risk:
In this blog post we will create two examples of Ockam nodes communicating with each other using Ockam Routing and Ockam Transports. We will use the Rust library to create these Ockam nodes and setup routing. Ockam Routing and transports enable other Ockam protocols to provide end-to-end guarantees like trust, security, privacy, reliable delivery, and ordering at the application layer.
An Ockam node is any running application that can communicate with other applications using various Ockam protocols like Routing, Relays, and Portals, Secure Channels, etc.
An Ockam node can be defined as any independent process which provides an API supporting the Ockam
Routing protocol. We can create Ockam nodes using the
Ockam command line interface (CLI) (ockam
command) or
using various Ockam programming libraries like our Rust and Elixir libraries. We will be using the
Rust library in this blog post.
π‘ To get started please follow this guide to get the Rust toolchain setup on your machine along with an empty project.
- The empty project is named
hello_ockam
.- This will be the starting point for all our examples in this blog post.
For our first example, we will create a simple Ockam node that will send a message over some hops (in the same node) to a worker (in the same node) that just echoes the message back. There are no TCP transports involved and all the messages are being passed back and forth inside the same node. This will give us a feel for building workers and routing at a basic level.
When a worker is started on a node, it is given one or more addresses. The node maintains a mailbox for each address and whenever a message arrives for a specific address it delivers that message to the corresponding registered worker.
π‘ For more information on creating nodes and workers using the Rust library, please refer to this guide.
We will need to create a Rust source file with a main()
program, and two other Rust source files
with two workers: Hopper
and Echoer
. We can then send a string message and see if we can get it
echoed back.
Before we begin letβs consider routing. When we send a message inside of a node it carries with it 2
metadata fields, onward_route
and return_route
, where a route
is simply a list of addresses
.
Each worker gets an address
in a node.
So, if we wanted to send a message from the app
address to the echoer
address, with 3 hops in
the middle, we can build a route like the following.
βββββββββββββββββββββββββ
β Node 1 β
βββββββββββββββββββββββββ€
β ββββββββββββββββββ β
β β Address: β β
β β 'app' β β
β βββ¬βββββββββββββ²ββ β
β βββΌβββββββββββββ΄ββ β
β β Address: β β
β β 'hopper1..3' βx3 β
β βββ¬βββββββββββββ²ββ β
β βββΌβββββββββββββ΄ββ β
β β Address: β β
β β 'echoer' β β
β ββββββββββββββββββ β
βββββββββββββββββββββββββ
Hereβs the Rust code to build this route.
/// Send a message to the echoer worker via the "hopper1", "hopper2", and "hopper3" workers.
let route = route!["hopper1", "hopper2", "hopper3", "echoer"];
Letβs add some source code to make this happen next. The first thing we will do is add one more
dependency to this empty hello_ockam
project. The
colored
crate will give us colorized console output
which will make the output from our examples so much easier to read and understand.
cargo add colored
Then we add the echoer
worker (in our hello_ockam
project) by creating a new /src/echoer.rs
file and copy / pasting the following code in it.
use colored::Colorize;
use ockam::{Context, Result, Routed, Worker};
pub struct Echoer;
/// When a worker is started on a node, it is given one or more addresses. The node
/// maintains a mailbox for each address and whenever a message arrives for a specific
/// address it delivers that message to the corresponding registered worker.
///
/// Workers can handle messages from other workers running on the same or a different
/// node. In response to a message, an worker can: make local decisions, change its
/// internal state, create more workers, or send more messages to other workers running on
/// the same or a different node.
#[ockam::worker]
impl Worker for Echoer {
type Context = Context;
type Message = String;
async fn handle_message(&mut self, ctx: &mut Context, msg: Routed<String>) -> Result<()> {
// Echo the message body back on its return_route.
let addr_str = ctx.address().to_string();
let msg_str = msg.as_body().to_string();
let new_msg_str = format!("π echo back: {}", msg);
// Formatting stdout output.
let lines = [
format!("π£ 'echoer' worker β Address: {}", addr_str.bright_yellow()),
format!(" Received: '{}'", msg_str.green()),
format!(" Sent: '{}'", new_msg_str.cyan()),
];
lines
.iter()
.for_each(|line| println!("{}", line.white().on_black()));
ctx.send(msg.return_route(), new_msg_str).await
}
}
Next we add the hopper
worker (in our hello_ockam
project) by creating a new /src/hopper.rs
file and copy / pasting the following code in it.
Note how this worker manipulates the onward_route
& return_route
fields of the message to send
it to the next hop. We will actually see this in the console output when we run this code soon.
use colored::Colorize;
use ockam::{Any, Context, Result, Routed, Worker};
pub struct Hopper;
#[ockam::worker]
impl Worker for Hopper {
type Context = Context;
type Message = Any;
/// This handle function takes any incoming message and forwards. it to the next hop
/// in it's onward route.
async fn handle_message(&mut self, ctx: &mut Context, msg: Routed<Any>) -> Result<()> {
// Cast the msg to a Routed<String>
let msg: Routed<String> = msg.cast()?;
let msg_str = msg.to_string().white().on_bright_black();
let addr_str = ctx.address().to_string().white().on_bright_black();
// Some type conversion.
let mut message = msg.into_local_message();
let transport_message = message.transport_mut();
// Remove my address from the onward_route.
let removed_address = transport_message.onward_route.step()?;
let removed_addr_str = removed_address
.to_string()
.white()
.on_bright_black()
.strikethrough();
// Formatting stdout output.
let lines = [
format!("π 'hopper' worker β Addr: '{}'", addr_str),
format!(" Received: '{}'", msg_str),
format!(" onward_route -> remove: '{}'", removed_addr_str),
format!(" return_route -> prepend: '{}'", addr_str),
];
lines
.iter()
.for_each(|line| println!("{}", line.black().on_yellow()));
// Insert my address at the beginning return_route.
transport_message
.return_route
.modify()
.prepend(ctx.address());
// Send the message on its onward_route.
ctx.forward(message).await
}
}
And finally letβs add a main()
to our hello_ockam
project. This will be the entry point for our
example.
π‘ When a new node starts and calls an
async
main
function, it turns that function into a worker with an address ofapp
. This makes it easy to send and receive messages from themain
function (i.e theapp
worker).
Create an empty file /examples/03-routing-many.hops.rs
(note this is in the examples/
folder and
not src/
folder like the workers above).
use colored::Colorize;
use hello_ockam::{Echoer, Hopper};
use ockam::{node, route, Context, Result};
#[rustfmt::skip]
const HELP_TEXT: &str =r#"
βββββββββββββββββββββββββ
β Node 1 β
βββββββββββββββββββββββββ€
β ββββββββββββββββββ β
β β Address: β β
β β 'app' β β
β βββ¬βββββββββββββ²ββ β
β βββΌβββββββββββββ΄ββ β
β β Address: β β
β β 'hopper1..3' βx3 β
β βββ¬βββββββββββββ²ββ β
β βββΌβββββββββββββ΄ββ β
β β Address: β β
β β 'echoer' β β
β ββββββββββββββββββ β
βββββββββββββββββββββββββ
"#;
/// This node routes a message through many hops.
#[ockam::node]
async fn main(ctx: Context) -> Result<()> {
println!("{}", HELP_TEXT.green());
print_title(vec![
"Run a node w/ 'app', 'echoer' and 'hopper1', 'hopper2', 'hopper3' workers",
"then send a message over 3 hops",
"finally stop the node",
]);
// Create a node with default implementations.
let mut node = node(ctx);
// Start an Echoer worker at address "echoer".
node.start_worker("echoer", Echoer).await?;
// Start 3 hop workers at addresses "hopper1", "hopper2" and "hopper3".
node.start_worker("hopper1", Hopper).await?;
node.start_worker("hopper2", Hopper).await?;
node.start_worker("hopper3", Hopper).await?;
// Send a message to the echoer worker via the "hopper1", "hopper2", and "hopper3" workers.
let route = route!["hopper1", "hopper2", "hopper3", "echoer"];
let route_msg = format!("{:?}", route);
let msg = "Hello Ockam!";
node.send(route, msg.to_string()).await?;
// Wait to receive a reply and print it.
let reply = node.receive::<String>().await?;
// Formatting stdout output.
let lines = [
"π Node 1 β".to_string(),
format!(" sending: {}", msg.green()),
format!(" over route: {}", route_msg.blue()),
format!(" and receiving: '{}'", reply.purple()), // Should print "π echo back: Hello Ockam!"
format!(" then {}", "stopping".bold().red()),
];
lines
.iter()
.for_each(|line| println!("{}", line.black().on_white()));
// Stop all workers, stop the node, cleanup and return.
node.stop().await
}
fn print_title(title: Vec<&str>) {
let line = format!("π {}", title.join("\n β ").white());
println!("{}", line.black().on_bright_black())
}
Now it is time to run our program to see what it does! π
In your terminal app, run the following command. Note that OCKAM_LOG=none
is used to disable
logging output from the Ockam library. This is done to make the output of the example easier to
read.
OCKAM_LOG=none cargo run --example 03-routing-many-hops
And you should see something like the following. Our example program creates multiple hop workers
(three hopper
workers) between the app
and the echoer
and route our message through them π.
π‘ This example continues from the simple example above, we are going to reuse all the dependencies and workers in this example so please make sure to complete the simple example before working on this one.
In this example, we will introduce TCP transports in between the hops. Instead of passing messages around between workers in the same node, we will spawn multiple nodes. Then we will have a few TCP transports (TCP socket client and listener combos) that will connect the nodes.
An Ockam transport is a plugin for Ockam Routing. It moves Ockam Routing messages using a specific transport protocol like TCP, UDP, WebSockets, Bluetooth, etc.
We will have three nodes:
node_initiator
: The first node initiates sending the message over TCP to the middle node (port
3000
).node_middle
: Then middle node simply forwards this message on to the last node over TCP again
(port 4000
this time).node_responder
: And finally the responder node receives the message and sends a reply back to
the initiator node.The following diagram depicts what we will build next. In this example all these nodes are on the same machine, but they can easy just be nodes on different machines.
ββββββββββββββββββββββββ
βnode_initiator β
ββββββββββββββββββββββββ€
β ββββββββββββββββββββ β
β βAddress: β β βββββββββββββββββββββββββββββ
β β'app' β β βnode_middle β
β ββββ¬βββββββββββββ²βββ β βββββββββββββββββββββββββββββ€
β ββββΌβββββββββββββ΄βββ β β ββββββββββββββββββββ β
β βTCP transport βββΌββββββΌββΊTCP transport β β
β βconnect to 3000 βββΌββββββΌββlistening on 3000 β β
β ββββββββββββββββββββ β β ββββ¬βββββββββββββ²βββ β
ββββββββββββββββββββββββ β ββββΌβββββββββββββ΄ββββββββ β
β βAddress: β β ββββββββββββββββββββββββ
β β'forward_to_responder' β β βnode_responder β
β ββββ¬βββββββββββββ²ββββββββ β ββββββββββββββββββββββββ€
β ββββΌβββββββββββββ΄βββ β β ββββββββββββββββββββ β
β βTCP transport ββββββββΌββββΌββΊTCP transport β β
β βconnect to 4000 ββββββββΌββββΌββlistening on 4000 β β
β ββββββββββββββββββββ β β ββββ¬βββββββββββββ²βββ β
βββββββββββββββββββββββββββββ β ββββΌβββββββββββββ΄βββ β
β βAddress: β β
β β'echoer' β β
β ββββββββββββββββββββ β
ββββββββββββββββββββββββ
Letβs start by creating a new file /examples/04-routing-over-two-transport-hops.rs
(in the
/examples/
folder and not /src/
folder). Then copy / paste the following code in that file.
use colored::Colorize;
use hello_ockam::{Echoer, Forwarder};
use ockam::{
node, route, AsyncTryClone, Context, Result, TcpConnectionOptions, TcpListenerOptions,
TcpTransportExtension,
};
#[rustfmt::skip]
const HELP_TEXT: &str =r#"
ββββββββββββββββββββββββ
βnode_initiator β
ββββββββββββββββββββββββ€
β ββββββββββββββββββββ β
β βAddress: β β βββββββββββββββββββββββββββββ
β β'app' β β βnode_middle β
β ββββ¬βββββββββββββ²βββ β βββββββββββββββββββββββββββββ€
β ββββΌβββββββββββββ΄βββ β β ββββββββββββββββββββ β
β βTCP transport βββΌββββββΌββΊTCP transport β β
β βconnect to 3000 βββΌββββββΌββlistening on 3000 β β
β ββββββββββββββββββββ β β ββββ¬βββββββββββββ²βββ β
ββββββββββββββββββββββββ β ββββΌβββββββββββββ΄ββββββββ β
β βAddress: β β ββββββββββββββββββββββββ
β β'forward_to_responder' β β βnode_responder β
β ββββ¬βββββββββββββ²ββββββββ β ββββββββββββββββββββββββ€
β ββββΌβββββββββββββ΄βββ β β ββββββββββββββββββββ β
β βTCP transport ββββββββΌββββΌββΊTCP transport β β
β βconnect to 4000 ββββββββΌββββΌββlistening on 4000 β β
β ββββββββββββββββββββ β β ββββ¬βββββββββββββ²βββ β
βββββββββββββββββββββββββββββ β ββββΌβββββββββββββ΄βββ β
β βAddress: β β
β β'echoer' β β
β ββββββββββββββββββββ β
ββββββββββββββββββββββββ
"#;
#[ockam::node]
async fn main(ctx: Context) -> Result<()> {
println!("{}", HELP_TEXT.green());
let ctx_clone = ctx.async_try_clone().await?;
let ctx_clone_2 = ctx.async_try_clone().await?;
let mut node_responder = create_responder_node(ctx).await.unwrap();
let mut node_middle = create_middle_node(ctx_clone).await.unwrap();
create_initiator_node(ctx_clone_2).await.unwrap();
node_responder.stop().await.ok();
node_middle.stop().await.ok();
println!(
"{}",
"App finished, stopping node_responder & node_middle".red()
);
Ok(())
}
fn print_title(title: Vec<&str>) {
let line = format!("π {}", title.join("\n β ").white());
println!("{}", line.black().on_bright_black())
}
This code wonβt actually compile, since there are 3 functions missing from this source file. We are just adding this file first in order to stage the rest of the code we will write next.
This main()
function creates the three nodes like we see in the diagram above, and it also stops
them after the example is done running.
So letβs write the function that creates the initiator node first. Copy the following into the
source file we created earlier (/examples/04-routing-over-two-transport-hops.rs
), and paste it
below the existing code there:
/// This node routes a message, to a worker on a different node, over two TCP transport
/// hops.
async fn create_initiator_node(ctx: Context) -> Result<()> {
print_title(vec![
"Create node_initiator that routes a message, over 2 TCP transport hops, to 'echoer' worker on node_responder",
"stop",
]);
// Create a node with default implementations.
let mut node = node(ctx);
// Initialize the TCP transport.
let tcp_transport = node.create_tcp_transport().await?;
// Create a TCP connection to the middle node.
let connection_to_middle_node = tcp_transport
.connect("localhost:3000", TcpConnectionOptions::new())
.await?;
// Send a message to the "echoer" worker, on a different node, over two TCP hops. Wait
// to receive a reply and print it.
let route = route![connection_to_middle_node, "forward_to_responder", "echoer"];
let route_str = format!("{:?}", route);
let msg = "Hello Ockam!";
let reply = node
.send_and_receive::<String>(route, msg.to_string())
.await?;
// Formatting stdout output.
let lines = [
"π node_initiator β".to_string(),
format!(" sending: {}", msg.green()),
format!(" over route: '{}'", route_str.blue()),
format!(" and received: '{}'", reply.purple()), // Should print "π echo back: Hello Ockam!"
format!(" then {}", "stopping".bold().red()),
];
lines
.iter()
.for_each(|line| println!("{}", line.black().on_white()));
// Stop all workers, stop the node, cleanup and return.
node.stop().await
}
This (initiator) node will send a message to the responder using the following route.
let route = route![connection_to_middle_node, "forward_to_responder", "echoer"];
π‘ Note the use of a mix of TCP transport routes as well as addresses for other workers. Also note that this node does not have to be aware of the full topology of the network of nodes. It just knows that it has to jump over the TCP transport
connection_to_middle_node
and then have its message routed toforward_to_responder
address followed byechoer
address.
Letβs create the middle node next, which will run the worker Forwarder
on this address:
forward_to_responder
.
Copy and paste the following into the source file we created above
(/examples/04-routing-over-two-transport-hops.rs
).
3000
) to port 4000
.Forwarder
worker on address forward_to_responder
, so thatβs how the initiator
can reach this address specified in its route at the start of this example./// - Starts a TCP listener at 127.0.0.1:3000.
/// - This node creates a TCP connection to a node at 127.0.0.1:4000.
/// - Starts a forwarder worker to forward messages to 127.0.0.1:4000.
/// - Then runs forever waiting to route messages.
async fn create_middle_node(ctx: Context) -> Result<ockam::Node> {
print_title(vec![
"Create node_middle that listens on 3000 and forwards to 4000",
"wait for messages until stopped",
]);
// Create a node with default implementations.
let node = node(ctx);
// Initialize the TCP transport.
let tcp_transport = node.create_tcp_transport().await?;
// Create a TCP connection to the responder node.
let connection_to_responder = tcp_transport
.connect("127.0.0.1:4000", TcpConnectionOptions::new())
.await?;
// Create a Forwarder worker.
node.start_worker(
"forward_to_responder",
Forwarder {
address: connection_to_responder.into(),
},
)
.await?;
// Create a TCP listener and wait for incoming connections.
let listener = tcp_transport
.listen("127.0.0.1:3000", TcpListenerOptions::new())
.await?;
// Allow access to the Forwarder via TCP connections from the TCP listener.
node.flow_controls()
.add_consumer("forward_to_responder", listener.flow_control_id());
// Don't call node.stop() here so this node runs forever.
Ok(node)
}
Finally, we will create the responder node. This node will run the worker echoer
which actually
echoes the message back to the initiator. Copy and paste the following into the source file above
(/examples/04-routing-over-two-transport-hops.rs
).
Echoer
worker on address echoer
, so thatβs how the initiator can reach this
address specified in its route at the start of this example./// This node starts a TCP listener and an echoer worker. It then runs forever waiting for
/// messages.
async fn create_responder_node(ctx: Context) -> Result<ockam::Node> {
print_title(vec![
"Create node_responder that runs tcp listener on 4000 and 'echoer' worker",
"wait for messages until stopped",
]);
// Create a node with default implementations.
let node = node(ctx);
// Initialize the TCP transport.
let tcp_transport = node.create_tcp_transport().await?;
// Create an echoer worker.
node.start_worker("echoer", Echoer).await?;
// Create a TCP listener and wait for incoming connections.
let listener = tcp_transport
.listen("127.0.0.1:4000", TcpListenerOptions::new())
.await?;
// Allow access to the Echoer via TCP connections from the TCP listener.
node.flow_controls()
.add_consumer("echoer", listener.flow_control_id());
Ok(node)
}
Letβs run this example to see what it does π.
In your terminal app, run the following command. Note that OCKAM_LOG=none
is used to disable
logging output from the Ockam library. This is done to make the output of the example easier to
read.
cargo run --example 04-routing-over-two-transport-hops
This should produce output similar to the following. Our example program creates a route that
traverses multiple nodes and TCP transports from the app
to the echoer
and routes our message
through them π.
Ockam Routing and transports are extremely powerful and flexible. They are one of the key features that enables Ockam Secure Channels to be implemented. By layering Ockam Secure Channels and other protocols over Ockam Routing, we can provide end-to-end guarantees over arbitrary transport topologies that span many networks and clouds.
π Please star and fork / clone the Ockam repo π.
In a future blog post we will be covering Ockam Secure Channels and how they can be used to provide end-to-end guarantees over arbitrary transport topologies. So stay tuned!
In the meantime here are some good jumping off points to learn more about Ockam:
ockam
command) on your computer and try to
create end-to-end encrypted communication
between two apps. That will give you a taste of the experience of using Ockam on the command line
in addition to our Rust library.This tutorial is a guide to parsing with nom. It covers the basics of parsing and how to use nom to parse a string into a data structure. We will cover a variety of different examples ranging from parsing simple CSS like syntax to a full blown Markdown parser.
This tutorial has 2 examples in it:
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
For more information on general Rust type system design (functional approach rather than object oriented), please take a look at this paper by Will Crichton demonstrating Typed Design Patterns with Rust.
nom is a huge topic. This tutorial takes a hands on approach to learning nom. However, the resources listed below are very useful for learning nom. Think of them as a reference guide and deep dive into how the nom library works.
nom is a parser combinator library for Rust. You can write small functions that parse a specific part of your input, and then combine them to build a parser that parses the whole input. nom is very efficient and fast, it does not allocate memory when parsing if it doesnβt have to, and it makes it very easy for you to do the same. nom uses streaming mode or complete mode, and in this tutorial & code examples provided we will be using complete mode.
Roughly the way it works is that you tell nom how to parse a bunch of bytes in a way that matches some pattern that is valid for your data. It will try to parse as much as it can from the input, and the rest of the input will be returned to you.
You express the pattern that youβre looking for by combining parsers. nom has a whole bunch of these that come out of the box. And a huge part of learning nom is figuring out what these built in parsers are and how to combine them to build a parser that does what you want.
Errors are a key part of it being able to apply a variety of different parsers to the same input. If a parser fails, nom will return an error, and the rest of the input will be returned to you. This allows you to combine parsers in a way that you can try to parse a bunch of different things, and if one of them fails, you can try the next one. This is very useful when you are trying to parse a bunch of different things, and you donβt know which one you are going to get.
Letβs dive into nom using a simple example of parsing hex color codes.
//! This module contains a parser that parses a hex color string into a [Color] struct.
//! The hex color string can be in the following format `#RRGGBB`.
//! For example, `#FF0000` is red.
use std::num::ParseIntError;
use nom::{bytes::complete::*, combinator::*, error::*, sequence::*, IResult, Parser};
#[derive(Debug, PartialEq)]
pub struct Color {
pub red: u8,
pub green: u8,
pub blue: u8,
}
impl Color {
pub fn new(red: u8, green: u8, blue: u8) -> Self {
Self { red, green, blue }
}
}
/// Helper functions to match and parse hex digits. These are not [Parser]
/// implementations.
mod helper_fns {
use super::*;
/// This function is used by [map_res] and it returns a [Result], not [IResult].
pub fn parse_str_to_hex_num(input: &str) -> Result<u8, std::num::ParseIntError> {
u8::from_str_radix(input, 16)
}
/// This function is used by [take_while_m_n] and as long as it returns `true`
/// items will be taken from the input.
pub fn match_is_hex_digit(c: char) -> bool {
c.is_ascii_hexdigit()
}
pub fn parse_hex_seg(input: &str) -> IResult<&str, u8> {
map_res(
take_while_m_n(2, 2, match_is_hex_digit),
parse_str_to_hex_num,
)(input)
}
}
/// These are [Parser] implementations that are used by [hex_color_no_alpha].
mod intermediate_parsers {
use super::*;
/// Call this to return function that implements the [Parser] trait.
pub fn gen_hex_seg_parser_fn<'input, E>() -> impl Parser<&'input str, u8, E>
where
E: FromExternalError<&'input str, ParseIntError> + ParseError<&'input str>,
{
map_res(
take_while_m_n(2, 2, helper_fns::match_is_hex_digit),
helper_fns::parse_str_to_hex_num,
)
}
}
/// This is the "main" function that is called by the tests.
fn hex_color_no_alpha(input: &str) -> IResult<&str, Color> {
// This tuple contains 3 ways to do the same thing.
let it = (
helper_fns::parse_hex_seg,
intermediate_parsers::gen_hex_seg_parser_fn(),
map_res(
take_while_m_n(2, 2, helper_fns::match_is_hex_digit),
helper_fns::parse_str_to_hex_num,
),
);
let (input, _) = tag("#")(input)?;
let (input, (red, green, blue)) = tuple(it)(input)?; // same as `it.parse(input)?`
Ok((input, Color { red, green, blue }))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_valid_color() {
let mut input = String::new();
input.push_str("#2F14DF");
input.push('π
');
let result = dbg!(hex_color_no_alpha(&input));
let Ok((remainder, color)) = result else { panic!(); };
assert_eq!(remainder, "π
");
assert_eq!(color, Color::new(47, 20, 223));
}
#[test]
fn parse_invalid_color() {
let result = dbg!(hex_color_no_alpha("π
#2F14DF"));
assert!(result.is_err());
}
}
Please note that:
#2F14DFπ
.π
#2F14DF
.So what is going on in the source code above?
The intermediate_parsers::hex_color_no_alpha()
function is the main function that orchestrates
all the other functions to parse an input: &str
and turn it into a (&str, Color)
.
tag
combinator function is used to match the #
character. This means that if the input
doesnβt start with #
, the parser will fail (which is why π
#2F14DF
fails). It returns the
remaining input after #
. And the output is #
which we throw away.tuple
is created that takes 3 parsers, which all do the same exact thing, but are written
in 3 different ways just to demonstrate how these can be written.
helper_fns::parse_hex_seg()
function is added to a tuple.intermediate_parsers::gen_hex_seg_parser_fn()
is added to the
tuple.map_res
combinator is directly added to the tuple.parse()
is called w/ the input
(thus far). This
is used to parse the input hex number.
#2F14DFπ
returns π
as
the first item in the tuple.Color
struct.Letβs look at the helper_fns::parse_hex_seg
(the other 2 ways shown above do the same exact
thing). The signature of this function tells nom that you can call the function w/ input
argument and it will return IResult<Input, Output, Error>
. This signature is the pattern that
we will end up using to figure out how to chain combinators together. Hereβs how the map_res
combinator is used by parse_hex_seg()
to actually do the parsing:
take_while_m_n
: This combinator takes a range of characters (2, 2
) and applies the
function match_is_hex_digit
to determine whether the char
is a hex digit (using
is_ascii_hexdigit()
on the char
). This is used to match a valid hex digit. It returns a
&str
slice of the matched characters. Which is then passed to the next combinator.parse_str_to_hex_num
: This parser is used on the string slice returned from above. It simply
takes string slice and turns it into a Result<u8>, std::num::ParseIntError>
. The error is
important, since if the string slice is not a valid hex digit, then we want to return this
error.The key concept in nom is the Parser
trait which is implemented for any FnMut
that accepts an
input and returns an IResult<Input, Output, Error>
.
fn(input: Input) -> IResult<Input, Output, Error>
then you are good to go! You just need to
call parse()
on the Input
type and this will kick off the parsing.tuple
function directly via
nom::sequence::tuple(...)(input)?
. Or you can just call the parse()
method on the tuple
since this is an extension function on tuples provided by nom.IResult
is a very important type alias. It encapsulates 3 key types that are related to
parsing:
Input
type is the type of the input that is being parsed. For example, if you are
parsing a string, then the Input
type is &str
.Output
type is the type of the output that is returned by the parser. For example, if
you are parsing a string and you want to return a Color
struct, then the Output
type is
Color
.Error
type is the type of the error that is returned by the parser. For example, if
you are parsing a string and you want to return a nom::Err::Error
error, then the Error
type is nom::Err::Error
. This is very useful when you are developing your parser
combinators and you run into errors and have to debug them.After the really complicated walk through above, we could have just written the entire thing concisely like so:
pub fn parse_hex_seg(input: &str) -> IResult<&str, u8> {
map_res(
take_while_m_n(2, 2, |it: char| it.is_ascii_hexdigit()),
|it: &str| u8::from_str_radix(it, 16),
)(input)
}
fn hex_color_no_alpha(input: &str) -> IResult<&str, Color> {
let (input, _) = tag("#")(input)?;
let (input, (red, green, blue)) = tuple((
helper_fns::parse_hex_seg,
helper_fns::parse_hex_seg,
helper_fns::parse_hex_seg,
))(input)?;
Ok((input, Color { red, green, blue }))
}
This is a very simple example, but it shows how you can combine parsers together to create more complex parsers. You start w/ the simplest one first, and then build up from there.
parse_hex_seg()
which is used to parse a single hex segment.
Inside this function we call map_res()
w/ the supplied input
and simply return the result.
This is also a very common thing to do, is to wrap calls to other parsers in functions and then
re-use them in other parsers.hex_color_no_alpha()
function is used to parse a hex color w/o an alpha channel.
tag()
combinator is used to match the #
character.tuple()
combinator is used to match the 3 hex segments.?
operator is used to return the error if there is one.Ok()
is used to return the parsed Color
struct and the remaining input.π‘ You can get the source code for the Markdown parser shown in this article from the
r3bl-open-core
repo.π Please star this repo on github if you like it π.
The md_parser
module
in the r3bl-open-core
repo contains a fully functional Markdown parser (and isnβt written as a test
but a real module that you can use in your projects that need a Markdown parser). This parser
supports standard Markdown syntax as well as some extensions that are added to make it work w/ R3BL
products. It makes a great starting point to study how a relatively complex parser is written. There
are lots of tests that you can follow along to understand what the code is doing.
Here are some entry points into the codebase.
The main function
parse_markdown()
that does the parsing of a string slice into a
Document
.
The tests are provided alongside the code itself. And you can follow along to see how other
smaller parsers are used to build up this big one that parses the whole of the Markdown document.
parser_impl_metadata
.parser_impl_block
parser_impl_element
The types
that are used to represent the Markdown document model
(Document
)
and all the other intermediate types (Fragment
, Block
, etc) & enums required for parsing.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
This tutorial is part of a collection of tutorials on basic data structures and algorithms that are created using TypeScript. Information is depicted visually using diagrams and code snippets. This article may be useful if you are trying to get more fluency in TypeScript or need a refresher to do interview prep for software engineering roles.
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
The source for this tutorial can be found on GitHub. Please clone it to your computer.
In your terminal, go to the root directory of the cloned repo and run the following command
npm install
to install dependencies.
All the tests are in the main.test.ts
file. To run jest tests continuously, run
npx jest --watchAll
.
Design and implement a class that can represent a single HTML element and its children. The class should have the following methods for tree construction and pretty printing:
addClass
- takes a class name and adds it to the elementβs list of classesappendChild
- takes an HTML element and adds it to the elementβs list of childrenprintTree
- returns a string representation of the element and its children in HTML formatThe class should also have the following methods for tree traversal:
findFirstChildBFS
- a method that takes an array of two elements (tuple). The first element is a
parent element selector to start searching the child selector. The second element is the child
selector. If we find the first child element that has the selector, we return the child element.
The BFS stands for breadth-first search. This simply defines the order we traverse the tree.findFirstChildDFS
- the same as the findFirstChildBFS
method, but the DFS stands for
depth-first search. We traverse the tree using a depth-first search algorithm. If you are not
familiar with the difference between BFS and DFS, I have added two images below this section.findDescendantBFS
- this method also takes a tuple, except the first element is the ancestor
selector and the second element is the descendant selector. We traverse the tree using a
breadth-first search algorithm. If we find the first descendant element that has the selector, we
return the descendant element. We traverse the tree using BFS.findDescendantDFS
- same as the findDescendantBFS
method, except we traverse the tree using
DFS.The following image shows the difference between finding the first child vs finding a descendant element.
The following image shows the depth-first search (HTML tree)
The following image shows the depth-first search
The following image shows the breadth-first search
Letβs walk through our thought process on the journey to solving the problem statement.
Letβs start w/ the output first. Our program should produce the following output (which is a string):
"
<html>
<head></head>
<body>
<h1 class="blue-theme"></h1>
<ul class="blue-theme bold-text">
<li></li>
<li></li>
</ul>
</body>
</html>
"
Data is all the pieces of information that we need to represent the element. For example each element needs:
MyElement
as a type.<ul class="blue-theme bold-text">
<!-- Tag name is "ul". It is an element -->
<!-- with two classes and two child elements ("li") -->
<li>item 1</li>
//
<!-- child 1: "li" is an element and a child of "ul" -->
<li>item 2</li>
//
<!-- child 2 (same as the above) -->
</ul>
class MyElement implements MyElementInterface t{
tagName: string
private _children: MyElement[] = []
private _classList: string[] = []
// For formatting, we need a "depth" property. For each when adding a child to a parent,
// we need to increment the dept of the child by 1. This is how we will know how many
// spaces to add before the child element.
private _depth: number = 0
}
Note: when using classes, one doesnβt have to create an interface. I like to do that, so itβs easy to read which properties and methods are public and which are private.
MyElement
class using a constructor function:
new Element(element: string) => Element
To pretty print the HTML tree, we need to add a certain amount of spaces in front of each element. To know how many spaces, we need to track the depth of each element.
Every time a child element is nested inside the parent element (via the appendChild
method), we
take the parent elementβs depth and increment the child elementβs depth by one.
E.g. when we pretty print the HTML tree, each element will have depth * 2 spaces in front of it. This means the first level element will have 0 spaces in front of it, the second level element will have 2 spaces in front of it, the third level element will have 4 spaces in front of it, etc.
appendChild = (child: MyElement) => {
// Increment the dept of the child by taking the current
// element's depth and adding 1 to it.
child.depth = this.depth + 1
// We are also tracking the parent element of each element.
// This will help us to traverse to the root element.
child._parentElement = this
this._children.push(child)
return this
}
Method to print the HTML tree:
printTree = (): string => {
let spaces = " ".repeat(this.depth * 2)
const elementHasChildren = this._children.length > 0
const hasClasses = this._classList.size > 0
const classes = hasClasses ? ` class="${[...this._classList].join(" ")}"` : ""
let string: string = ""
if (elementHasChildren) {
const children = this._children.map((child) => child.printTree()).join("")
// Print start and end tags to different lines.
string = `${spaces}<${this.tagName}${classes}>\n${children}${spaces}</${this.tagName}>\n`
} else {
// Print the start and end tags to the same line.
string = `${spaces}<${this.tagName}${classes}></${this.tagName}>\n`
}
return string
}
Implement
findFirstChildBFS
findFirstChildDFS
findDescendantBFS
findDescendantDFS
Check out
main.ts
for
how to implement those methods.
All BFS methods use a queue data structure. In JavaScript queue is an array where we add items, one after another, and then remove them from the start of the array. Itβs called first-in-first-out (FIFO). You could think of it as a line of people waiting in the checkout line at the grocery store. The first person in line is the first person to be served.
All DFS methods use stack data structure. In JavaScript stack is an array where we add items one after another and then remove them from the end of the array. Itβs called last-in-first-out (LIFO). You could think of it as a stack of papers. The last paper you put on the stack is the first paper you take off the stack.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
This tutorial is part of a collection of tutorials on basic data structures and algorithms that are created using TypeScript. Information is depicted visually using diagrams and code snippets. This article may be useful if you are trying to get more fluency in TypeScript or need a refresher to do interview prep for software engineering roles.
This tutorial has 2 examples in it:
π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.
You can get the code for this and all the other tutorials in this collection from this github repo. Once you clone this repo locally, you can run the following:
cd ts-string-tokenizer
npm install
npx jest --watchAll
All the examples in this repo are written in the form of tests and you can see them running in there. If you change the code while the tests are running, the tests will re-run automatically w/ these new changes. This is nice to have when you are learning how the code works.
Create a function that mimics the
cd
command in the terminal.
function cd(currentPath: string, action: string): string
The function takes two argument, first argument is a current path and the second argument is an action to apply to the current path to change the path. The function modifies the path and returns the resulting path as a string.
| Current path | Action | Resulting path |
| -------------------- | ------------------ | ------------------------------------ |
| / | /folder | /folder |
| /folder | nestedFolder | /folder/nestedFolder |
| /folder | /home | /home |
| /folder/nestedFolder | .. | /folder |
| /folder/nestedFolder | ../folder2/folder3 | /folder/folder2/folder3 |
| /folder/nestedFolder | folder2/./folder3 | /folder/nestedFolder/folder2/folder3 |
Our currentPath
is a string and we need to manipulate this string and eventually return a new
string.
In order for us to manipulate a string easily, we chunk the string into smaller strings and store those smaller string chunks into an array.
"/home/user/folder"
would be turned into an array of
["home", "user", "folder"]
<- Data modelling.We have a string currentPath
that we know we need to manipulate. But how? Our second argument
is a action
. action
is a string that tells us how to manipulate our currentPath
.
action
string command: For example action
can look like this:
"../anotherFolder"
. This means we want to go up one folder (..) and then go into folder
called anotherFolder.
In order for us to understand what this action
string means, we need to break it down into
smaller strings and store those smaller strings in an array as well."../anotherFolder"
would be turned into an array of ["..", "anotherFolder"]
."."
, then ignore it and go to the next element in the array."/"
then this means that whatever comes after the "/"
is the
absolute path and the string output we want to return.action
into a smaller strings, we
need to think about how to treat the "/"
character, since we need to account for it (it tells
us the new path is an absolute path)."/"
, then donβt chunk the string into an
array and just return the string as output string, since itβs an absolute path.Now we have action
string array. In order to apply the action
to our currentPath
, we need
to go through each array element, check what it means and then manipulate our currentPath
accordingly. We need to define all of our action
cases:
"/"
, then we donβt chunk the string into an array and just return
the string as output string, since itβs an absolute path.".."
means go up one directory, which in terms or data structures means, remove the last
element of our currentPathArray
."anotherFolder"
means go into another folder, which in terms of data structures means, add
the name of the anotherFolder to the end of our currentPathArray
."."
means do nothing, which in terms of data structures means just go to the next element in
the array.Chunking up currentPath
and action
strings to arrays:
split()
method and pass in a delimiter "/"
.
shift()
(modifies the array, returns the removed element) or splice(0,1)
(0 is
start index, 1 means how many elements we need to remove from the start index. Splice
modifies an array, returns removed element).currentString
variable, when encounter "/"
character, then push the currentString
to
currentPathArray
and reset the currentString
to empty string.Manipulating currentPathArray
:
cdActionArray
, we need to loop over it and for each element we check our use
cases (see number 3) and manipulate our currentPathArray
accordingly.Done!
Implement a in-memory rate limiter that limits the number of tokens that can be produced in a given time period.
function acceptOrDenyRequest(
maxRequests: number,
timeWindowInMs: number,
successfulRequests: number[],
newRequestTimestamp: number
): "accept" | "deny"
| Request ## | Timestamp | Result |
| ---------- | --------- | ------ |
| 1 | 1100 | accept |
| 2 | 1200 | accept |
| 3 | 1300 | deny |
| 4 | 2100 | accept |
| 5 | 2150 | deny |
| 6 | 2200 | accept |
We have built and API and made itβs endpoint public. This means that in theory we could have millions of requests coming in to our API endpoint. This could be costly or we could have a malicious code that could potentially crash our server. We need to protect our API endpoint from malicious code and also from too many requests coming in at the same time. We need to implement a rate limiter that limits the number of tokens (requests) that can be produced in a given time period.
We are going to create two functions: outer and inner function. Outer function defines all the variables and data structures we need to keep track of and the inner function accepts all those variables as arguments and checks is there space left in the time window to allow another request in or not.
Things we need to keep track of in our outer function:
successfulRequests
to start tracking the successful request timestamps.maxRequests
.timeWindowInMs
.newRequestTimestamp
. This is the time window endpoint. We use this timestamp to calculate the
window start point to check if there are less than max requests currently in the time window.Now we are going to call a function acceptOrDenyRequest
which will accept or deny the request
to hit the API endpoint. This function will take in the arguments we just defined:
maxRequests: number,
timeWindowInMs: number,
successfulRequests: number[],
newRequestTimestamp: number)
We need to keep track of request count in the time window. We create a requestCount
variable.
When the request count is equal to the max requests, we need to deny the request since the time
window is full.
We will loop through the successfulRequests
array from back to front. We will check if the
previously successful timestamp is still within the time window. If it is, we will increment
the request count. If it is not, we will stop the loop since we know we have reached outside of
the window bounds.
Now that we know how many requests were successful in the current time window, we can compare successful requests number with the max requests allowed in the time window. If successful requests number is less than max request allowed, then we can accept the request. If successful requests number is equal to max requests allowed, then we need to deny the request since the time window is full with successful requests already.
In the outer function we check if the request was successful or not. It the request was
successful, we push the request timestamp to successfulRequests
array and return βacceptβ. If
the request was not successful, we return βdenyβ.
]]>π Please star the r3bl-open-core repo. You can get lots of useful Rust command line apps from the
r3bl-cmdr
crate in this repo:π¦ You can install them using
- π±
giti
: run interactive git commands with confidence in your terminal- π¦
edi
: edit Markdown with style in your terminalcargo install r3bl-cmdr
giti in action edi in actionIf you would like to get involved in an open source project and like Rust crates, we welcome your contributions to the r3bl-open-core repo.