Network Pipeline Scripts

Capture Agents

Here are the AntiNex Network Pipeline Capture Agents. These scripts allow for capturing traffic on a network device and flattening it into JSON dictionaries before publishing to the aggregation message broker. Please refer to the handle_packets method for more details.

Warning

These tools will capture network traffic. Please be careful where you deploy them.

ARP

network_pipeline.scripts.capture_arp.capture_arp_packets()

Capture ARP packets and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

ICMP

network_pipeline.scripts.capture_icmp.capture_icmp_packets()

Capture ICMP packets and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

TCP

network_pipeline.scripts.capture_ssh.capture_tcp_packets_over_ssh()

Capture TCP packets over ssh and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

network_pipeline.scripts.capture_tcp.capture_tcp_packets()

Capture TCP packets and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

network_pipeline.scripts.capture_telnet.capture_tcp_packets_over_telnet()

Capture TCP packets over telnet and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

UDP

network_pipeline.scripts.capture_udp.capture_udp_packets()

Capture UDP packets and call the handle_packets method

Change the network interface by export CAP_DEVICE=eth0

Publishers

These tools are designed to show how to save captured packet dictionaries to CSVs and how to publish them for live predictions using a pre-trained Deep Neural Network.

network_pipeline.scripts.packets_rabbitmq.recv_msg(body, message)

Handler method - fires when a messages is consumed from the FORWARD_QUEUE queue running in the FORWARD_BROKER_URL broker.

Parameters:

• body – message body
• message – message object can ack, requeue or reject

network_pipeline.scripts.packets_rabbitmq.consume_network_packet_messages_from_rabbitmq()

Setup a celery_connectors.KombuSubscriber to consume meessages from the FORWARD_BROKER_URL broker in the FORWARD_QUEUE queue.

network_pipeline.scripts.packets_redis.recv_msg(body, message)

Handler method - fires when a messages is consumed from the FORWARD_QUEUE queue running in the FORWARD_BROKER_URL broker.

Parameters:

• body – message body
• message – message object can ack, requeue or reject

network_pipeline.scripts.packets_redis.consume_network_packet_messages_from_redis()

Setup a celery_connectors.KombuSubscriber to consume meessages from the FORWARD_BROKER_URL broker in the FORWARD_QUEUE queue.

Test Tools

These will send mock traffic data to the targeted network device.

network_pipeline.scripts.base_capture.example_capture()

An example capture script

Change the network interface by export CAP_DEVICE=eth0

network_pipeline.scripts.arp_send_msg.send_arp_msg()

Send an ARP message to the network device (enp0s3 by default).

network_pipeline.scripts.tcp_send_large_msg.send_tcp_large_message()

Send a large TCP message to port 80 by default.

network_pipeline.scripts.tcp_send_msg.send_tcp_message()

Send a TCP message to port 80 by default.

network_pipeline.scripts.udp_send_msg.send_udp_message()

Send a UDP message to port 80 by default.

Environment variables:

UDP_SEND_TO_HOST - host ip address UDP_SEND_TO_PORT - send to this UDP port

network_pipeline.scripts.listen_tcp_port.listen_on_tcp_port()

Run a simple server for processing messages over TCP.

LISTEN_ON_HOST - listen on this host ip address

LISTEN_ON_PORT - listen on this TCP port

LISTEN_SIZE - listen on to packets of this size

LISTEN_SLEEP - sleep this number of seconds per loop

LISTEN_SHUTDOWN_HOOK - shutdown if file is found on disk

network_pipeline.scripts.listen_udp_port.listen_on_udp_port()

Run a simple server for processing messages over UDP.

UDP_LISTEN_ON_HOST - listen on this host ip address

UDP_LISTEN_ON_PORT - listen on this UDP port

UDP_LISTEN_SIZE - listen on to packets of this size

UDP_LISTEN_SLEEP - sleep this number of seconds per loop

UDP_LISTEN_SHUTDOWN_HOOK - shutdown if file is found on disk

network_pipeline.scripts.builders.prepare_dataset.find_all_headers(pipeline_files=[], label_rules=None)

Parameters:

• pipeline_files – files to process
• label_rules – labeling rules

network_pipeline.scripts.builders.prepare_dataset.build_csv(pipeline_files=[], fulldata_file=None, clean_file=None, post_proc_rules=None, label_rules=None, metadata_filename='metadata.json')

Parameters:

• pipeline_files – files to process
• fulldata_file – output all columns to this csv file
• clean_file – output all numeric-ready columns to this csv file
• post_proc_rules – rules after building the DataFrame
• label_rules – labeling rules
• metadata_filename – metadata

network_pipeline.scripts.builders.prepare_dataset.find_all_pipeline_csvs(csv_glob_path='/opt/antinex/datasets/**/*.csv')

Parameters:csv_glob_path – path to csvs

network_pipeline.scripts.builders.prepare_dataset.prepare_new_dataset()

class network_pipeline.scripts.tools.arp_send_msg.Ethernet

Generic Ethernet Frame class

class network_pipeline.scripts.tools.arp_send_msg.Arp

Generic ARP Frame class