components.md

Components

Malcolm leverages the following excellent open source tools, among others:

• Arkime (formerly Moloch) - for PCAP file processing, browsing, searching, analysis, and carving/exporting. Arkime consists of two main components:
  • capture - a tool for traffic capture and offline PCAP parsing with metadata insertion into OpenSearch
  • viewer - a browser-based interface for data visualization and payload analysis
• OpenSearch - a search and analytics engine for indexing and querying network traffic session metadata (Elasticsearch is also supported)
  • OpenSearch Dashboards - for creating additional ad-hoc visualizations, dashboards, and reports beyond those provided by Arkime Viewer (Kibana is also supported)
    • Vega is used to create many of Malcolm's custom visualizations in Dashboards
  • OpenSearch anomaly detection engine - for detecting anomalies in network traffic in near-real time using Random Cut Forests (RCF)
  • OpenSearch ML Commons plugin - for machine-learning-powered search using pretrained models provided by OpenSearch, uploading custom models, or connecting to externally hosted models
  • OpenSearch Alerting plugin - for monitoring logs and creating alerts based on thresholds or other triggers
• Logstash and Filebeat - for ingesting, parsing, enriching, and indexing Zeek log files into OpenSearch
• Zeek - a network analysis framework and IDS
• Suricata - an IDS and threat detection engine
• netsniff-ng or tcpdump - for simpler capture-only deployments where initial traffic parsing and metadata forwarding are not required
• Strelka - a real-time, container-based file scanning system used for threat hunting, threat detection, and incident response (used for scanning files extracted by Zeek)
  • Yara - a tool used to identify and classify malware samples
  • Capa - a tool for detecting capabilities in executable files
  • ClamAV - an antivirus engine
• Threat intelligence feeds - indicators of compromise can be pulled from MISP, TAXII, Google, and Mandiant for use with the Zeek intelligence framework
• CyberChef - a "Swiss Army Knife" data conversion tool
• evtx - a fast and safe parser for the Windows XML Event Log (EVTX) format
• FilePond - for uploading PCAP files and Zeek logs for processing
• For application containerization and orchestration enabling simple, reproducible deployment of Malcolm and coordination of its components, either of the following may be used for local deployment (details):
  • Docker
  • Podman
    • Note: when using rootless Podman, Malcolm cannot perform traffic capture on local network interfaces, though it can accept metadata forwarded from a network sensor appliance
• For distributed or cloud deployment, Malcolm can be deployed using Kubernetes (on-prem or, for example, on AWS) via standard manifests or, for scalable deployments, the Malcolm Helm chart (currently in beta)
• NetBox - a suite for modeling and documenting modern networks; used to enrich network log data with asset inventory information
• PostgreSQL - a relational database used for storing configuration and state for several Malcolm components
• Valkey - an in-memory data store for caching session information for various Malcolm components
• Keycloak - an identity and access management (IAM) tool
• OpenResty - a dynamic web platform based on Nginx and LuaJIT for HTTPS and reverse proxying Malcolm components
• nginx-auth-ldap - an LDAP authentication module for Nginx
• Fluent Bit - for forwarding metrics to Malcolm from network sensors
• Mark Baggett's freq - a tool for calculating entropy of strings (e.g., domain names observed in DNS traffic)
• These YARA rulesets:
  • advanced-threat-research/Yara-Rules
  • bartblaze/Yara-rules
  • elastic/protections-artifacts
  • eset/malware-ioc
  • kevoreilly/CAPEv2
  • Neo23x0/signature-base
  • reversinglabs/reversinglabs-yara-rules
  • SEKOIA-IO/Community
  • volexity/threat-intel
• These [Zeek packages]({{ site.github.repository_url }}/blob/{{ site.github.build_revision }}/zeek/scripts/zeek_install_plugins.sh):
  • Amazon.com, Inc.'s ICS protocol analyzers
  • Andrew Klaus's Sniffpass plugin for detecting cleartext passwords in HTTP POST requests
  • Andrew Klaus's zeek-httpattacks plugin for detecting noncompliant HTTP requests
  • CERT.LV's (the Cyber Incident Response Institution of the Republic of Latvia) spicy-iec104 parser, which is a fork of George Makrakis' work.
  • ICS protocol analyzers for Zeek published by Idaho National Lab and DHS CISA
  • Numerous packages from Corelight, Inc.
  • FoxIO's JA4+ network fingerprinting algorithms. JA4+ algorithms (used by the Zeek and Arkime components of Malcolm) have licensing requirements under the FoxIO License. See also the FoxIO license FAQ.
  • Johanna Amann's CVE-2020-0601 ECC certificate validation plugin and CVE-2020-13777 GnuTLS unencrypted session ticket detection plugin
  • Lexi Brent's EternalSafety plugin
  • MITRE Cyber Analytics Repository's Bro/Zeek ATT&CK®-Based Analytics (BZAR) scripts
  • NCSA's bro-is-darknet and bro-simple-scan
  • ATT&CK-based Control-system Indicator Detection (ACID) indicators from DHS and MITRE
  • Salesforce's gQUIC analyzer
  • Seiso's zeek-kafka, a Zeek log writer that publishes to Kafka
  • Zeek's Spicy plugin framework
• GeoLite2 - Malcolm includes GeoLite2 data created by MaxMind
• Debian Live - the framework used to build the x86_64 installer ISOs and the Raspberry Pi sensor image