[TMLCyber-f1fd] Details ============================ Generated On: 2024-11-04 19:25:57 UTC TML Solution DAG Parameters' Details: User Chosen Parametets ---------------------------- STEP 1: Get TML Core Params: `tml_system_step_1_getparams_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - solutionname - TMLCyber-f1fd * - solutiontitle - My Solution Title * - solutiondescription - This is an awesome real-time solution built by TSS * - brokerhost - 127.0.0.1 * - brokerport - 9092 * - cloudusername - None * - ingestdatamethod - MQTT STEP 2: Create Kafka Topics: `tml_system_step_2_kafka_createtopic_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - companyname - Otics * - myname - Sebastian * - myemail - Sebastian.Maurice * - mylocation - Toronto * - replication - 1 * - numpartitions - 1 * - enabletls - 1 * - microserviceid - * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - prediction_data_topic - prediction-data STEP 3: `Produce to Kafka Topics `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PRODUCETYPE - MQTT * - TOPIC - cisco-network-mainstream * - PORT - 39987 * - IDENTIFIER - MQTT Subscription Topic: tml/cybersecurity * - HTTPADDR - https:// * - FROMHOST - hammed-VMware-Virtual-Platform,127.0.1.1 * - TOHOST - 127.0.1.1 * - CLIENTPORT - 8883 * - TSS_CLIENTPORT - 8883 * - TML_CLIENTPORT - 8883 STEP 4: Preprocesing Data: `tml-system-step-4-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - cisco-network-mainstream * - preprocess_data_topic - cisco-network-preprocess * - preprocessconditions - * - delay - 70 * - array - 0 * - saveasarray - 1 * - topicid - -999 * - rawdataoutput - 1 * - asynctimeout - 120 * - timedelay - 0 * - preprocesstypes - anomprob,trend,avg * - pathtotmlattrs - --pathtotmlattrs-- * - identifier - IoT device performance and failures * - jsoncriteria - uid=metadata.dsn,filter:allrecords~subtopics=metadata.property_name~values=datapoint.value~identifiers=metadata.display_name~datetime=datapoint.updated_at~msgid=datapoint.id~latlong=lat:long STEP 4b: Preprocesing Data: `tml-system-step-4b-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - --raw_data_topic2-- * - preprocess_data_topic - --preprocess_data_topic2-- * - preprocessconditions - --preprocessconditions2-- * - delay - --delay2-- * - array - --array2-- * - saveasarray - --saveasarray2-- * - topicid - --topicid2-- * - rawdataoutput - --rawdataoutput2-- * - asynctimeout - --asynctimeout2-- * - timedelay - --timedelay2-- * - preprocesstypes - --preprocesstypes2-- * - pathtotmlattrs - --pathtotmlattrs2-- * - identifier - --identifier2-- * - jsoncriteria - --jsoncriteria2-- STEP 5: Entity Based Machine Learning : `tml-system-step-5-kafka-machine-learning-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_data_topic - ml-data * - modelruns - --modelruns-- * - offset - -1 * - islogistic - --islogistic-- * - networktimeout - --networktimeout-- * - modelsearchtuner - --modelsearchtuner-- * - processlogic - --processlogic-- * - dependentvariable - --dependentvariable-- * - independentvariables - --independentvariables-- * - rollbackoffsets - --rollbackoffsets-- * - topicid - -999 * - consumefrom - cisco-network-preprocess * - fullpathtotrainingdata - --fullpathtotrainingdata-- * - transformtype - --transformtype-- * - sendcoefto - --sendcoefto-- * - coeftoprocess - --coeftoprocess-- * - coefsubtopicnames - --coefsubtopicnames-- STEP 6: Entity Based Predictions: `tml-system-step-6-kafka-predictions-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - cisco-network-preprocess * - ml_prediction_topic - --ml_prediction_topic-- * - streamstojoin - --streamstojoin-- * - inputdata - --inputdata-- * - consumefrom - --consumefrom2-- * - offset - -1 * - delay - 70 * - usedeploy - --usedeploy-- * - networktimeout - --networktimeout-- * - maxrows - --maxrows-- * - topicid - -999 * - pathtoalgos - --pathtoalgos-- STEP 7: Real-Time Visualization: `tml-system-step-7-kafka-visualization-dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - vipervizport - 9005 * - topic - cisco-network-preprocess,cisco-network-privategpt * - dashboardhtml - tml-cisco-network-privategpt-monitor.html * - secure - 1 * - offset - -1 * - append - 0 * - chip - amd64 * - rollbackoffset - 400 STEP 8: `tml_system_step_8_deploy_solution_to_docker_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Docker Container - hooluwole/TMLCyber-f1fd-amd64 (https://hub.docker.com/r/hooluwole/TMLCyber-f1fd-amd64) * - Docker Run Command - docker run -d -p 38407:38407 -p 46561:46561 -p 52953:52953 -p 8883:8883 \-\-env TSS=0 \-\-env SOLUTIONNAME=TMLCyber-f1fd \-\-env SOLUTIONDAG=solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd \-\-env GITUSERNAME=Dynamo7001 \-\-env GITREPOURL=https://github.com/Dynamo7001/raspberrypi \-\-env SOLUTIONEXTERNALPORT=38407 \-\-env CHIP=amd64 \-\-env SOLUTIONAIRFLOWPORT=46561 \-\-env SOLUTIONVIPERVIZPORT=52953 \-\-env DOCKERUSERNAME='hooluwole' \-\-env CLIENTPORT=8883 \-\-env EXTERNALPORT=39987 \-\-env KAFKACLOUDUSERNAME='' \-\-env VIPERVIZPORT=9005 \-\-env MQTTUSERNAME='hammed' \-\-env AIRFLOWPORT=9000 \-\-env GITPASSWORD='' \-\-env KAFKACLOUDPASSWORD='' \-\-env MQTTPASSWORD='' \-\-env READTHEDOCS='' hooluwole/TMLCyber-f1fd-amd64 STEP 9: `tml_system_step_9_privategpt_qdrant_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PrivateGPT Container - maadsdocker/tml-privategpt-with-gpu-nvidia-amd64 * - PrivateGPT Run Command - docker run -d -p 8001:8001 --net=host --gpus all --env PORT=8001 --env GPU=1 --env COLLECTION=tml --env WEB_CONCURRENCY=1 --env CUDA_VISIBLE_DEVICES=0 maadsdocker/tml-privategpt-with-gpu-nvidia-amd64 * - Qdrant Container - qdrant/qdrant * - Qdrant Run Command - docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant * - Consumefrom - cisco-network-preprocess * - pgpt_data_topic - cisco-network-privategpt * - offset - -1 * - rollbackoffset - 400 * - topicid - -999 * - enabletls - 1 * - partition - -1 * - prompt - Do any of the values of the inbound or outbound packets look abnormal? * - context - These data are anomaly probabilities of suspicious data traffic.A high probability of over 0.80 is likely suspicious.’ * - jsonkeytogather - hyperprediction * - keyattribute - outboundpackets,inboundpackets * - keyprocesstype - anomprob * - vectordbcollectionname - tml * - concurrency - 1 * - CUDA_VISIBLE_DEVICES - 0 * - pgpthost - http://127.0.0.1 * - pgptport - 8001 * - hyperbatch - _0 STEP 10: `tml_system_step_10_documentation_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Solution Documentation URL - https://TMLCyber-f1fd.readthedocs.io