[TMLCyber-f1fd] Operating Details

Generated On: 2024-11-04 19:25:57 UTC

Note

THIS DOCUMENTATION CREATION WAS AUTOMATICALLY TRIGGERED BY: TSS Development Environment Container

If this documentation was triggered by the TSS - then your solution is running in the TSS Development environment. All ports and links will point to your TSS development environment.

If this documentation was triggered by your TML solution - then your solution is running in your TML solution container. All ports and links will point to your TML solution container.

If you have NOTHING running - most of the links and ports below WILL NOT WORK.

If you have BOTH RUNNING - all of the links and ports below WILL WORK.

Also, this documentation is updated by the LATEST run of either the TSS or TML.

Important

These are the operating details for your TML solution.

It is important to note that this documentation will

dynamically update with the Runtime Ports for your solution.

The ports below were valid for the last run of your solution, and will

not be valid if your [TMLCyber-f1fd] container is NOT running.

Tip

You must have your [TMLCyber-f1fd] container running before connecting to the Visualization and Airflow URLs.

Github Logs

This is your main TSS Github logs. All TSS processes are committed to Github and logged.

Important

https://github.com/Dynamo7001/raspberrypi/blob/main/tml-airflow/logs/logs.txt

TSS Docker Run Command

This is the TML Solution Studio Docker Run command. Note for MAC users change amd64 to arm64 in the container name.

Important

docker run -d --net=host --env AIRFLOWPORT=9000 -v <change to your local folder>:/dagslocalbackup:z -v /var/run/docker.sock:/var/run/docker.sock:z --env GITREPOURL=https://github.com/Dynamo7001/raspberrypi --env CHIP=amd64 --env TSS=1 --env SOLUTIONNAME=TSS --env EXTERNALPORT=39987 --env VIPERVIZPORT=9005 --env GITUSERNAME='Dynamo7001' --env DOCKERUSERNAME='hooluwole' --env MQTTUSERNAME='hammed' --env KAFKACLOUDUSERNAME='' --env KAFKACLOUDPASSWORD='<Enter your API secret>' --env READTHEDOCS='<Enter your readthedocs token>' --env GITPASSWORD='<Enter personal access token>' --env DOCKERPASSWORD='<Enter your docker hub password>' --env MQTTPASSWORD='<Enter your mqtt password>' maadsdocker/tml-solution-studio-with-airflow-amd64

TSS Docker Run Command: Parameter Explanation

Parameter

Explanation

docker

This calls the docker engine

-d

This runs your container in detached mode

--net=host

This give your container access to your host operating system

-v

This stands for volume mapping. It maps a local folder

in your host machine to the folder in the container. The value z

means the container has shared access to your local folder.

For example, -v /mylocal/folder:/dagslocalbackup:z, means map

/mylocal/folder (on my host machine) to /dagslocalbackup

in the contaner. This allows files generated in the container

to be automatically written to your local folder.

--env GITREPOURL

This is your Github repo, that you cloned from https://github.com/smaurice101/raspberrypi

--env CHIP=AMD64

This is the chip if your are running the TSS on windows/linux.

If you are running MAC, use CHIP=ARM64

--env TSS=1

This is the TSS value and MUST be 1.

--env AIRFLOWPORT=9000

This is the airflow port for TSS. Connect to TSS from your browser:

http://localhost:9000

--env SOLUTIONNAME=TSS

This is the solution name.

--env VIPERVIZPORT=9005

This is the port the Viperviz binary will listen on for connections.

Note: If VIPERVIZPORT=-1, a random free port is selected by TSS.

--env EXTERNALPORT=39987

This is the external port that will be assigned to your TSS solution for external access.

You will need this port in the REST, and gRPC clients.

Note: if EXTERNALPORT=-1, TSS will choose a free port randomly.

This external port is used by Viper binary: Viper will be listening on this port

for a connection as shown here: Your Solution TML Binaries

In the TMUX window Viper-produce: Your Solution TMUX Windows

--env READTHEDOCS

This is the readthedocs API token you created.

Refer to: Set up readthedocs

--env GITUSERNAME

This is your Githib username.

--env GITPASSWORD

This is the Github Personal Access Token you created.

Refer to: Creating Github Token:

--env DOCKERUSERNAME

This is your Docker Hub username.

--env DOCKERPASSWORD

This is your Docker Hub password.

--env MQTTUSERNAME

This is your MQTT username. See Set up HiveMQ

--env MQTTPASSWORD

This is your MQTT password.

--env KAFKACLOUDUSERNAME

This is your API key from Confluent Cloud

--env KAFKACLOUDPASSWORD

This is your API Secret from Confluent Cloud.

maadsdocker/tml-solution-studio-with-airflow-amd64

This is the TSS container name for AMD64

If using MAC/Unix use: maadsdocker/tml-solution-studio-with-airflow-arm64

TSS Dashboard URL

This is the visualization URL for your TSS dashboard. Note ports may change at runtime. The solution documentation will update automatically.

Important

http://localhost:9005/tml-cisco-network-privategpt-monitor.html?topic=cisco-network-preprocess,cisco-network-privategpt&offset=-1&groupid=&rollbackoffset=400&topictype=prediction&append=0&secure=1

TSS Airflow Port

This is the airflow port in your TSS solution container.

It can be accessed by entering: http://localhost:9000

TSS Log File Dashboard

This is the log file dashboard for your development TML solution running in TSS.

Important

http://localhost:9005/viperlogs.html?topic=viperlogs&append=0

Note

It should be noted that your solution is running in the TSS Development Environment. This gives TML developers a very good way to test their TML solutions before deploying it.

The solution ports and links below may not work because they will require your to RUN your solution container first. After, you run your solution container the links and ports will automatically update in the documentation.

Your Solution Docker Container

Important

hooluwole/TMLCyber-f1fd-amd64 (https://hub.docker.com/r/hooluwole/TMLCyber-f1fd-amd64)

Your Solution Docker Run Command

This is the Docker Run command for your solution container. Note ports may change at runtime. The solution documentation will update automatically.

Important

docker run -d -p 38407:38407 -p 46561:46561 -p 52953:52953 -p 8883:8883 --env TSS=0 --env SOLUTIONNAME=TMLCyber-f1fd --env SOLUTIONDAG=solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd --env GITUSERNAME=Dynamo7001 --env GITREPOURL=https://github.com/Dynamo7001/raspberrypi --env SOLUTIONEXTERNALPORT=38407 --env CHIP=amd64 --env SOLUTIONAIRFLOWPORT=46561 --env SOLUTIONVIPERVIZPORT=52953 --env DOCKERUSERNAME='hooluwole' --env CLIENTPORT=8883 --env EXTERNALPORT=39987 --env KAFKACLOUDUSERNAME='' --env VIPERVIZPORT=9005 --env MQTTUSERNAME='hammed' --env AIRFLOWPORT=9000 --env GITPASSWORD='<Enter Github Password>' --env KAFKACLOUDPASSWORD='<Enter API secret>' --env MQTTPASSWORD='<Enter mqtt password>' --env READTHEDOCS='<Enter Readthedocs token>' hooluwole/TMLCyber-f1fd-amd64

Tip

Use the above Docker Run command to run your solution. Make sure to UPDATE the GITPASSWORD and READTHEDOCS parameters.

Your Solution Docker Run Command: Parameter Explanation

Parameter

Explanation

docker

This calls the docker engine

-d

This runs your container in detached mode

--net=host

This give your container access to your host operating system

--env TSS=0

Internal TSS variable. MUST be 0.

--env SOLUTIONNAME

This is the name of your TML solution.

--env SOLUTIONDAG

This is the name of the DAG that comprises your solution.

This DAG is triggered automatically when you run this container.

--env SOLUTIONVIPERVIZPORT=52953

This is the port Viperviz is listening.

You point your browser to this port. See Your Solution Dashboard URL

--env CLIENTPORT=8883

Use this port if you are externally connecting to the TML/TSS solution using

REST API or gRPC clients. You will need this port in the REST, and gRPC clients.

This external port is used by Viper binary: Viper will be listening on this port

for a connection as shown here: Your Solution TML Binaries

In the TMUX window Viper-produce: Your Solution TMUX Windows

--env VIPERVIZPORT=9005

This is the port Viperviz is listening in TSS.

You point your browser to this port. See Your Solution Dashboard URL

--env AIRFLOWPORT=9000

This is the port for Airflow in TSS solution studio container.

--env SOLUTIONAIRFLOWPORT=46561

This is the port for Airflow in TML solution container.

Note: This is provided mainly for debugging and testing purposes only.

--env GITUSERNAME

This is your Github username.

--env GITPASSWORD

This is the Github Personal Access Token you created.

Refer to: Creating Github Token

--env GITREPOURL

This is your Github repo, that you cloned from https://github.com/smaurice101/raspberrypi

--env DOCKERUSERNAME

This is your Docker username.

--env READTHEDOCS

This is the readthedocs API token you created.

Refer to: Set up readthedocs

--env CHIP=amd64

This is the chip family of your OS.

--env EXTERNALPORT=39987

This is the external port that you can use when making an external

connection to your TML solution running in TSS Dev environment.

--env SOLUTIONEXTERNALPORT=38407

This is the external port that you can use when making an external connection to your TML solution

for external data ingestion. if SOLUTIONEXTERNALPORT=-1, TSS selects a free port randomly.

--env MQTTUSERNAME

This is your MQTT username

--env MQTTPASSWORD

This is your MQTT password.

--env KAFKACLOUDUSERNAME

This is your API key from Confluent Cloud

--env KAFKACLOUDPASSWORD

This is your API Secret from Confluent Cloud.

hooluwole/TMLCyber-f1fd-amd64

Your solution container name.

Your Solution Airflow Port

This is the airflow port in your solution container.

It can be accessed by entering: http://localhost:46561

Important

46561

Note: This port will change when SOLUTIONAIRFLOWPORT=-1, you can set it to

particular number.

Your Solution External Port

This is the Docker Run command for your solution container. Note ports may change at runtime. The solution documentation will update automatically.

Important

38407

This is the external port that you can use when making an external connection to your TML solution for external data ingestion. You will need this port in the REST, and gRPC clients.

Note: if SOLUTIONEXTERNALPORT=-1, TSS will choose a free port randomly.

This external port is used by Viper binary: Viper will be listening on this port

for a connection as shown here Your Solution TML Binaries

In the TMUX window Viper-produce: Your Solution TMUX Windows

Non-Solution vs Solution Ports

Non-solution ports are only for TSS, this is because TSS includes a TML Dev environment to allow TML solution developers to test their solutions.

Solution ports are for your TML solution that you created and will deploy.

Important

It is important to note the difference between the following ports:

  • AIRFLOWPORT and SOLUTIONAIRFLOWPORT

  • EXTERNALPORT and SOLUTIONEXTERNALPORT

  • VIPERVIZPORT and SOLUTIONVIPERVIZPORT

The reason is because TSS includes a Development environment for TML

solutions, many times you will want to run your solution in Dev and run

it in its own solution container for testing before you deploy your

solution. But, since ONLY ONE application can listen on a port,

we must assign a different port to the solutions so there is no

port conflict between applications in DEV and PROD.

However, if you set all port to -1, TSS will randomly choose

free ports for you. The reason for setting the ports with an

actual number that is NOT -1, is if you want to scale your TML solution

with Kubernetes and producing data using REST or gRPC and do not want

ports to keep changing and breaking your app.

Your Solution Dashboard URL

This is the visualization URL for your TML dashboard. Note ports may change at runtime. The solution documentation will update automatically.

Important

http://localhost:52953/tml-cisco-network-privategpt-monitor.html?topic=cisco-network-preprocess,cisco-network-privategpt&offset=-1&groupid=&rollbackoffset=400&topictype=prediction&append=0&secure=1

Your Solution Log File Dashboard

This is the log file dashboard for your TML solution running.

Important

http://localhost:52953/viperlogs.html?topic=viperlogs&append=0

Your Solution Dashboard URL: Parameter Explanation

Parameter

Explanation

http://localhost:52953/<html file>

This is the URL pointing to an html file running inside your solution container.

Refer to: TML Real-time dashboards

SOLUTIONVIPERVIZPORT=52953

This is the port Viperviz is listening on.

topic

This is the topic that the TML binary Viperviz

is reading (consuming) in Apache Kafka and sending it to your broweser over websockets.

offset

This value tells the Viperviz binary to read the latest real-time data.

offset=-1, means to go to the end of the data stream and get the latest record.

groupid

This can be empty.

rollbackoffset

This is the number of offsets to rollback the data stream from the offset value.

Note: If you increase this number, Viperviz will send more data to your browser.

But be carefull, too much data may crash your browser or computer.

topictype

Leave as is.

append

This tells your html file whether to append or not the data streaming to your browser.

If append=0, the html will not apend, if append=1, then data will accumulate in your browser.

secure

This tells Viperviz whether to encrypt your data to the browser.

If secure=1, data are encrypted, secure=0 no encryption.

[TMLCyber-f1fd] Github Repo

This is the Github repo for all your solution code

Important

https://github.com/Dynamo7001/raspberrypi/tree/main/tml-airflow/dags/tml-solutions/TMLCyber-f1fd

Readthedocs URL

This is this URL.

Important

https://TMLCyber-f1fd.readthedocs.io

Solution Trigger DAG

This is the name of the solution DAG you chose to trigger.

Important

solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd

Your Solution TML Binaries

These are the ports the TML binaries are listening on.

Important

VIPERHOST_PRODUCE=127.0.1.1, VIPERPORT_PRODUCE=38407, VIPERHOST_PREPOCESS=127.0.1.1, VIPERPORT_PREPROCESS=34601, VIPERHOST_PREPOCESS2=127.0.1.1, VIPERPORT_PREPROCESS2=33097, VIPERHOST_PREPOCESS_PGPT=127.0.1.1, VIPERPORT_PREPROCESS_PGPT=43233, VIPERHOST_ML=127.0.1.1, VIPERPORT_ML=46461, VIPERHOST_PREDCT=127.0.1.1, VIPERPORT_PREDICT=36455, HPDEHOST=127.0.1.1, HPDEPORT=34835, HPDEHOST_PREDICT=127.0.1.1, HPDEPORT_PREDICT=41757

Your Solution TMUX Windows

Important

python-produce-9563-TMLCyber-f1fd,solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd, python-preprocess-2626-TMLCyber-f1fd,solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd, python-ai-3001-TMLCyber-f1fd,solution_preprocessing_ai_mqtt_dag-TMLCyber-f1fd, viper-produce, viper-preprocess, viper-preprocess-pgpt, viper-ml, viper-predict

  • Your solution is running in these

    TMUX windows:

    • To view windows, type:

      tmux ls

    • To go inside window, type:

      tmux a -t <window name>

    • To exit window, type:

      CTLR+b, d

    • To scroll window, type:

      CTLR+b, [

    • To un-scroll window, type:

      CTLR+[