Scribd Logo
Upload

Welcome to Scribd!

  • 4 views

    Obf Troubleshooting

    Uploaded by

    Shibani Sahu

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd

    Obf Troubleshooting

    Uploaded by

    Shibani Sahu
    4 views10 pages

    Original Title

    obf troubleshooting

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    4 views10 pages

    Obf Troubleshooting

    Uploaded by

    Shibani Sahu

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    of 10

    Obf troubleshooting | mop

    If any application data is not available in grafana dashboard :-


     Connect the totsuka VPN
     Enable foxy proxy
     Open the grafana - http://obf.prod.rmn.local/grafana/login
     Tap on search button go to manage section and open the
    application
     In any application dashboard go into not ready node drop down
    button
     Click on edit (to see the backend configuration)
     Here you able to select the data source where the data is
    coming

     We can also go on inspect then Json then search expr and copy
    the query and in the section of explore we have to run it
     We also must copy data source query which is in the inspect
    section
     At the top of explore we put the data source
     And in metrics browser we put the explore query also we have
    to remove the backslash and we also have to check the cluster
    name in the setting of dashboard and change it

     If after applying this query we get 0 series returned, then it


    means whatever we are looking into dashboards are good
     If we remove == 0 from the query, then we get

     We get condition is ready that means my nodes are fine


    How we start debugging in OBF End
     we will go in Kibana
     select the tenant in custom section
     then we go specific microservices and check the recent logs
     in case if data is not available in Kibana then we check the
    responsible spark job which is processing this data
     then we move to Grafana dashboard to check the job is running
    or not

     In driver id we put the tenant name and select the spark and
    check the job id
     If job id is running fine but still data is not find in Kibana then
    we have check the spark jobs logs (is any issue with Kafka or
    not consuming data like that )

     In terminal we can check the logs of spark jobs


     If we get time out exception it because of the KAFKA ,
    and Kafka is not responding to the spark job so our spark
    job is not able to communicate with Kafka , that’s why
    its not consuming the data from the Kafka , so in that
    case we are not able to see the data in Kibana
     Same steps will perform if any FM data is not found in
    OBF
    Why The Data is not showing in any dashboard
     Go to the application in OBF Grafana

     Click on any one element like load average and go into


    inspect then panel json search expr here and copy the
    query which show blue collared in below picture , and
    also see the data source name

     Then go to explore section
     In the top put the data source name
     And in metrics browser put the expr query

     We get application name in app section


     If there is our app is not mentioned then go to the
    metrics browser and add {app = ~ “* *”}

     If there is no data coming from the application then we
    check if data is getting scrapped for this application or
    not then we remove the added query in {}

    Press on add query and put the job name into new metric
    browser like mentioned in below picture

    Then here you can see your application is listed


    Here if you scroll in left side then you will see the value
    column then there is value like 0 & 1 if your value is 0 it
    means your application is not sending the metrics to java -
    melody and 1 means your jobs are running or scrapping the
    data from applications

    POD PERFORMANCE

    If in pod status value is 1 it means pods is running in pod


    container we can see pods are ready or not how many times
    pods are started
    Apache spark
    We look here driver status or also number of executor if
    driver status is one then it’s running and if 0 then it’s down
    We also check cluster detail dashboard

    In this figure there is so many nodes are unscheduled then


    we have to open the node by the clicking in the section of
    unschedulable node name then we need to share the
    attention to the cloud team
    >> if fault led is on then its take care by cloud team only we
    can’t do anything here

    You might also like